Method and apparatus for distilling visual transformation model, electronic device and medium

Abstract

This disclosure provides a distillation method, apparatus, electronic device, and medium for visual transformation models, relating to the field of artificial intelligence technology, particularly computer vision and deep learning, and applicable to image classification or object detection scenarios. The specific implementation scheme is as follows: The dimensions of the student encoding parameters in the image encoder of the student model are processed to align with the dimensions of the teacher encoding parameters; the teacher encoding parameters belong to the image encoder of the teacher model; a distillation loss function for the student model is determined based on the student encoding parameters and the teacher encoding parameters; student prediction parameters for the image processing task are determined based on the student encoding parameters, and a task loss function for the student model is determined based on the student prediction parameters; the student model is trained by distillation based on the distillation loss function and the task loss function. This disclosure can improve the distillation effect of visual transformation models and enhance the predictive performance of student models.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to computer vision, deep learning and other technical fields, and can be applied to image classification or object detection scenarios. Background Technology

[0002] The Vision Transformer (ViT) model has achieved great success in the field of computer vision due to its outstanding performance, and network models with the ViT structure as the backbone network have become a research hotspot.

[0003] It is known that the performance of a visual transformation model is generally positively correlated with the number of network parameters. However, due to limitations in inference speed and memory resources, large visual transformation models cannot be deployed directly in some scenarios and need to be compressed. Summary of the Invention

[0004] This disclosure provides a distillation method, apparatus, electronic device, and medium for visual transformation models.

[0005] According to one aspect of this disclosure, a distillation method for a visual transformation model is provided, the method comprising:

[0006] The student coding parameters in the image encoder of the student model are dimensionally processed to align with the dimensions of the teacher coding parameters; the teacher coding parameters belong to the image encoder of the teacher model.

[0007] The distillation loss function of the student model is determined based on the student coding parameters and the teacher coding parameters;

[0008] Based on the student coding parameters, the student prediction parameters of the student model for the image processing task are determined, and based on the student prediction parameters, the task loss function of the student model is determined.

[0009] The student model is trained by distillation based on the distillation loss function and the task loss function.

[0010] According to another aspect of this disclosure, a distillation apparatus for a visual transformation model is provided, comprising:

[0011] The dimension processing module is used to perform dimension processing on the student encoding parameters in the image encoder of the student model, so that the dimensions of the student encoding parameters are aligned with those of the teacher encoding parameters; the teacher encoding parameters belong to the image encoder of the teacher model.

[0012] The distillation loss determination module is used to determine the distillation loss function of the student model based on the student coding parameters and the teacher coding parameters;

[0013] The business loss determination module is used to determine the student prediction parameters of the student model for the image processing task based on the student coding parameters, and to determine the task loss function of the student model based on the student prediction parameters.

[0014] The distillation training module is used to perform distillation training on the student model based on the distillation loss function and the task loss function.

[0015] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform the distillation method for the visual transformation model as described in any embodiment of this disclosure.

[0019] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform a distillation method for a visual transformation model as described in any embodiment of this disclosure.

[0020] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the distillation method for the visual transformation model described in any embodiment of this disclosure.

[0021] According to the technology disclosed herein, the distillation effect of visual transformation models can be improved, and the predictive performance of student models can be enhanced.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0023] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0024] Figure 1 This is a flowchart of a distillation method for a visual transformation model provided according to an embodiment of the present disclosure;

[0025] Figure 2 This is a flowchart of a distillation method for another visual transformation model provided according to an embodiment of the present disclosure;

[0026] Figure 3 This is a flowchart of a distillation method for another visual transformation model provided according to an embodiment of the present disclosure;

[0027] Figure 4 This is a schematic diagram of the structure of a distillation apparatus for a visual transformation model provided in an embodiment of the present disclosure;

[0028] Figure 5 This is a block diagram of an electronic device used to implement the distillation method of the visual transformation model of the embodiments of this disclosure. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] Figure 1 This is a flowchart of a distillation method for a visual transformation model according to an embodiment of this disclosure. This disclosure is applicable to the compression of visual transformation models. The method can be executed by a distillation device for the visual transformation model, which can be implemented in software and / or hardware and integrated into an electronic device that carries the distillation function of the visual transformation model. Figure 1 As shown, the distillation method for the visual transformation model in this embodiment may include:

[0031] S101, Perform dimensionality processing on the student coding parameters in the image encoder of the student model to align the dimensions of the student coding parameters with those of the teacher coding parameters; the teacher coding parameters belong to the image encoder of the teacher model.

[0032] S102, Determine the distillation loss function of the student model based on the student coding parameters and the teacher coding parameters;

[0033] S103, determine the student prediction parameters of the student model for the image processing task based on the student coding parameters, and determine the task loss function of the student model based on the student prediction parameters;

[0034] S104, Based on the distillation loss function and the task loss function, perform distillation training on the student model.

[0035] The teacher model, pre-trained, meets business requirements for prediction performance. However, teacher models are typically large in size, and deploying them requires significant inference speed and GPU memory, making direct deployment to edge devices like mobile phones difficult. The student model, obtained by compressing the pre-trained visual transformation model using knowledge distillation, is smaller in size, thus reducing the requirements for inference speed and GPU memory, and can be directly deployed to edge devices such as mobile phones.

[0036] Knowledge distillation is the process of using the knowledge learned by the teacher model to guide the student model in feature learning. Through knowledge distillation, the student model can achieve the computational performance of the teacher model. The student model obtained after knowledge distillation is essentially a compressed version of the teacher model.

[0037] The teacher model and the student model are both built with a visual transformation model as their backbone network. Both models possess the structural characteristics of the visual transformation model. Each model includes at least an image encoder and an image decoder. The image encoder is used to understand the local and global features of the image. The image decoder works in conjunction with the image encoder to complete the image processing task. As a crucial feature learning component in the visual transformation model, the feature learning capability of the image encoder directly affects the performance of the visual transformation model.

[0038] In related technologies, knowledge distillation typically involves using the final output layer of the teacher model to distill the final output layer of the student model. However, the image encoder in the intermediate layer of the student model cannot replicate the performance of the image encoder in the teacher model, resulting in the student model not being able to fully replicate the computational performance of the teacher model and affecting the distillation effect of the visual transformation model.

[0039] This embodiment utilizes the image encoder of the teacher model at an intermediate level to distill the image encoder of the student model, thereby improving the distillation effect on the visual transformation model. Since the model sizes of the teacher and student models differ, the dimensions of their intermediate-level output parameters will also differ. In other words, distilling the student model's image encoder using the teacher model's image encoder requires aligning the output parameter dimensions.

[0040] The teacher model is pre-trained, and the teacher encoding parameters in the image encoder of the teacher model are fixed. It is necessary to align the student encoding parameters in the image encoder of the student model with the teacher encoding parameters.

[0041] Based on the dimension-aligned student and teacher encoding parameters, a distillation loss function is determined for the student model. The student encoding parameters determine the feature representation capability of the image encoder in the student model, while the teacher encoding parameters determine the feature representation capability of the image encoder in the teacher model. The distillation loss function measures the difference in feature representation between the image encoders of the student and teacher models. Training the student model using the distillation loss function forces the student model's image encoder to fit the teacher model's image encoder, enabling the student model's image encoder to achieve the feature representation of the teacher model's image encoder.

[0042] The distillation loss function is only one component of the training loss function, which also includes the business loss function. Model distillation training aims to improve the predictive performance of the student model. The business loss function measures the predictive performance of the student model itself. The business loss function is related to the image processing task and is determined based on the student model's predictions for the task and the actual information presented in the image processing task. Optionally, the image processing task can be either object recognition or image classification. The specific type of image processing task is not limited here and is determined based on actual business needs.

[0043] The student encoding parameters belong to the image encoder of the student model. The image encoder is an intermediate layer in the student model. The student encoding parameters need to be processed by other layers in the student model, such as the image decoder, to obtain the student prediction parameters for the image processing task. The student prediction parameters refer to the predicted probability output of the student model for the image processing task. These student prediction parameters are used to construct the task loss function of the student model and correspond to the prediction information in the data basis of the task loss parameters.

[0044] After the distillation loss function and task loss function are constructed, the training loss function is determined based on them. Optionally, the sum of the distillation loss function and the task loss function can be used as the training loss function. The student model is then trained using the training loss function through distillation, and the composition structure of the student model is updated based on the training loss function. Optionally, a model optimization algorithm such as Adamw can be used to update the composition structure of the student model based on the training loss function.

[0045] This embodiment aligns the parameter dimensions of student and teacher encoding parameters. Based on the aligned student and teacher encoding parameters, a distillation loss function for the student model is determined. This distillation loss function is then used to train the student model, making it possible to distill the image encoder of the student model using the image encoder of the teacher model. This embodiment uses the image encoder in the teacher model to guide the image encoder in the student model for feature learning, allowing the image encoder in the student model to fully fit the image encoder in the teacher model. The image encoder, as a key structure for feature learning in the visual transformation model, is distilled in a fine-grained manner using the image encoder in the teacher model. This improves the feature learning capabilities and computational performance of the student model. This embodiment trains the student model using distillation loss function and business loss function, focusing not only on the local performance of the image encoder in the student model but also on the global performance of the student model during distillation training. This improves the distillation effect of the visual transformation model and enhances the predictive performance of the student model.

[0046] In an optional embodiment, the student model is distilled and trained based on the distillation loss function and the task loss function, including: distilling and training the image encoder in the student model based on the distillation loss function; and distilling and training all components in the student model based on the task loss function.

[0047] The distillation loss function is determined based on the student's encoding parameters and the teacher's encoding parameters. The student's encoding parameters belong to the image encoder of the student model; the teacher's encoding parameters belong to the image encoder of the teacher model. The distillation loss function is related to the image encoder and has local properties. Distillation training is performed on the image encoder in the student model based on the distillation loss function. Specifically, a model optimization algorithm is used to update the network parameters of the image encoder in the student model based on the distillation loss function, enabling the image encoder in the student model to realize the feature representation of the image encoder in the teacher model.

[0048] The task loss function is determined based on the student model's encoding parameters for the image processing task. The task loss function is related to all components of the student model and possesses global properties. That is, the task loss function is related not only to the image encoder but also to components such as the image decoder. Distillation training is performed on all components of the student model based on the task loss function. Specifically, a model optimization algorithm is used to update the network parameters of all components in the student model based on the task loss function, enabling the student model to achieve the predictive performance of the teacher model.

[0049] The above technical solution provides a practical distillation training method by distilling the image encoder in the student model based on the distillation loss function and distilling all components in the student model based on the task loss function, thus providing technical support for the distillation of visual transformation models.

[0050] Figure 2 This is a flowchart of another distillation method for a visual transformation model provided according to an embodiment of this disclosure; this embodiment is an optional solution proposed based on the above embodiments. Specifically, this disclosure embodiment refines the operation of "performing dimensional processing on the student encoding parameters in the image encoder of the student model to align the dimensions of the student encoding parameters with those of the teacher encoding parameters".

[0051] See Figure 2 The distillation method for the visual transformation model provided in this embodiment includes:

[0052] S201, the student coding parameters in the image encoder of the student model are upgraded in dimension so that the dimensions of the student coding parameters are aligned with those of the teacher coding parameters; the teacher coding parameters belong to the image encoder of the teacher model.

[0053] The teacher model is larger, and the parameter dimension of the teacher's encoding parameters is greater than that of the student's encoding parameters in the student model. To distill the image encoder of the student model using the image encoder in the teacher model, it is necessary to align the parameter dimensions of the student and teacher encoding parameters.

[0054] Specifically, the dimensionality of the student attention parameter in the student encoding parameters is increased while keeping the dimensionality of the teacher encoding parameters unchanged. This ensures that the dimensionality of the student attention parameter is consistent with that of the teacher attention parameter in the teacher encoding parameters, allowing the distillation loss function to be determined based on both the teacher encoding parameters and the increased-dimensional student encoding parameters.

[0055] S202, reduce the dimensionality of the student coding parameters after dimensionality increase, so that the student coding parameters are restored to the initial dimension before dimensionality increase.

[0056] The various components in the student model work together, and each component has its own requirements for the dimensionality of the parameters input into it. Therefore, it is necessary to reduce the dimensionality of the student coding parameters after dimensionality increase so that the student coding parameters are restored to the initial dimension before dimensionality increase, so as to avoid affecting the data flow between different components in the student model.

[0057] S203, determine the distillation loss function of the student model based on the student coding parameters and the teacher coding parameters.

[0058] It is worth noting that the data basis for determining the distillation loss function is the student coding parameters after dimensionality upgrade.

[0059] S204, determine the student prediction parameters of the student model for the image processing task based on the student coding parameters, and determine the task loss function of the student model based on the student prediction parameters.

[0060] It is worth noting that the data basis for determining the task loss function is the dimensionality-reduced student coding parameters.

[0061] S205, Based on the distillation loss function and the task loss function, perform distillation training on the student model.

[0062] This disclosed technical solution involves increasing the dimensionality of student encoding parameters in the image encoder of a student model to align the dimensions of these parameters with those of the teacher encoding parameters. A distillation loss function is then determined based on the teacher encoding parameters and the increased-dimensional student encoding data. Next, the increased-dimensional student encoding parameters are reduced to their initial dimensions before the increase. These reduced-dimensional student encoding parameters are then used to determine the business loss function. Finally, the student model is trained using both the distillation loss function and the task loss function. This disclosed embodiment provides a practical dimensionality processing scheme, offering technical support for the distillation of visual transformation models.

[0063] In an optional embodiment, the student encoding parameters in the image encoder of the student model are dimensionally increased to align with the dimensions of the teacher encoding parameters. This includes: increasing the dimensionality of the student attention parameters in the student encoding parameters through a first linear mapping module in the multi-head attention layer of the student model to align with the dimensions of the teacher attention parameters in the teacher encoding parameters; and increasing the dimensionality of the student perceptual ability parameters in the student encoding parameters through a first fully connected layer in the multilayer perceptual layer of the student model to align with the dimensions of the teacher perceptual ability parameters in the teacher encoding parameters. The multi-head attention layer and the multilayer perceptual layer belong to the image encoder.

[0064] Both the teacher and student models are constructed using the visual transformation model as the backbone network. In other words, the main structure of the teacher and student models is consistent with that of the visual transformation model. The image encoders of both the teacher and student models include a multi-head attention layer and a multi-layer perceptual (MLP) block.

[0065] Accordingly, student coding parameters include: student attention parameters and student perception parameters. Teacher coding parameters include: teacher attention parameters and teacher perception parameters. Student attention parameters belong to the multi-head attention layer of the student model, and student perception parameters belong to the multi-layer perception layer of the student model. Teacher attention parameters belong to the multi-head attention layer of the teacher model, and teacher perception parameters belong to the multi-layer perception layer of the teacher model.

[0066] The multi-head attention layer of the student model is configured with a first linear mapping module (projector1) to increase the dimensionality of the student attention parameters in the student encoding parameters, so that the dimensions of the student attention parameters are aligned with the dimensions of the teacher attention parameters in the teacher encoding parameters.

[0067] The student model has a first fully connected layer in its multi-layer perceptual layer to increase the dimensionality of the student perceptual ability parameter in the student encoding parameters, so that the student perceptual ability parameter is aligned with the dimensionality of the teacher perceptual ability parameter in the teacher encoding parameters.

[0068] The above technical solution provides a practical and feasible dimensionality-upgrading method, realizing the dimensional alignment of student attention parameters and teacher attention parameters in a multi-head attention layer, and providing technical support for the distillation of visual transformation models.

[0069] In an optional embodiment, the dimensionality reduction of the student attention parameters and the student perception parameters after dimensionality increase is performed respectively, so that the student attention parameters and the student perception parameters are restored to their initial dimensions before dimensionality increase. This includes: reducing the dimensionality of the student attention parameters after dimensionality increase through the second linear mapping module in the multi-head attention layer of the student model, so that the student attention parameters are restored to their initial dimensions before dimensionality increase; and reducing the dimensionality of the student perception parameters after dimensionality increase through the second fully connected module in the multi-layer perception layer of the student model, so that the student perception parameters are restored to their initial dimensions before dimensionality increase.

[0070] The multi-head attention layer of the student model is also equipped with a second linear mapping module (projector2). The second linear mapping module is used to reduce the dimensionality of the student attention parameters after dimensionality increase, so that the student attention parameters are restored to the initial dimension before dimensionality increase.

[0071] The student model's multi-layer perception layer is also equipped with a second fully connected module, which is used to reduce the dimensionality of the student's perception parameters after dimensionality increase, so that the student's perception parameters are restored to the initial dimension before dimensionality increase.

[0072] It is worth noting that the multi-head attention module of the image encoder in the visual transformation model only includes one type of linear mapping module. This embodiment expands the linear mapping module in the multi-head attention module to two types: a first linear mapping module and a second linear mapping module. The first and second linear mapping modules are used for dimensionality upscaling and dimensionality reduction of the student attention parameters, respectively. The multilayer perception layer in the visual transformation model includes two fully connected modules, one configured before the activation function module and the other after the activation function module. This embodiment expands the fully connected module after the activation function module to two types: a first fully connected module and a second fully connected module. The first and second fully connected modules are used for dimensionality upscaling and dimensionality reduction of the student's perceptual parameters, respectively.

[0073] For example, the student attention parameters can be dimensionally restored based on the following steps: The dimension-upgraded student attention parameters are denoted as W1, and the parameter dimension of W1 is represented by student_embed_dim × teacher_embed_dim. student_embed_dim is the actual student dimension, and teacher_embed_dim is the reference teacher dimension. The dimension-reduced student attention parameters are denoted as W2, and the parameter dimension of W2 is represented by teacher_embed_dim × student_embed_dim. A matrix multiplication operation is then performed on W1 and W2 based on W = W1W2, thus restoring the student attention parameters to their initial dimensions before the dimension upgrade.

[0074] For example, the dimensions of student perception parameters can be reduced based on the following steps: the increased-dimensional student attention parameters are represented by W1. ′ It is stated that W1 ′ The parameter dimension is represented by student_embed_dim × teacher_embed_dim. student_embed_dim is the actual student dimension, and teacher_embed_dim is the reference teacher dimension. The dimensionality-reduced student attention parameter is represented by W2. ′ It is stated that W2 ′ The parameter dimension is represented by teacher_embed_dim × student_embed_dim. This will be based on W... ′ =W1 ′ W2 ′ For W1 ′ and W2 ′ Perform matrix multiplication to restore the students' perception parameters to their initial dimensions before dimensionality increase.

[0075] The above technical solution provides a practical dimensionality reduction scheme, realizing the dimensional alignment of student perception parameters and teacher perception parameters in multi-layer perception layers, and providing technical support for the distillation of visual transformation models.

[0076] Figure 3 This is a flowchart of another distillation method for a visual transformation model provided according to an embodiment of this disclosure; this embodiment is an optional solution proposed based on the above embodiments. Specifically, this disclosure refines the operation "determining the distillation loss function of the student model based on the student encoding parameters and the teacher encoding parameters".

[0077] See Figure 3 The distillation method for the visual transformation model provided in this embodiment includes:

[0078] S301, Perform dimensionality processing on the student coding parameters in the image encoder of the student model to align the dimensions of the student coding parameters with those of the teacher coding parameters; the teacher coding parameters belong to the image encoder of the teacher model.

[0079] S302, determine the attention difference between the student model and the teacher model based on the student attention parameter in the student coding parameters and the teacher attention parameter in the teacher coding parameters.

[0080] Attention difference is used to measure the difference in feature representation between the multi-head attention layer in the student model and the multi-head attention layer in the teacher model.

[0081] The parameter dimensions of student attention parameters and teacher attention parameters used to determine attention differences are consistent.

[0082] Optionally, the mean loss (MSELoss) between the student attention parameters and the teacher attention parameters can be calculated, and the MSELoss can be used to quantify the attention difference between the student model and the teacher model.

[0083] S303, determine the perceptual difference between the student model and the teacher model based on the student perceptual ability parameter in the student coding parameters and the teacher perceptual ability parameter in the teacher coding parameters.

[0084] Perceptual difference is used to measure the difference in feature representation between the multi-layer perceptual layer in the student model and the multi-layer perceptual layer in the teacher model.

[0085] The parameter dimensions of student perception parameters and teacher perception parameters used to determine differences in perception are consistent.

[0086] Optionally, the mean loss (MSELoss) between the student's perception parameters and the teacher's perception parameters can be calculated, and the MSELoss can be used to quantify the difference in perception between the student model and the teacher model.

[0087] S304, Determine the distillation loss function of the student model based on the attention difference and the perception difference.

[0088] Optionally, the sum of attentional and perceptual differences can be calculated and used as the distillation loss function for the student model. It is worth noting that, in determining the distillation loss function, the attentional and perceptual differences can be weighted according to actual business needs.

[0089] As we know, a visual transformation model includes multiple image encoders; similarly, the student model and the teacher model also include multiple image encoders. The number of image encoders in the student model is the same as that in the teacher model. Each image encoder has a corresponding distillation loss function. The distillation loss of the entire student model is the sum of the distillation loss functions corresponding to all image encoders.

[0090] Specifically, the distillation loss function for the student model can be determined based on the following formula.

[0091]

[0092] in, This represents the attention difference between the i-th image encoder in the student model and the teacher model; mse_loss() is used to calculate the mean loss. For student attention parameters; This represents the teacher attention parameters. The subscript 's' represents the student model, and the subscript 't' represents the teacher model. N represents the total number of image encoders in the student model and the teacher model, and proj represents a multi-head attention layer.

[0093] This represents the difference in perceptual ability between the i-th image encoder in the student model and the teacher model; Indicates parameters of students' perception. This represents the teacher's perception parameters. fc represents multiple layers of perception.

[0094] S305, determine the student prediction parameters of the student model for the image processing task based on the student coding parameters, and determine the task loss function of the student model based on the student prediction parameters.

[0095] The student coding parameters used to determine the task loss function are dimensionality-reduced so that their dimensionality parameters are consistent with the initial dimensionality before dimensionality increase.

[0096] S306, Based on the distillation loss function and the task loss function, perform distillation training on the student model.

[0097] This disclosed technical solution determines the attention difference based on student and teacher attention parameters, respectively, and uses this difference to determine the distillation loss function; it also determines the perceptual difference based on student and teacher perceptual parameters, and uses this difference to determine the distillation loss function. Then, the distillation loss data is used to train the student model through distillation, achieving distillation of the multi-head attention layer and multi-layer perceptual layer in the image encoder. This embodiment improves the distillation effect of the visual transformation model and enhances the predictive performance of the student model by performing fine-grained distillation on the image encoder.

[0098] In an optional embodiment, determining the student prediction parameters of the student model for the image processing task based on the student coding parameters, and determining the task loss function of the student model based on the student prediction parameters, includes: decoding the student coding parameters through the image decoder of the student model to obtain the student prediction information of the student model for the image processing task; and determining the task loss function of the student model based on the student prediction information and the sample annotation information of the training sample images in the image processing task.

[0099] The image decoder and image encoder work together to decode the student encoding parameters in the image encoder, thereby obtaining the student model's prediction information for the image processing task. This student prediction information is related to the image processing task; in the case of an image classification task, it represents the predicted class probabilities output by the student model for the training sample images. The student prediction information may or may not match the sample annotation information of the training sample images in the image processing task. Based on the difference between the student prediction information and the sample annotation information, the task loss function of the student model can be determined.

[0100] The above technical solution provides a practical and feasible method for determining task loss parameters, and provides data support for subsequently using the business loss function to distill and train student models.

[0101] Figure 4 This is a schematic diagram of a distillation apparatus for a visual transformation model according to an embodiment of this disclosure. This disclosure is applicable to situations involving the compression of visual transformation models. The apparatus can be implemented using software and / or hardware, and can implement the distillation method for visual transformation models described in any embodiment of this disclosure. Figure 4 As shown, the distillation apparatus 400 of the visual transformation model includes:

[0102] The dimension processing module 401 is used to perform dimension processing on the student encoding parameters in the image encoder of the student model, so that the dimensions of the student encoding parameters are aligned with those of the teacher encoding parameters; the teacher encoding parameters belong to the image encoder of the teacher model.

[0103] The distillation loss determination module 402 is used to determine the distillation loss function of the student model based on the student coding parameters and the teacher coding parameters;

[0104] The business loss determination module 403 is used to determine the student prediction parameters of the student model for the image processing task based on the student coding parameters, and to determine the task loss function of the student model based on the student prediction parameters;

[0105] The distillation training module 404 is used to perform distillation training on the student model based on the distillation loss function and the task loss function.

[0106] This embodiment aligns the parameter dimensions of student and teacher encoding parameters. Based on the aligned student and teacher encoding parameters, a distillation loss function is determined for the student model. This distillation loss function is then used to train the student model, enabling the image encoder of the student model to be distilled using the image encoder of the teacher model. This allows the image encoder in the student model to fit the image encoder in the teacher model, improving the computational performance of the student model compared to the teacher model. This embodiment trains the student model using both the distillation loss function and the business loss function. It not only focuses on the local performance of the image encoder in the student model but also considers the global performance of the student model during distillation training, thus improving the distillation effect of the visual transformation model and enhancing the predictive performance of the student model.

[0107] Optionally, the dimension processing module 401 includes: a dimension-up processing submodule, used to increase the dimension of the student coding parameters in the image encoder of the student model, so that the dimensions of the student coding parameters are aligned with those of the teacher coding parameters; and a dimension-reduction processing submodule, used to reduce the dimension of the increased student coding parameters, so that the student coding parameters are restored to their initial dimension before the increase.

[0108] Optionally, the dimensionality-upgrading submodule includes: an attention layer dimensionality-upgrading unit, used to upgrade the student attention parameter in the student encoding parameters through the first linear mapping module in the multi-head attention layer of the student model, so that the dimension of the student attention parameter is aligned with the dimension of the teacher attention parameter in the teacher encoding parameters; and a multi-layer perception layer dimensionality-upgrading unit, used to upgrade the student perception parameter in the student encoding parameters through the first fully connected layer in the multi-layer perception layer of the student model, so that the dimension of the student perception parameter is aligned with the dimension of the teacher perception parameter in the teacher encoding parameters; wherein, the multi-head attention layer and the multi-layer perception layer belong to the image encoder.

[0109] Optionally, the dimensionality reduction processing submodule includes: an attention layer dimensionality reduction unit, used to reduce the dimensionality of the student attention parameters after dimensionality increase through the second linear mapping module in the multi-head attention layer of the student model, so that the student attention parameters are restored to the initial dimension before dimensionality increase; and a multi-layer perception layer dimensionality reduction unit, used to reduce the dimensionality of the student perception parameters after dimensionality increase through the second fully connected module in the multi-layer perception layer of the student model, so that the student perception parameters are restored to the initial dimension before dimensionality increase.

[0110] Optionally, the distillation loss determination module 402 includes: an attention difference determination submodule, used to determine the attention difference between the student model and the teacher model based on the student attention parameter in the student encoding parameters and the teacher attention parameter in the teacher encoding parameters; a perception difference determination submodule, used to determine the perception difference between the student model and the teacher model based on the student perception parameter in the student encoding parameters and the teacher perception parameter in the teacher encoding parameters; and a distillation loss determination submodule, used to determine the distillation loss function of the student model based on the attention difference and the perception difference.

[0111] Optionally, the business loss determination module 403 includes: a student prediction determination submodule, used to decode the student encoding parameters through the image decoder of the student model to obtain student prediction information of the student model for the image processing task; and a task loss determination submodule, used to determine the task loss function of the student model based on the student prediction information and the sample annotation information of the training sample images in the image processing task.

[0112] Optionally, the distillation training module 404 includes: a first distillation training submodule, used to perform distillation training on the image encoder in the student model based on the distillation loss function; and a second distillation training submodule, used to perform distillation training on all components of the student model based on the task loss function.

[0113] The distillation apparatus for the visual transformation model provided in this disclosure can execute the distillation method for the visual transformation model provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the distillation method for the visual transformation model.

[0114] The collection, storage, use, processing, transmission, provision, and disclosure of user information involved in the technical solution disclosed herein all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0115] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0116] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0117] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0118] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the distillation method of a visual transformation model. For example, in some embodiments, the distillation method of a visual transformation model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the distillation method of the visual transformation model described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the distillation method of the visual transformation model by any other suitable means (e.g., by means of firmware).

[0120] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a distillation apparatus of a general-purpose computer, special-purpose computer, or other programmable visual transformation model, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0124] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0125] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0126] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0127] Cloud computing refers to a technology system that enables access to a shared pool of physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.

[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A distillation method for a visual transformation model, the method comprising: The dimensions of the student encoding parameters in the image encoder of the student model are processed to align the dimensions of the student encoding parameters with those of the teacher encoding parameters. The teacher coding parameters belong to the image encoder of the teacher model; The distillation loss function of the student model is determined based on the student coding parameters and the teacher coding parameters. Based on the student coding parameters, the student prediction parameters of the student model for the image processing task are determined, and based on the student prediction parameters, the task loss function of the student model is determined. The student model is trained by distillation based on the distillation loss function and the task loss function; wherein the task loss function is determined based on the prediction information of the student model for the image processing task and the real information of the image processing task; the image processing task is an object recognition task or an image classification task. Specifically, the dimensionality processing of student coding parameters in the image encoder of the student model to align the dimensions of the student coding parameters with those of the teacher coding parameters includes: increasing the dimensionality of the student coding parameters in the image encoder of the student model to align the dimensions of the student coding parameters with those of the teacher coding parameters; and reducing the dimensionality of the increased student coding parameters to restore them to their initial dimensions before the increase. Specifically, determining the distillation loss function of the student model based on the student encoding parameters and the teacher encoding parameters includes: determining the attention difference between the student model and the teacher model based on the student attention parameter in the student encoding parameters and the teacher attention parameter in the teacher encoding parameters; determining the perception difference between the student model and the teacher model based on the student perception parameter in the student encoding parameters and the teacher perception parameter in the teacher encoding parameters; and determining the distillation loss function of the student model based on the attention difference and the perception difference.

2. The method according to claim 1, wherein, Upsizing the student encoding parameters in the image encoder of the student model to align the dimensions of the student encoding parameters with those of the teacher encoding parameters includes: The student attention parameters in the student encoding parameters are increased in dimensionality by the first linear mapping module in the multi-head attention layer of the student model, so that the dimensions of the student attention parameters are aligned with the dimensions of the teacher attention parameters in the teacher encoding parameters. The student perception parameter in the student coding parameters is upgraded by the first fully connected layer in the multi-layer perceptual layer of the student model, so that the dimension of the student perception parameter is aligned with the dimension of the teacher perception parameter in the teacher coding parameters; wherein, the multi-head attention layer and the multi-layer perceptual layer belong to the image encoder.

3. The method according to claim 2, wherein, The dimensionality of the student coding parameters after dimensionality increase is reduced to their initial dimension before dimensionality increase, including: The second linear mapping module in the multi-head attention layer of the student model reduces the dimensionality of the student attention parameters after dimensionality increase, so that the student attention parameters are restored to the initial dimension before dimensionality increase. The second fully connected module in the multi-layer perception layer of the student model performs dimensionality reduction on the student's perception parameters after dimensionality increase, so that the student's perception parameters are restored to the initial dimension before dimensionality increase.

4. The method according to claim 1, wherein, Based on the student coding parameters, the student prediction parameters of the student model for the image processing task are determined, and based on the student prediction parameters, the task loss function of the student model is determined, including: The student encoding parameters are decoded by the image decoder of the student model to obtain the student prediction information of the student model for the image processing task; The task loss function of the student model is determined based on the student prediction information and the sample annotation information of the training sample images in the image processing task.

5. The method according to claim 1, wherein, Based on the distillation loss function and the task loss function, the student model is trained by distillation, including: The image encoder in the student model is trained by distillation based on the distillation loss function. Distillation training is performed on all components of the student model based on the task loss function.

6. A distillation apparatus for a visual transformation model, the apparatus comprising: The dimension processing module is used to perform dimension processing on the student encoding parameters in the image encoder of the student model, so that the dimensions of the student encoding parameters are aligned with those of the teacher encoding parameters; the teacher encoding parameters belong to the image encoder of the teacher model. The distillation loss determination module is used to determine the distillation loss function of the student model based on the student coding parameters and the teacher coding parameters; The business loss determination module is used to determine the student prediction parameters of the student model for the image processing task based on the student coding parameters, and to determine the task loss function of the student model based on the student prediction parameters. The distillation training module is used to perform distillation training on the student model based on the distillation loss function and the task loss function; wherein, the task loss function is determined based on the prediction information of the student model for the image processing task and the real information of the image processing task; the image processing task is an object recognition task or an image classification task; The dimension processing module includes: a dimension-upgrading submodule, used to upgrade the student coding parameters in the image encoder of the student model so that the dimensions of the student coding parameters are aligned with those of the teacher coding parameters; and a dimension-reduction submodule, used to reduce the dimensions of the upgraded student coding parameters so that the student coding parameters are restored to their initial dimensions before the upgrade. The distillation loss determination module includes: an attention difference determination submodule, used to determine the attention difference between the student model and the teacher model based on the student attention parameter in the student encoding parameters and the teacher attention parameter in the teacher encoding parameters; a perception difference determination submodule, used to determine the perception difference between the student model and the teacher model based on the student perception parameter in the student encoding parameters and the teacher perception parameter in the teacher encoding parameters; and a distillation loss determination submodule, used to determine the distillation loss function of the student model based on the attention difference and the perception difference.

7. The apparatus according to claim 6, wherein, The dimension-up processing submodule includes: The attention layer dimension-upgrading unit is used to upgrade the student attention parameters in the student encoding parameters through the first linear mapping module in the multi-head attention layer of the student model, so that the dimensions of the student attention parameters are aligned with the dimensions of the teacher attention parameters in the teacher encoding parameters. The multi-layer perceptual layer dimensionality-upgrading unit is used to upgrade the student perceptual ability parameter in the student encoding parameters through the first fully connected layer in the multi-layer perceptual layer of the student model, so that the student perceptual ability parameter is aligned with the dimension of the teacher perceptual ability parameter in the teacher encoding parameters; wherein, the multi-head attention layer and the multi-layer perceptual layer belong to the image encoder.

8. The apparatus according to claim 7, wherein, The dimensionality reduction processing submodule includes: The attention layer dimensionality reduction unit is used to reduce the dimensionality of the student attention parameters after dimensionality increase through the second linear mapping module in the multi-head attention layer of the student model, so that the student attention parameters are restored to the initial dimension before dimensionality increase. The multi-layer perception layer dimensionality reduction unit is used to reduce the dimensionality of the student's perception parameters after dimensionality increase through the second fully connected module in the multi-layer perception layer of the student model, so that the student's perception parameters are restored to the initial dimension before dimensionality increase.

9. The apparatus according to claim 6, wherein, The business loss determination module includes: The student prediction determination submodule is used to decode the student encoding parameters through the image decoder of the student model to obtain the student prediction information of the student model for the image processing task; The task loss determination submodule is used to determine the task loss function of the student model based on the student prediction information and the sample annotation information of the training sample images in the image processing task.

10. The apparatus according to claim 6, wherein, The distillation training module includes: The first distillation training submodule is used to perform distillation training on the image encoder in the student model based on the distillation loss function. The second distillation training submodule is used to perform distillation training on all components of the student model based on the task loss function.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the distillation method of the visual transformation model according to any one of claims 1-5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the distillation method of the visual transformation model according to any one of claims 1-5.

13. A computer program product comprising a computer program that, when executed by a processor, implements the distillation method for a visual transformation model according to any one of claims 1-5.

Images