Knowledge distillation method, device, equipment, storage medium and program product

Abstract

The invention discloses a knowledge distillation method, device and equipment, a storage medium and a program product, and the method comprises the steps: carrying out the model compression of a first teacher model generated through the training of a target sample set, and obtaining a first student model; wherein the model parameter quantity of the first student model is smaller than the model parameter quantity of the first teacher model; performing random masking on a first feature map output by the first student model to obtain a first mask feature map; performing feature reconstruction on the first mask feature map to obtain a first reconstructed feature map; determining a first distillation loss based on a difference between the first reconstructed feature map and a second feature map output by the first teacher model; determining a first loss function based on the first distillation loss and a first predicted loss of the first student model; optimizing the first student model based on the first loss function, and performing performance test on the optimized first student model based on the target sample set; wherein the first teacher model and the first student model are used for a target detection task.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a knowledge distillation method, apparatus, device, storage medium, and program product. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technologies, computer vision has been widely used in household robots such as sweepers. Due to the stringent computing and power requirements of sweepers, visual models must be lightweight while maintaining high detection performance. Therefore, a knowledge distillation method has been proposed to improve the student model's ability to simulate the teacher model. However, this knowledge distillation method in related technologies suffers from a conflict between lightweight models and high model accuracy, resulting in poor knowledge transfer. Summary of the Invention

[0003] To solve the above technical problems, the embodiments of the present application provide a knowledge distillation method, device, equipment, storage medium and program product.

[0004] The knowledge distillation method provided in the embodiments of the present application includes:

[0005] Performing model compression on a first teacher model generated by training the target sample set to obtain a first student model; wherein the model parameters of the first student model are smaller than the model parameters of the first teacher model;

[0006] Randomly masking the first feature map output by the first student model to obtain a first masked feature map; performing feature reconstruction on the first masked feature map to obtain a first reconstructed feature map;

[0007] Determining a first distillation loss based on a difference between the first reconstructed feature map and a second feature map output by the first teacher model; determining a first loss function based on the first distillation loss and a first prediction loss of the first student model;

[0008] The first student model is optimized based on the first loss function, and the performance of the optimized first student model is tested based on the target sample set; wherein the first teacher model and the first student model are used for target detection tasks.

[0009] The knowledge distillation device provided in the embodiment of the present application includes:

[0010] a processing unit, configured to perform model compression on a first teacher model generated by training a target sample set to obtain a first student model; wherein the model parameter quantity of the first student model is smaller than the model parameter quantity of the first teacher model; perform random masking on a first feature map output by the first student model to obtain a first masked feature map; and perform feature reconstruction on the first masked feature map to obtain a first reconstructed feature map;

[0011] a determining unit, configured to determine a first distillation loss based on a difference between the first reconstructed feature map and a second feature map output by the first teacher model; and determine a first loss function based on the first distillation loss and a first prediction loss of the first student model;

[0012] The processing unit is also used to optimize the first student model based on the first loss function, and perform performance testing on the optimized first student model based on the target sample set; wherein the first teacher model and the first student model are used for target detection tasks.

[0013] The processing device provided in an embodiment of the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any one of the above-mentioned knowledge distillation methods.

[0014] The computer-readable storage medium provided in an embodiment of the present application is used to store a computer program, which enables a computer to execute any one of the above-mentioned knowledge distillation methods.

[0015] The computer program product provided in an embodiment of the present application includes computer program instructions, which enable a computer to execute any of the above-mentioned knowledge distillation methods.

[0016] In the technical solution of the embodiment of the present application, the first teacher model generated by training the target sample set is compressed to obtain a first student model; the first feature map output by the first student model is randomly masked to obtain a first masked feature map; the first masked feature map is feature reconstructed to obtain a first reconstructed feature map; based on the difference between the first reconstructed feature map and the second feature map output by the first teacher model, a first distillation loss is determined; based on the first distillation loss and the first prediction loss of the first student model, a first loss function is determined; the first student model is optimized based on the first loss function, and the performance of the optimized first student model is tested based on the target sample set; wherein, the model parameter amount of the first student model is less than the model parameter amount of the first teacher model; the first teacher model and the first student model are used for target detection tasks. In this way, by randomly masking a certain proportion of the student model features and restoring the original feature points of the masked features through a two-layer convolutional network to imitate the teacher model features, the distillation process can be made more focused on the effective feature transfer of the target area, and the student model is forced to learn how to extract and restore key features from limited information during training. This not only improves the student model's ability to simulate the teacher model's features, but also maintains the high precision of the student model while keeping it lightweight, thereby effectively improving the performance of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process of the knowledge distillation method provided in the embodiment of the present application;

[0018] Figure 2 This is a flow chart of a knowledge distillation framework for a sweeper target detection model based on random masking and feature reconstruction provided by an embodiment of the present application;

[0019] Figure 3 This is a schematic diagram of the result of obstacle detection performed on an input image by the teacher model provided in an embodiment of the present application;

[0020] Figure 4 Schematic diagram of the structure of the knowledge distillation device provided in the embodiment of the present application;

[0021] Figure 5 It is a structural diagram of the processing equipment provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and being indicated, configuration and being configured, etc.

[0024] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following relevant technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all fall within the protection scope of the embodiments of the present application.

[0025] With the rapid development of artificial intelligence and deep learning technologies, computer vision algorithms have been widely used in various smart devices, especially in household robots such as sweepers. During operation, sweepers must not only detect obstacles in real time and accurately, but also identify furniture and distinguish floor materials to ensure efficient and safe operation. However, in actual deployments, due to the strict computing resource and power requirements of sweepers, vision models must be lightweight while maintaining high detection performance. To address this, model compression techniques such as model pruning are widely used to reduce model parameters, but model compression often leads to a decrease in model detection performance.

[0026] Knowledge distillation technology was proposed in this context. It achieves the goal of compressing the model while maintaining detection performance as much as possible by transferring knowledge from a large-scale, high-precision teacher model to a student model with fewer parameters. However, related knowledge distillation methods directly align all features of the teacher and student. In actual object detection tasks, this can lead to poor knowledge transfer due to an imbalance in foreground and background pixel information. Specific limitations include:

[0027] (1) The contradiction between lightweight and high precision: When deploying a sweeper, it is necessary to achieve high-precision target detection with limited computing resources. Although traditional pruning methods can reduce the number of parameters, they are usually accompanied by performance degradation.

[0028] (2) Limitations of direct feature alignment: Directly aligning all features of the teacher model and the student model has the problem of interference from foreground and background information, which may lead to insufficient knowledge transfer from the teacher model.

[0029] To solve the above technical problems, this application proposes a knowledge distillation framework based on random masking and feature reconstruction. This method randomly blocks a certain proportion (such as 65%) of the student model features and restores the original feature points of the blocked features through a two-layer convolutional network to imitate the teacher features. Since the features in computer vision tasks usually have high redundancy, even if a large number of pixels are blocked, the remaining information is still sufficient to restore the overall features. Therefore, the two-layer convolution module is sufficient to complete the feature reconstruction task, so that the distillation process focuses more on the effective feature transfer of the target area, and improves the student model's ability to simulate the teacher model features.

[0030] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0031] This application embodiment proposes a knowledge distillation method. Figure 1 This is a flow chart of the knowledge distillation method provided in the embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0032] Step 101: compress the first teacher model generated by training the target sample set to obtain a first student model.

[0033] Among them, the model parameter amount of the first student model is smaller than the model parameter amount of the first teacher model.

[0034] In an embodiment of the present application, the first teacher model generated by training the target sample set is a fully trained and well-performing model that can achieve a high level of performance on a specific task. The first teacher model is usually highly complex and large in scale, requiring a large amount of computing resources and training time. Therefore, by compressing the first teacher model, the storage and transmission overhead of the model can be reduced to streamline the model and reduce the model size, thereby obtaining a smaller and simpler first student model. Among them, the target sample set can be an image sample set, and the image sample set can be an image sample set in the field of medical imaging, or an image sample set in the field of autonomous driving, or an image sample set in the field of safety and control, etc., which are not limited here.

[0035] Here, the first student model has fewer model parameters than the first teacher model, and its structure is simpler than that of the first teacher model, which helps reduce its computational complexity and memory usage. Due to the smaller size of the first student model, its inference speed is generally faster, enabling it to complete prediction tasks in a shorter time. Furthermore, by learning by imitating the output or feature representation of the first teacher model, the first student model can, to a certain extent, achieve or approach the performance of the first teacher model.

[0036] In some implementations, step 101 may specifically include:

[0037] Perform parameter pruning on the first teacher model to obtain the first student model; or,

[0038] Quantize the parameters of the first teacher model to obtain a first student model; or,

[0039] Perform parameter pruning on the first teacher model to obtain a pruned first teacher model; perform parameter quantization on the pruned first teacher model to obtain a first student model; or,

[0040] The target distillation knowledge is determined based on the first teacher model, and the model is trained based on the target distillation knowledge and the target sample set to obtain the first student model.

[0041] Here, since there are many model compression methods, one or more of these model compression methods can be used to compress the first teacher model. Model compression methods mainly include pruning, quantization, knowledge distillation, low-rank decomposition and other methods.

[0042] Specifically, the first student model can be obtained by performing parameter pruning on the first teacher model, thereby reducing the size and computational complexity of the model by removing unimportant parameters in the first teacher model. Unimportant parameters generally refer to those with small weights or small impact on the model output. The parameter pruning process may include:

[0043] (1) Determine the pruning criteria: Select an appropriate pruning criterion, such as based on the size of the weight, the size of the gradient, or other importance indicators.

[0044] (2) Pruning operation: According to the determined pruning criteria, unimportant parameters in the first teacher model are set to zero or directly removed.

[0045] (3) Fine-tuning the model: The pruned model may cause performance degradation, so it is necessary to fine-tune the pruned model to restore the model performance and obtain the first student model.

[0046] Alternatively, the first student model can be obtained by quantizing the parameters of the first teacher model and converting the parameter representation in the first teacher model from high precision (such as 32-bit floating point numbers) to low precision (such as 8-bit integers or less) to reduce the storage size and computational complexity of the model. The parameter quantization process may include:

[0047] (1) Determine the quantization method: Select an appropriate quantization method, such as linear quantization, nonlinear quantization, etc.

[0048] (2) Quantization parameters: Quantize the parameters of the first teacher model according to the selected quantization method.

[0049] (3) Fine-tuning the model: The quantized model may need to adjust the model structure and retrain or fine-tune it to adapt to the changes brought about by the quantization operation, thereby obtaining the first student model.

[0050] Alternatively, the first teacher model can be pruned first, and then the pruned first teacher model can be quantized to obtain the first student model. By combining parameter pruning and parameter quantization, unimportant parameters can be pruned first, and then the remaining parameters can be quantized. This can further reduce the size and computational complexity of the model while improving the inference speed of the model. The execution process may include:

[0051] (1) Pruning operation: Parameter pruning is performed on the first teacher model according to the parameter pruning method to obtain the pruned first teacher model.

[0052] (2) Quantization operation: Quantize the parameters of the pruned first teacher model and convert the remaining parameters into low-precision representation.

[0053] (3) Fine-tuning the model: Fine-tune the pruned and quantized model to restore the performance of the model, thereby obtaining the first student model.

[0054] Alternatively, knowledge distillation can be performed on the first teacher model to obtain a first student model. By allowing the first student model to learn the output of the first teacher model, the knowledge and performance of the first teacher model can be retained in a smaller model. The execution process may include:

[0055] (1) Design the student model structure: Determine the structure of the first student model, which is usually smaller than the first teacher model in order to be more efficient in terms of computation and storage.

[0056] (2) Select the distillation target: Determine the target knowledge to be distilled, such as the output label distribution of the first teacher model, the feature map of the intermediate layer, etc.

[0057] (3) Train the student model: Based on the target distilled knowledge and the target sample set, and combined with the output of the first teacher model as an additional supervision signal, the first student model is trained. During the training process, different loss functions, such as mean squared error and cross entropy, can be used to measure the difference between the output of the first student model and the first teacher model.

[0058] Alternatively, the first teacher model can be compressed in blocks, that is, the first teacher model is divided into multiple modules, each of which is relatively independent in function. Each module is then compressed separately, and each compressed module is combined to obtain the first student model. For example, the first teacher model can be divided according to different functional modules, such as feature extraction module, classification module, etc., and then a corresponding compression method is used according to the characteristics of each module. For the feature extraction module, feature graph clustering and quantization methods can be used to reduce the amount of calculation and storage. For the classification module, weight sharing and pruning methods can be used to optimize the model.

[0059] Alternatively, the structure of the first teacher model can be rearranged and compressed. That is, the computational graph of the first teacher model is analyzed to determine rearrangeable structural patterns, and then the first teacher model is optimized through rearrangement to obtain the first student model. For example, two adjacent convolutional layers in the first teacher model can be fused into one convolutional layer, or a loaded convolutional layer can be decomposed into multiple smaller convolutional layers.

[0060] Alternatively, the first teacher model can be subjected to low-rank decomposition to obtain a first student model by decomposing the weight matrix of the first teacher model into a product of low-rank matrices to reduce the parameter size of the model. Low-rank decomposition can remove redundant information in the weight matrix of the first teacher model while retaining important features and patterns. The execution process may include:

[0061] (1) Select decomposition method: Determine the appropriate matrix decomposition method, such as singular value decomposition, truncated singular value decomposition, etc.

[0062] (2) Decomposition of weight matrix: Decompose the weight matrix in the first teacher model to obtain a low-rank approximate matrix.

[0063] (3) Reconstructing the model: Use the decomposed low-rank matrix to reconstruct the first student model, and adjust and train the reconstructed model.

[0064] Alternatively, the first student model can be obtained by performing a network structure search on the first teacher model. By automatically searching for the optimal network structure, the performance of the model can be improved as much as possible while satisfying the constraints of model size and computational complexity. By evaluating and selecting different network structures in the search space, a structure suitable for the first student model is found. The execution process may include:

[0065] (1) Define the search space: determine the possible combinations and variation ranges of the network structure, including the number of network layers, number of channels, convolution kernel size, etc.

[0066] (2) Select a search strategy: Select an appropriate search strategy, such as reinforcement learning, genetic algorithm, gradient descent, etc.

[0067] (3) Evaluation and structure selection: Generate candidate network structures in the search space, use the knowledge or data of the first teacher model to evaluate their performance, and select the optimal structure as the first student model.

[0068] Step 102: Randomly mask the first feature map output by the first student model to obtain a first masked feature map.

[0069] In an embodiment of the present application, after obtaining the first student model, the first feature map output by the first student model can be randomly masked, and some features in the first feature map can be shielded at a certain ratio to obtain a first mask feature map.

[0070] In some implementations, before step 102, the following steps may also be included:

[0071] Input the target sample set into the first student model for forward propagation, and extract the first feature map output by the target layer during the forward propagation process;

[0072] The target sample set is input into the first teacher model for forward propagation, and the second feature map output by the target layer is extracted during the forward propagation process.

[0073] Here, the hook function can be added to the target layer in the first student model and the first teacher model, and then the target sample set is input into the first student model for forward propagation. When it runs to the target layer during the forward propagation of the first student model, the hook function is automatically triggered to extract the output of the target layer, which is the first feature map. Similarly, the target sample set is input into the first teacher model for forward propagation. When it runs to the target layer during the forward propagation of the first teacher model, the hook function is automatically triggered to extract the output of the target layer, which is the second feature map.

[0074] Hook functions are a programming technique that extends or modifies existing behavior by inserting custom code at specific points in program execution. The core principle is to pre-set, mountable "hook points" that allow callback functions to be registered at these locations, automatically triggering these functions when the program reaches the corresponding stage.

[0075] In some implementations, step 102 may specifically include:

[0076] Determine, based on the first feature map, a binary mask map having the same size as the first feature map; wherein each element in the binary mask map is selected with a first probability, and each element in the binary mask map is not selected with a second probability;

[0077] The first feature map and the binary mask map are element-wise multiplied to obtain the first mask feature map.

[0078] Here, first, the first feature map output by the first student model is input into the pre-trained segmentation model, and the mask area is predicted from the first feature map by the segmentation model, thereby obtaining a binary mask map of the same size as the first feature map; secondly, a random sampling technique is used to set each element in the binary mask map to be selected with a first probability (selected is 1), and each element in the binary mask map is set to be unselected with a second probability (unselected is 0), wherein the sum of the first probability and the second probability is equal to 1, and the magnitude of the second probability is greater than the magnitude of the first probability; then, the first feature map is feature-masked according to the binary mask map, that is, the first feature map and the binary mask map are element-by-element multiplied to obtain the first mask feature map. The feature masking process formula is as follows:

[0079]

[0080] in, represents the Hadamard product (element-wise multiplication), F masked Represents the first mask feature map, F S Represents the first feature map, and M represents the binary mask map.

[0081] It should be noted that the segmentation model can be any artificial intelligence model for image segmentation, such as Fully Convolutional Networks (FCN), Mask Region-based Convolutional Neural Network (Mask R-CNN), etc., and is not limited here.

[0082] Step 103: reconstruct the first mask feature map to obtain a first reconstructed feature map.

[0083] In an embodiment of the present application, after obtaining the first mask feature map, feature reconstruction can be performed on the first mask feature map. Based on the unmasked features in the first mask feature map, the masked features in the first mask feature map can be restored through a preset algorithm (such as a two-layer convolutional network) to obtain a first reconstructed feature map.

[0084] In some implementations, step 103 may specifically include:

[0085] Perform a convolution operation on each convolution kernel included in the first convolution layer and the first mask feature map to obtain a third feature map;

[0086] Performing a nonlinear transformation on the third feature map to obtain a first activation feature map;

[0087] Based on the second convolutional layer, feature reconstruction mapping is performed on the first activated feature map to obtain a first reconstructed feature map.

[0088] Here, we first design the first convolutional layer and determine the number and size of convolution kernels. A convolution kernel is typically a small two-dimensional matrix such as 3×3 or 5×5. The kernel weights are randomly initialized, typically using methods such as Xavier initialization or He initialization, to ensure that the network can effectively propagate gradients and learn. Each convolution kernel in the first convolutional layer is convolved with the first masked feature map to generate a third feature map. For each convolution kernel, a sub-feature map is generated by sliding it across the image and calculating the dot product between the kernel and the local region of the image. The sum of the sub-feature maps generated by convolving each convolution kernel with the first masked feature map is the third feature map.

[0089] For example, assuming that the size of the first mask feature map is H×W×C (height×width×number of channels), the size of the convolution kernel is K×K×C, and the convolution step is S, then the size of the sub-feature map output after the convolution kernel is convolved with the first mask feature map is [(HK) / S+1]×[(WK) / S+1].

[0090] Then, an activation function such as the Rectified Linear Unit (ReLU) is applied to the third feature map to perform a nonlinear transformation to obtain the first activated feature map. This introduces nonlinear factors, enabling the network to learn more complex feature representations. The ReLU function sets all negative values ​​to zero and keeps positive values ​​unchanged. Its calculation formula can be expressed as:

[0091] f(x)=max(0,1) (2)

[0092] Then, feature reconstruction mapping is performed on the first activated feature map according to the second convolutional layer to obtain a first reconstructed feature map.

[0093] In some embodiments, performing feature reconstruction mapping on the first activation feature map based on the second convolutional layer to obtain the first reconstructed feature map may specifically include:

[0094] Performing a convolution operation on each convolution kernel included in the second convolution layer and the first activation feature map to obtain a fourth feature map; wherein the number of convolution kernels included in the second convolution layer is greater than the number of convolution kernels included in the first convolution layer;

[0095] Performing a nonlinear transformation on the fourth feature map to obtain a second activation feature map;

[0096] Perform feature reconstruction mapping on the second activated feature map to obtain a first reconstructed feature map.

[0097] Here, first set up the second convolution layer and determine the number and size of the convolution kernels. The number of convolution kernels in the second convolution layer is usually greater than that in the first convolution layer to extract higher-level features, and the size of the convolution kernels in the second convolution layer may be the same as or different from that of the convolution kernels in the first convolution layer. Similarly, the weight parameters of the convolution kernels in the second convolution layer are initialized to ensure that the network can effectively propagate gradients and learn. Each convolution kernel included in the second convolution layer is convolved with the first activation feature map to obtain a fourth feature map, wherein for each convolution kernel, sliding on the image and calculating the dot product of the convolution kernel and the local area of ​​the image will generate a sub-feature map. The sum of the sub-feature maps generated after the convolution kernel is convolved with the first activation feature map is the fourth feature map.

[0098] Secondly, an activation function such as ReLU is applied again to perform nonlinear transformation on the fourth feature map to obtain the second activated feature map, so as to further enhance the expression ability and nonlinear fitting ability of the model.

[0099] The second activated feature map is then mapped and spatially transformed. Upsampling methods, such as bilinear interpolation or deconvolution, can be used to gradually restore the size of the second activated feature map to a size similar to that of the input image (the first mask feature map) while maintaining the semantic information of the feature. In addition, structures such as skip connections can be used to fuse the features output by the previous and next layers to retain more detailed information, which helps to reconstruct the features more accurately.

[0100] In the final stage of feature reconstruction, an output layer is designed, typically a convolutional layer. This convolutional layer performs a final convolution operation on the mapped and transformed feature map to generate the first reconstructed feature map. An activation function such as Sigmoid or Tanh can be used in the output layer to constrain the output values ​​(the first reconstructed feature map) to a reasonable range. For example, the Sigmoid function maps output values ​​to the range [0, 1], making it suitable for image reconstruction with pixel values ​​between 0 and 255.

[0101] Step 104: Determine a first distillation loss based on the difference between the first reconstructed feature map and the second feature map output by the first teacher model.

[0102] In an embodiment of the present application, after obtaining the first reconstructed feature map, in order to determine the accuracy of feature reconstruction, the difference between the first reconstructed feature map and the second feature map output by the first teacher model can be compared, and the error measurement method can be used to align the first reconstructed feature map and the second feature map, and the first distillation loss between the first reconstructed feature map and the second feature map is calculated.

[0103] For example, the mean-square error (MSE) loss can be used to align the first reconstructed feature map and the second feature map to calculate the first distillation loss. The corresponding calculation formula is as follows:

[0104]

[0105] Among them, N = H × W × C represents the total number of elements in the feature map, L distill represents the first distillation loss, F rec Denotes the first reconstructed feature map, F T The second feature map representing the output of the first teacher model.

[0106] Step 105: Determine a first loss function based on the first distillation loss and the first prediction loss of the first student model.

[0107] In an embodiment of the present application, after obtaining the first distillation loss, the target sample set is input into the first student model, and the first prediction loss of the first student model is calculated based on the error between the prediction result output by the first student model and the true label in the target sample set. Then, a first loss function is constructed based on the first distillation loss and the first prediction loss of the first student model.

[0108] For example, the first loss function constructed according to the first distillation loss and the first prediction loss of the first student model is defined as follows:

[0109] L total =L stu +λ·L distill (4)

[0110] Where λ represents the knowledge distillation weight, which is used to balance the first prediction loss L of the first student model stu and distillation loss L distill , L total Represents the first loss function, that is, the first loss function of the knowledge distillation framework.

[0111] Step 106: Optimize the first student model based on the first loss function, and perform a performance test on the optimized first student model based on the target sample set.

[0112] Among them, the first teacher model and the first student model are used for target detection tasks.

[0113] In an embodiment of the present application, after obtaining the first loss function, the first student model is optimized according to the first loss function to obtain the optimized first student model, and then the target sample set is input into the optimized first student model for the prediction task, and the performance index of the optimized first student model is evaluated based on the error between the output prediction result and the true label in the target sample set. When the performance index does not reach the preset threshold, the optimized first student model needs to be optimized again (or iteratively optimized) according to the first loss function until the performance index of the first student model after N times (N is greater than or equal to 1) reaches the preset threshold. Among them, the preset threshold is flexibly adjusted according to different application scenarios and is not limited here.

[0114] It should be noted that in addition to the above-mentioned situation of using a teacher model to guide the learning of the student model, multiple teacher models can also be used to form an integrated model to guide the learning of the student model. Different teacher models can provide knowledge of different types or with different focuses, so that the student model can more comprehensively simulate the behavior of the teacher model.

[0115] In some embodiments, the above method may further include:

[0116] Integrate multiple second teacher models generated by training the target sample set to obtain an integrated model;

[0117] Performing model compression on the integrated model to obtain a second student model; wherein the model parameters of the second student model are smaller than the model parameters of the integrated model;

[0118] Randomly masking the fifth feature map output by the second student model to obtain a second masked feature map; performing feature reconstruction on the second masked feature map to obtain a second reconstructed feature map;

[0119] Determine a second distillation loss based on the second reconstructed feature map and the sixth feature map output by the integrated model; determine a second loss function based on the second distillation loss and the second prediction loss of the second student model;

[0120] The second student model is optimized based on the second loss function, and the performance of the optimized second student model is tested based on the target sample set.

[0121] Here, first, the target sample set is used to train and generate multiple different second teacher models, each of which will show different performance and characteristics on a specific task, and multiple second teacher models are integrated to obtain an integrated model; secondly, the integrated model is compressed to obtain a second student model whose model parameters are smaller than those of the integrated model, and the target sample set is input into the second student model and the integrated model respectively, and the fifth feature map and the sixth feature map output by the target layer are extracted during the forward propagation of the second student model and the integrated model respectively; then the fifth feature map output by the second student model is randomly masked to obtain a second masked feature map, and the second masked feature map is feature reconstructed to obtain a second reconstructed feature map; then the difference between the second reconstructed feature map and the sixth feature map output by the integrated model is compared to calculate the second distillation loss between the second reconstructed feature map and the sixth feature map, and a second loss function is jointly constructed based on the second distillation loss and the second prediction loss of the second student model; finally, the second student model is optimized according to the second loss function, and the performance of the optimized second student model is tested based on the target sample set.

[0122] It should be noted that the detailed process of model compression, random masking and feature reconstruction here can refer to the above method process and will not be elaborated here.

[0123] In the technical solution of the embodiment of the present application, the first teacher model generated by training the target sample set is compressed to obtain a first student model; the first feature map output by the first student model is randomly masked to obtain a first masked feature map; the first masked feature map is feature reconstructed to obtain a first reconstructed feature map; based on the difference between the first reconstructed feature map and the second feature map output by the first teacher model, a first distillation loss is determined; based on the first distillation loss and the first prediction loss of the first student model, a first loss function is determined; the first student model is optimized based on the first loss function, and the performance of the optimized first student model is tested based on the target sample set; wherein, the model parameter amount of the first student model is less than the model parameter amount of the first teacher model; the first teacher model and the first student model are used for target detection tasks. In this way, by randomly masking a certain proportion of the student model features and restoring the original feature points of the masked features through a two-layer convolutional network to imitate the teacher model features, the distillation process can be made more focused on the effective feature transfer of the target area, and the student model is forced to learn how to extract and restore key features from limited information during training. This not only improves the student model's ability to simulate the teacher model's features, but also maintains the high precision of the student model while keeping it lightweight, thereby effectively improving the performance of the student model.

[0124] This embodiment of the application also proposes a knowledge distillation framework for a sweeper robot object detection model based on random masking and feature reconstruction. The key technical points of this knowledge distillation framework include:

[0125] (1) Random masking strategy. A random masking technique is used on the feature map of the student model to mask a certain proportion (e.g., 65%) of the features, thereby selectively retaining key information and removing redundant content. This strategy exploits the inherent feature redundancy in computer vision tasks and ensures that even if a large number of features are masked, sufficient information can be retained for subsequent feature recovery and learning.

[0126] (2) Feature reconstruction module. A module consisting of two convolutional layers is designed to recover the masked features. This module is responsible for completing the masked feature information so that the student model can be as close as possible to the teacher model’s feature expression during the reconstruction process, thereby effectively achieving knowledge transfer and imitation of the teacher model.

[0127] (3) Knowledge distillation loss design. By comparing the differences between the reconstructed student features and the features of the corresponding layers of the teacher model, a special distillation loss function is constructed to guide training. This loss function is mainly used to constrain the performance of the student model in recovering key features, thereby maintaining or improving the detection performance of the student model while making it lightweight.

[0128] Figure 2 This is a flow chart of the knowledge distillation framework of the sweeper target detection model based on random mask and feature reconstruction provided by the embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0129] Step 201: Train a teacher model based on a large-scale sweeper obstacle detection dataset.

[0130] The YOLO model is trained using a large-scale robot vacuum obstacle detection dataset to obtain a teacher model. The teacher model can achieve high accuracy in tasks such as obstacle detection, furniture recognition, and ground material recognition. Figure 3 As shown in the figure, the teacher model detects obstacles in the input image and outputs the predicted box, category, and confidence information corresponding to the obstacle.

[0131] Step 202: Perform model compression based on the teacher model to obtain a student model.

[0132] After training the teacher model, model compression techniques such as channel pruning are used to prune a lightweight student model based on the teacher model. During the pruning process, the core feature extraction channels are retained to reduce redundant parameters.

[0133] Step 203: Randomly mask the feature map output by the student model to obtain a masked feature map.

[0134] Let the feature maps output by the teacher model and the student model at the target layer be F T and F S , F T , F S ∈R H×W×C Specifically, input the above detection data set into the teacher model and the student model respectively, and extract the feature map F T output by the teacher model at the target layer and the feature map F S output by the student model at the target layer during the forward propagation process, where H×W×C represents the total number of elements in the feature map.

[0135] After obtaining the feature maps F T and F S output by the teacher model and the student model at the target layer, generate a binary mask map M with the same size as it. In this binary mask map M, use the random sampling technique to set each element to be selected with probability p and not to be selected with probability 1 - p. The specific formula is as follows:

[0136] M(i,j,k)~Bernoulli(p), p = 0.35 (5)

[0137] where the Bernoulli method is a random sampling technique based on the Bernoulli distribution. i, j, k represent the values in the three size dimensions of H×W×C, and 0≤i<H, 0≤j<W, 0≤k<C. In this method, each sample or element will be selected with a given probability p (selected is recorded as 1) and not selected with a probability of 1 - p (not selected is recorded as 0). In the embodiment of this application, p = 0.35, that is, each element in the binary mask M will be selected with a probability of 35%, and each element will not be selected with a probability of 65%, that is, 65% of the features will be masked.

[0138] Step 204: Use a two-layer convolutional network to reconstruct the features of the masked feature map to obtain a reconstructed feature map.

[0139] After obtaining the binary mask map M, perform feature masking on the feature map F S :The process formula of feature masking is as follows:

[0140]

[0141] where represents the Hadamard product (element-wise multiplication), and F masked ​Represents the mask feature map.

[0142] After obtaining the mask feature map F masked After that, a two-layer convolutional network is used to mask the feature map F masked Perform feature recovery. The feature recovery process formula is as follows:

[0143] F rec =Conv2(σ(Conv1(F masked ))) (7)

[0144] Among them, Conv1 represents the first layer of convolution operation (such as using a 3×3 convolution kernel), σ() represents the activation layer (such as ReLU), Conv2 represents the second layer of convolution operation (such as using a 3×3 convolution kernel), F rec Represents the reconstructed feature map.

[0145] Step 205: Build a knowledge distillation framework based on the feature map output by the teacher model, the reconstructed feature map, and the basic loss of the detection task of the student model.

[0146] After obtaining the reconstructed feature map F rec After that, the MSE is used to reconstruct the feature map F rec And the feature map F output by the teacher model at the target layer T Alignment constraints are applied to calculate the distillation loss, which is defined as follows:

[0147]

[0148] Among them, N = H × W × C represents the total number of elements in the feature map, L distill Represents distillation loss.

[0149] The distillation loss L distill Then, let the basic loss of the detection task of the student model be L stu Specifically, the above detection data set is input into the student model, and the basic loss L of the detection task can be calculated based on the difference between the prediction results output by the student model and the real labels in the detection data set. stu .

[0150] The basic loss of the detection task of the student model is L stu After that, according to the distillation loss L distill The basic loss of the detection task of the student model is L stu Determine the total loss function of the entire sweeper target detection training framework, which is defined as follows:

[0151] L total =L stu +λ·L distill (9)

[0152] Among them, λ represents the knowledge distillation weight, which is used to balance the basic loss L of the detection task of the student model stu and distillation loss L distill , L total Represents the total loss function of the sweeper object detection training framework, that is, the total loss function of the knowledge distillation framework.

[0153] In the total loss function L of the knowledge distillation framework total After that, according to the total loss function L total The student model is optimized and its performance is evaluated based on the aforementioned detection dataset. If the performance metric does not reach the preset threshold, the optimized student model is iteratively optimized multiple times using the aforementioned method until its performance meets the target. When deployed in a robot vacuum for object detection, the optimized student model maintains high detection performance while maintaining a lightweight design, ensuring efficient and safe operation.

[0154] The technical solution of the embodiment of the present application utilizes random masks to shield a large number of redundant features of the student model, and uses a two-layer convolutional network to restore key features, enabling the student model to efficiently imitate the features of the teacher model. Through the design of random masking and reconstruction of features, this method forces the student model to learn how to extract and restore key features from limited information during training. This not only improves the student model's ability to simulate the teacher model's features, but also helps the student model maintain high accuracy while being lightweight, providing an efficient solution for sweeper target detection in complex scenarios.

[0155] The present application also proposes a knowledge distillation device. Figure 4 This is a schematic diagram of the structure of the knowledge distillation device provided in the embodiment of the present application. Figure 4 As shown, the device includes:

[0156] The processing unit 401 is used to perform model compression on the first teacher model generated by training the target sample set to obtain a first student model; wherein the model parameter amount of the first student model is less than the model parameter amount of the first teacher model; randomly mask the first feature map output by the first student model to obtain a first masked feature map; and perform feature reconstruction on the first masked feature map to obtain a first reconstructed feature map.

[0157] A determination unit 402 is configured to determine a first distillation loss based on a difference between the first reconstructed feature map and a second feature map output by the first teacher model; and to determine a first loss function based on the first distillation loss and a first prediction loss of the first student model.

[0158] The processing unit 401 is also used to optimize the first student model based on the first loss function, and perform performance testing on the optimized first student model based on the target sample set; wherein the first teacher model and the first student model are used for target detection tasks.

[0159] In some embodiments, the processing unit 401 is further used to input the target sample set into the first student model for forward propagation, and extract the first feature map output by the target layer during the forward propagation process; input the target sample set into the first teacher model for forward propagation, and extract the second feature map output by the target layer during the forward propagation process.

[0160] In some implementations, the processing unit 401 is specifically configured to:

[0161] Determine, based on the first feature map, a binary mask map having the same size as the first feature map; wherein each element in the binary mask map is selected with a first probability, and each element in the binary mask map is not selected with a second probability;

[0162] The first feature map and the binary mask map are element-wise multiplied to obtain the first mask feature map.

[0163] In some implementations, the processing unit 401 is further configured to:

[0164] Perform a convolution operation on each convolution kernel included in the first convolution layer and the first mask feature map to obtain a third feature map;

[0165] Performing a nonlinear transformation on the third feature map to obtain a first activation feature map;

[0166] Based on the second convolutional layer, feature reconstruction mapping is performed on the first activated feature map to obtain a first reconstructed feature map.

[0167] In some implementations, the processing unit 401 is further configured to:

[0168] Performing a convolution operation on each convolution kernel included in the second convolution layer and the first activation feature map to obtain a fourth feature map; wherein the number of convolution kernels included in the second convolution layer is greater than the number of convolution kernels included in the first convolution layer;

[0169] Performing a nonlinear transformation on the fourth feature map to obtain a second activation feature map;

[0170] Perform feature reconstruction mapping on the second activated feature map to obtain a first reconstructed feature map.

[0171] In some implementations, the processing unit 401 is further configured to:

[0172] Perform parameter pruning on the first teacher model to obtain the first student model; or,

[0173] Quantize the parameters of the first teacher model to obtain a first student model; or,

[0174] Perform parameter pruning on the first teacher model to obtain a pruned first teacher model; perform parameter quantization on the pruned first teacher model to obtain a first student model; or,

[0175] The target distillation knowledge is determined based on the first teacher model, and the model is trained based on the target distillation knowledge and the target sample set to obtain the first student model.

[0176] In some embodiments, the processing unit 401 is further used to integrate multiple second teacher models generated by training the target sample set to obtain an integrated model; perform model compression on the integrated model to obtain a second student model; wherein the model parameter amount of the second student model is less than the model parameter amount of the integrated model; perform random masking on the fifth feature map output by the second student model to obtain a second masked feature map; perform feature reconstruction on the second masked feature map to obtain a second reconstructed feature map.

[0177] In some embodiments, the determination unit 402 is further configured to determine a second distillation loss based on the second reconstructed feature map and the sixth feature map output by the integrated model; and determine a second loss function based on the second distillation loss and the second prediction loss of the second student model.

[0178] In some embodiments, the processing unit 401 is further configured to optimize the second student model based on the second loss function, and perform a performance test on the optimized second student model based on the target sample set.

[0179] In the technical solution of the embodiment of the present application, the first teacher model generated by training the target sample set is compressed to obtain a first student model; the first feature map output by the first student model is randomly masked to obtain a first masked feature map; the first masked feature map is feature reconstructed to obtain a first reconstructed feature map; based on the difference between the first reconstructed feature map and the second feature map output by the first teacher model, a first distillation loss is determined; based on the first distillation loss and the first prediction loss of the first student model, a first loss function is determined; the first student model is optimized based on the first loss function, and the performance of the optimized first student model is tested based on the target sample set; wherein, the model parameter amount of the first student model is less than the model parameter amount of the first teacher model; the first teacher model and the first student model are used for target detection tasks. In this way, by randomly masking a certain proportion of the student model features and restoring the original feature points of the masked features through a two-layer convolutional network to imitate the teacher model features, the distillation process can be made more focused on the effective feature transfer of the target area, and the student model is forced to learn how to extract and restore key features from limited information during training. This not only improves the student model's ability to simulate the teacher model's features, but also maintains the high precision of the student model while keeping it lightweight, thereby effectively improving the performance of the student model.

[0180] It should be understood by those skilled in the art that Figure 4 The implementation functions of each unit in the knowledge distillation device shown can be understood by referring to the relevant description of the aforementioned method. Figure 4 The functions of each unit in the knowledge distillation device shown can be implemented by a program running on a processor, or by a specific logic circuit.

[0181] Figure 5 This is a schematic diagram of the structure of the processing device provided in the embodiment of the present application. The processing device may be a terminal device or a network device. Figure 5 The processing device shown includes a processor 501, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0182] Alternatively, as Figure 5 As shown, the processing device may further include a memory 502. The processor 501 may call and execute a computer program from the memory 502 to implement the method in the embodiment of the present application.

[0183] The memory 502 may be a separate device independent of the processor 501 , or may be integrated into the processor 501 .

[0184] Alternatively, as Figure 5As shown, the processing device may further include a transceiver 503 , and the processor 501 may control the transceiver 503 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices.

[0185] The transceiver 503 may include a transmitter and a receiver. The transceiver 503 may further include an antenna, and the number of antennas may be one or more.

[0186] The processing device can specifically be the knowledge distillation device of the embodiment of the present application, and the processing device can implement the corresponding processes of the various methods implemented in the embodiment of the present application. For the sake of brevity, it will not be repeated here.

[0187] It should be understood that the processor of the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented as a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0188] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0189] It should be understood that the above-mentioned memories are exemplary but not restrictive. For example, the memories in the embodiments of the present application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM RAM (DR RAM), etc. In other words, the memories in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0190] The present application also provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to the processing device in the present application, and the computer program causes a computer to execute the corresponding processes of the various methods implemented in the present application. For the sake of brevity, the details are not repeated here.

[0191] The present application also provides a computer program product including computer program instructions. The computer program product can be applied to the processing device in the present application, and the computer program instructions cause a computer to execute the corresponding processes of the various methods implemented in the present application. For the sake of brevity, the details are not repeated here.

[0192] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0193] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0195] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0197] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0198] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A knowledge distillation method, characterized in that: The method comprises: Performing model compression on a first teacher model generated by training the target sample set to obtain a first student model; wherein the model parameters of the first student model are smaller than the model parameters of the first teacher model; Randomly masking the first feature map output by the first student model to obtain a first masked feature map; performing feature reconstruction on the first masked feature map to obtain a first reconstructed feature map; Determining a first distillation loss based on a difference between the first reconstructed feature map and a second feature map output by the first teacher model; determining a first loss function based on the first distillation loss and a first prediction loss of the first student model; The first student model is optimized based on the first loss function, and the performance of the optimized first student model is tested based on the target sample set; wherein the first teacher model and the first student model are used for target detection tasks.

2. The method according to claim 1, characterized in that The randomly masking the first feature map output by the first student model to obtain a first masked feature map includes: Determine, based on the first feature map, a binary mask map having the same size as the first feature map; wherein each element in the binary mask map is selected with a first probability, and each element in the binary mask map is not selected with a second probability; Performing element-by-element multiplication on the first feature map and the binary mask map to obtain the first mask feature map.

3. The method according to claim 1, characterized in that The performing feature reconstruction on the first mask feature map to obtain a first reconstructed feature map includes: Performing a convolution operation on each convolution kernel included in the first convolution layer and the first mask feature map to obtain a third feature map; Performing a nonlinear transformation on the third feature map to obtain a first activation feature map; Perform feature reconstruction mapping on the first activated feature map based on the second convolutional layer to obtain the first reconstructed feature map.

4. The method according to claim 3, characterized in that The performing feature reconstruction mapping on the first activation feature map based on the second convolutional layer to obtain the first reconstructed feature map includes: Performing a convolution operation on each convolution kernel included in the second convolution layer and the first activation feature map to obtain a fourth feature map; wherein the number of convolution kernels included in the second convolution layer is greater than the number of convolution kernels included in the first convolution layer; Performing a nonlinear transformation on the fourth feature map to obtain a second activation feature map; Perform feature reconstruction mapping on the second activated feature map to obtain the first reconstructed feature map.

5. The method according to claim 1, wherein The method further comprises: Inputting the target sample set into the first student model for forward propagation, and extracting the first feature map output by the target layer during the forward propagation process; The target sample set is input into the first teacher model for forward propagation, and the second feature map output by the target layer is extracted during the forward propagation process.

6. The method according to claim 1, characterized in that The first teacher model generated by training the target sample set is compressed to obtain a first student model, including: Perform parameter pruning on the first teacher model to obtain the first student model; or, quantizing the parameters of the first teacher model to obtain the first student model; or Performing parameter pruning on the first teacher model to obtain a pruned first teacher model; performing parameter quantization on the pruned first teacher model to obtain the first student model; or Target distillation knowledge is determined based on the first teacher model, and a model is trained based on the target distillation knowledge and the target sample set to obtain the first student model.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Integrate multiple second teacher models generated by training the target sample set to obtain an integrated model; Performing model compression on the integrated model to obtain a second student model; wherein the model parameter amount of the second student model is smaller than the model parameter amount of the integrated model; Randomly masking the fifth feature map output by the second student model to obtain a second masked feature map; performing feature reconstruction on the second masked feature map to obtain a second reconstructed feature map; Determining a second distillation loss based on the second reconstructed feature map and the sixth feature map output by the ensemble model; determining a second loss function based on the second distillation loss and the second prediction loss of the second student model; The second student model is optimized based on the second loss function, and the performance of the optimized second student model is tested based on the target sample set.

8. A knowledge distillation device, characterized in that: The device comprises: a processing unit, configured to perform model compression on a first teacher model generated by training a target sample set to obtain a first student model; wherein the model parameter quantity of the first student model is smaller than the model parameter quantity of the first teacher model; perform random masking on a first feature map output by the first student model to obtain a first masked feature map; and perform feature reconstruction on the first masked feature map to obtain a first reconstructed feature map; a determining unit, configured to determine a first distillation loss based on a difference between the first reconstructed feature map and a second feature map output by the first teacher model; and determine a first loss function based on the first distillation loss and a first prediction loss of the first student model; The processing unit is also used to optimize the first student model based on the first loss function, and perform performance testing on the optimized first student model based on the target sample set; wherein the first teacher model and the first student model are used for target detection tasks.

9. A processing device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 7.

Images