A model distillation method, a target detection method and related devices

Abstract

This application discloses a model distillation method, an object detection method, and related equipment. The method includes: acquiring a teacher model and a student model; acquiring sample images and inputting the sample images into the teacher model to obtain a first output result, the first output result including first detection information of each target object in the sample image; inputting the sample images into the student model to obtain a second output result, the second output result including second detection information of each target object in the sample image; calculating an inter-class attention consistency loss based on the first detection information and corresponding second detection information of at least two target objects; calculating the current loss value of the student model based on the inter-class attention consistency loss; and updating the student model based on the current loss value. Through the above methods, this application can improve the performance of the student model.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, specifically to a model distillation method, an object detection method, and related equipment. Background Technology

[0002] Model distillation is a technique used to compress the computational overhead of large models. It uses the large model as the teacher model and constructs a small model as the student model. The student model is then trained by optimizing the teacher model so that it can achieve or exceed the inference performance of the teacher model. However, distillation techniques in this field do not guarantee good performance of the compressed model in all data domains. Summary of the Invention

[0003] This application provides a model distillation method, a target detection method, and related equipment, which can improve the performance of student models.

[0004] To address the aforementioned technical problems, this application adopts the following technical solution: A model distillation method is provided, comprising: acquiring a teacher model and a student model; acquiring sample images and inputting the sample images into the teacher model to obtain a first output result, the first output result including first detection information of each target object in the sample image; inputting the sample images into the student model to obtain a second output result, the second output result including second detection information of each target object in the sample image; calculating an inter-class attention consistency loss based on the first detection information and corresponding second detection information of at least two target objects; calculating the current loss value of the student model based on the inter-class attention consistency loss; and updating the student model based on the current loss value.

[0005] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a target detection method, which includes: acquiring an image to be detected; inputting the image to be detected into an updated student model to obtain a target detection result; wherein the updated student model is obtained through the model distillation method in the above technical solution.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a model distillation apparatus, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the model distillation method or the target detection method in the above-mentioned technical solution.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing a computer program. When the computer program is executed by a processor, it is used to implement the model distillation method or the target detection method in the above-mentioned technical solution.

[0008] The beneficial effects of this application through the above scheme are as follows: First, the teacher model is pre-trained, then sample images are acquired and simultaneously input into the student model and the teacher model to obtain first detection information and second detection information of at least two target objects; then, the inter-class attention consistency loss is calculated using the first detection information and corresponding second detection information of at least two target objects; by processing the inter-class attention consistency loss, the current loss value of the student model is obtained; the student model is updated using the current loss value to complete the distillation training of the student model; because the inter-class attention consistency supervised distillation learning process of the model enables the student model to learn better from the teacher model, improves the performance of the student model, and enables the training of a student model suitable for hardware devices with limited computing power, so that small-scale models can also meet the performance requirements of real-world scenarios, avoiding the problem of deploying large-scale models on devices with limited computing power, and has a wide range of applications. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0010] Figure 1 This is a schematic flowchart of an embodiment of the model distillation method provided in this application;

[0011] Figure 2 This is a schematic flowchart of another embodiment of the model distillation method provided in this application;

[0012] Figure 3 This is a schematic diagram of the processing of two-dimensional feature information provided in this application;

[0013] Figure 4 This is a flowchart illustrating an embodiment of the target detection method provided in this application;

[0014] Figure 5 This is a schematic diagram of the structure of an embodiment of the model distillation apparatus provided in this application;

[0015] Figure 6 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0016] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0017] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0018] It should be noted that the terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0019] Please see Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the model distillation method provided in this application, the method including:

[0020] S11: Obtain the teacher model and student model.

[0021] The teacher model and the student model use the same reasoning methods and perform the same functions, such as object detection, object recognition, or segmentation. The teacher model is either pre-trained or uses an existing model. Specifically, a teacher model can be obtained from a model database, a student model can be built, and the student model can be initially trained or not trained at all. Alternatively, a student model and a teacher model can be built, and the teacher model can be trained in a regular manner, resulting in superior performance, such as an accuracy rate greater than the preset accuracy rate.

[0022] In one implementation, a deep learning model with a large-scale backbone network is constructed as a teacher model based on a two-stage object detection model to complete the training of a conventional teacher model; then, a small-scale deep learning model with the same reasoning method is constructed as a student model according to the model's time consumption requirements.

[0023] S12: Obtain the sample image and input it into the teacher model to obtain the first output result.

[0024] A camera is used to capture images of the target scene, resulting in sample images. The information contained in these sample images is related to the function implemented by the teacher model. For example, if the teacher model's function is object detection, then the sample images must contain at least one target object, which can be a person, other animals, or objects. After acquiring the sample images, the teacher model processes them to obtain the corresponding output results (denoted as the first output results). The first output results include the first detection information for each target object in the sample images. The first detection information can include the target object's position, category, attributes, or pose.

[0025] In one embodiment, it is assumed that the teacher model is used for object detection. The teacher model includes a first backbone network and a first detector. The sample image is input into the first backbone network to obtain a third output result; the first output result is input into the first detector to obtain a first output result.

[0026] S13: Input the sample image into the student model to obtain the second output result.

[0027] After obtaining the sample image, the student model is used to process the sample image to obtain the corresponding output result (referred to as the second output result). The second output result includes the second detection information of each target object in the sample image. The information contained in the second detection information is the same as the information contained in the first detection information. That is, the second detection information may include the position of the target object, the category of the target object, the attribute of the target object, or the pose of the target object.

[0028] In one embodiment, it is assumed that the student model is used for object detection. The student model includes a second backbone network and a second detector. The sample image is input into the second backbone network to obtain a fourth output result; the fourth output result is input into the second detector to obtain a second output result.

[0029] S14: Calculate the inter-class attention consistency loss based on the first detection information and the corresponding second detection information of at least two target objects.

[0030] After obtaining the first and second output results, they are processed to evaluate the detection performance of the student model and the teacher model on different target objects, thus obtaining the inter-class attention consistency loss. Specifically, at least two target objects are selected from the target objects common to both the first and second output results. The first detection information of these target objects is selected from the first output result, and the second detection information of these target objects is selected from the second output result. The inter-class attention consistency loss is calculated using the selected first and second detection information.

[0031] S15: Based on the inter-class attention consistency loss, calculate the current loss value of the student model and update the student model based on the current loss value.

[0032] After calculating the inter-class attention consistency loss, the current loss value of the student model can be calculated using this inter-class attention consistency loss. Specifically, the inter-class attention consistency loss can be used as the current loss value; or, it can be combined with other types of losses to calculate the current loss value. For example, if the first detection information includes the first detection box of the target object and the second detection information includes the second detection box of the target object, the label data of the sample image can be obtained. This label data includes the labeled box of each target object in the sample image. The loss between the labeled box of the target object in the label data and the corresponding second detection box in the second output result is calculated to obtain the bounding box regression loss. Based on the inter-class attention consistency loss and the bounding box regression loss, the current loss value is calculated.

[0033] In one embodiment, after updating the student model, it can be determined whether the student model meets the preset update termination condition; if the student model meets the preset update termination condition, the update of the student model is ended and the updated student model is obtained; if the student model does not meet the preset update termination condition, the step of obtaining sample images is returned, that is, the execution of S12 is returned until the student model meets the preset update termination condition.

[0034] Furthermore, the preset update termination conditions include: loss value convergence, i.e., the difference between the previous loss value and the current loss value is less than a set value; determining whether the current loss value is less than a preset loss value, which is a pre-set loss threshold. If the current loss value is less than the preset loss value, then the preset update termination condition is determined to be met; the number of training iterations reaches a set value (e.g., 10,000 training iterations); or the accuracy obtained when testing with the test set reaches a set condition (e.g., exceeding the preset accuracy), etc.

[0035] This embodiment trains the teacher model in advance to enable the student model to learn in a targeted manner. The learning process is distilled under the supervision of the inter-class attention consistency of the model. Because the inter-class attention consistency loss is taken into account, the current loss value is more in line with the actual situation, the calculation of the current loss value is more accurate, the performance of the student model is improved, and a student model suitable for hardware devices with limited computing power is trained. This allows small-scale models to meet the performance requirements of real-world scenarios and avoids the problem of deploying large-scale models on devices with limited computing power. The application scope is relatively wide.

[0036] The following example uses a student model and a teacher model as object detection models to illustrate the model distillation scheme provided in this application. Please refer to [link / reference needed]. Figure 2 , Figure 2 This is a schematic flowchart of another embodiment of the model distillation method provided in this application, the method including:

[0037] S21: Obtain the teacher model and student model.

[0038] S22: Obtain sample images and input them into the teacher model to obtain the first output result; obtain sample images and input them into the student model to obtain the second output result.

[0039] The first output includes first detection information for each target object in the sample image, which includes the location and category of the target object; the second output includes second detection information for each target object in the sample image, which includes the location and category of the target object. Understandably, S21 and S22 are the same as in the above embodiments and will not be repeated here.

[0040] The following explains the consistency of attention between classes.

[0041] The overall process of an object detection model can be understood as a high-dimensional sliding window that slides across an image to find a set of feature combinations that the model is most interested in, generating a large response to them. This process reflects the model's global attention tendency. The model can output different categories of target objects, indicating that the model has multiple attention mechanisms, employing different attention to capture corresponding feature combinations for different categories. By analyzing the differences in attention between different categories, we can better demonstrate how the model distinguishes different types of targets, guiding student models to learn from the teacher model's attention differences for different target categories.

[0042] The following section explains how to generate inter-class attention consistency loss, as shown in S23 to S25.

[0043] S23: Obtain the label data corresponding to the sample image, and generate the first pairing result and the second pairing result based on the label data.

[0044] The sample images are first labeled to obtain label data, which includes a bounding box for each target object in the sample image. The bounding box represents the actual location of the target object and can be a rectangle. Then, the first pairing result and the second pairing result are generated using the following scheme:

[0045] 1) Match the target object's bounding box with the first detection box to obtain the first pairing result. The first pairing result includes the bounding box and the first detection box that matches the bounding box.

[0046] First, calculate the intersection over union (IOU) ratio between the target object's bounding box and each first detection box to obtain multiple IOU ratios; the formula for calculating the IOU ratio is shown below:

[0047] IOU(box1,box2)=(box1∩box2) / (box1∪box2) (1)

[0048] Where box1 is the label box, box2 is the first detection box, ∩ is the intersection, and ∪ is the union.

[0049] Then, the maximum value of multiple first parallel ratios is calculated, and it is determined whether the maximum value of the first parallel ratio is greater than the first preset threshold. When the maximum value of the first parallel ratio is greater than the first preset threshold, it is determined that the first detection box corresponding to the maximum value matches the annotation box.

[0050] For example, if the first preset threshold is 0.5, the bounding box of the target object F is denoted as C1, and the first detection boxes are denoted as D1, D2, and D3, then the intersection-union ratio (IU) of bounding box C1 and the first detection box D1 is calculated (denoted as H1), the intersection-union ratio (IU) of bounding box C1 and the first detection box D2 is calculated (denoted as H2), and the intersection-union ratio (IU) of bounding box C1 and the first detection box D3 is calculated (denoted as H3). Assuming that the maximum value among H1 to H3 is H3, and H3 is greater than 0.5, then the bounding box C1 is paired with the first detection box D3.

[0051] 2) Match the target object's bounding box with the second detection box to obtain the second pairing result, which includes the bounding box and the second detection box that matches the bounding box.

[0052] Calculate the intersection-union ratio (IUGR) of the target object's bounding box with each second detection box to obtain multiple second IUGRs; calculate the maximum value of the multiple second IUGRs and determine whether the maximum value of the second IUGR is greater than a first preset threshold; when the maximum value of the second IUGR is greater than the second preset threshold, determine that the first detection box corresponding to the maximum value matches the bounding box, and the second preset threshold can be the same as the first preset threshold.

[0053] S24: Based on the first pairing result, select the first detection information of at least two target objects from the first output result to obtain a first candidate detection information set; based on the second pairing result, select the second detection information corresponding to the first detection information in the first candidate detection information set from the second output result to obtain a second candidate detection information set.

[0054] The first detection information also includes the first feature information of the target object, and the second detection information also includes the second feature information of the target object; the following scheme is used to generate the first candidate detection information set and the second candidate detection information set:

[0055] 1) Select at least two label boxes from the label data to obtain a label box set.

[0056] A set of annotation boxes includes at least two annotation boxes.

[0057] 2) Generate a first candidate detection box set based on the labeled box set, the first pairing result, and the first detection information.

[0058] Since the sizes of the first detection boxes may differ, the scales of the first feature information of their corresponding regions may also differ, making comparison impossible. Therefore, a first attention model is used to convert feature information of different scales into feature information of the same scale. Specifically, first detection boxes that match the labeled boxes in the labeled box set are selected from the first pairing results to obtain a first candidate detection box set; the first feature information corresponding to the first detection boxes in the first candidate detection box set is input into the first attention model to obtain third feature information; each first detection box in the first candidate detection box set and its corresponding third feature information are placed into a first candidate detection information set, that is, the first candidate detection box set includes first detection boxes that match the labeled boxes in the labeled box set.

[0059] 3) Generate a second candidate detection box set based on the labeled box set, the second pairing results, and the second detection information.

[0060] The second detection box that matches the labeled box in the labeled box set is selected from the second pairing results to obtain the second candidate detection box set; the second feature information corresponding to the second detection box in the second candidate detection box set is input into the first attention model to obtain the fourth feature information, which has the same dimension as the third feature information; each second detection box in the second candidate detection box set and the corresponding fourth feature information are put into the second candidate detection information set, that is, the second candidate detection box set includes the second detection box that matches the labeled box in the labeled box set.

[0061] S25: Generate inter-class attention consistency loss based on the first candidate detection information set and the second candidate detection information set.

[0062] After obtaining the first and second candidate detection information sets, the following approach can be used to generate the inter-class attention consistency loss:

[0063] Two third feature information are selected from the first candidate detection information set to obtain the first candidate feature information and the second candidate feature information; the difference between the first candidate feature information and the second candidate feature information is calculated to obtain the first type of inter-class attention difference information, for example: the difference between the first candidate feature information and the second candidate feature information.

[0064] Select the fourth feature information corresponding to the first candidate feature information from the second candidate detection information set to obtain the third candidate feature information; select the fourth feature information corresponding to the second candidate feature information from the second candidate detection information set to obtain the fourth candidate feature information. Calculate the difference between the third and fourth candidate feature information to obtain the inter-class attention difference information, for example, by subtracting the third and fourth candidate feature information.

[0065] Based on the inter-class attentional difference information and the inter-class attentional consistency information, the inter-class attentional consistency loss is calculated.

[0066] In one embodiment, when the number of first feature information in the first candidate detection information set is a preset value, the difference between the first inter-class attention difference information and the second inter-class attention difference information is calculated to obtain the inter-class attention consistency loss.

[0067] For example, suppose the first attention model is a Region of Interest (ROIAlign) layer with a preset value of 2, meaning the first candidate detection information set includes two first feature information sets, and the second candidate detection information set includes two second feature information sets; the first output result is denoted as R. t The second output result is denoted as R. s .

[0068] Randomly select two label boxes G1 and G2 of different categories; determine R. t The first detection box paired with G1 and G2 is denoted as P. t1 and P t2 Determine R s The second detection box paired with G1 and G2 is denoted as P. s1 and P s2 Based on P t1 P t2 The corresponding region feature F is extracted from the feature layer of the last layer of the teacher model using the Roi Align layer. t1 and F t2 Based on P s1 P s2The corresponding region features F are extracted from the feature layer of the last layer of the student model using the Roi Align layer. s1 and F s2 F t1 F t2 F s1 and F s2 The dimensions are all 7*7. The difference in category attention (Dis) between the teacher model and the student model is calculated separately. t and Dis s :

[0069] Dis t =|F t1 -F t2 | (2)

[0070] Dis s =|F s1 -F s2 | (3)

[0071] Analyzing Dis using KL divergence (Kullback-Leibler Divergence) t and Dis s The difference is used to obtain the inter-class attention consistency loss Dis1, which is calculated as follows:

[0072]

[0073] Where, x ij The features of the i-th row and j-th column are given.

[0074] In another embodiment, when the number of first feature information in the first candidate detection information set is greater than a preset value, the difference between the first inter-class attention difference information and each second inter-class attention difference information is calculated to obtain multiple inter-class attention losses; the average value of multiple inter-class attention losses is calculated to obtain inter-class attention consistency loss.

[0075] Furthermore, when the number of first feature information in the first candidate detection information set is greater than a preset value, the difference between the first class attention difference information and the second class attention difference information is calculated; the step of selecting two first feature information from the first candidate detection information set is returned until the first candidate detection information set is traversed, and multiple subclass attention losses are obtained.

[0076] S26: Calculate the current loss value of the student model based on the inter-class attention consistency loss.

[0077] The current loss value can be calculated using the reference class inter-class attention consistency loss and the feature attention consistency loss. The feature attention consistency is explained below.

[0078] The network features are reconstructed using the encoder in the second self-attention model. The encoder can capture the dependencies between features and reconstruct them based on these dependencies, thus reflecting the attention-oriented nature of the deep learning model and representing the features the model focuses on. By comparing the reconstructed features of the teacher model and the student model within the same module, the differences in their attention mechanisms can be observed. To enable the student model to possess the detection capabilities of the teacher model, the attention parameters of the student model need to be guided to be as close as possible to those of the teacher model. The following section explains how to apply feature attention consistency to knowledge distillation.

[0079] 1) Calculate the feature attention consistency loss based on the third and fourth output results.

[0080] The third output includes the third detection information of the target object, and the fourth output includes the fourth detection information of the target object. The third detection information is split into multiple first sub-feature information, and the first sub-feature information is input into the second attention model to obtain the second sub-feature information. The fourth detection information is split into multiple third sub-feature information, and the third sub-feature information is input into the second attention model to obtain the fourth sub-feature information. The number of first sub-feature information is the same as the number of second sub-feature information. The difference between the third sub-feature information and the corresponding fourth sub-feature information is calculated to obtain the sub-feature attention consistency loss. All sub-feature attention consistency losses are summed to obtain the feature attention consistency loss.

[0081] In one implementation, for each forward computation during training, the feature information F of the last layer of the backbone networks (i.e., the first backbone network and the second backbone network) of the teacher model and the student model is extracted respectively. t and F s F t and F s It can be decomposed into two-dimensional feature information with C channels, and each two-dimensional feature information is encoded separately. Then, an encoder based on a transformer network is constructed to encode F. t and F s Each two-dimensional feature information in the algorithm is divided into multiple windows, and each window corresponds to a sub-feature information. An encoder is used to reconstruct each sub-feature information. For example, ... Figure 3 As shown, the transformer includes a self-attention module and a multilayer perceptron (MLP) module, which decomposes the two-dimensional feature information into f1~f2. N A self-attention module is used to process the sub-feature information f. h(1≤h≤N) are processed to obtain attention feature information; MLP is used to process each attention feature information to obtain the reconstructed feature information t. h .

[0082] Then, F is calculated using KL divergence. t and F s The corresponding feature differences Dis2 of the sub-feature information are shown below:

[0083]

[0084] Where m represents the number of windows, F t (t i ) represents F t The tth i The third sub-feature information corresponding to each window, F s (t i ) represents F s The tth i The fourth sub-feature information corresponding to each window.

[0085] 2) Calculate the loss value based on feature attention consistency loss, inter-class attention consistency loss, bounding box regression loss, and classification loss.

[0086] The loss value, i.e., the loss function of the student model, is obtained by weighted summing of the feature attention consistency loss, inter-class attention consistency loss, bounding box regression loss, and classification loss:

[0087] Loss = Loss box +Loss class +Dis1+Dis2 (6)

[0088] Among them, Loss box Loss is the loss for bounding box regression. class For classification loss.

[0089] S27: Update the student model based on the current loss value.

[0090] Based on the attention consistency and prediction consistency in the object detection model, the student model is trained. Specifically, based on the current loss value, the network weights of the student model are updated through gradient backpropagation, while the parameters of the teacher model remain unchanged, ultimately resulting in a high-quality student model.

[0091] This embodiment provides a model distillation method based on an attention mechanism. It constructs and trains a student model suitable for computationally limited hardware. The feature attention difference between the student and teacher models is calculated in the last layer of the backbone network to obtain the feature attention consistency loss. An ROI Align layer is used to calculate the inter-class attention consistency loss, enabling simultaneous supervision of the distillation learning process from both the model's feature attention consistency and inter-class attention consistency. Furthermore, the model distillation training is completed using the inter-class attention consistency loss, feature attention consistency loss, and prediction loss together, resulting in better performance of the student model after distillation. Compared to schemes using large-scale convolutional neural networks as the backbone network of object detection models, this method has lower hardware requirements and can be applied to the compression of complex object detection models, improving the performance of small-scale models in image parsing and prediction. Moreover, compared to model distillation methods that only focus on the output similarity of fixed layers, this scheme considers the specificity of the detection task, distilling large-scale models from the perspective of model attention to extract deeper semantic information to support the learning of the student model, resulting in better detection performance.

[0092] Please see Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of the target detection method provided in this application. The method includes:

[0093] S41: Acquire the image to be detected.

[0094] The target scene is captured by a camera to obtain an image to be detected. The image to be detected includes the target object, which can be a person, an object, or another animal.

[0095] S42: Input the image to be detected into the updated student model to obtain the target detection result.

[0096] The updated student model is used to process the image to be detected to obtain the target detection result. The updated student model is obtained by the model distillation method in the above embodiment.

[0097] This embodiment uses the model distillation method to perform object detection tasks, constructs a teacher model and a student model, and pre-trains the teacher model to enable the student model to learn in a targeted manner. This allows a small-scale object detection model (i.e., the updated student model) to meet the performance requirements of real-world scenarios, avoiding the problem of limited computing power when deploying large-scale models. It enables object detection on hardware devices with limited computing power, and the accuracy of object detection is relatively good.

[0098] Please see Figure 5 , Figure 5This is a schematic diagram of an embodiment of the model distillation apparatus provided in this application. The model distillation apparatus 50 includes a memory 51 and a processor 52 connected to each other. The memory 51 is used to store computer programs. When the computer programs are executed by the processor 52, they are used to implement the model distillation method or the target detection method in the above embodiments.

[0099] Please see Figure 6 , Figure 6 This is a schematic diagram of an embodiment of a computer-readable storage medium provided in this application. The computer-readable storage medium 60 is used to store a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the model distillation method or the target detection method in the above embodiments.

[0100] The computer-readable storage medium 60 can be any medium capable of storing program code, such as a server, USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0105] The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A model distillation method, characterized by, include: Obtain the teacher model and student model; A sample image is acquired and input into the teacher model to obtain a first output result, wherein the first output result includes the first detection information of each target object in the sample image; The sample image is input into the student model to obtain a second output result, which includes the second detection information of each target object in the sample image. The step of calculating the inter-class attention consistency loss based on first detection information and corresponding second detection information of at least two different categories of target objects includes: acquiring label data corresponding to the sample image; generating a first pairing result and a second pairing result based on the label data; selecting first detection information of at least two target objects from the first output result based on the first pairing result to obtain a first candidate detection information set; selecting second detection information corresponding to the first detection information in the first candidate detection information set from the second output result based on the second pairing result to obtain a second candidate detection information set; and generating the inter-class attention consistency loss based on the first candidate detection information set and the second candidate detection information set, wherein the inter-class attention consistency loss is used to characterize the attention difference between the teacher model and the student model on different categories of target objects. The first detection information further includes first feature information of the target object, and the second detection information further includes second feature information of the target object. The step of selecting first detection information of at least two target objects from the first output result to obtain a first candidate detection information set includes: selecting first detection boxes that match the annotation boxes in the annotation box set from the first pairing result to obtain a first candidate detection box set; inputting the first feature information corresponding to the first detection boxes in the first candidate detection box set into a first attention model to obtain third feature information; and placing each first detection box in the first candidate detection box set and its corresponding third feature information into the first candidate detection information set; the annotation box set is determined based on at least two annotation boxes selected from the label data. The step of selecting second detection information from the second output result that corresponds to the first detection information in the first candidate detection information set to obtain a second candidate detection information set includes: selecting second detection boxes that match the annotation boxes in the annotation box set from the second pairing result to obtain a second candidate detection box set; inputting the second feature information corresponding to the second detection boxes in the second candidate detection box set into the first attention model to obtain fourth feature information, wherein the fourth feature information has the same dimension as the third feature information; and placing each second detection box in the second candidate detection box set and its corresponding fourth feature information into the second candidate detection information set. The step of generating the inter-class attention consistency loss based on the first candidate detection information set and the second candidate detection information set includes: selecting two third feature information from the first candidate detection information set to obtain first candidate feature information and second candidate feature information; calculating the difference between the first candidate feature information and the second candidate feature information to obtain first inter-class attention difference information; selecting a fourth feature information corresponding to the first candidate feature information from the second candidate detection information set to obtain third candidate feature information; selecting a fourth feature information corresponding to the second candidate feature information from the second candidate detection information set to obtain fourth candidate feature information; calculating the difference between the third candidate feature information and the fourth candidate feature information to obtain second inter-class attention difference information; and calculating the inter-class attention consistency loss based on the first inter-class attention difference information and the second inter-class attention difference information. Based on the inter-class attention consistency loss, calculate the current loss value of the student model; The student model is updated based on the current loss value.

2. The model distillation method of claim 1, wherein, The label data includes bounding boxes for each target object in the sample image, the first detection information includes a first detection box for the target object, the second detection information includes a second detection box for the target object, and the step of generating a first pairing result and a second pairing result based on the label data includes: The target object's bounding box is matched with the first detection box to obtain the first pairing result, which includes the bounding box and the first detection box that matches the bounding box. The target object's bounding box is matched with the second detection box to obtain the second pairing result, which includes the bounding box and the second detection box that matches the bounding box.

3. The model distillation method of claim 1, wherein, The step of calculating the inter-class attention consistency loss based on the first inter-class attention difference information and the second inter-class attention difference information includes: When the number of first feature information in the first candidate detection information set is a preset value, the difference between the first inter-class attention difference information and the second inter-class attention difference information is calculated to obtain the inter-class attention consistency loss.

4. The model distillation method of claim 1, wherein, The step of calculating the inter-class attention consistency loss based on the first inter-class attention difference information and the second inter-class attention difference information further includes: When the number of first feature information in the first candidate detection information set is greater than a preset value, the difference between the first inter-class attention difference information and each second inter-class attention difference information is calculated to obtain multiple inter-class attention losses. The average value of the attention loss among the multiple subclasses is calculated to obtain the interclass attention consistency loss.

5. The model distillation method of claim 2, wherein, The step of matching the bounding box of the target object with the first detection box to obtain the first pairing result includes: Calculate the intersection-union ratio (IUU) of the target object's bounding box with each of the first detection boxes to obtain multiple IUUs; Calculate the maximum value of the plurality of first union ratios, and when the maximum value is greater than a first preset threshold, determine that the first detection box corresponding to the maximum value matches the annotation box; The step of matching the bounding box of the target object with the second detection box to obtain the second pairing result includes: Calculate the intersection-union ratio (IUU) of the target object's bounding box with each of the second detection boxes to obtain multiple second IUU ratios; Calculate the maximum value of the plurality of second parallel ratios, and when the maximum value is greater than a second preset threshold, determine that the first detection box corresponding to the maximum value matches the annotation box.

6. The model distillation method of claim 1, wherein, The step of calculating the current loss value of the student model based on the inter-class attention consistency loss includes: Calculate the loss between the bounding box of the target object in the label data and the corresponding second detection box in the second output result to obtain the bounding box regression loss; Calculate the loss between the category label of the target object in the label data and the corresponding category prediction value of the target object in the second output result to obtain the classification loss; The loss value is calculated based on the inter-class attention consistency loss, the bounding box regression loss, and the classification loss.

7. The model distillation method of claim 6, wherein, The teacher model includes a first backbone network and a first detector, and the student model includes a second backbone network and a second detector; the step of inputting the sample image into the teacher model to obtain a first output result includes: The sample image is input into the first backbone network to obtain the third output result; The third output result is input into the first detector to obtain the first output result; The step of inputting the sample image into the student model to obtain the second output result includes: The sample image is input into the second backbone network to obtain the fourth output result; The fourth output result is input into the second detector to obtain the second output result; The step of calculating the loss value based on the inter-class attention consistency loss, the bounding box regression loss, and the classification loss includes: Based on the third and fourth output results, the feature attention consistency loss is calculated. The loss value is calculated based on the feature attention consistency loss, the inter-class attention consistency loss, the bounding box regression loss, and the classification loss.

8. The model distillation method of claim 7, wherein, The third output result includes the third detection information of the target object, and the fourth output result includes the fourth detection information of the target object. The step of calculating the feature attention consistency loss based on the third output result and the fourth output result includes: The third detection information is split into multiple first sub-feature information, and the first sub-feature information is input into the second attention model to obtain the second sub-feature information; The fourth detection information is split into multiple third sub-feature information, and the third sub-feature information is input into the second attention model to obtain the fourth sub-feature information. The number of the first sub-feature information is the same as the number of the second sub-feature information. The difference between the second sub-feature information and the corresponding fourth sub-feature information is calculated to obtain the sub-feature attention consistency loss; The feature attention consistency loss is obtained by summing up all the sub-feature attention consistency losses.

9. The model distillation method of claim 7, wherein, The step of calculating the loss value based on the feature attention consistency loss, the inter-class attention consistency loss, the bounding box regression loss, and the classification loss includes: The loss value is obtained by weighted summing of the feature attention consistency loss, the inter-class attention consistency loss, the bounding box regression loss, and the classification loss.

10. A target detection method characterized by, include: Acquire the image to be detected; The image to be detected is input into the updated student model to obtain the target detection result; The updated student model is obtained by the model distillation method according to any one of claims 1-9.

11. A model distillation apparatus, characterized by, It includes an interconnected memory and a processor, wherein the memory is used to store a computer program, which, when executed by the processor, is used to implement the model distillation method of any one of claims 1-9 or the target detection method of claim 10.

12. A computer readable storage medium for storing a computer program, characterized in that, When executed by a processor, the computer program is used to implement the model distillation method of any one of claims 1-9 or the target detection method of claim 10.

Images