DETR knowledge distillation method based on teacher prediction optimization and target detection method

Abstract

The application provides a DETR knowledge distillation method and a target detection method based on teacher prediction optimization, and the distillation method comprises the following steps: inputting a training image into a teacher DETR model and a student DETR model, and obtaining classification and bounding box prediction results of each decoding stage; in the teacher DETR model, matching the prediction results of each stage with real labels, and dividing positive sample prediction and negative sample prediction; for the teacher prediction result of the current decoding stage, screening candidate predictions from historical decoding stages, performing a prediction correction operation based on the positive sample prediction, performing a prediction suppression operation based on the negative sample prediction, and generating an optimized teacher prediction result; and constructing a distillation loss based on the optimized teacher prediction result, and guiding the student DETR model to complete training. The distillation supervision of the application is no longer limited to the instantaneous output of the teacher model at a certain moment, but can reflect the detection ability gradually formed by the teacher model in the multi-stage reasoning process, thereby improving the reliability and consistency of the supervision signal.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to a DETR knowledge distillation method and object detection method based on teacher prediction optimization. Background Technology

[0002] Object detection is one of the fundamental and critical tasks in the field of computer vision, and it is a crucial technical support for applications such as smart city security, autonomous driving, intelligent transportation, and video surveillance. In smart security and intelligent sensing scenarios, object detection models need to accurately locate and identify multiple categories of targets in complex environments, while balancing real-time performance and system deployment costs. Therefore, how to reduce model complexity and improve the deployability of models on resource-constrained devices while ensuring detection accuracy has become a critical issue that urgently needs to be addressed in the engineering implementation of object detection technology.

[0003] In recent years, detection models based on the Transformer architecture, represented by DETR (Detection Transformer), have avoided the complex candidate box generation and post-processing steps of traditional detection frameworks by introducing ensemble prediction and binary matching mechanisms. This enables an end-to-end object detection process, demonstrating significant advantages in model structure simplicity and global modeling capabilities. However, DETR and its improved models typically rely on deep encoders and multi-stage decoder structures, resulting in large model parameter scales and high computational overhead. Deployment in edge devices or scenarios with limited computing power faces significant challenges, which to some extent restricts their widespread application in practical engineering systems.

[0004] To maintain detection performance while reducing model size, knowledge distillation is widely used for compressing and accelerating object detection models. Given the structural characteristics of DETR, existing distillation methods mainly focus on constructing distillation alignment points, such as through shared queries, Hungarian matching, or fixed matching strategies, to achieve consistency constraints between teacher and student model predictions. These methods alleviate the difficulties caused by the disordered nature of predictions in DETR to some extent, but they generally assume that the prediction results output by the teacher model in the current decoding stage can be directly used as reliable distillation supervision signals.

[0005] However, the DETR decoder employs a multi-stage cascaded update mechanism, with prediction results evolving continuously across different decoding stages. Research has found that in actual training, the prediction quality of the teacher model does not always monotonically improve with each decoding stage. Instead, the following situations may occur: some positive sample predictions show decreased localization accuracy or classification confidence in subsequent stages; some negative sample predictions produce overconfident misclassification results in later stages. This results in high-quality predictions generated in earlier decoding stages not being fully utilized in subsequent distillation processes. Under these circumstances, directly using the teacher predictions from the current decoding stage as distillation supervision can easily introduce noise or unstable signals, thereby weakening the role of knowledge distillation in improving the student model's performance. Therefore, how to fully extract the effective information contained in the teacher model's predictions at different stages during the multi-stage decoding process of DETR, and how to construct a high-quality distillation supervision mechanism while ensuring the stability and reliability of the supervision signal, have become key technical issues restricting further performance improvements of existing DETR knowledge distillation methods.

[0006] A search revealed Chinese patent application number 202311259031.4, which discloses a knowledge distillation method applicable to DETR-type detectors. This method first trains a convergent teacher model and freezes its parameters. The training set is then input into both the teacher and student models, and the student model is trained through three levels of joint distillation: logits-level, feature-level, and assignment-level. This approach solves the problems of uncertain matching and poor generalization in traditional distillation, achieving adaptive foreground / background segmentation and stable knowledge transfer. However, it fails to consider situations where the DETR model is not the optimal prediction result for the current stage, resulting in insufficiently accurate distillation supervision signals. Summary of the Invention

[0007] To address one of the shortcomings of existing technologies, the purpose of this application is to provide a DETR knowledge distillation method based on teacher prediction optimization, so as to improve the training stability and detection performance of student models in object detection tasks.

[0008] A first aspect of this application provides a DETR knowledge distillation method based on teacher prediction optimization, comprising: The training images are input into the teacher DETR model and the student DETR model to obtain the classification and bounding box prediction results at each decoding stage. In the teacher DETR model, the prediction results of each decoding stage are matched with the real annotations, and the prediction results of each stage of the teacher DETR model are divided into a positive sample prediction set and a negative sample prediction set. For the teacher prediction results at each decoding stage, candidate predictions are selected from the historical decoding stages before that decoding stage. A prediction correction operation is performed based on the positive sample prediction set, and a prediction suppression operation is performed based on the negative sample prediction set to generate optimized teacher prediction results. Distillation loss is constructed based on the optimized teacher prediction results of each decoding stage and the prediction results of the student DETR model at the corresponding decoding stage, guiding the student DETR model to complete training.

[0009] Optionally, performing the prediction correction operation based on the positive sample prediction set includes: Iterate through the historical decoding stages preceding the current decoding stage; If the prediction result of the same index position in a certain historical stage is better than the prediction result of the current decoding stage in both classification confidence and bounding box localization accuracy, then the prediction result of the historical stage is marked as a candidate prediction. The best candidate prediction is determined from the candidate predictions and replaces the prediction result in the current decoding stage.

[0010] Optionally, determining the best candidate prediction from the candidate predictions includes: Calculate the overall quality score for each candidate prediction, whereby the overall quality score is a weighted combination of classification confidence and bounding box localization accuracy; The candidate prediction with the highest overall quality score is determined as the best candidate prediction.

[0011] Optionally, performing prediction suppression based on the negative sample prediction set includes: Iterate through the historical decoding stages preceding the current decoding stage; When the predicted category of the same index position in a certain historical decoding stage is consistent with the predicted category in the current decoding stage, and the classification confidence of the same index position in the historical decoding stage is lower than the classification confidence of the current decoding stage, the prediction result of the historical decoding stage is marked as a candidate prediction. The best candidate prediction is determined from the candidate predictions and replaces the negative sample prediction result in the current decoding stage.

[0012] Optionally, determining the best candidate prediction from the candidate predictions includes: Among the candidate predictions, the prediction with the lowest classification confidence is selected as the best candidate prediction.

[0013] Optionally, generating the optimized teacher prediction results includes: For positive samples, only the classification component and bounding box prediction component corresponding to the target category in the current decoding stage prediction result are replaced, while the original classification component of non-target categories is retained; For negative samples, only the classification component corresponding to the target category in the prediction result of the current decoding stage is replaced, while the original classification component of the non-target category is retained.

[0014] Optionally, the step of constructing a distillation loss based on the optimized teacher prediction results and the corresponding decoding stage prediction results of the student DETR model to guide the student DETR model to complete training includes: A decoupled knowledge distillation strategy is adopted to construct a distillation loss function, which includes a target category distillation term and a dark knowledge distillation term for non-target categories. The target category distillation term is calculated based on the difference in classification confidence between the teacher DETR model and the student DETR model in the target category, and is weighted by a weight coefficient α; The non-target category dark knowledge distillation term is calculated based on the difference in normalized probability distributions between the teacher DETR model and the student DETR model after removing the target category, and is weighted by the weight coefficient β. The weighting coefficients α and β are used to balance the contributions of target category distillation and non-target category dark knowledge distillation. The sum of the distillation loss functions is used to guide the student DETR model to complete training.

[0015] A second aspect of this application provides a target detection method based on a distillation model, applied to a terminal device, comprising: Load the detection model, wherein the detection model is a student DETR model trained by any of the DETR knowledge distillation methods based on teacher prediction optimization described in the above-mentioned articles; The image or video frame to be detected is acquired through the image acquisition device built into the terminal device. The image or video frame to be detected is input into the detection model for forward inference calculation; Output the target detection results, which include the target category information, bounding box location information, and confidence score in the image; The terminal device includes one of a mobile terminal, an embedded device, a drone, or a network camera.

[0016] A third aspect of this application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described above.

[0017] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the methods described above.

[0018] This application presents a DETR knowledge distillation method based on teacher prediction optimization. It proposes a teacher prediction optimization mechanism tailored to the characteristics of multi-stage decoding, enabling the distillation process to more fully and stably utilize the superior prediction information of the teacher model throughout the entire decoding evolution. Through this mechanism, distillation supervision is no longer limited to the instantaneous output of the teacher model at a single moment, but rather reflects the detection capabilities gradually developed by the teacher model during multi-stage reasoning, thus ensuring the reliability and consistency of the supervision signal.

[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram illustrating a prior distillation method according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a DETR knowledge distillation method based on teacher prediction optimization according to an exemplary embodiment; Figure 3 This is a general framework diagram of a DETR knowledge distillation method based on teacher prediction optimization, according to an exemplary embodiment. Figure 4 This is a schematic diagram illustrating the identification of better predicted samples according to an exemplary embodiment; Figure 5 This is a schematic diagram illustrating a maximum dark knowledge retention strategy according to an exemplary embodiment; Figure 6 This is a distillation process diagram illustrated according to an exemplary embodiment. Detailed Implementation

[0021] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0022] In the DETR model, the decoder generates predictions progressively through multi-stage cascaded updates. The distillation process typically relies on the predictions output by the teacher model at each decoding stage as supervision signals (e.g., ...). Figure 1(See the previous distillation method shown). However, existing methods generally assume that the predictions generated by the teacher model at the current decoding stage have high reliability, without fully considering the non-monotonic changes in prediction quality that may occur during multi-stage decoding. In actual training, because the classification confidence and bounding box regression results are affected by feature updates and matching mechanisms at different decoding stages, teacher predictions may experience problems such as decreased localization accuracy and amplified false prediction confidence. Directly using such predictions as distillation supervision can easily introduce noise information, thereby limiting further improvement in the performance of the student model. Based on the above problems, this application provides a DETR knowledge distillation method based on teacher prediction optimization to solve the aforementioned problems.

[0023] Reference Figure 2-6 In one embodiment of this application, a DETR knowledge distillation method based on teacher prediction optimization includes the following steps: S100: Input the training image into the teacher DETR model and the student DETR model to obtain the classification and bounding box prediction results at each decoding stage; S200, in the teacher DETR model, matches the prediction results of each decoding stage with the real annotations, and divides the prediction results of each stage of the teacher DETR model into a positive sample prediction set and a negative sample prediction set. S300: For the teacher prediction results of each decoding stage, candidate predictions are selected from the historical decoding stages before the current decoding stage. Based on the positive sample prediction set, a prediction correction operation is performed, and based on the negative sample prediction set, a prediction suppression operation is performed to generate optimized teacher prediction results. Specifically, this step can also be called teacher prediction optimization, which includes collecting multi-stage predictions from the teacher model, identifying better predictions from historical stages, and optimizing the teacher model's predictions for the current stage.

[0024] S400 constructs a distillation loss based on the optimized teacher prediction results at each decoding stage and the corresponding decoding stage prediction results of the student DETR model, guiding the student DETR model to complete training.

[0025] The above embodiments fully utilize the prediction information generated by the teacher model during the multi-stage decoding process to construct a more reliable distillation supervision signal.

[0026] In some specific embodiments of this application, in S100, the training image is input into the teacher DETR model and the student DETR model to obtain the classification and bounding box prediction results at each decoding stage. This can be achieved by the following steps: S101, Obtain the teacher DETR model and the student DETR model; S102, Obtain training images.

[0027] In S103, the training images are input into the two models in S101 respectively. The feature representations are extracted by the backbone network and encoder respectively, and the decoder performs multi-stage decoding to obtain the classification prediction results and bounding box prediction results corresponding to each decoding stage.

[0028] Specifically, the prediction for each decoding stage consists of a set of predictions corresponding to a set of queries, where the prediction for the i-th query in the j-th decoding stage can be represented as... ,in Indicates the classification prediction results. This indicates the predicted bounding box of the target object.

[0029] This embodiment provides the necessary data foundation for subsequent prediction optimization based on historical stages by synchronously acquiring multi-stage prediction results from teacher and student models.

[0030] Multi-stage prediction presents various challenges. Some positive sample predictions may lead to decreased localization accuracy or classification confidence in subsequent stages; others may result in overconfident misclassifications in later stages. Therefore, it is necessary to differentiate the prediction results. In some specific embodiments of this application, in step S200, in the teacher DETR model, the prediction results of each decoding stage are matched with the ground truth annotations. The prediction results of each stage of the teacher DETR model are divided into a positive sample prediction set and a negative sample prediction set. This can be achieved through the following steps: Figure 3 and Figure 4 As shown: S201, The matching algorithm is determined to be the Hungarian algorithm.

[0031] Specifically, the Hungarian Algorithm (also known as the Kuhn-Munkres algorithm) is a classic multinomial-time algorithm in graph theory / combinatorial optimization for solving the maximum matching and optimal matching (assignment problem) problems in bipartite graphs.

[0032] S202 uses the Hungarian algorithm from S201 to perform a one-to-one matching between the predictions of the teacher DETR model and the ground truth labels, thereby dividing the predictions of the teacher model obtained in S103 into a set of positive sample predictions. With negative sample prediction set .

[0033] Specifically, in the matching process, firstly, based on the classification prediction results and target bounding box prediction results of each query output of the teacher's DETR model, and combined with the category and bounding box information of the real labels, a cost matrix between the prediction results and the real labels is constructed. Then, the Hungarian algorithm is used to perform optimal binary matching on the cost matrix to achieve a one-to-one allocation between the prediction results and the real labels. Finally, the prediction results that successfully match with the real labels are determined as positive sample predictions, and the prediction results that do not form a matching relationship with any real labels are determined as negative sample predictions.

[0034] All positive sample predictions constitute the positive sample prediction set. All negative sample predictions constitute the negative sample prediction set. .

[0035] For example, Figure 4 In China, the real label T 1, T 2 represents the ground truth of objects in the image. Decoding layer. i This represents the current decoding stage (current layer) of the teacher model.

[0036] Hungarian matching means that the current layer i The prediction results ( , , , This involves performing a one-to-one match with the real labels to determine which samples are positive (successful matches are indicated by red text). Which are negative samples (failed matches, such as those in blue)? ).

[0037] This embodiment achieves accurate separation of positive and negative samples through Hungarian matching, providing a basis for subsequent differentiated optimization processing for different types of predictions.

[0038] Furthermore, such as Figure 4 As shown in some specific embodiments of this application, after obtaining the sample type for each stage through Hungarian matching, the samples of the current layer are replaced. Specifically, this is done by selecting the best sample from historical stages. Decoding layer j represents a historical decoding stage preceding the teacher model (j <i)。

[0039] The expression `for j in [1, i-1]` means iterating through all historical layers preceding the current layer.

[0040] The red / blue dashed arrows indicate the prediction of historical layer j. The prediction of the current layer i ( ) to make comparisons.

[0041] Figure 4 In the middle, the red dashed line indicates that the positive sample correction condition is met. (For example, historical layer localization is more accurate). The blue dashed line indicates that the negative sample suppression condition is met. (For example, the confidence level of the historical layer is more reasonable).

[0042] If the correction or suppression conditions are met, a better candidate set is predicted, resulting in the optimal candidate (which will be used as the "optimized teacher prediction result" for distillation at this query position).

[0043] To address the potential decrease in localization accuracy or instability in classification confidence during multi-stage decoding of positive sample predictions, this application proposes a positive sample correction strategy. This strategy prioritizes using prediction results from historical decoding stages that simultaneously possess higher localization accuracy and classification confidence for supervised construction, thereby avoiding the impact of prediction degradation on distillation quality. In some specific embodiments of this application, the step of S300, which involves selecting candidate predictions from historical decoding stages and performing prediction correction operations based on positive sample predictions, can employ the following steps: S311, For positive sample prediction, when a historical prediction outperforms the current prediction in both target localization accuracy and classification confidence, the historical prediction is considered a better prediction. The criterion for this determination is defined as follows: in, This represents the prediction of the j-th historical stage for the i-th query. Better than current stage forecast The judgment criteria , These are the prediction results for the i-th query in the j-th historical decoding stage and the current decoding stage, respectively. , These are the corresponding predicted target bounding boxes; The true target bounding box that matches the i-th query; The intersection-union ratio (IUU) is used to measure the accuracy of target positioning. , These represent the corresponding predicted classification probability vectors; , These represent the confidence scores of the true class in the corresponding predictions, used to measure classification confidence. This indicates that the i-th query belongs to the set of positive samples. This represents the index of the true target corresponding to the prediction, where "true" represents the index of the true category.

[0044] S312, Based on the above judgment conditions, a candidate stage set for positive sample prediction can be constructed: S313. Since optimal predictions in both localization and classification are rare in actual training, a comprehensive quality score is introduced to screen candidate predictions. The best candidate prediction is determined by maximizing a weighted combination of classification confidence and localization accuracy, defined as follows: in This is a hyperparameter used to balance the contribution of classification confidence to positioning accuracy.

[0045] The above embodiments ensure that distillation supervision consistently reflects the optimal detection level achieved by the teacher model in previous stages. Unlike existing methods that directly use the prediction at the current decoding stage as supervision, distillation supervision comprehensively compares and filters the prediction results of the teacher model at different decoding stages, enabling it to reflect the stable detection capabilities that the teacher model has developed throughout the multi-stage inference process. To address the overconfidence that easily occurs in negative sample prediction, this application proposes a negative sample suppression strategy. This strategy reduces the cumulative effect of noise supervision in the multi-stage distillation process by suppressing overconfident erroneous predictions in subsequent stages. In some specific embodiments of this application, the prediction suppression operation based on negative sample prediction in step S300 can be performed using the following steps: S321, For negative sample prediction, when the historical prediction and the current prediction have the same prediction category, and the historical prediction has a lower classification confidence than the current prediction, the historical prediction is considered a better prediction. The criterion for this is defined as follows: in This indicates that in the context of negative samples, historical predictions Better than current forecasts The criteria for determination; .

[0046] S322, among the historical predictions that meet the above conditions, the prediction with the lowest classification confidence is selected as the best candidate, defined as follows: The above embodiments effectively suppress the adverse effects of noise supervision on student model training during multi-stage propagation and accumulation, thereby significantly enhancing the stability of distillation training under complex sample distributions and difficult training stages.

[0047] After obtaining the best candidate predictions for both positive and negative samples, the teacher predictions are further optimized. For example... Figure 4As shown, to avoid the loss of teacher model dark knowledge due to direct replacement of predictions, some specific embodiments of this application adopt a maximum dark knowledge preservation strategy, that is, only the prediction component corresponding to the target category is replaced, while the predictions of other categories remain unchanged. Specifically, the teacher prediction optimization operation is defined as: in Let represent the classification logits before activation, and t represent the target category index. After the above optimization operations, construct the optimized teacher prediction for the current decoding stage: This yields an optimized set of teacher predictions for distillation training. .

[0048] For example, Figure 5 middle, The representative teacher model in the j Predictions from each historical decoding stage. This is a categorical distribution, where the bar chart displays the probability of classification. Dark green represents the target category, and light green represents the non-target category. These are bounding boxes; the red solid lines represent predicted bounding boxes, and the green dashed lines represent actual bounding boxes. The figure shows an IoU of 0.8, indicating high positioning accuracy in historical periods. This is a prediction for the current stage, representing the teacher model in the [stage / phase]. i The prediction result of the current decoding stage. It is also a classification distribution, representing the classification probability distribution at the current stage. This is the bounding box. The figure shows IoU:0.6, indicating that the positioning accuracy at the current stage is relatively low (degradation has occurred). This represents the optimized teacher prediction, i.e., the final monitoring signal. (This is an optimized bounding box), and we can see that its IoU is 0.8, indicating that it directly inherited the historical stage. j High-precision positioning. It optimizes the classification distribution, meaning that the target category (dark green) maintains a high confidence level (inherited from historical stages), while the distribution structure of non-target categories (light green) is preserved (dark knowledge preservation).

[0049] The specific execution process of the maximum dark knowledge preservation strategy is as follows: Compared to the current stage i With historical stages j Boundary box prediction. Due to historical stages. j The IoU (0.8) is better than the current stage. i The IoU (0.6) indicates that the historical period is considered a “better prediction”.

[0050] Correct the target category: adjust the optimized prediction results The target category classification logits and bounding box coordinates in the data should be replaced with historical stages. j The value ensures high accuracy of the monitoring signal.

[0051] Non-target categories are also preserved: during generation At that time, retain the current stage. i The distribution of logits for the non-target category. This means that although the location uses historical data, the contextual relationship (tacit knowledge) that "it is not like other categories" still uses the current data.

[0052] The final generated It possesses both high-precision positioning from historical periods and retains the category relationship structure of the current period, serving as the target value for distillation loss calculation.

[0053] The above embodiments avoid replacing the teacher's output entirely when optimizing the teacher's prediction. Instead, they adjust the key information in a way that minimizes perturbation. This improves the effectiveness of distillation supervision while preserving the inter-category relationship information and dark knowledge structure contained in the teacher model to the greatest extent. This allows the student model to fully inherit the discriminative ability of the teacher model under stable supervision, thereby improving the training stability and detection performance in the target detection task.

[0054] To further reduce the impact of prediction replacement on inter-category relationships, this application employs a decoupled knowledge distillation strategy to supervise the optimization of teacher predictions. A distillation loss function is constructed, comprising target category distillation terms and non-target category dark knowledge distillation terms. The target category distillation terms are calculated based on the difference in classification confidence between the teacher's DETR model and the student's DETR model for the target category, and are weighted by a weighting coefficient α. The non-target category dark knowledge distillation terms are calculated based on the difference in normalized probability distribution between the teacher's DETR model and the student's DETR model after removing the target category, and are weighted by a weighting coefficient β.

[0055] Specifically, the distillation loss function is defined as: in, It is the target category (the correct classification label for positive samples and the category with the highest classification score for negative samples) distillation weight, used to adjust the importance of the student model learning the "correct category" prediction from the teacher model; The dark knowledge distillation weights for non-target categories (labels other than the correct label) are used to adjust the importance of the student model learning the "relationships between incorrect categories" (i.e., dark knowledge) from the teacher model. t represents the target category index (i.e., the true labeled category). It is the predicted probability of the teacher model on the target category. It is the predicted probability of the student model on the target category. This represents the normalized probability distribution of the teacher model on the i-th non-target category; Let represent the normalized probability distribution of the student model on the i-th non-target category.

[0056] The above embodiments decouple the target category from the non-target category, unify the prediction of positive and negative samples into the same distillation framework, and preserve the probability distribution structure corresponding to the non-target category, thereby achieving full preservation and effective utilization of the teacher model's dark knowledge.

[0057] In some specific embodiments of this application, S400, a distillation loss is constructed based on the optimized teacher prediction results of each decoding stage and the prediction results of the student DETR model corresponding to the decoding stage, guiding the student DETR model to complete training. The specific process is as follows: S401, perform distillation point alignment.

[0058] The optimized teacher prediction results are matched with the prediction results of the corresponding decoding stage of the student DETR model to establish a one-to-one correspondence between teacher and student queries, ensuring that the distillation supervision signal is transmitted to the correct student node and achieving distillation point alignment.

[0059] S402, based on the aligned prediction results, employs a decoupled knowledge distillation strategy to construct a loss function. Specifically, this includes calculating the classification confidence difference for the target category, the dark knowledge distribution difference for non-target categories, and the bounding box regression difference, and then weighted summing of these losses to obtain the total distillation loss. S403, backpropagation is performed based on the total distillation loss to calculate the gradient and update the parameters of the student DETR model. The above steps are repeated until the model converges to obtain the trained student DETR model.

[0060] The above embodiments completed the training of the student model.

[0061] Based on the same technical concept, other embodiments of this application provide a target detection method based on a distillation model, applied to a terminal device, including: S1, Load the detection model, which is a student DETR model trained by any of the DETR knowledge distillation methods based on teacher prediction optimization; Specifically, the distillation method includes the following steps: S11, input the training image into the teacher DETR model and the student DETR model to obtain the classification and bounding box prediction results at each decoding stage; S12, in the teacher DETR model, the prediction results of each decoding stage are matched with the real annotations, and the prediction results of each stage of the teacher DETR model are divided into a positive sample prediction set and a negative sample prediction set. S13, for the teacher prediction results of each decoding stage, candidate predictions are selected from the historical decoding stages before the current decoding stage, prediction correction operation is performed based on the positive sample prediction set, prediction suppression operation is performed based on the negative sample prediction set, and optimized teacher prediction results are generated. S14. Based on the optimized teacher prediction results of each decoding stage and the corresponding decoding stage prediction results of the student DETR model, a distillation loss is constructed to guide the student DETR model to complete the training.

[0062] The specific implementation process, technical details and preferred solutions of steps S11 to S14 are completely consistent with the aforementioned DETR knowledge distillation method embodiment based on teacher prediction optimization in this application. For detailed description, please refer to the relevant records above, and they will not be repeated here.

[0063] S2 acquires the image or video frame to be detected through the image acquisition device built into the terminal device; S3, input the image or video frame to be detected into the detection model and perform forward inference calculation; S4 outputs the target detection results, which include the target category information, bounding box location information, and confidence score in the image. The terminal device includes one of the following: a mobile terminal, an embedded device, a drone, or a network camera.

[0064] The distillation method described herein achieves a systematic improvement in the quality of distillation supervision without introducing additional inference overhead or altering the original DETR inference structure. This allows student models to more efficiently inherit the discriminative ability and class relationship structure of teacher models, demonstrating significant advantages in detection accuracy, training robustness, and adaptability to different model sizes and structures. This distillation method exhibits good versatility for existing DETR models and their various improved versions, possessing high engineering application value and promising prospects for widespread adoption.

[0065] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0066] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0067] In the description of the embodiments in this application, "multiple" means two or more, unless otherwise explicitly specified. In this application, unless otherwise explicitly specified and limited, the terms "installed," "connected," "linked," "fixed," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0068] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0069] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-mentioned DETR knowledge distillation method and target detection method based on teacher prediction optimization.

[0070] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described DETR knowledge distillation method and target detection method based on teacher prediction optimization.

[0071] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0072] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0073] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0074] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0075] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A DETR knowledge distillation method based on teacher prediction optimization, characterized in that, include: The training images are input into the teacher DETR model and the student DETR model to obtain the classification and bounding box prediction results at each decoding stage. In the teacher DETR model, the prediction results of each decoding stage are matched with the real annotations, and the prediction results of each stage of the teacher DETR model are divided into a positive sample prediction set and a negative sample prediction set. For the teacher prediction results at each decoding stage, candidate predictions are selected from the historical decoding stages before that decoding stage. A prediction correction operation is performed based on the positive sample prediction set, and a prediction suppression operation is performed based on the negative sample prediction set to generate optimized teacher prediction results. Distillation loss is constructed based on the optimized teacher prediction results of each decoding stage and the prediction results of the student DETR model at the corresponding decoding stage, guiding the student DETR model to complete training.

2. The DETR knowledge distillation method based on teacher prediction optimization according to claim 1, characterized in that, The step of performing prediction correction operation based on the positive sample prediction set includes: Iterate through the historical decoding stages preceding the current decoding stage; If the prediction result of the same index position in a certain historical stage is better than the prediction result of the current decoding stage in both classification confidence and bounding box localization accuracy, then the prediction result of the historical stage is marked as a candidate prediction. The best candidate prediction is determined from the candidate predictions and replaces the prediction result in the current decoding stage.

3. The DETR knowledge distillation method based on teacher prediction optimization according to claim 2, characterized in that, The step of determining the best candidate prediction from the candidate predictions includes: Calculate the overall quality score for each candidate prediction, whereby the overall quality score is a weighted combination of classification confidence and bounding box localization accuracy; The candidate prediction with the highest overall quality score is determined as the best candidate prediction.

4. The DETR knowledge distillation method based on teacher prediction optimization according to claim 1, characterized in that, The prediction suppression operation based on the negative sample prediction set includes: Iterate through the historical decoding stages preceding the current decoding stage; When the predicted category of the same index position in a certain historical decoding stage is consistent with the predicted category in the current decoding stage, and the classification confidence of the same index position in the historical decoding stage is lower than the classification confidence of the current decoding stage, the prediction result of the historical decoding stage is marked as a candidate prediction. The best candidate prediction is determined from the candidate predictions and replaces the negative sample prediction result in the current decoding stage.

5. The DETR knowledge distillation method based on teacher prediction optimization according to claim 4, characterized in that, The step of determining the best candidate prediction from the candidate predictions includes: Among the candidate predictions, the prediction with the lowest classification confidence is selected as the best candidate prediction.

6. The DETR knowledge distillation method based on teacher prediction optimization according to claim 1, characterized in that, The generated optimized teacher prediction results include: For positive samples, only the classification component and bounding box prediction component corresponding to the target category in the current decoding stage prediction result are replaced, while the original classification component of non-target categories is retained; For negative samples, only the classification component corresponding to the target category in the prediction result of the current decoding stage is replaced, while the original classification component of the non-target category is retained.

7. The DETR knowledge distillation method based on teacher prediction optimization according to claim 1, characterized in that, The distillation loss is constructed based on the optimized teacher prediction results at each decoding stage and the prediction results of the student DETR model at the corresponding decoding stage, guiding the student DETR model to complete training, including: A decoupled knowledge distillation strategy is adopted to construct a distillation loss function, which includes a target category distillation term and a dark knowledge distillation term for non-target categories. The target category distillation term is calculated based on the difference in classification confidence between the teacher DETR model and the student DETR model in the target category, and is weighted by a weight coefficient α; The non-target category dark knowledge distillation term is calculated based on the difference in normalized probability distributions between the teacher DETR model and the student DETR model after removing the target category, and is weighted by the weight coefficient β. The weighting coefficients α and β are used to balance the contributions of target category distillation and non-target category dark knowledge distillation. The sum of the distillation loss functions is used to guide the student DETR model to complete training.

8. A target detection method based on a distillation model, applied to a terminal device, characterized in that, include: Load the detection model, wherein the detection model is a student DETR model trained by the method described in any one of claims 1-7; The image or video frame to be detected is acquired through the image acquisition device built into the terminal device. The image or video frame to be detected is input into the detection model for forward inference calculation; Output the target detection results, which include the target category information, bounding box location information, and confidence score in the image; The terminal device includes one of a mobile terminal, an embedded device, a drone, or a network camera.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, this program can be used to perform the method of any one of claims 1-8.

Images