Lightweight target detection model-oriented small sample increment updating method

Abstract

The invention discloses a lightweight target detection model-oriented small sample increment updating method. The method comprises the following steps of: (1) using a lightweight optimization version of YOLO as a backbone network for target detection; (2) the problem of disastrous forgetting is relieved by adopting a multi-granularity knowledge distillation module; and (3) a selective structure freezing module is used for keeping stability during incremental learning, and unification is realized through an innovative fusion fine tuning optimization scheme. According to the method, the recognition rate of the new class of samples can be improved while the recognition rate of the basic class of samples is kept.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a lightweight method for incremental update of target detection models using small samples. Background Technology

[0002] Few-shot incremental object detection (FSCIOD) is a challenging computer vision task that requires models to continuously learn new object classes from an extremely limited number of training samples (typically 1-10 per class) while maintaining performance on previously learned base classes. Unlike traditional object detection, FSCIOD must overcome two key challenges simultaneously: (1) the data scarcity inherent in few-shot learning; and (2) the catastrophic forgetting problem in incremental learning.

[0003] This task is typically evaluated using metrics that assess the model's ability to acquire new knowledge (mean average accuracy, mAP) and retain old knowledge (mean average accuracy and forgetting rate of the base classes). This makes FSCIOD particularly suitable for real-world applications where the system must scale its detection capabilities with minimal additional training data.

[0004] However, incremental learning on-device faces significant computational challenges due to hardware limitations. Current few-shot incremental object detection (FSCIOD) methods primarily rely on heavyweight backbone networks (e.g., ResNet-50 with 25.5 million parameters), which incur high computational and memory overhead during updates. Furthermore, most existing methods prioritize accuracy at the expense of increased algorithmic complexity, further exacerbating computational demands. These limitations collectively hinder practical deployment on resource-constrained embedded systems where efficiency and real-time performance are critical.

[0005] Existing network models suffer from high memory and computational overhead during updates, catastrophic forgetting of old classes after learning new ones, and unstable incremental learning. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a lightweight incremental update method for object detection models using small samples. (1) A lightweight optimized version of YOLO is used as the backbone network for object detection; (2) a multi-granularity knowledge distillation module is employed to mitigate the catastrophic forgetting problem; (3) a selective structure freezing module is used to maintain stability during incremental learning, achieving unification through an innovative fusion fine-tuning optimization scheme. This invention can improve the recognition rate of new types of samples while maintaining the recognition rate of basic class samples.

[0007] The technical solution adopted by this invention to solve its technical problem is as follows:

[0008] Step 1: Offline pre-training;

[0009] The object detection model is trained using a dataset containing a large number of base class data, with the training loss being:

[0010] L basic =L box +L cls

[0011] Where L box It is the bounding box loss, including the bounding box regression loss function L. cIou It is used to measure the overlap between the predicted bounding box and the ground truth bounding box, taking into account the distance between center points, aspect ratio, and coverage area; L cls It is the classification loss, namely the binary cross-entropy loss, which is calculated independently for each class.

[0012] Step 2: Incremental learning online;

[0013] Step 2-1: Use the original target detection model obtained through offline pre-training as the teacher model and student model; the student model uses a mixture of new class samples and old class samples as input and is trained and updated; the teacher model only uses old class samples for inference and does not update the model; when using historical category images for training, the teacher model performs multi-granularity knowledge distillation on the student model; the teacher model extracts the features of the hidden layer as guidance signals to cultivate the feature learning ability of the corresponding layer of the student model; by minimizing the mean square error or cosine similarity, the difference between the teacher model and the student model at the feature level is reduced, prompting the student model to gradually master the feature expression ability similar to that of the teacher model;

[0014] Step 2-2: Calculate the multi-particle size knowledge distillation loss, including the following process:

[0015] (1) Feature extraction: Extract feature maps from the corresponding hidden layers of the teacher model and the student model;

[0016] (2) Feature normalization: The extracted feature map is normalized so that the feature map has a uniform proportion in the channel dimension;

[0017] (3) Calculate the loss: Calculate the difference between the normalized feature maps using the specified loss function. The difference reflects the knowledge difference between the teacher model and the student model at the feature level.

[0018] (4) Optimization: By minimizing the multi-granularity knowledge distillation loss function, the student model learns the feature representation of the teacher model, thereby improving its own performance; the optimization method is as follows:

[0019]

[0020] in, Let i represent the characteristics of the teacher and student at position i in channel c, respectively. Here, C represents the number of channels in the feature map, W·H represents the spatial size of the feature map, and φ(.) is the softmax normalization of the feature map.

[0021] Steps 2-3: Calculate the response loss;

[0022] In the object detection model, each location in the output feature map corresponds to the probability of a certain class of object appearing at that location; the response loss compares the responses of the teacher model and the student model on these feature maps to ensure that the student model can learn a class activation pattern similar to that of the teacher model. The calculation formula is as follows:

[0023]

[0024] in, Let represent the teacher's and student's logistic values ​​at spatial location i, respectively; σ(·) is the spatial softmax normalization; KL(.) represents the KL divergence; and N represents the number of samples.

[0025] Steps 2-4: The total distillation loss is:

[0026]

[0027] Where α and β are two hyperparameters used to control the weights of conditional weighted distillation and response distillation losses; D base Represents a historical category dataset. As an indicator function, when the input sample (x,y) belongs to D base When the function is active, its value is 1; otherwise, it is 0.

[0028] Step 3: Fine-tune the blend;

[0029] A selective structure freezing method was adopted, in which layers 1, 3, 5, and 7 of YOLOv8n, i.e. the layer before each C2f module, were frozen for training, thereby enhancing the object detection model's ability to learn new samples.

[0030] During the fine-tuning of the target detection model, two complementary strategies are combined: (1) multi-granularity knowledge distillation and (2) selective structure freezing. The stability-plasticity dilemma in incremental learning is solved by alternating optimization schemes: multi-granularity knowledge distillation is applied in odd training iterations to retain existing knowledge, and selective structure freezing is implemented in even iterations to promote the acquisition of new knowledge.

[0031] Preferably, the specified loss function is the KL divergence.

[0032] An electronic device includes: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the above-described small sample incremental update method.

[0033] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described small-sample incremental update method.

[0034] A chip includes a processor for retrieving and running a computer program from a memory, causing a device equipped with the chip to perform the aforementioned small-sample incremental update method.

[0035] A computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the above-described small-sample incremental update method.

[0036] The beneficial effects of this invention are as follows:

[0037] Compared to existing methods, this invention combines multi-granularity knowledge distillation and selective backbone freezing strategies, enabling the ELFSDet model to improve the recognition rate of new classes while maintaining the recognition rate of basic class samples. The technical indicators achieved are: a target recognition rate of at least 35% higher on new sample test sets compared to the original model; a target recognition rate of at least 35% lower on historical sample test sets compared to training directly on new datasets; and a 50% reduction in the time from generating a model update requirement to deploying the new model. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method of the present invention;

[0039] Figure 2 These are the detection results of ELFSDet on the COCO and MAR20 datasets (when K=10). Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0041] This invention addresses the problems of slow updates and poor performance in existing technologies by proposing a virtual-real interoperability mechanism for digital twins, combining incremental learning and model caching. It achieves this by adding a storage component to cache the model, thus separating model updates from online invocation. Specifically: 1) The update component uses incremental learning to fully learn and retain every change in the target object. 2) The storage component dynamically controls the amount of model cached based on real-time computation requirements. 3) The model cache and computation component can achieve delayed invocation to meet real-time needs. Ultimately, this achieves iterative model updates while effectively ensuring the timeliness and stability of computation results.

[0042] This invention proposes a highly efficient and lightweight incremental target detection framework using a small number of samples, combining multi-granularity knowledge distillation and selective structure freezing techniques. This framework is implemented through three modules: (1) a lightweight optimized version of YOLO is used as the backbone network for target detection; (2) a multi-granularity knowledge distillation module is employed to mitigate the catastrophic forgetting problem; and (3) a selective structure freezing module is used to maintain stability during incremental learning, achieving unification through an innovative fusion fine-tuning optimization scheme.

[0043] This invention includes the following steps:

[0044] Step 1: Offline pre-training;

[0045] During the offline pre-training phase, the entire object detection model is trained using a dataset containing a large amount of base class data. The training loss is:

[0046] L basic =L box +L cls

[0047] Where L box It is the bounding box loss, which includes L cIou , used to measure the overlap between the predicted bounding box and the ground truth bounding box, taking into account the distance between center points, aspect ratio, and coverage area. L cls It is the classification loss, namely the binary cross-entropy loss, which is calculated independently for each class.

[0048] Step 2: Incremental online learning;

[0049] Knowledge distillation is a widely adopted technique used to reduce forgetting of historical classes as object detection models learn new categories.

[0050] First, during the incremental learning phase, a small number of new class samples arrive. The original object detection model, obtained through offline pre-training, is used as the teacher model. When training with historical class images, the teacher model performs knowledge distillation on the student model. Features extracted from the hidden layers by the teacher model serve as guidance signals, cultivating the student model's ability to learn features at the corresponding layers. By minimizing the mean squared error or cosine similarity, the differences between the teacher and student models at the feature level are reduced, enabling the student model to gradually master feature representation capabilities similar to the teacher model.

[0051] Secondly, the calculation of channel-based knowledge distillation loss includes the following process:

[0052] 1. Feature extraction: Extract feature maps from the corresponding hidden layers of the teacher and student models;

[0053] 2. Feature normalization: The extracted feature maps are normalized to ensure that the feature maps have a uniform proportion in the channel dimension;

[0054] 3. Calculate the loss: Use a specific loss function (such as KL divergence) to calculate the difference between the normalized feature maps. This difference reflects the knowledge difference between the two models at the feature level;

[0055] 4. Optimization: By minimizing the channel-based knowledge distillation loss function, the student model learns the feature representations of the teacher model, thereby improving its performance. The optimization method is as follows:

[0056]

[0057] Then, the response loss is calculated. In the object detection model, each position in the output feature map (usually a classification probability map) corresponds to the probability that a target of a certain class is present at that position. Response Loss ensures that the student model learns class activation patterns similar to those of the teacher model by comparing the responses of the teacher model and the student model on these feature maps. The calculation formula is as follows:

[0058]

[0059] Finally, a sample-level gating mechanism is implemented on the regularization term to maintain the object detection model's adaptability to new classes. This adaptive controller selectively disables regularization when processing samples from new classes. The overall distillation loss is obtained as follows:

[0060]

[0061] Symbol explanation: α and β are two hyperparameters used to control the weights of conditional weighted distillation and response distillation losses.

[0062] Step 3: Fine-tuning and integration;

[0063] The selective structure freezing method freezes appropriate layers of the backbone network and gradually unfreezes the network during training. The C2f module (CSP Bottleneck with 2 Convolutions) in YOLOv8 is a key network component for feature extraction and fusion. It processes the input data in two branches: one branch directly passes the data to the output, and the other branch is processed through multiple Bottleneck modules. In YOLOv8n, layers 1, 3, 5, and 7 (the layer preceding each C2f module) are frozen for training to enhance the object detection model's ability to learn from new samples.

[0064] Two complementary strategies are combined during the fine-tuning of the object detection model: (1) multi-granularity knowledge distillation (similar to memory reinforcement) and (2) selective structure freezing (similar to focused learning). This dual approach addresses the stability-plasticity dilemma in incremental learning through an alternating optimization scheme: knowledge distillation is applied in odd-numbered training iterations to preserve existing knowledge, while selective freezing is implemented in even-numbered iterations to facilitate the acquisition of new knowledge.

[0065] Example:

[0066] Reference Figure 1 A highly efficient and lightweight incremental target detection framework for few-sample classes;

[0067] To verify the effectiveness of this invention, this embodiment uses the MS COCO 2017 dataset and the MAR20 dataset. The MS COCO 2017 dataset contains 80 different object categories; the training set is used for training, and the validation set is used for testing. For COCO, 60 categories that do not intersect with PASCAL VOC are considered as base categories; the remaining 20 categories are considered as new categories. For the MAR20 dataset, A1 and A2 are considered as new categories, and the remaining 18 categories are considered as base categories. The object detection model achieved state-of-the-art performance in both 5-shot and 10-shot settings. In the 10-shot setting, the object detection model significantly improved the new category recognition performance by 40% while maintaining excellent overall accuracy across all categories, as shown in the results. Figure 2 The following is the implementation process of the ELFSDet model:

[0068] Step 1: Offline pre-training;

[0069] The entire object detection model is trained directly using a dataset containing a large amount of base class data, with the training loss being:

[0070] L basic =L box +L cls

[0071] A basic original target detection model is obtained;

[0072] Step 2: Incremental learning online;

[0073] Online incremental learning is performed based on the original object detection model. The process is as follows:

[0074] (1) Calculate the loss of channel-based knowledge distillation:

[0075] Feature extraction: Extract feature maps from the corresponding hidden layers of the teacher and student models;

[0076] Feature normalization: The extracted feature map is normalized so that the feature map has a uniform proportion in the channel dimension;

[0077] Loss calculation: A specific loss function (such as KL divergence) is used to calculate the difference between the normalized feature maps. This difference reflects the knowledge difference between the two models at the feature level;

[0078] Optimization: By minimizing the channel-based knowledge distillation loss function, the student model learns the feature representation of the teacher model, thereby improving its own performance.

[0079] The calculation formula is:

[0080]

[0081] (2) Calculate the loss of channel-based knowledge distillation:

[0082] In object detection models, Response Loss ensures that the student model learns category activation patterns similar to the teacher model by comparing the responses of the teacher and student models on the output feature maps. The calculation formula is as follows:

[0083]

[0084] (3) Obtain the total distillation loss:

[0085] The regularization term implements a sample-level gating mechanism to maintain the object detection model's adaptability to new classes. This adaptive controller selectively disables regularization when processing samples from new classes. The calculation formula is:

[0086]

[0087] The formula yields an overall loss.

[0088] Step 3: Fine-tune the integration;

[0089] Step 3 - Thread 1: Selective structure freezing;

[0090] During training, the network is gradually unfrozen. The C2f module in YOLOv8 processes the input data into two branches. One branch is directly passed to the output, while the other branch is processed by multiple Bottleneck modules. Layers 1, 3, 5, and 7 (the layer before each C2f module) are frozen in YOLOv8n for training to enhance the object detection model's ability to learn from new samples.

[0091] Step 3 - Thread 2: Merge Fine-tuning;

[0092] Knowledge distillation is applied in odd-numbered training iterations to preserve existing knowledge, while selective freezing is implemented in even-numbered iterations to facilitate the acquisition of new knowledge.

Claims

1. A lightweight method for incremental update of object detection models using small samples, characterized in that, Includes the following steps: Step 1: Offline pre-training; The object detection model is trained using a dataset containing a large number of base class data, with the training loss being: L basic L box +L cls Where L box It is the bounding box loss, including the bounding box regression loss function L. cIou It is used to measure the overlap between the predicted bounding box and the ground truth bounding box, taking into account the distance between center points, aspect ratio, and coverage area; L cls It is the classification loss, namely the binary cross-entropy loss, which is calculated independently for each class; Step 2: Incremental learning online; Step 2-1: Use the original target detection model obtained through offline pre-training as the teacher model and student model; the student model uses a mixture of new class samples and old class samples as input and is trained and updated; the teacher model only uses old class samples for inference and does not update the model; when using historical category images for training, the teacher model performs multi-granularity knowledge distillation on the student model; the teacher model extracts the features of the hidden layer as guidance signals to cultivate the feature learning ability of the corresponding layer of the student model; by minimizing the mean square error or cosine similarity, the difference between the teacher model and the student model at the feature level is reduced, prompting the student model to gradually master the feature expression ability similar to that of the teacher model; Step 2-2: Calculate the multi-particle size knowledge distillation loss, including the following process: (1) Feature extraction: Extract feature maps from the corresponding hidden layers of the teacher model and the student model; (2) Feature normalization: The extracted feature map is normalized so that the feature map has a uniform proportion in the channel dimension; (3) Calculate the loss: Calculate the difference between the normalized feature maps using the specified loss function. The difference reflects the knowledge difference between the teacher model and the student model at the feature level. (4) Optimization: By minimizing the multi-granularity knowledge distillation loss function, the student model learns the feature representation of the teacher model, thereby improving its own performance; the optimization method is as follows: in, Let i represent the characteristics of the teacher and student at position i in channel c, respectively. Here, C represents the number of channels in the feature map, W·H represents the spatial size of the feature map, and φ(.) is the softmax normalization of the feature map. Steps 2-3: Calculate the response loss; In the object detection model, each location in the output feature map corresponds to the probability of a certain class of object appearing at that location; the response loss compares the responses of the teacher model and the student model on these feature maps to ensure that the student model can learn a class activation pattern similar to that of the teacher model. The calculation formula is as follows: in, Let represent the teacher's and student's logistic values ​​at spatial location i, respectively; σ(·) is the spatial softmax normalization; KL(.) represents the KL divergence; and N represents the number of samples. Steps 2-4: The total distillation loss is: Where α and β are two hyperparameters used to control the weights of conditional weighted distillation and response distillation losses; D base Represents a historical category dataset. As an indicator function, when the input sample (x,y) belongs to D base When the function is active, its value is 1; otherwise, it is 0. Step 3: Fine-tune the blend; A selective structure freezing method was adopted, in which layers 1, 3, 5, and 7 of YOLOv8n, i.e. the layer before each C2f module, were frozen for training to enhance the learning ability of the object detection model for new samples. During the fine-tuning of the target detection model, two complementary strategies are combined: (1) multi-granularity knowledge distillation and (2) selective structure freezing. The stability-plasticity dilemma in incremental learning is solved by alternating optimization schemes: multi-granularity knowledge distillation is applied in odd training iterations to retain existing knowledge, and selective structure freezing is implemented in even iterations to promote the acquisition of new knowledge.

2. The lightweight object detection model small-sample incremental update method according to claim 1, characterized in that, The specified loss function is the KL divergence.

3. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 2.

5. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 / 2.

6. A computer program product, characterized in that, The computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the method as described in any one of claims 1 to 2.

Images