An incremental object detection method based on multi-network mean value distillation loss function

Abstract

The application provides an incremental target detection method based on a multi-network mean value distillation loss function, and comprises the following steps: S1, obtaining to-be-trained image data; S2, inputting the to-be-trained image data obtained in the step S1 into a training model for training to obtain a target model; and S3, inputting to-be-processed image data into the target model in the step S2 to obtain an image result. According to the Faster-RCNN detector, the distillation output of the backbone network at the input end and the RoI_Head network at the output end is strengthened, the RPN network at the intermediate stage is adaptively distilled to retain the past knowledge, the meta learning mode in the application is adopted to relieve the influence of the model learning performance decline caused by the knowledge distillation, and in addition, the output data of the backbone network and the RPN network of the new and old models is zero-meaned to relieve the output data distribution deviation problem of the new and old models.

Description

Technical Field

[0001] This invention relates to the field of incremental target detection technology, and in particular to an incremental target detection method based on a multi-network mean distillation loss function. Background Technology

[0002] The emergence of neural networks has significantly improved the performance of deep learning models on visual tasks, such as object detection and semantic segmentation. However, existing supervised models can only learn from task data that includes labels already present in the current training data. When a new task appears, the model needs to adjust its network parameters to adapt to it, which leads to a decline in the model's performance on past tasks and a forgetting of old knowledge. This makes it difficult for existing object detection models to adapt to dynamic real-world environments.

[0003] In this work, we investigate incremental multi-network object detection based on the catastrophic forgetting mechanism. In the incremental setting, task queues are sequentially introduced into the object detector, and a high-performance agent should maintain the performance of old tasks during the learning process of new tasks. Therefore, at the model parameter level, the adaptive updating of model parameters due to new task inputs is gradually proposed by setting knowledge distillation constraints. However, for the multi-network structure Faster R-CNN, distilling only a single network is insufficient to alleviate the catastrophic forgetting problem. A more effective approach is to use multi-network distillation to maintain the entire network's retention of old knowledge. The choice of multi-network distillation method directly affects model performance. Furthermore, existing incremental object detection methods based on multi-network knowledge distillation, due to the uncompromising nature of knowledge distillation, tend to favor learning old tasks, thus reducing the effectiveness of learning new tasks. Summary of the Invention

[0004] This invention aims to at least solve the technical problems existing in the prior art, and in particular, it innovatively proposes an incremental target detection method based on the multi-network mean distillation loss function.

[0005] To achieve the above-mentioned objectives of this invention, this invention provides an incremental target detection method based on a multi-network mean distillation loss function, comprising the following steps:

[0006] S1, Obtain the image data to be trained;

[0007] S2, input the image data to be trained obtained in step S1 into the training model for training to obtain the target model;

[0008] S3, input the image data to be processed into the target model in step S2 to obtain the image result.

[0009] In a preferred embodiment of the present invention, the training image data obtained in step S1 includes the PASCALVOC 2007 and / or MS COCO 2014 datasets.

[0010] In a preferred embodiment of the present invention, the method for obtaining the target model in step S2 includes:

[0011]

[0012] in, This represents the total loss of the RoI_Head network;

[0013] This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network.

[0014] p represents the predicted classification probability of the RoI_Head network;

[0015] p * The tag value represents the actual class;

[0016] λ represents the balance coefficient between classification loss and regression loss;

[0017] [p * ≥1] indicates the condition for bounding box regression calculation;

[0018] This represents the Smooth L1 bounding box regression loss of the RoI_Head network;

[0019] l indicates the position of the predicted class bounding box by the model;

[0020] l * Indicates the actual border position of the class;

[0021]

[0022] in, This represents the total loss of the RPN network;

[0023] This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network.

[0024] o indicates whether the region contains the predicted score for the class;

[0025] o * This indicates whether the region contains the true score for the class;

[0026] λ represents the balance coefficient between classification loss and regression loss;

[0027] This represents the Smooth L1 bounding box regression loss of the RoI_Head network;

[0028] r represents the predicted bounding box position;

[0029] r * Indicates the actual bounding box location.

[0030] In a preferred embodiment of the present invention, it further includes:

[0031]

[0032] in, This represents the value of the i-th network output after being softened and normalized by the temperature coefficient T;

[0033] x i This represents the unnormalized output value of the i-th network;

[0034] x i This represents the unnormalized output value of the j-th network;

[0035] n represents the number of network output values;

[0036] T represents the temperature coefficient;

[0037]

[0038] in, This represents the value of a pixel after zero-mean normalization.

[0039] y i Represents pixel value;

[0040] n represents the number of pixels.

[0041] In a preferred embodiment of the present invention, it further includes:

[0042]

[0043] in, This represents the distillation loss of the backbone feature network;

[0044] min means to find the minimum value;

[0045] express Each pixel feature;

[0046] express Each pixel feature;

[0047] This represents the feature layer after zero-mean normalization of the teacher network.

[0048] This represents the feature layer of the student network after zero-mean normalization.

[0049] In a preferred embodiment of the present invention, it further includes:

[0050]

[0051]

[0052] in, This represents the distillation loss of the RPN network;

[0053] min means to find the minimum value;

[0054] Represents the teacher RPN network;

[0055] Represents the student RPN network;

[0056] Indicates an adaptive control valve;

[0057] r te,j express The predicted bounding box position;

[0058] r st,j express The predicted bounding box position;

[0059] ξ represents the set empirical threshold.

[0060] In a preferred embodiment of the present invention, it further includes:

[0061]

[0062] in, This represents the mean value across the channel;

[0063] c represents the number of channels;

[0064] h i,j This represents the i-th original value of the j-th channel;

[0065]

[0066] express Distillation loss;

[0067] min means to find the minimum value;

[0068] Represents the teacher RoI_Head network;

[0069] Represents the student RoI_Head network;

[0070] l te,j express The predicted bounding box value;

[0071] l st,j express The predicted bounding box value.

[0072] In a preferred embodiment of the present invention, it further includes:

[0073]

[0074] in, This indicates the total losses incurred during the first phase of the mission;

[0075] α represents the balance coefficient between stability and plasticity;

[0076] Indicates the RoI_Head network loss;

[0077] Indicates RPN network loss;

[0078] This represents the backbone eigenvalue network distillation loss;

[0079] This represents the distillation loss in the RPN network;

[0080] This represents the distillation loss of the RoI_Head network.

[0081] In a preferred embodiment of the present invention, it further includes:

[0082]

[0083]

[0084] This indicates the warp loss in the second phase;

[0085] This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network.

[0086] p represents the predicted classification probability of the RoI_Head network;

[0087] p * This represents the true classification value of the RoI_Head network;

[0088] [p *≥1] indicates the condition for bounding box regression calculation;

[0089] This represents the Smooth L1 bounding box regression loss of the RoI_Head network;

[0090] l represents the bounding box position prediction value of the RoI_Head network;

[0091] l* represents the true value of the bounding box position in the RoI_Head network;

[0092] st indicates that it is restricted;

[0093] This represents the RoI_Head network.

[0094] In this invention, the first step is to acquire the training image data T1;

[0095] Step 2: Input the image data T1 to be trained obtained in Step 1 into the training model for training, and obtain a model that can recognize image data T1.

[0096] Step 3: Obtain the training image data T2;

[0097] Step 4: Input the training image data T2 obtained in step 3 into the training model for training, and obtain a model that can recognize image data T1 and T2.

[0098] Step 5: Obtain the training image data T3;

[0099] Step 6: Input the training image data T3 obtained in step 5 into the training model for training, and obtain a model that can recognize image data T1, T2 and T3;

[0100] Step 7: Obtain the training image data T4;

[0101] Step 8: Input the training image data T4 obtained in step 7 into the training model for training, and obtain a model that can recognize image data T1, T2, T3 and T4;

[0102] ...;

[0103] In step 2N+1, the image data to be processed, T1, T2, T3, T4, ..., TN, are input into the model in step 2N to obtain the image result. N = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ...

[0104] When N is 2, the first step is to obtain the training image data T1;

[0105] Step 2: Input the image data T1 to be trained obtained in Step 1 into the training model for training, and obtain a model that can recognize image data T1.

[0106] Step 3: Obtain the training image data T2;

[0107] Step 4: Input the training image data T2 obtained in step 3 into the training model for training, and obtain a model that can recognize image data T1 and T2.

[0108] Step 5: Input the image data T1 and T2 to be processed into the model in step 4 to obtain the image results.

[0109] When N is 4, the first step is to obtain the training image data T1;

[0110] Step 2: Input the image data T1 to be trained obtained in Step 1 into the training model for training, and obtain a model that can recognize image data T1.

[0111] Step 3: Obtain the training image data T2;

[0112] Step 4: Input the training image data T2 obtained in step 3 into the training model for training, and obtain a model that can recognize image data T1 and T2.

[0113] Step 5: Obtain the training image data T3;

[0114] Step 6: Input the training image data T3 obtained in step 5 into the training model for training, and obtain a model that can recognize image data T1, T2 and T3;

[0115] Step 7: Obtain the training image data T4;

[0116] Step 8: Input the training image data T4 obtained in step 7 into the training model for training, and obtain a model that can recognize image data T1, T2, T3 and T4;

[0117] Step 9: Input the image data T1, T2, T3, and T4 to be processed into the model in step 8 to obtain the image results.

[0118] In a preferred embodiment of the present invention, the image result includes image category classification and / or category borders.

[0119] In summary, by employing the aforementioned technical solutions, this invention, targeting the Faster R-CNN detector, strengthens the distillation outputs of the input backbone network and the output RoI_Head network. For the intermediate RPN network, adaptive distillation is used to retain past knowledge. Simultaneously, a meta-learning approach is employed to mitigate the impact of knowledge distillation on the model's performance in learning new tasks. Furthermore, to address the issue of data distribution discrepancies between the old and new models, zero-mean normalization is applied to the output data in both the backbone and RPN networks to alleviate this problem. Therefore, we propose a novel and generalizable multi-network distillation scheme for the Faster R-CNN detector, employing methods such as strengthening the distillation values ​​of the backbone and RoI_Head networks and adaptive distillation for the RPN network to mitigate the catastrophic forgetting problem.

[0120] To mitigate the network output instability caused by the output deviation between the old and new networks, we performed zero-mean processing on the outputs of the backbone networks and RPN networks of the old and new models, and considered the mean of the RoI_Head network across classes, resulting in a new set of distillation losses.

[0121] Our extensive evaluation on the PASCAL VOC and COCO benchmark datasets, and comparison with two state-of-the-art baseline methods, demonstrate the superior performance of our method across a variety of incremental scenarios.

[0122] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0123] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0124] Figure 1 This is a schematic block diagram illustrating the implementation principle of the present invention.

[0125] Figure 2 This is a schematic block diagram illustrating the gradient element learning mechanism of this invention.

[0126] Figure 3 This is a schematic diagram illustrating the impact of the learning of the incremental task of this invention on the base class, the old class, and all classes.

[0127] Figure 4 This is a schematic diagram of the box of the present invention. Detailed Implementation

[0128] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0129] Incremental object detectors do not require all classes of data to be available beforehand. When new data is input, the detector can mitigate catastrophic forgetting through its special structure. For example... Figure 1 As shown, our method mitigates catastrophic forgetting by employing multi-network knowledge distillation and experience replay. Specifically, we reinforce the focus on past tasks at the beginning and end of the model and consider the poor performance of the RoI Head training in the next stage due to the lack of training of the RPN network on old tasks. We conditionally maintain the model's focus on the old task proposal region by using adaptive distillation in the middle RPN stage. Simultaneously, during knowledge distillation, we perform averaging on the input to improve the stability of the model output. Furthermore, to prevent over-protection of past tasks by knowledge distillation from limiting the learning of new tasks, we use the gradient preprocessing meta-learning method. Specifically, our model learning process can be divided into two stages: the first stage is the initial new task learning stage (new task loss), during which distillation loss constrains the changes in model parameters related to old tasks (multi-network mean distillation loss); the second stage uses the meta-learning gradient matrix to correct the model learning gradient direction (warp loss).

[0130] 4.1 Problem Modeling

[0131] For continuous task flow There is a task at time t. Transported to target detector T t It is composed of incremental subtask group T i (i = 1, 2, ..., n; T) i ∈T t Composed of, T t The task is to use a labeled image dataset D at time t. t The composition consists of images containing multiple objects of different classes, but the labels are only for task T. i The object is valid. At the same time, we set... The update rule is determined by θ, and is defined as follows: The parameter θ is divided into the task parameter ψ and the warp parameter. That is and In terms of the learning process, the model learns the task parameters in the first stage. ψ The second stage involves learning warp parameters. ∪ represents the union of sets. To represent the empty set, This represents the empty set.

[0132] The specific learning process can be described as follows: At time t, there is task T. t At that time, the target detector We need to process the input image I (I∈D) t Learning can be viewed as a set function. For the two-stage Faster R-CNN, This can be described as being composed of a backbone network. Regional Recommendation Network and RoI_Head Composition of set functions, i.e. ο represents function composition. The input I passes through... Feature extraction is performed to generate a feature map (F). These features are used to generate N candidate regions that may contain objects and their corresponding scores. Each candidate region is then processed... The calculation is assigned as task T. i≤t The probability of a certain class in the data is calculated, and regression is performed on its bounding box position. For incremental object detection, in a continuous task flow... Maintaining performance on previous tasks without accessing all data is challenging. Compared to ordinary incremental classification problems, incremental object detectors need to consider the memory of old knowledge from multiple network classifications and bounding box regressions, such as in Faster R-CNN. Old class feature extraction, and This addresses the classification and regression problem of prior knowledge. Our method employs a knowledge distillation strategy, freezing past network models as teacher networks to guide the current task model, which serves as the student model. For ease of subsequent theoretical explanation, elements marked with "te" are defined as related to the teacher network, such as the teacher object detector. Elements marked with "st" are defined as being related to student networks, such as student target detectors.

[0133] 4.2 Losses from New Tasks

[0134] The model's learning of new tasks can be considered as learning model parameters using a loss function. Specifically, the object detector uses a loss function to minimize classification error and bounding box localization error during learning. Let p = (p0, p1, ..., p...). k () represents the predicted probabilities of k+1 classes (k true classes and 1 background class); l = (l0, l1, ..., ln(k+1)) k ) indicates by This represents the predicted bounding box position after feature pooling for each RoI. The ground truth labels are p* (ground truth class) and l* (ground truth class bounding box position). The definition is as follows:

[0135]

[0136] in, This represents the total loss of the RoI_Head network;

[0137] This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network.

[0138] p represents the predicted classification probability of the RoI_Head network;

[0139] p * The label value represents the real class, when p * =0 indicates the background class;

[0140] λ represents the balance coefficient between classification loss and regression loss, and here we take λ = 1;

[0141] [p * ≥1] indicates that the bounding box regression calculation condition is met, that is, the selected bounding box regression calculation only includes non-background classes;

[0142] This represents the Smooth L1 bounding box regression loss of the RoI_Head network;

[0143] l indicates the position of the predicted class bounding box by the model;

[0144] l * Indicates the actual border position of the class;

[0145] and To predict the cross-entropy loss between the classified class and the true class, The loss is Smooth L1, where when p* = 0, it represents the background, and no bounding box regression loss is applied. Training is performed in a similar manner. The loss produces a prediction score o∈[0; 1] for whether the selected region contains an instance, and the corresponding bounding box prediction r. The loss is defined as follows:

[0146]

[0147] in, This represents the total loss of the RPN network;

[0148] This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network.

[0149] o indicates whether the region contains the predicted score of the class; 1 indicates yes, 0 indicates no.

[0150] o * This indicates whether the region contains the true score of the class; 1 indicates if it does, and 0 otherwise.

[0151] λ represents the balance coefficient between classification loss and regression loss, and here we take λ = 1;

[0152] This represents the Smooth L1 bounding box regression loss of the RoI_Head network;

[0153] r represents the predicted bounding box position;

[0154] r * Indicates the actual bounding box location;

[0155] o* is defined as whether the region contains the true label; o* = 1 if it does, and 0 otherwise. r* is the regression target of the true bounding box. The weight parameter λ is set to 1 in the subsequent experiments.

[0156] 4.3 Losses from Multi-Network Mean Distillation

[0157] Similar to the new task learning approach, our model maintains performance on old tasks while learning new tasks by calculating the mean distillation loss of multiple networks. Similar to Faster ILOD and iOD, we use knowledge distillation to introduce a temperature factor T on the log output using Equation (3) to soften the softmax output, thus maintaining the model's performance on past tasks in a continuous task flow. In our method, we take the empirical value T = 6. However, unlike Faster ILOD's multi-network adaptive distillation and iOD's distillation only for feature maps and RoI_Head, we strengthen both ends (…). and The distillation output is used to ensure the accuracy of the backbone feature extraction at the initial input and the RoI_Head detection at the final output. On the other hand, it is also to ensure... The input anchor has past memory. We use an adaptive zero-mean distillation method in the middle RPN layer to alleviate the overprotection of past knowledge by the distillation method, but at the same time, we retain the RPN's memory of past knowledge by adaptively increasing the distillation loss.

[0158]

[0159] in, This represents the value of the i-th network output after being softened and normalized by the temperature coefficient T;

[0160] x i This represents the unnormalized output value of the i-th network;

[0161] x i This represents the unnormalized output value of the j-th network;

[0162] n represents the number of network output values;

[0163] T represents the temperature coefficient;

[0164] Furthermore, we consider that directly introducing knowledge distillation would cause the new model to adaptively update its network parameters during training to adapt to the new task, resulting in a significant deviation between its output and the data distribution of the old model. This would make the network output difficult to fit and unstable. We... and The input is first zero-mean processed to obtain a new output before the distillation loss is calculated. Specifically, zero-mean is achieved by applying zero-mean to each pixel f. i Subtract the mean of all pixels, as shown in Equation (5). After zero-mean subtraction, all pixels in the model output are distributed with the origin as the center point, avoiding the situation where the data distribution is all negative or all positive for a certain period of time. At the same time, the mean of all pixels after zero-mean subtraction is zero, and the original data distribution shape is not changed. This is conducive to the convergence of model weights during backpropagation, making the model output more stable.

[0165]

[0166] in, This represents the value of a pixel after zero-mean normalization.

[0167] y i Represents pixel value;

[0168] n represents the number of pixels;

[0169] Therefore, to address the issue of memorizing prior knowledge, we performed multi-network distillation on the Backbone network, RPN network, and RoI_Head network, and incorporated a mean-based approach to ensure that the model maintains its performance on older tasks while learning new tasks.

[0170] 4.3.1 Backbone distillation loss

[0171] F is the image after... The extracted layer containing object pixel features, for each feature pixel f, contains F. i To obtain the object characteristics related to the old and new classes, it is necessary to... We impose distillation loss constraints. We do this by freezing... The weight parameters are used as the teacher network to guide the student network. In learning, for the same input I, the teacher network and the student network respectively obtain the output F. te and F st .at the same time, Accurate descriptions of the features of the old and new classes are crucial for subsequent classification and regression steps; therefore, we strengthen the description of these features. To achieve faster convergence, we distill the obtained features using equation (4) to obtain the zero-mean value. and Characteristic distillation loss is defined as:

[0172]

[0173] in, This represents the distillation loss of the backbone feature network;

[0174] min means to find the minimum value;

[0175] express Each pixel feature;

[0176] express Each pixel feature;

[0177] This represents the feature layer after zero-mean normalization of the teacher network.

[0178] This represents the feature layer of the student network after zero-mean normalization.

[0179] and Defined as and Each pixel feature, i.e.

[0180] 4.3.2 RPN distillation loss

[0181] For the previous stage The extracted new and old class features of I propose a region r = (r1, r2, ..., r j ) and the score o = (o1, o2, ..., o) for whether the corresponding region is a class. i As the first stage of a two-stage target detector, Does the extracted suggested area network contain information about the old and new classes for the next stage? The classification and regression of new and old categories is particularly important, but at the same time, excessive emphasis on... Applying distillation constraint will lead to Increasing focus on old candidate districts can negatively impact the learning process for new tasks, therefore we... Using the teacher network as the lower bound, adaptive selection is made to determine whether to proceed. The distillation constraint. In contrast, we... and The output score is subjected to distillation softening processing according to equation (3) and zero-mean processing according to equation (4), and KL discrete loss is used as the classification loss. For For anchor regression, we calculate the value of l in each dimension and control the regression by setting a threshold ξ. We take an empirical value of ξ = 0.1. Higher than The sum of the o value and ξ is used as the activation value in the calculation of distillation loss; while higher values... It may be more important for learning new tasks, therefore it is not included in the calculation of distillation loss. The RPN distillation loss is defined as follows:

[0182]

[0183]

[0184] in, This represents the distillation loss of the RPN network;

[0185] min means to find the minimum value;

[0186] Indicates the total number of anchors;

[0187] anchor represents the anchor frame produced by the RPN network;

[0188] express After zero-mean processing according to equation (4), the score is obtained after distillation logarithmic softmax processing similar to that of equation (3).

[0189] Represents the teacher RPN network;

[0190] express The score is obtained after processing with equations (3) and (4);

[0191] Represents the student RPN network;

[0192] Indicates an adaptive control valve;

[0193] r te,j express The predicted bounding box position;

[0194] r st,j express The predicted bounding box position;

[0195] ξ represents the set empirical threshold;

[0196] in, for The output score is processed by zero mean normalization in equation (4) and distillation softening in equation (3). In our experiment, we take the empirical value T = 6. This represents the total number of anchors.

[0197] 4.3.3 RoI_Head Distillation Loss

[0198] Proposals for both new and old classes are generated and passed to the pooling layer. Obtain the final classification probability (p) of the teacher network and the student network. te ,p st ) and bounding box regression values ​​(l te ,l st In our method, taking into account As the final stage of the two-stage target detector, we therefore place greater emphasis on classification and regression of the old class. In addition to the normal distillation loss calculation, we calculate the mean of each channel with respect to the class using Equation (7) to increase... We focus on the overall trend of the final classification probability and bounding box regression, and also introduce a temperature factor T to soften the softmax output using formula (3) for the log output, so as to obtain more probabilities about past knowledge.

[0199]

[0200] in, This represents the mean value across the channel;

[0201] c represents the number of channels;

[0202] h i,j This represents the i-th original value of the j-th channel. Therefore... The loss is defined as follows:

[0203]

[0204] express Distillation loss;

[0205] min means to find the minimum value;

[0206] This represents the variable after the mean is calculated using equation (7);

[0207] express The classification prediction value after softmax softening by equation (3);

[0208] Represents the teacher RoI_Head network;

[0209] express The classification prediction value after softmax softening by equation (3);

[0210] Represents the student RoI_Head network;

[0211] l te,j express The predicted bounding box value;

[0212] l st,j express The predicted bounding box value;

[0213] in, and All variables are variables that have undergone mean processing using equation (7). Parameters marked with T are variables that have undergone processing using equation (3).

[0214] 4.4 Total mission losses in Phase 1

[0215] Our model's learning process for the task parameter ψ in the first stage can be described as first learning the current task through each stage, and then correcting the model parameters through distillation loss at each stage to maintain past task performance. Therefore, the overall task loss can be defined as a linear combination of the loss for the new task and the multi-network mean distillation loss. In our method, we employ a similar convex combination, setting a stability-plasticity tradeoff parameter α to balance the model's performance on past and current tasks. The overall loss is defined as:

[0216]

[0217] in, This indicates the total losses incurred during the first phase of the mission;

[0218] α represents the balance coefficient between stability and plasticity;

[0219] Indicates the RoI_Head network loss;

[0220] Indicates RPN network loss;

[0221] This represents the backbone eigenvalue network distillation loss;

[0222] This represents the distillation loss in the RPN network;

[0223] This represents the distillation loss of the RoI_Head network;

[0224] In our experiment, α is defined as 0.1, as shown in Section 4.4.

[0225] 4.5 Gradient Matrix Warp Loss

[0226] For the warp parameters in the second stage Learning, such as Figure 2 As shown, we set A dedicated warp layer is used to learn this preprocessing matrix. During the distillation learning process, by setting up an image storage device (I... Store Store a small number of images (N) for each class. img ), I Store The stored images have been processed After feature extraction, the data is stored in the designated feature memory (F). Store In this context, it is worth noting that F Store Set a fixed-size queue (N) for each class. feat This mitigates class imbalance by directly embedding stored queue features into the task learner for meta-learning parameterization. It distorts the gradient to the steepest direction, causing parameter updates to be directed to the most suitable direction for different tasks.

[0227] I Store Each image in the video is processed through... and This generates the pooled features and associated labels for the RoI, and then these features are queued into F. Store Let f be the feature of RoI after pooling, and f is obtained through Generate predicted classification values ​​p and bounding box predicted values ​​l. Then, the warp loss can be obtained through F. Store The features and labels stored in the database are used for calculation. The calculation formula is:

[0228]

[0229]

[0230] This indicates the warp loss in the second phase;

[0231] (f,p * ,l* )~F Store Describing f, p * ,l * All three values ​​come from F Store middle;

[0232] This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network.

[0233] p represents the predicted classification probability of the RoI_Head network;

[0234] p * This represents the true classification value of the RoI_Head network;

[0235] [p * ≥1] indicates that the bounding box regression calculation condition is met, that is, the selected bounding box regression calculation only includes non-background classes;

[0236] This represents the Smooth L1 bounding box regression loss of the RoI_Head network;

[0237] l represents the bounding box position prediction value of the RoI_Head network;

[0238] l * This represents the true value of the bounding box position in the RoI_Head network;

[0239] st indicates that it is restricted;

[0240] Represents the RoI_Head network;

[0241] in, Let cross-entropy be the loss function. The loss is for smooth L1 regression.

[0242] We conducted extensive incremental experiments on two well-known datasets, PASCAL VOC 2007 and MS COCO 2014, and compared them with state-of-the-art methods. Our method outperformed them in all incremental scenarios. Below, we will further describe the datasets used in the experiments, the incremental experiment settings, details of various incremental experiments, and the experimental data and conclusions.

[0243] 5.1 Dataset and Evaluation Metrics

[0244] In our experiments, we evaluated our method on the PASCAL VOC 2007 and MS COCO 2014 datasets. PASCAL VOC 2007 contains 9963 images with 24640 annotated instances and 20 classes. Based on the setup, 50% of the dataset was split into training and validation sets, with the remainder used for testing. MS COCO 2014 contains objects from 80 different categories, with 83000 images in the training set and 41000 images in the validation set. Since the MS COCO test set was unavailable, we used the validation set for evaluation.

[0245] For the evaluation metrics, we used the average precision at the 50% IoU threshold (mAP@50) as the primary evaluation metric for both datasets. For MS COCO, we set multiple IoU (AP, AP50, AP-.75) and sizes (APs - small, APm - medium, and AP1 - large) as evaluation metrics.

[0246] 5.2 Incremental Experiment Scenario Setup

[0247] Similarly, we simulated incremental scenarios for PASCAL VOC and MS COCO, as described in Section 4.1, using dataset D. t A set of selected classes C is provided as task T. t And it is passed to the learner at time t. For dataset D... t Each image in the process may contain multiple classes. Instances belonging to one or more classes in C will be used as the target class for learning, while instances that do not belong to classes in C will not be marked for learning.

[0248] Based on different forms of task difficulty classification, we considered the impact of the learning intensity of the initial base class task and incremental task on the model output, and defined flow-like tasks and batch tasks. A flow-like task can be interpreted as a task that, after learning the base class task T0, has an incremental task flow T. i The input model adds 1 to 2 classes to each task; while the batch task, after learning the base class task, only contains one incremental task with more classes. As shown in Table 1, based on the division of class-flow tasks and batch tasks, we set up five incremental scenarios according to the difficulty level of the incremental experimental scenarios: Experiments a to e. Specifically, the class-flow tasks include incremental experiments a and b; the batch tasks include experiments c to d. Experiments a to d use the first 20 classes of the PASCAL VOC dataset, while experiment e uses the 80 classes of the MS COCO dataset.

[0249] Table 1. Experimental Setup

[0250]

[0251] 5.3 Incremental Step Experiment

[0252] We used stochastic gradient descent (SGD) with a momentum of 0.9. The initial learning rate was set to 0.02, then reduced to 0.0002. For base class training, each task was trained iterated 18,000 times on the PASCAL VOC dataset, followed by 100 iterations per image thereafter. For the MS COCO dataset, each task was trained 90,000 times. Model training was performed on a single GPU (a 2080Ti), with each GPU processing two images at a time; therefore, the batch size was 2. Feature storage (F...) Store ) and image storage (I Store The queue size N feat and N img Set to 10. During the evaluation process, each image is considered for 100 detections, and the NMS threshold is 0.4. The stability coefficient α is 0.1.

[0253] 5.3.1 Streaming Tasks

[0254] We simulated incremental scenarios where the model, having already learned the first 10 or 15 classes from the PASCAL VOC dataset as base classes, sequentially input one or two classes into the detector. Tables 2 and 3 report the experimental results for the class flow task. Specifically, the first row shows the joint learning of 20 classes as the upper bound for incremental learning; the second row shows the model learning the base classes, where experiment a learned the first class, and experiment b learned the first 15 classes; the following rows represent the corresponding index of each class in the class flow task. Tables 2 and 3 report the changes in AP values ​​for each class and the mAP values ​​for all classes observed so far during the class increment process. Figure 3 This illustrates the changing trends of the model's impact on the base class, old class, and all classes during each incremental task in the class flow incremental task process. The red line represents our method, and the green line represents the iOD method; base_class represents the base class, old_class represents all classes seen so far except for the newly learned class, and all_class represents all classes seen so far.

[0255] In the incremental scenario of experiment a, we will... i The number of classes learned is set to 2. As can be seen from Table 1, our model in each T... i The mAP values ​​in the task were all higher than those of the iOD method, with an average mAP difference of 4.5% and a maximum mAP difference of 5.1%. Figure 3In the detailed experimental results of Experiment a shown in (a), our model outperforms the iOD method in mAP for the base class, the old class, and all classes. Specifically, when learning the T2 task, the old class mAP gap reaches 5.5%, the base class mAP gap reaches 5.2%, and the all class gap reaches 5.1%.

[0256] In experiment b, we set T0 = 15, T i Incremental scenarios with a base class size of 1. Table 2 shows that when the number of base classes increased to 15, the difference between our model's mAP and iOD gradually increased with the learning of new classes, reaching a maximum of 3.9% and an average mAP difference of 2.42%. Figure 3 In the details of experiment b shown in (b), our model outperforms iOD on mAP for the base class, the old class, and all classes, while with T i The input of the model gradually increases for the base class, the old class and all classes. Specifically, the largest difference is shown at T5, where the old class mAP difference reaches 3.1%, the base class mAP difference reaches 3.5%, and the all classes difference reaches 3.0%.

[0257] Table 2. Experiment b: Number of base classes is 10, number of incremental task learning classes is 2 (%)

[0258]

[0259]

[0260] Table 3. Experiment c: Number of base classes is 15, number of incremental task learning classes is 1 (%)

[0261]

[0262] 5.3.1 Batch Tasks

[0263] In batch learning scenarios, we consider the class batch learning performance of our model on the PASCAL VOC and MS COCO datasets with different numbers of base classes and incremental classes. To verify the accuracy of our method, for the PASCAL VOC dataset, we report our results compared with those of Shmelkov et al., Faster ILOD, and iOD on mAP. For the MS COCO dataset, we compare our method with iOD using standard COCO dataset evaluation methods. Tables 4 to 6 show our comparative experimental results.

[0264] In Experiment c, we set up a batch task incremental scenario with T0:T1 = 10:10. As shown in Table 5, our model achieves the best learning performance among all experiments compared to other methods, with an mAP of 65.0%. It also achieves the best performance among several methods in learning new tasks, with a new task mAP of 68.3%. Furthermore, our method outperforms iOD in maintaining performance on old tasks, achieving an old class mAP of 61.7%.

[0265] In Experiment d, we increased the number of classes to be learned for T0 to 19. In the results reported in Table 6, our model maintains comparable, but is still optimal, performance to other methods in both the overall task and the old class task, achieving an overall task mAP of 68.9% and an old class task mAP of 68.9%.

[0266] Table 4 presents our results on the COCO dataset in Experiment e. The first row represents the results of jointly learning 80 classes of COCO. Specifically, we set up an incremental scenario with T0=40 and T1=40, and used the standard COCO dataset evaluation method to comprehensively evaluate at multiple IOUs (AP, AP50, AP-.75) and sizes (APs-small, APm-medium, and AP1-large). As can be seen from Table 4, even in the scenario of large-scale class learning on the complex COCO dataset, our model still exhibits excellent performance. It outperforms the iOD method by more than 2% at different scales, and the AP50 reaches 37.706%, which is 4.684% higher than the iOD method.

[0267] Table 4. Experiment g: CO2 increment experiment (%)

[0268]

[0269] Table 5. Experiment c: 10⁻¹⁰

[0270]

[0271] Table 6. Experiment d: 19-1

[0272]

[0273] 5.4 Stability Analysis (Box Diagram Analysis, Experiment on the Stability-Plasticity Trade-off Parameter α)

[0274] To verify the impact of zero mean on the stability of the model output, we repeated experiments c and d five times and plotted the output results as shown below. Figure 4The box plot is shown. As can be seen from the figure, our method outperforms the iOD method in terms of output stability for all iOD methods without zero-mean, and our accuracy is higher than that of the iOD method in all five experiments across the three scenarios.

[0275] To find the optimal stability equilibrium parameter α for Equation 9, we conducted multiple experiments on the value of α based on the incremental task approach of Experiment c. The specific experimental results are shown in Table 7. The table shows that as α gradually increases, the accuracy of our model gradually decreases. Finally, based on the experimental results, we set α to 0.1.

[0276] Table 7. α Stability Analysis

[0277]

[0278] 5.5 Ablation Experiment

[0279] We conducted ablation experiments to verify the accuracy improvement effect of the modules (RPN, Zero_meaning) introduced in our method. Specifically, as shown in Table 8, we still used the task format based on incremental experiment c. The final experimental results were tested on the mAP values ​​of the base class T0 (the first 10 classes), the incremental task T1 (the last 10 classes), and the total 20 classes. The table shows that when only Zero_meaning was introduced, the learning ability of the new task was improved, with the mAP of T1 reaching 67.33%. When the RPN adaptive distillation loss was introduced, the stability of past knowledge was improved, with the mAP of T0 reaching 62.20%. When both methods were introduced, the mAP values ​​of the new task T1 and all 20 classes reached the highest (68.28% and 65.00%, respectively), and the experimental results are consistent with the theoretical analysis conclusions in Section 5.3.

[0280] Table 8. Ablation Test (%)

[0281]

[0282] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An incremental target detection method based on multi-network mean distillation loss function, characterized in that, Includes the following steps: S1, Obtain the image data to be trained; S2, input the image data to be trained obtained in step S1 into the training model for training to obtain the target model; The training model consists of a teacher network and a student network, both of which include a backbone network, an RPN network, and a RoI_Head network to form a multi-network structure. The output of the backbone network in the teacher network is connected to the input of the RPN network in the teacher network, and the output of the RPN network in the teacher network is connected to the input of the RoI_Head network in the teacher network. The output of the backbone network in the student network is connected to the input of the RPN network in the student network, and the output of the RPN network in the student network is connected to the input of the RoI_Head network in the student network. The outputs of the backbone network and RPN network are zero-mean processed, and the outputs of the backbone network, RPN network and RoI_Head network are trained with distillation loss to obtain the target model; The training model learning process consists of two stages: the first stage is the initial new task learning stage, during which distillation loss is used to constrain the changes in the model parameters related to the old task; the second stage uses the meta-learning gradient matrix to correct the model learning gradient direction. The model's learning process for task parameters in the first stage can be described as first learning the current task through each stage, and then correcting the model parameters through the distillation loss of each stage to maintain past task performance. Therefore, the overall task loss is defined as a linear combination of the loss for the new task and the multi-network mean distillation loss. A stability and plasticity tradeoff parameter α is set to balance the model's performance on past and current tasks. The overall loss is defined as: (9) in, This indicates the total losses incurred during the first phase of the mission; This represents the coefficient that balances stability and plasticity. Indicates the RoI_Head network loss; Indicates RPN network loss; This represents the backbone eigenvalue network distillation loss; This represents the distillation loss in the RPN network; This represents the distillation loss of the RoI_Head network; For the second stage of learning warp parameters, by setting... A dedicated network layer is used to learn the preprocessing matrix. During the distillation learning process, an image storage device was set up. Store a small number of images for each class , The stored images have been processed After feature extraction, the features are stored in the designated feature memory. middle, Set a fixed-size queue for each class. To mitigate the class imbalance problem, the stored queue features are directly embedded into the task learner for meta-learning parameterization. It distorts the gradient to the steepest direction, causing parameter updates to be directed to the most suitable direction for different tasks. Each image in the video is processed through... and This generates the pooled features and associated labels for the RoI, and then these features are queued into... ,make Features after RoI pooling pass Generate predicted classification values and bounding box prediction values ,Then, pass The features and labels stored in the database are used for calculation. The calculation formula is: (10) This indicates the warp loss in the second phase; express All three values ​​come from middle; This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network. This represents the predicted classification probability of the RoI_Head network; This represents the true classification value of the RoI_Head network; Indicates the conditions for bounding box regression calculation; This represents the Smooth L1 bounding box regression loss of the RoI_Head network; This represents the predicted bounding box position value of the RoI_Head network; This represents the true value of the bounding box position in the RoI_Head network; Indicates that it is subject to; Represents the RoI_Head network; Let cross-entropy be the loss function. The loss is for smooth L1 regression. S3, input the image data to be processed into the target model in step S2 to obtain the image result.

2. The incremental target detection method based on multi-network mean distillation loss function according to claim 1, characterized in that, The training image data obtained in step S1 includes the PASCAL VOC 2007 and / or MS COCO 2014 datasets.

3. The incremental target detection method based on multi-network mean distillation loss function according to claim 1, characterized in that, The method for calculating the loss of RoI_Head networks is as follows: (1) in, This represents the total loss of the RoI_Head network; This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network. This represents the predicted classification probability of the RoI_Head network; The tag value represents the actual class; This represents the balance coefficient between classification loss and regression loss; Indicates the conditions for bounding box regression calculation; This represents the Smooth L1 bounding box regression loss of the RoI_Head network; Indicates the position of the predicted class bounding box by the model; Indicates the actual border position of the class; The method for calculating the loss of an RPN network is as follows: （2） in, Indicates RPN network loss; This represents the cross-entropy loss between the predicted classification and the true class in the RoI_Head network. This indicates whether the region contains a predicted score for the class. This indicates whether the region contains the true score for the class; This represents the balance coefficient between classification loss and regression loss; This represents the Smooth L1 bounding box regression loss of the RoI_Head network; Indicates the predicted border position; Indicates the actual bounding box location.

4. The incremental target detection method based on multi-network mean distillation loss function according to claim 1, characterized in that, The method for calculating the distillation loss of the Backbone eigenvalue network is as follows: (5) in, This represents the distillation loss of the backbone feature network; This indicates finding the minimum value; express Each pixel feature; express Each pixel feature; This represents the feature layer after zero-mean normalization of the teacher network. This represents the feature layer of the student network after zero-mean normalization.

5. The incremental target detection method based on multi-network mean distillation loss function according to claim 1, characterized in that, The method for calculating the distillation loss of an RPN network is as follows: (6) in, This represents the distillation loss in the RPN network; This indicates finding the minimum value; Represents the teacher RPN network; Represents the student RPN network; Indicates an adaptive control valve; express The predicted bounding box position; express The predicted bounding box position; This represents the set experience threshold; Indicates the total number of anchors; express After zero-mean processing according to equation (4), the score is obtained after distillation logarithmic softmax processing similar to that of equation (3). express The score is obtained after processing by equations (3) and (4); (4) in, This represents the value of a pixel after zero-meaning. Represents pixel value; Indicates the number of pixels; (3) in, This represents the value of the i-th network output after being softened and normalized by the temperature coefficient T; This represents the unnormalized output value of the i-th network; This represents the unnormalized output value of the j-th network; Indicates the number of network output values; This represents the temperature coefficient.

6. The incremental target detection method based on multi-network mean distillation loss function according to claim 5, characterized in that, The method for calculating the distillation loss of the RoI_Head network is as follows: (8) This represents the distillation loss of the RoI_Head network; This indicates finding the minimum value; Represents the teacher RoI_Head network; Represents the student RoI_Head network; express The predicted bounding box value; express The predicted bounding box value; , , , This represents the variable after the mean is calculated using equation (7); express The classification prediction value after softmax softening by equation (3); express The classification prediction value after softmax softening by equation (3); （7） in, This represents the mean value across the channel; Indicates the number of channels; This represents the i-th original value of the j-th channel.

7. The incremental target detection method based on multi-network mean distillation loss function according to claim 1, characterized in that, Image results include image category classification and / or category borders.

Images