Lifelong learning based object detection method and apparatus

Abstract

The application provides a target detection method and device based on lifelong learning, which simulates human target detection behavior criteria, so that the model can continuously learn new target detection tasks without forgetting old target detection tasks. Furthermore, the model can not only use the knowledge learned from old target detection tasks to assist new target detection tasks, but also use new target detection tasks to strengthen old target detection tasks. The target detection model in the application is a bidirectional learning model, which can not only use the knowledge learned from existing target detection tasks to speed up and improve new target detection tasks, but also use new target detection tasks to strengthen old target detection tasks to avoid the problem of "catastrophic forgetting".

Description

Technical Field

[0001] This invention belongs to the field of image processing and pattern recognition, and specifically relates to target detection technology methods and systems. Background Technology

[0002] In computer vision, where deep learning models are most widely used, real-world image object detection and classification databases are vast and diverse, while the incremental image data generated via the internet is also rapidly increasing. Therefore, building a deep network model capable of continuous learning has gradually become a research focus. It is well known that deep learning models suffer from "catastrophic forgetting" after being trained with new data. This means that learning new tasks or data significantly degrades the performance of older tasks. In contrast, humans begin learning object detection and recognition from birth, and as they age, their detection and recognition abilities improve, and they can detect and recognize an increasing number of objects without experiencing the "catastrophic forgetting" problem seen in deep learning. Existing object detection methods require retraining the detection model from scratch when encountering new object detection tasks, which is time-consuming and laborious. Furthermore, when learning for new object detection tasks, they cannot effectively utilize knowledge learned from previous tasks; although transfer training methods exist to utilize old object detection images, they suffer from the "catastrophic forgetting" problem—learning the new task but forgetting the old one. Summary of the Invention

[0003] To address the above issues, this invention proposes a target detection method based on lifelong learning. This method mimics human target detection behavior principles, enabling the model to continuously learn new target detection tasks without forgetting old ones. Furthermore, the model can not only use the knowledge learned from old target detection tasks to assist new target detection tasks, but also use new target detection tasks to enhance old target detection tasks.

[0004] To achieve the above objectives, the technical solution provided by this invention is: a target detection method based on lifelong learning, comprising the following steps:

[0005] Step 1: Construct a basic target detection model based on lifelong learning to enable it to perform basic target detection tasks; the target detection learning model based on lifelong learning includes four main parts: backbone network, multi-scale fusion network, multi-task learning, and lifelong learning.

[0006] Step 2, Known Task Learning: The basic target detection model is trained and learned using the backbone network and multi-scale fusion network to enable it to complete the known target detection task and the corresponding samples.

[0007] Step 3, new task learning, when encountering a new target detection task, using multi-task learning and lifelong learning part to transfer learning on new task samples and known task knowledge, optimizing the basic target detection model, and training the optimized target detection model, so that the optimized target detection model can not only complete the new target detection task well, but also not forget the original target detection task;

[0008] Step 4, repeat S3, so that the target detection model becomes more and more perfect, and completes various different target detection tasks.

[0009] Further, the multi-scale fusion network first performs down-sampling on the final feature map of the previous scale layer to obtain a feature map consistent with the size of the current scale layer at each scale; then, the feature map of the previous scale layer and the feature map input by the current scale layer are spliced to obtain a new feature map of the current scale layer; then, the new feature map is subjected to convolution and pooling operation to obtain the final feature map of the current scale layer; finally, the head prediction network is used for target detection, and the detected target class and position information of the current scale layer are output, wherein the head prediction network refers to the CenterNet strategy, and the network head with Anchor-free idea is used for target detection frame regression and class identification.

[0010] Further, the head prediction network includes a heat map classification part, a center point offset and a bounding box length-width regression part, and the corresponding loss function includes three loss functions: heat map classification, offset regression and bounding box size regression; the heat map classification loss function is:

[0011]

[0012] Wherein, N is the number of training samples, xyc represents the coordinate value and class respectively, represents the predicted key point class confidence, Y xyc ∈[0,1] represents the real key point class confidence, Y xyc =1 represents the target, Y xyc =0 represents the background, and alpha=2 and beta=4 are constants;

[0013] The offset regression loss function is:

[0014]

[0015] Wherein, P represents the real bounding box center point coordinate on the original image, R represents the scaling multiple of the feature map compared with the original image, represents the floor, represents the predicted bounding box center point offset;

[0016] The bounding box size regression loss function is:

[0017]

[0018] wherein, is the real value of the length and width of the target frame at the center point k, that is, the real value of the area, is the predicted value of the length and width of the target frame at the center point k, and the final loss function is:

[0019] L = λ k L k + λ size L size + λ off L off

[0020] wherein, λ k , λ size , λ off are the weights of each loss function.

[0021] Further, the target detection model training and learning in step 2 includes two processes of model training and testing;

[0022] The purpose of model training is to learn network parameters using a training data set, including data preprocessing, detection network, and label matching and loss calculation;

[0023] 21. Data preprocessing

[0024] Data augmentation operations are performed on the training data set, including flipping, scaling, mean normalization, hue change, and using part of the sub-region erased from the image, different images and labels are represented by difference value, and other training images are pasted to the erased sub-region;

[0025] 22. Detection network

[0026] The detection network includes three parts of the basic backbone network, the feature fusion network and the prediction network, wherein the basic backbone network is the backbone network in step 1, EfficientNet B7 is used for feature extraction, the feature fusion network is the feature fusion part of the multi-scale fusion network in step 1, and the specific feature fusion method adopts the multi-scale feature fusion based on the Transformer encoding and decoding method, and the prediction network is the head prediction network part of the multi-scale fusion network in step 1, and the classification and regression adopt the full convolution method;

[0027] 23. Label assignment and loss calculation

[0028] The label assignment is to provide a true value for the detection network prediction, and the anchor box is assigned to the corresponding object according to the intersection over union between the anchor box and the true box of the target object by using the intersection over union criterion. Based on the result of label classification, the loss of classification and regression tasks is calculated by using a loss function, and the weight of the detection network is updated by using a back propagation algorithm, wherein the loss function includes three loss functions of heat map classification, offset regression and box size regression.

[0029] The heat map classification loss function is:

[0030]

[0031] Wherein, N is the number of training samples, xyc represents coordinate values and categories respectively, represents the predicted category confidence at the key point, Y xyc represents the true category confidence at the key point, Y xyc = 1 represents the target, Y xyc = 0 represents the background, and alpha = 2 and beta = 4 are constants.

[0032] The offset regression loss function is:

[0033]

[0034] Wherein, P represents the true box center point coordinate on the original image, and R represents the scaling multiple of the feature map compared with the original image, represents the down rounding, represents the predicted box center point offset;

[0035] The box size regression loss function is:

[0036]

[0037] Wherein, is the true value of the length and width of the target box at the center point k, that is, the true value of the area, is the predicted value of the length and width of the target box at the center point k, and the final loss function is:

[0038] L = lambda k L k + lambda size L size + lambda off L off

[0039] Wherein, lambda k , lambda size , lambda off are the weights of the loss functions;

[0040] In the test phase, the class and location information of the object existing in the given image are output, including the input image, the detection network, and post-processing; for a given image, the classification and regression results are generated by using the trained detection network, and then the Soft-NMS is used for post-processing to suppress redundant detection by reducing the classification score of the detection result with a high intersection over union; the target detection model is tested by using the collected test set, if the test result reaches the preset threshold, it means that the target detection model has the ability of the existing target detection task, and the model training is stopped; otherwise, the training is continued.

[0041] Further, the specific implementation process of knowledge transfer in step 3 is as follows:

[0042] For each target detection task, after the task is trained, the importance of each parameter in the target detection model for the task is calculated ij , that is, the proportion of the value of the i-th row and the j-th column parameter to the entire parameter value, and is used in the subsequent task training, Ω ij is added to the loss function in the form of a regularization term, and whenever a new task is trained: for Ω ij , the larger parameter, the smaller the change amplitude in gradient descent, because the parameter is important to a certain task in the past, and needs to retain its value to avoid catastrophic forgetting; and for Ω ij , the smaller parameter, the larger the gradient update amplitude, to get better performance on the new task, so the loss function of the n-th task is:

[0043]

[0044] Where L n (θ) is the loss function of the current task, λ is the balance factor, θ ij is the i-th row and j-th column parameter in the current model parameter matrix, is the model parameter obtained after training the first n-1 tasks, and Ω ij is updated after each task is trained, when the first task is performed, 0 is taken;

[0045] The specific method for training the optimized target detection model is: assuming that there are M target detection tasks T 1 , T 2 , …, T M , each task T m =(f m (θ), X m , y m ), that is, the label of each task is generated by a real implicit function f m(θ) is determined, wherein θ is a network parameter of the target detection model, which is continuously updated and optimized with the increase of the detection tasks; each task is given n m training samples and corresponding labels y m , d is the dimension of the extracted sample features; at any time point, the optimized target detection model receives a batch of labeled samples from task m and makes a prediction of the learned tasks according to the requirements; the training target is to learn the target detection model of each task through the given training samples as close to the real target function f m as possible.

[0046] Further, the specific implementation of step 4 is as follows:

[0047] 61. A first task-based neural network is constructed using a basic target detection network as the first column, and one or a batch of tasks are trained for known old tasks;

[0048] 62. A second task-based neural network is constructed using the basic target detection network, and then the first column, i.e., the previous task-based neural network, is fixed, and each layer of the neural network in the previous column is connected to each layer of the neural network in the second column as additional input through dimension reduction processing, that is, each layer of the second task-based neural network adds the input of the corresponding layer of the previous task-based neural network processed by a as an additional input in addition to the original input;

[0049] 63. A third task-based neural network is constructed using the basic target detection network, and the third task is trained, and the neural networks in the previous two columns are fixed, and then connected to the third task-based neural network in the same way as step 62, and so on, to construct N task-based neural networks and learn N new tasks.

[0050] The application also provides a target detection device based on lifelong learning, which is used to implement any of the methods in the above solutions.

[0051] Compared with the prior art, the application has the following advantages and beneficial effects:

[0052] A new target detection model method is proposed, which can continuously learn new target detection tasks on the basis of existing target detection tasks without starting training the network model from scratch; the detection model is a bidirectional learning model, which not only uses the knowledge learned from existing target detection tasks to speed up and improve new target detection tasks, but also uses new target detection tasks to strengthen old target detection tasks to avoid the problem of catastrophic forgetting. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1A schematic diagram of a lifelong learning based target detection learning model.

[0054] Figure 2 A schematic diagram of a multi-task multi-scale classification / regression network.

[0055] Figure 3 A schematic diagram of a head prediction network.

[0056] Figure 4 A schematic diagram of a lifelong learning network.

[0057] Figure 5 Model training learning of a known target detection task. DETAILED DESCRIPTION

[0058] The technical solutions of the present application will be further described below in combination with the drawings and examples.

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer and more explicit, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0060] In order to solve the above problems, the present application regards the multi-target detection problem as a multi-task learning problem, and proposes a target detection method based on lifelong learning in reference to the human object detection and recognition method, mainly including the following steps:

[0061] S1, basic target detection model construction: a lifelong learning based target detection learning model is constructed, which has the basic ability of target detection task.

[0062] The lifelong learning based target detection learning model construction method is as shown in Figure 1 It includes four parts of backbone network, multi-scale fusion network, multi-task learning and lifelong learning.

[0063] The backbone network part adopts EfficientNet(B0-B7) series, and the appropriate backbone network model is selected according to the GPU configuration during model training. Because different detection tasks have different needs for the receptive field of CNN network model, for example, for the house detection task, globality is more important than locality, so a larger receptive field is helpful to improve the accuracy. And for the face detection task, local details are more important than globality, so more attention should be paid to the features of low receptive field network level. Therefore, the present application designs a multi-scale fusion network after the backbone network, which performs classification and regression tasks at different scale layers to preliminarily obtain target detection candidate frame and target category.

[0064] The detailed model of the multi-scale fusion network part is as shown in Figure 2As shown, the first two numbers in the target regression part represent the feature map size, and the last number 4 represents the center point coordinate offset and the target box length and width 4 information; the first two numbers in the target classification part represent the feature map size, and the last number 2 (where 0 represents not belonging to this type of target, and 1 represents belonging to this type of target) represents the number of categories. The multi-scale fusion network includes two parts of feature fusion and head prediction network, and the processing process of the feature fusion part is: at each scale, first, the final feature map of the last scale layer is down-sampled to obtain a feature map consistent with the size of the current scale layer; then, the feature map of the last scale layer and the feature map input by the current scale layer are spliced to obtain a new feature map of the current scale layer; then, the new feature map is subjected to convolution and pooling operation to obtain the final feature map of the current scale layer; finally, the head prediction network is used for target detection, and the category and position information of the detected target of the current scale layer are output. The head prediction network refers to the CenterNet strategy, and uses the network head of the Anchor-free idea to perform target detection box regression and category identification, and its network structure is as shown in Figure 3 As shown, including the heat map classification part, the center point offset and the box length and width regression part, and the corresponding model training loss function includes: heat map classification, offset regression and box size regression three loss functions. The heat map classification loss function is:

[0065]

[0066] Wherein, N is the number of training samples, xyc represents the coordinate value and category, respectively, Y xyc ∈[0,1] represents the true key point category confidence, Y xyc =1 represents the target, Y xyc =0 represents the background, and α=2, β=4 are constants.

[0067] The offset regression loss function is:

[0068]

[0069] Wherein, P represents the center point coordinate of the real box on the original image, R represents the scaling multiple of the feature map compared with the original image, ( representing the floor), representing the predicted box center point offset.

[0070] The box size regression loss function is:

[0071]

[0072] Wherein, The true value of the length and width of the target frame at the center point k, that is, the true value of the area, The predicted value of the length and width of the target frame at the center point k. The final loss function is:

[0073] L = λ k L k + λ size L size + λ off L off

[0074] Wherein, λ k , λ size , λ off are the weights of each loss function.

[0075] The multi-task learning part changes according to the actual target detection task, and a selection gate is added between the multi-task learning to control which type and which type of target detection task is performed.

[0076] The lifelong learning part mainly learns new target detection tasks, and does not forget old tasks while learning new tasks, and the network result is as shown in Figure 4 In order to not forget the previous task, the network of all previous tasks is retained and fixed, and a new network (a column) is newly built every time a new task is added. In order to use the experience of the past, the input of this task is input into all previous networks, and the output of each layer of the previous network is input together with the output of each layer of the current task network into the next layer. Every time a new task is added, a new column is added, and the outputs of the previous columns are fused into the current column. For example, if the low-level features of two tasks are similar, the first few layers in the current task network may be completely useless, and only the output of the previous task is needed.

[0077] S2, known task learning: using known target detection tasks and corresponding learning samples to learn and train the basic target detection model, so that it can complete the detection task of known targets.

[0078] Take the 80-class target detection task in the COCO2017 dataset as an example to train the model. The COCO dataset is an open-source dataset provided by the Microsoft team that can be used for image target detection and recognition, and the target detection task is annotated with 12 large categories and 80 small categories. The invention takes the annotated 80 categories as known target detection tasks, and uses the 80-class training samples to train the basic target detection model, so that the model has the target detection task of 80 categories.

[0079] Model training and learning mainly includes model training and testing processes, as shown in Figure 5The main purpose of model training is to learn the parameters of the detection network using the training dataset, which mainly includes data preprocessing, detection network, and label matching and loss calculation, etc.

[0080] 1. Data preprocessing

[0081] The training dataset contains a large number of visual images and annotation information (object position and category). The purpose of preprocessing is to enhance the diversity of training data and improve the detection ability of the detection network. In addition to using common data enhancement methods such as flipping, scaling, mean normalization, and color change, etc. Also use the method of erasing part of the sub-region from the image, the method of difference representation of different images and labels, and the method of pasting other training images to the erased sub-region, etc.

[0082] 2. Detection network

[0083] The detection network generally includes three parts: basic backbone network, feature fusion network, and prediction network. Among them, the basic backbone network is the backbone network in S1, which uses EfficientNet B7 for feature extraction. The feature fusion network is the feature fusion part of the multi-scale fusion network in S1 (such as Figure 2 shown), and the specific feature fusion method uses a Transformer-based encoding and decoding method for multi-scale feature fusion. The prediction network is the head prediction network part of the multi-scale fusion network in S1 (such as Figure 3 shown), which uses full convolution for classification and regression.

[0084] 3. Label assignment and loss calculation

[0085] Label assignment is mainly to provide true values for detector prediction. Using the intersection over union criterion, the anchor box is assigned to the corresponding object according to the intersection over union between the anchor box and the object true box. Based on the label classification result, the loss function is used to calculate the loss of classification and regression tasks, and the back propagation algorithm is used to update the weights of the detection network. The loss function includes: heat map classification, offset regression and bounding box size regression, which are the same as S1.

[0086] Based on the learning of the training phase, the detection network outputs the category and position information of the existing objects in the given image in the test phase. As Figure 5(b) as shown, mainly includes input image, detection network, post-processing, etc. For a given image, first use the trained detection network to generate classification and regression results. Generally speaking, most target detection methods will generate multiple detection results around the same object. Therefore, most target detection methods require a post-processing step to retain one detection result for each object and remove other redundant detection results. The present application improves the Non-Maximum Suppression (NMS) method and proposes Soft-NMS, which suppresses redundant detection by reducing the classification score of detection results with high intersection over union.

[0087] Test the learning model by collecting the test set, if the test result reaches the preset threshold, it means that the learning model has the ability to complete the existing target detection task, and the model training is stopped; otherwise, continue training.

[0088] S3, new task learning: when encountering a new target detection task, transfer learning is used with new task samples and known task knowledge to optimize the detection model, so that the optimized detection model can not only complete the new target detection task well, but also remember the original target detection task.

[0089] When encountering a new target detection task, the detection model does not need to start training from scratch, but only needs to perform knowledge transfer learning training based on the trained model.

[0090] In order to make full use of previous tasks, all previous task networks are retained and fixed, and a new network (a column) is created each time a new task is added. In order to use past experience, the input of this task is input into all previous networks, and the output of each layer of the previous network is input into the next layer together with the output of each layer of the current task network. Each time a new task is added, a new column is added, and the outputs of the previous columns are fused into the current column. For example, if the low-level features of two tasks are similar, the first few layers of the current task network may not be useful at all, and only the output of the previous task is needed.

[0091] In order to perform knowledge transfer, for each target detection task, after the task is trained, the importance of each parameter in the network to the task is calculated ij (i.e. the proportion of the value of the i-th row and j-th column parameter to the entire parameter value), and is used in subsequent task training. Ω ij is added to the loss function in the form of a regularization term, and each time a new task is trained: for Ω ij larger parameters, try to reduce its change amplitude in gradient descent, because the parameter is important to the past task, and needs to retain its value to avoid catastrophic forgetting; and for Ω ijThe smaller parameters can be updated with a larger gradient to achieve better performance on the new task. Thus, the loss function of the nth task is:

[0092]

[0093] where L n (θ) is the loss function of the current task, λ is the balance factor, θ ij is the parameter in the i-th row and j-th column of the current model parameter matrix, is the model parameter obtained after training the first n-1 tasks, and Ω ij is updated every time a task is trained. When the first task is trained, 0 is taken.

[0094] The model training method during new task learning is as follows: suppose there are M target detection tasks T 1 ,T 2 ,…T M , each task T m =(f m (θ),X m ,y m ). That is, the label of each task is determined by a real hidden function f m (θ), where θ is the network parameter of the learning model, which is constantly updated and optimized as the detection task increases; each task is given n m training samples (d is the extracted sample feature dimension) and the corresponding label y m , n m indicates that the total number of samples of the mth task is n m . At any time point, when the detection model receives a batch of labeled samples from task m, the detection model will make a prediction of the learned task according to the requirement. The training goal is: for each target detection task, through the given training samples, the detection model function is as close as possible to the real target function f m , and the training effect is determined by calculating the error on all test samples.

[0095] S4, lifelong learning: continuously performing S3, so that the target detection model becomes more and more robust and can complete various detection tasks.

[0096] The function of lifelong learning is to enable the model to continuously learn new tasks while not forgetting old tasks; in addition, it is also necessary to transfer the knowledge learned in the old task to the new task to improve the performance of the new task. The main process is as follows:

[0097] 1. Use the base target detection network to build a task-based neural network, train one or a batch of tasks (usually known old tasks), Figure 4 The first column.

[0098] 2. Use the base target detection network to build a second task-based neural network, then fix the first column, that is, the last task-based neural network, and connect each layer of the neural network in the last column to each layer of the neural network in the second column as additional input through a processing. That is, each layer of the second task-based neural network adds the input of the corresponding layer of the previous task-based neural network processed by a to the original input.

[0099] 3. Use the base target detection network to build a third task-based neural network, train the third task, fix the neural networks in the first two columns, and then connect them to the third task-based neural network in the same way as the previous step. By analogy, N task-based neural networks can be built to learn N new tasks.

[0100] Wherein, the role of a is mainly to reduce the dimension and unify the dimension of the input (match the original input), which is realized by a simple MLP. The new task learning is to extract the information of the previous neural network and fuse it with the current input information, and then train and learn. The training effect can be compared with the method without adding the previous neural network. If the effect is much better, it means that the previous neural network is useful and the knowledge is transferred. The advantage of the method of the present application is that the previous training is preserved, so as not to change the original network like fine tune, and the feature information of each layer can be transferred and better analyzed.

[0101] The following will illustrate the process and effect of target detection of the present application through a specific experiment:

[0102] First, the 80 categories labeled in the COCO2017 dataset are used as the initial target detection task, and the 80 categories are used to train the base target detection model of the present application. The training samples (training set: 118287; validation set: 5000; total: 123287) are used to train the base target detection model. After model training, the base target detection model reaches 61.28 on the (COCO Minival set) test set mAP@0.5.

[0103] Then, the Open Images Challenge 2018 dataset is used for new target detection task learning. The Open Images Challenge 2018 dataset has 1743042 pictures, 12195144 object frames and 500 object categories. Three categories with appropriate quantities and not in the COCO2017 dataset are selected as new target detection task categories for experiments, which are: flower, glasses and face. The new target detection task experiment adopts two methods, one of which is a traditional target detection method, that is, without using the multi-task learning and lifelong learning proposed in the application, directly using the new target detection task training samples for fine-tune transfer learning based on the original training model. The other is the target detection method based on lifelong learning proposed in the application, that is, using the training samples of the new target detection task and the knowledge learned from the existing task for multi-task and lifelong learning, so that the target detection model can not only detect the new task well but also not forget the old task.

[0104] The experimental results are shown in Table 1. In the table, “old category-face” refers to the accuracy of the new model trained and learned on the old category when the face new category is added. The meanings of “old category-glasses” and “old category-flower” follow the above rules. The accuracy of the basic target detection model on the old category test set (COCO Minival set) is 61.28, and the unit is mAP@0.5. After traditional training and learning with three new category pictures, the average accuracy of the new model on the new category is 76.6, but the accuracy on the old category test set is reduced to 5.7, 15.2 and 8.4 respectively, and the average is only 9.8, which is 51.48 lower than the initial accuracy of 61.28. It can be seen that the traditional method completely forgets the old target detection task after learning the new target detection task. That is, after traditional training and learning, the detection ability of the trained network on the old category at the initial time will be lost most of the time, that is, catastrophic forgetting occurs. The mAP@0.5 of the method of the application on the new test set is 74.8, which is only 1.8 lower than that of the traditional method 76.6. But the mAP@0.5 on the old category test set is 59.5, which is only 1.78 lower than the initial 61.28, and 49.7 higher than the traditional method 9.8. It shows that through the multi-task and lifelong learning of the application, the detection ability of the detection network on the old category is greatly retained, and the influence on the detection ability of the new category is not great.

[0105] Table 1 Comparison of effects of two methods for new target detection task (the values in the table represent Map@0.5)

[0106]

[0107] The specific embodiments described herein are merely illustrative of the spirit of the application. Various modifications or changes in the specific embodiments described herein can occur to those skilled in the art to which the application pertains without departing from the spirit of the application, and it is understood that such modifications or changes are to be considered as within the scope of the application as defined by the appended claims.

Claims

1. A method for object detection based on lifelong learning, characterized by, It comprises the following steps: Step 1, constructing a lifelong learning-based basic target detection model to have basic target detection tasks; the lifelong learning-based target detection learning model comprises a backbone network, a multi-scale fusion network, multi-task learning and lifelong learning four parts; Wherein, the backbone network adopts EfficientNet B7 for feature extraction; The multi-scale fusion network firstly performs down-sampling on the final feature map of the previous scale layer to obtain a feature map consistent with the size of the current scale layer; then, the feature map of the previous scale layer and the feature map input by the current scale layer are spliced to obtain a new feature map of the current scale layer; subsequently, the new feature map is subjected to convolution and pooling operation to obtain the final feature map of the current scale layer; finally, the head prediction network is used for target detection, and the target class and position information detected by the current scale layer are output, wherein the head prediction network refers to the CenterNet strategy, and the network head adopting the Anchor-free idea is used for target detection frame regression and class identification; Step 2, known task learning, using the backbone network and the multi-scale fusion network to train and learn the known target detection task and the corresponding sample, so that the basic target detection model can complete the known target detection task; Step 3, new task learning, when encountering a new target detection task, using the multi-task learning and lifelong learning part to perform transfer learning on the new task sample and known task knowledge, optimizing the basic target detection model, and training the optimized target detection model, so that the optimized target detection model can not only complete the new target detection task, but also not forget the original target detection task; Step 4, repeating S3, so that the target detection model becomes more and more perfect, and completes various different target detection tasks.

2. The goal detection method based on lifelong learning of claim 1, wherein: The head prediction network comprises a heat map classification part, a center point offset and a frame length-width regression part, and the corresponding loss function comprises three loss functions: heat map classification, offset regression and frame size regression; the heat map classification loss function is: wherein, is the number of training samples, respectively represent coordinate values and classes, represents the predicted class confidence at a keypoint, represents the true class confidence at a keypoint, = 1 represents a target, represents background, is a constant; The offset regression loss function is: wherein, represents the real bounding box center point coordinate on the original image, represents the scaling multiple of the feature map compared with the original image, , represents the floor operation, represents the predicted bounding box center point offset. The frame size regression loss function is: wherein, is the center point is the length and width of the target frame at the real value, i.e., the real value of the area, is the center point is the length and width of the target frame at the predicted value, and the final loss function is: wherein, are the weights for each loss function.

3. The goal detection method based on lifelong learning of claim 1, wherein: The target detection model training and learning in step 2 comprises two processes of model training and testing; The purpose of model training is to learn network parameters by using training data set, including data preprocessing, detection network, and label matching and loss calculation; Step 21, data preprocessing The training data set is subjected to data enhancement operation, including flipping, scaling, mean normalization, hue change, and using part of the sub-region erased from the image, different images and labels for difference representation, and other training images pasted to the erased sub-region; Step 22, detection network The detection network comprises a basic backbone network, a feature fusion network and a prediction network, wherein the basic backbone network is the backbone network in step 1, adopts EfficientNet B7 for feature extraction, the feature fusion network is the feature fusion part of the multi-scale fusion network in step 1, the specific feature fusion mode adopts a multi-scale feature fusion mode based on Transformer encoding and decoding, and the prediction network is the head prediction network part of the multi-scale fusion network in step 1, and classification and regression are performed in a full convolution manner; Step 23, label assignment and loss calculation Label assignment is to provide a true value for the detection network prediction, adopts an intersection over union criterion, assigns anchor boxes to corresponding objects according to the intersection over union between anchor boxes and real object boxes, based on the label classification result, calculates the loss of classification and regression tasks by using a loss function, and updates the weights of the detection network by using a back propagation algorithm, wherein the loss function comprises three loss functions: heat map classification, offset regression and bounding box size regression; The heat map classification loss function is: wherein, is the number of training samples, respectively represent coordinate values and classes, represents the predicted class confidence at a keypoint, represents the true class confidence at a keypoint, = 1 represents a target, represents background, is a constant; The offset regression loss function is: wherein, represents the real bounding box center point coordinate on the original image, represents the scaling multiple of the feature map compared with the original image, , represents the floor operation, represents the predicted bounding box center point offset. The bounding box size regression loss function is: wherein, is the center point is the length and width of the target frame at the real value, i.e., the real value of the area, is the center point is the length and width of the target frame at the predicted value, and the final loss function is: wherein, are the weights for each loss function; In the test stage, the class and position information of the existing objects in a given image are output, including an input image, a detection network and post-processing; for a given image, first, the classification and regression results are generated by using the trained detection network, then, the Soft-NMS is used for post-processing, redundant detection is suppressed by reducing the classification score of the detection result with a high intersection over union; the test result of the target detection model is tested by collecting a test set, if the test result reaches a preset threshold, it means that the target detection model has the ability of the existing target detection task, and the model training is stopped; otherwise, the training is continued.

4. The goal detection method based on lifelong learning of claim 1, wherein: The specific implementation process of knowledge transfer in step 3 is as follows: For each target detection task, after training the task, the importance of each parameter in the target detection model for the task is calculated , i.e., the proportion of the parameter value in the first i row and the first j column to the entire parameter value, and is used in the subsequent task training, is added to the loss function in the form of a regularization term, so the loss function of the first task is: in, It is the loss function for the current task. It is a balancing factor. These are the parameters in the i-th row and j-th column of the current model parameter matrix. It is from the front The model parameters are obtained after training each task. They will all be updated, especially when performing the first task. Set to 0; The specific method for training the optimized target detection model is as follows: assuming that there are target detection tasks , each task , that is, the label of each task is determined by a real implicit function , wherein is the network parameter of the target detection model, which is continuously updated and optimized with the increase of the detection tasks; each task is given training samples and corresponding labels , d is the dimension of the extracted sample features; at any time point, when the optimized target detection model receives a batch of labeled samples from task m, it makes a prediction of the learned tasks according to the requirements; the training goal is to learn the target detection model of each task through the given training samples as close as possible to the real target function .

5. The goal detection method based on lifelong learning of claim 1, wherein: The specific implementation mode of step 4 is as follows: Step 61, a first task type neural network is constructed by using the basic target detection network as the first column, and a task or a batch of tasks are trained for the known old task; Step 62, a second task type neural network is constructed by using the basic target detection network, then the first column, that is, the previous task type neural network is fixed, and each layer of the neural network of the previous column is connected to each layer of the neural network of the second column as additional input by dimension reduction processing, that is, each layer of the second task type neural network adds the input of the corresponding layer of the previous task type neural network processed by a as an additional input in addition to the original input; Step 63, a third task type neural network is constructed by using the basic target detection network, and the third task is trained, and the neural networks of the previous two columns are fixed, then the method in step 62 is used to connect to the third task type neural network, and N task type neural networks are constructed to learn N new tasks.

6. A life-long learning based object detection apparatus, characterized by: The target detection device based on lifelong learning is used to realize the method in any one of claims 1-5.

Images