Target detection model optimization method, device, electronic device and storage medium

By adding a new side branch network and forgetful priority sampling strategy to the DINO model, the catastrophic forgetting problem of deep learning models in dynamic environments is solved, and rapid adaptation in the new task domain and historical knowledge are achieved, reducing the Computing resource requirements.

CN118350417BActive Publication Date: 2024-11-19HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410780844.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2024-11-19
Estimated Expiration
2044-06-18

AI Technical Summary

Technical Problem

Existing deep learning models are difficult to continuously adaptively learn in dynamic environments, resulting in catastrophic forgetting when tasks change, performance degradation, and high maintenance costs in multiple new task domains, making it difficult to quickly adapt to new tasks and retain historical knowledge.

Method used

A new side branch network was added to the DINO model, and the side branch network parameters were initially warmed up through historical task data, and the forgetful priority sampling strategy was used for continuous learning and training, and incremental training was used for ER technology to reduce the computing complexity and memory requirements.

Benefits of technology

It effectively alleviates the catastrophic forgetting problem of the model, reduces the demand for data and computing resources, improves the model's generalization ability in the new task domain and the memory retention ability of historical tasks, and significantly improves the training speed and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118350417B_ABST
    Figure CN118350417B_ABST
Patent Text Reader

Abstract

The present invention discloses an optimization method, device, electronic device and storage medium of a target detection model, which is applied to a DINO model and relates to the field of artificial intelligence technology. The optimization method of the target detection model comprises the following steps: S10, LST structure improvement: adding a side branch network to the original DINO model trained by historical tasks to obtain a new DINO model; S20, preliminary warm-up: fixing the network parameters of the original DINO model, and using historical tasks to train the parameters of the side branch network; S30, continuous learning incremental training: using ER technology to continuously learn and train the new DINO model, and using a forgetfulness priority sampling strategy to select historical task samples during the training process; S40, reasoning test. The beneficial effects of the present invention: it can not only alleviate the catastrophic forgetting of the model, but also quickly adapt to new tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to an optimization method, device, electronic device and storage medium for a target detection model. Background Art

[0002] In recent years, with the rapid development of deep learning, deep learning has achieved brilliant results in natural image classification, object detection, image segmentation and other fields. However, in the real world, such methods can only achieve high performance when the probability distribution of the test data is similar to that of the training data. Because most of the existing deep learning methods are methods of fitting training data, the performance of the model depends on the quality and characteristics of the training data used. In other words, current deep learning algorithms cannot continuously learn adaptively in dynamic environments, where tasks may change significantly.

[0003] When an existing deep learning model is incrementally updated using a gradient-based update method, the model will experience catastrophic interference or forgetting problems, which will directly lead to a rapid decline in model performance. That is, after learning a new task, the model will forget how to solve the old task due to the interference caused by parameter updates. This is different from the way humans and animals learn. While learning different knowledge, humans and animals do not always forget the knowledge they have learned, but have good memory. Inspired by this learning method of humans and animals, it is natural to apply this idea to the field of deep learning: the model can continuously learn new knowledge while retaining previously learned knowledge. This ability to continuously learn is called continuous learning.

[0004] The research on continuous learning is mainly in the field of image classification, and has achieved certain results, while there are fewer studies on continuous learning for other visual tasks. In actual industrial production, the target detection task is one of the most important tasks, so the research on continuous learning for target detection tasks has great research prospects and application value. A trained target detector often shows obvious performance degradation (catastrophic forgetting) when the task domain changes significantly. If a model for a new task domain is retrained, first, a large amount of training data in the new task domain is required, second, a certain amount of computing resources are required, and third, an additional new model needs to be maintained for the new task domain. The demand for data and computing resources is difficult to achieve in many real industrial applications (especially on edge devices). What's more serious is that once multiple new task domains appear, the maintenance cost of the model increases linearly, and when the domain information is unknown, it is difficult to determine which model to use to perform the detection task. Therefore, how to reduce the demand for data and computing resources so that a unified model can quickly adapt to the new task domain and not forget the historical task domain knowledge has become a key to solving the problem. This is also the advantage of continuous learning technology.

[0005] To this end, the present invention provides an optimization method, device, electronic device and storage medium for a target detection model, which can not only alleviate the catastrophic forgetting of the model, but also quickly adapt to new tasks. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides an optimization method, device, electronic device and storage medium for a target detection model, which can not only alleviate the catastrophic forgetting of the model, but also quickly adapt to new tasks.

[0007] The technical solution adopted by the present invention to solve the technical problem is: an optimization method of a target detection model, which is applied to a DINO model, and the improvement thereof is that the optimization method of the target detection model comprises the following steps:

[0008] S10, LST structure improvement: Add a side branch network to the original DINO model trained by historical tasks to obtain a new DINO model;

[0009] S20, preliminary warm-up: fix the network parameters of the original DINO model and use historical tasks to train the parameters of the side network;

[0010] S30, continuous learning incremental training: ER technology is used to continuously learn and train the new DINO model, and the forgetfulness priority sampling strategy is used to select historical task samples during the training process;

[0011] S40, reasoning test: after training is completed, perform target detection tasks: fire detection, reflective clothing detection and outdoor advertising detection respectively. By evaluating the test set accuracy of the new DINO model for historical tasks, the degree of forgetting of the new DINO model for historical tasks is obtained. By evaluating the test set accuracy of new tasks, the generalization ability of the new DINO model for new tasks is obtained.

[0012] Further, in step S10, the original DINO model includes a backbone network, several encoder layers, several decoder layers and an object detection element; the output end of the backbone network is connected to the input end of the first encoder layer; the output end of the previous encoder layer between adjacent encoder layers is connected to the input end of the next encoder layer, wherein the encoder layer close to the backbone network is the previous encoder layer, and the encoder layer far from the backbone network is the next encoder layer; the output end of the last encoder layer is connected to the input ends of all decoder layers; the output end of the object detection element is connected to the input end of the first decoder layer; the output end of the previous decoder layer between adjacent decoder layers is connected to the input end of the next decoder layer, wherein the decoder layer close to the object detection element is the previous decoder layer, and the decoder layer far from the object detection element is the next decoder layer.

[0013] Furthermore, in step S10, the side network includes a plurality of LST encoder layers, an LST decoder layer, a plurality of dimensionality reduction modules, a plurality of dimensionality increase modules and a detection head module; the number of the LST encoder layers is the same as the number of the encoder layers, and the input end of the first LST encoder layer is connected to the output end of the backbone network through the dimensionality reduction module, and the output end of the previous LST encoder layer between adjacent LST encoder layers is connected to the input end of the next LST encoder layer, wherein the LST encoder layer close to the backbone network is the previous LST encoder layer, and the LST encoder layer far away from the backbone network is the next LST encoder layer; the output end of the encoder layer at the same position is connected to the input end of the LST encoder layer through the dimensionality reduction module; the number of the LST decoder layers is the same as the number of the decoder layers, and the first LST decoder The input end of the layer is connected to the output end of the object detector through a dimensionality reduction module, and the output end of the previous LST decoder between adjacent LST decoder layers is connected to the input end of the next LST decoder layer through a dimensionality increase module, wherein the LST decoder layer close to the object detector is the previous LST decoder layer, and the LST decoder layer far away from the object detector is the next LST decoder layer; the output end of the decoder layer at the same position is connected to the input end of the LST decoder layer through a dimensionality reduction module, and the output end of the last decoder layer is directly connected to the input end of the last LST decoder layer; the output end of the last LST encoder layer is connected to the input end of the LST decoder layer through a dimensionality increase module, and is directly connected to the input ends of the remaining LST decoder layers; the output end of the last LST decoder layer is connected to the input end of the detection head module.

[0014] Furthermore, in step S20, the specific method of the preliminary preheating is: the parameters of the newly added side network are trained by the training data of the historical tasks using the original DINO model, and the network parameters of the original DINO model are fixed; after updating the parameters of the side network by gradient descent, the trained side network can be well adapted to the historical tasks.

[0015] Furthermore, in step S30, the specific implementation steps of adopting the forgetfulness priority sampling strategy to select historical task samples include:

[0016] S301, before training, use the new DINO model to be trained to calculate the loss function for all historical task samples, and record the loss function value of each historical task sample , and assign an identical sampling weight coefficient , the initial value is set to 1, that is ;

[0017] S302, in the first training iteration, each historical task sample is uniformly sampled, and the loss function is calculated for the sampled historical samples using the current new DINO model , the loss function value of the historical task samples that have not been sampled remains unchanged;

[0018] S303, at the nth training iteration, each historical task sample is The probability sampling is used to calculate the loss function value of the sampled historical task samples using the current new DINO model , the loss function value of the historical task samples that have not been sampled remains unchanged; and the weight coefficient is used to update the historical task samples ,in:

[0019] ; In the formula, Indicates positive temperature coefficient;

[0020] S304, repeat S303 until the continuous learning training is completed.

[0021] Furthermore, in step S40, the specific method of the reasoning test is: input the pictures in the test set of the historical task or the pictures in the test set of the new task as the target detection pictures into the new DINO model, the new DINO model will generate the prediction results corresponding to the target detection pictures, and perform mAP calculation on the prediction results and the target detection pictures to obtain the accuracy of the target detection pictures.

[0022] An optimization device for a target detection model, applying the optimization method for a target detection model as described above, wherein the optimization device for the target detection model comprises:

[0023] The model optimization module is used to add a side branch network to the original DINO model trained by historical tasks to obtain a new DINO model;

[0024] The model training module is used to fix the network parameters of the original DINO model, use historical tasks to train the parameters of the side network, and use ER technology to continuously learn and train the new DINO model;

[0025] The data processing module is used to select historical task samples using a forgetfulness-first sampling strategy during training;

[0026] The test evaluation module is used to obtain the degree of forgetting of the new DINO model for historical tasks by evaluating the test set accuracy of the new DINO model for historical tasks, and to obtain the generalization ability of the new DINO model for new tasks by evaluating the test set accuracy of new tasks.

[0027] An electronic device, the improvement of which is that it comprises at least one processor and at least one memory, wherein:

[0028] The memory has computer-readable instructions stored thereon;

[0029] The computer-readable instructions are executed by one or more processors, so that the electronic device implements the optimization method of the target detection model as described above.

[0030] A storage medium having computer-readable instructions stored thereon, wherein the improvement is that the computer-readable instructions are executed by one or more processors to implement the optimization method of the target detection model as described above.

[0031] The beneficial effects of the present invention are as follows: the present invention obtains a new DINO model by adding a side branch network to the trained original DINO model, and firstly trains the parameters of the side branch network using historical task data, and then uses the ER technology to perform continuous learning training on the new DINO model, and adopts a forgetfulness-first sampling strategy to select historical task samples during the training process, wherein the memory playback scheme of the forgetfulness-first sampling strategy has a better effect of alleviating catastrophic forgetting than the continuous learning method based on regularization, and has a lower complexity than the continuous learning method based on gradient memory; the continuous learning training based on LST is more efficient in calculating the inverse When propagating gradients, only the gradients of the side networks need to be calculated, which significantly reduces the computational complexity and video memory requirements compared to fine-tuning all parameters. During training, a small amount of more helpful historical data is selected through the forgetfulness-first sampling strategy instead of all historical data, which greatly speeds up the training speed. After the training is completed, the degree of forgetting of the new DINO model for historical tasks is obtained by evaluating the test set accuracy of the new DINO model for historical tasks, and the generalization ability of the new DINO model for new tasks is obtained by evaluating the test set accuracy of new tasks, so as to evaluate the performance of the new DINO model. This can not only alleviate the catastrophic forgetting of the model, but also quickly adapt to new tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is an overall flow chart of an optimization method of a target detection model of the present invention;

[0033] Figure 2 The overall structure diagram of the new DINO model is shown as an exemplary embodiment;

[0034] Figure 3 A block diagram of an optimization device for a target detection model of the present invention;

[0035] Figure 4 A hardware structure diagram of an electronic device is shown in an exemplary embodiment;

[0036] Figure 5The present invention is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0037] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0038] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0039] Reference Figure 1 As shown, the present invention discloses an optimization method for a target detection model, which is applied to a DINO model. The improvement thereof is that the optimization method for the target detection model comprises the following steps:

[0040] S10, LST structure improvement: Add a side network to the original DINO model trained by historical tasks to obtain a new DINO model; wherein the original DINO model includes a backbone network, several encoder layers, several decoder layers and object detection elements; the output end of the backbone network is connected to the input end of the first encoder layer; the output end of the previous encoder layer between adjacent encoder layers is connected to the input end of the next encoder layer, wherein the encoder layer close to the backbone network is the previous encoder layer, and the encoder layer far from the backbone network is the next encoder layer; the output end of the last encoder layer is connected to the input ends of all decoder layers; The output end of the object detection unit is connected to the input end of the first decoder layer; the output end of the previous decoder layer between adjacent decoder layers is connected to the input end of the next decoder layer, wherein the decoder layer close to the object detection unit is the previous decoder layer, and the decoder layer far away from the object detection unit is the next decoder layer; the side network includes a plurality of LST encoder layers, an LST decoder layer, a plurality of dimensionality reduction modules, a plurality of dimensionality increase modules and a detection head module; the number of the LST encoder layers is the same as the number of the encoder layers, and the input end of the first LST encoder layer is connected to the output end of the backbone network through the dimensionality reduction module, and the adjacent LST encoder layers are connected. The output end of the previous LST encoder layer is connected to the input end of the next LST encoder layer, wherein the LST encoder layer close to the backbone network is the previous LST encoder layer, and the LST encoder layer far away from the backbone network is the next LST encoder layer; the output end of the encoder layer at the same position is connected to the input end of the LST encoder layer through a dimensionality reduction module; the number of LST decoder layers is the same as the number of decoder layers, and the input end of the first LST decoder layer is connected to the output end of the object detector through the dimensionality reduction module, and the output end of the previous LST decoder between adjacent LST decoder layers is connected to the input end of the next LST decoder layer The ends are connected through a dimension-raising module, wherein the LST decoder layer close to the object detector is the previous LST decoder layer, and the LST decoder layer far from the object detector is the next LST decoder layer; the output end of the decoder layer at the same position is connected to the input end of the LST decoder layer through a dimension-reducing module, and the output end of the last decoder layer is directly connected to the input end of the last LST decoder layer; the output end of the last LST encoder layer is connected to the input end of the LST decoder layer through a dimension-raising module, and is directly connected to the input ends of the remaining LST decoder layers; the output end of the last LST decoder layer is connected to the input end of the detection head module;

[0041] When a target detection model is fully trained in a task domain, it can often perform well in the domain and in domains with similar probability distributions. However, if the domain changes significantly, the model performance will usually degrade significantly. In a new task domain, a limited amount of new data is usually available. Using continuous learning technology to perform incremental training on these new data can allow the model to acquire detection capabilities in the new task domain, while retaining the old task domain capabilities. In order to achieve incremental continuous learning and training of new task data, this embodiment proposes an implementation method for improving the target detection model (original DINO model), namely, adding a side network to the original DINO model to form a new DINO model to correct the output results of the original DINO model. As a preferred embodiment, refer to Figure 2 As shown, the original DINO model in this embodiment is composed of a backbone network, 2 encoder layers, 2 decoder layers and an object detector; the side network is composed of 2 LST encoder layers, 2 LST decoder layers, 5 dimension reduction modules, 2 dimension increase modules and a detection head module; wherein, after the input image is characterized by the backbone network, it enters the encoding area formed by stacking 2 encoder layers for processing; and enters the LST encoding area formed by stacking 2 LST encoder layers. The structure of the LST encoding area is similar to that of the encoding area of ​​the original DINO model. The only difference is that the feature dimension is lower (such as 1 / 8), and the input and output of the encoder layer of the original DINO model are input into the LST encoder layer for processing after dimension reduction; the input image passes through the original encoder layer and the LST encoder After the layer processing, two output features with different dimensions will be obtained. These two output features enter the decoder layer and the LST decoder layer respectively. The design of the LST decoder layer is similar to that of the encoder layer, that is, the same number of LST decoder layers are added in the position of the original decoder. Similarly, the structure of the LST decoder layer is similar to that of the original decoder layer, but the feature dimension is lower (such as 1 / 8, except for the last LST decoder layer). It is worth noting that the feature dimension of the last LST decoder layer is the same as that of the last decoder layer. Therefore, the dimension reduction module from the decoder layer to the LST decoder layer is discarded, and the outputs of the LST encoder layer and the previous LST decoder layer are respectively processed by dimension increase, and then used as the output of the last LST decoder layer. After being processed by the last LST decoder layer, the features are output through a detection head module to output the detection box prediction results. Since the side network is much narrower than the original network, the additional overhead introduced is limited, which is a design for efficient parameter fine-tuning;

[0042] S20, preliminary preheating: fixing the network parameters of the original DINO model, and using the historical tasks to train the parameters of the side network; specifically, the specific method of the preliminary preheating is: training the parameters of the newly added side network by the training data of the historical tasks using the original DINO model, and the network parameters of the original DINO model are fixed; after updating the parameters of the side network by gradient descent, the trained side network can be well adapted to the historical tasks; in this embodiment, after the original DINO model that has been fully trained by the historical tasks is improved by LST, the parameters of the newly added side network are trained again using the training data of the historical tasks, while the parameters of the main network are fixed, and after updating the parameters of the side network by the gradient descent method, the output of the side network can also obtain the same task accuracy as the main network;

[0043] S30, continuous learning incremental training: using ER technology to continuously learn and train the new DINO model, and using the forgetfulness priority sampling strategy to select historical task samples during the training process; further, the specific implementation steps of using the forgetfulness priority sampling strategy to select historical task samples include:

[0044] S301, before training, use the new DINO model to be trained to calculate the loss function for all historical task samples, and record the loss function value of each historical task sample , and assign an identical sampling weight coefficient , the initial value is set to 1, that is ;

[0045] S302, in the first training iteration, each historical task sample is uniformly sampled, and the loss function is calculated for the sampled historical samples using the current new DINO model , the loss function value of the historical task samples that have not been sampled remains unchanged;

[0046] S303, at the nth training iteration, each historical task sample is The probability sampling is used to calculate the loss function value of the sampled historical task samples using the current new DINO model , the loss function value of the historical task samples that have not been sampled remains unchanged; and the weight coefficient is used to update the historical task samples ,in:

[0047] ; In the formula, Indicates positive temperature coefficient;

[0048] S304, repeatedly executing S303 until the continuous learning training is completed;

[0049] In this embodiment, in order to enable the new DINO model to process data in the new task domain, incremental training is performed, that is, the DINO model is fine-tuned using the new task data and part of the historical task data. During fine-tuning, the model parameters of the original DINO model that remembers the learned historical task knowledge remain unchanged, which significantly improves the problem of catastrophic forgetting of historical tasks. The new task is learned through learnable side network parameters and will not interfere with historical knowledge. In this incremental training process, the memory replay (ER) technology is used, that is, the data used contains a small amount of historical task data in addition to the training data of the new task. ER is also an effective typical method of continuous learning, which is used to improve training efficiency and task accuracy and meet the requirements of limited computing resources. Furthermore, when optimizing the training objectives of new tasks, it will inevitably affect the prediction results of historical task samples. If it is expressed as the loss function of historical task samples, If the value increases, it means that the new task has a great adverse effect on the sample (catastrophic forgetting); if it is manifested as a decrease in the loss function of the historical sample, it means that the new task has a favorable effect on the sample, that is, the gradient update direction of the historical task target and the new task target is consistent; then, the historical samples with adverse effects should be more involved in training to make up for the catastrophic forgetting caused by the new task; the historical task samples with favorable effects can be used less, because optimizing the new task target will not cause catastrophic forgetting, but will be beneficial to performance; therefore, the idea of ​​the forgetfulness priority sampling strategy is to sample historical samples based on the severity of forgetfulness, and the sampling weight coefficient of severe forgetfulness is high, and vice versa; the memory replay scheme of this forgetfulness priority sampling strategy has a better effect of alleviating catastrophic forgetting than the continuous learning method based on regularization, and has a lower complexity than the continuous learning method of gradient memory;

[0050] S40, reasoning test: after the training is completed, the target detection tasks are performed: fire detection, reflective clothing detection and outdoor advertising detection are performed respectively, and the degree of forgetting of the new DINO model for the historical tasks is obtained by evaluating the test set accuracy of the new DINO model for the historical tasks, and the generalization ability of the new DINO model for the new tasks is obtained by evaluating the test set accuracy of the new tasks; specifically, the specific method of the reasoning test is: input the pictures in the test set of the historical tasks or the pictures in the test set of the new tasks as the target detection pictures into the new DINO model, and the new DINO model will generate the prediction results corresponding to the target detection pictures, and calculate the mAP between the prediction results and the target detection pictures to obtain the accuracy of the target detection pictures;

[0051] As a preferred embodiment, this embodiment experiments on three target detection tasks, specifically fire detection, reflective clothing detection and outdoor advertising detection, where each task contains two data sets representing different domains, data set 1 and data set 2. The difference between the two is that the scenes and specific shapes of the objects to be detected are not exactly the same; the fires in the fire detection data set 1 often occur in construction site scenes, while the fires in the data set 2 are distributed in various life scenes, including roads, kitchens, and the wild; the main scene of the reflective clothing detection data set 1 is the construction site, and the color of the reflective clothing is only yellow, while the data set includes various colors and covers more occurrence scenes; the outdoor advertising detection data set 1 is mainly vertical billboards, while the data set 2 contains more styles, including banners, posters, graffiti, etc.

[0052] Assuming that the original DINO detection model has been fully trained on the training set of dataset 1 (historical tasks) of the three tasks, this experiment will conduct continuous learning training on the training set of dataset 2 (new tasks) (the training set of dataset 1 is the historical task samples, and the training set of dataset 2 is the new task samples); after the model training is completed, in order to evaluate the degree of forgetting of the model, mAP is tested on the test set of dataset 1; in order to evaluate the generalization ability of the model on new tasks, mAP is tested on the test set of dataset 2; the higher the two indicators, the higher the better; specifically, assuming that before training, the model is tested on the test set of historical tasks and the test set of new tasks The mAP measured on the test set is A0 and B0. After training, the mAP measured is A1 and B1. The expected result is that A1 is not lower than A0 (higher is better), and the less A1 decreases, the better; B1 is as high as possible from B0, and the higher B1 increases, the better; the former reflects the degree of forgetting on historical tasks, the less it decreases, the less forgetting; the latter reflects the generalization of the model on new tasks, the higher it is, the stronger the generalization; at the same time, the accuracy on these two data sets before training is listed. For comparison, the difference in accuracy before and after training reflects the change in the model's ability after continuous learning and training; the experimental data of the three target detection tasks are as follows:

[0053] Table 1: Fire detection mAP

[0054]

[0055] Table 2: mAP for reflective clothing detection

[0056]

[0057] Table 3: Outdoor advertising detection mAP

[0058]

[0059] From Tables 1 to 3, we can see that the method proposed in the present invention has the most obvious improvement on new tasks after training, and has the least effect of forgetting on historical tasks. In addition, the training time required for direct fine-tuning instead of the LST structure increases by about 40%, and the video memory usage increases by 50%.

[0060] Reference Figure 3 As shown, the present invention further provides an optimization device 600 for a target detection model, which applies the optimization method for the target detection model as described above, and the improvement thereof is that the optimization device 600 for the target detection model comprises:

[0061] A model optimization module 601 is used to add a side branch network to the original DINO model trained by historical tasks to obtain a new DINO model;

[0062] Model training module 602, used to fix the network parameters of the original DINO model, and use historical tasks to train the parameters of the side network, and use ER technology to continuously learn and train the new DINO model;

[0063] The data processing module 603 is used to select historical task samples by adopting a forgetfulness-first sampling strategy during the training process;

[0064] The test evaluation module 604 is used to obtain the degree of forgetting of the new DINO model for historical tasks by evaluating the test set accuracy of the new DINO model for historical tasks, and to obtain the generalization ability of the new DINO model for new tasks by evaluating the test set accuracy of new tasks.

[0065] It should be noted that the optimization strategy of the target detection model provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the optimization device 600 of the target detection model will be divided into different functional modules to complete all or part of the functions described above.

[0066] In addition, the target detection model optimization device 600 provided in the above embodiment and the target detection model optimization method embodiment belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiment and will not be repeated here.

[0067] Figure 4 The present invention schematically illustrates the structure of an electronic device according to an exemplary embodiment.

[0068] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as requiring dependence on or having Figure 4One or more components of exemplary electronic device 2000 are shown.

[0069] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 4 As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0070] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

[0071] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this is not intended to be a specific limitation.

[0072] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0073] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0074] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 4 (not shown), each module may respectively include computer-readable instructions for the electronic device 2000. For example, the device for assessing task priority may be regarded as an application 253 deployed on the electronic device 2000.

[0075] The data 255 may be signal information, etc., stored in the memory 250 .

[0076] The central processor 270 may include one or more processors and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the optimization method of the target detection model is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.

[0077] In addition, the present invention can also be implemented by hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or a combination of the two.

[0078] See also Figure 5 In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 400 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capability.

[0079] exist Figure 5 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0080] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0081] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0082] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0083] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to this.

[0084] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0085] The computer-readable instructions are executed by one or more processors 4001 to implement the optimization method of the target detection model in the above-mentioned embodiments.

[0086] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the optimization method of the target detection model as described above.

[0087] Compared with the related art, the beneficial effects of the present invention are as follows: the present invention obtains a new DINO model by adding a side branch network to the trained original DINO model, and first trains the parameters of the side branch network using historical task data, and then uses the ER technology to perform continuous learning training on the new DINO model, and adopts the forgetfulness priority sampling strategy to select historical task samples during the training process, wherein the memory playback scheme of the forgetfulness priority sampling strategy has a better effect of alleviating catastrophic forgetting than the continuous learning method based on regularization, and has a lower complexity than the continuous learning method based on gradient memory; continuous learning training based on LST, When calculating the back-propagation gradient, only the gradient of the side network needs to be calculated, which significantly reduces the computational complexity and video memory requirements compared to fine-tuning all parameters. During training, a small amount of more helpful historical data is selected through the forgetfulness-first sampling strategy, rather than all historical data, which greatly speeds up the training speed. After the training is completed, the degree of forgetting of the new DINO model for historical tasks is obtained by evaluating the test set accuracy of the new DINO model for historical tasks, and the generalization ability of the new DINO model for new tasks is obtained by evaluating the test set accuracy of new tasks, so as to evaluate the performance of the new DINO model. This can not only alleviate the catastrophic forgetting of the model, but also quickly adapt to new tasks.

[0088] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An optimization method for a target detection model, applied to a DINO model, characterized in that: The optimization method of the target detection model comprises the following steps: S10, LST structure improvement: Add a side network to the original DINO model trained by historical tasks to obtain a new DINO model; wherein the original DINO model includes a backbone network, several encoder layers, several decoder layers and object detection elements; the output end of the backbone network is connected to the input end of the first encoder layer; the output end of the previous encoder layer between adjacent encoder layers is connected to the input end of the next encoder layer, wherein the encoder layer close to the backbone network is the previous encoder layer, and the encoder layer far from the backbone network is the next encoder layer; the output end of the last encoder layer is connected to the input ends of all decoder layers; The output end of the object detection unit is connected to the input end of the first decoder layer; the output end of the previous decoder layer between adjacent decoder layers is connected to the input end of the next decoder layer, wherein the decoder layer close to the object detection unit is the previous decoder layer, and the decoder layer far away from the object detection unit is the next decoder layer; the side network includes a plurality of LST encoder layers, an LST decoder layer, a plurality of dimensionality reduction modules, a plurality of dimensionality increase modules and a detection head module; the number of the LST encoder layers is the same as the number of the encoder layers, and the input end of the first LST encoder layer is connected to the output end of the backbone network through the dimensionality reduction module, and the adjacent LST encoder layers are connected. The output end of the previous LST encoder layer is connected to the input end of the next LST encoder layer, wherein the LST encoder layer close to the backbone network is the previous LST encoder layer, and the LST encoder layer far away from the backbone network is the next LST encoder layer; the output end of the encoder layer at the same position is connected to the input end of the LST encoder layer through a dimensionality reduction module; the number of LST decoder layers is the same as the number of decoder layers, and the input end of the first LST decoder layer is connected to the output end of the object detector through the dimensionality reduction module, and the output end of the previous LST decoder between adjacent LST decoder layers is connected to the input end of the next LST decoder layer The ends are connected through a dimension-raising module, wherein the LST decoder layer close to the object detector is the previous LST decoder layer, and the LST decoder layer far from the object detector is the next LST decoder layer; the output end of the decoder layer at the same position is connected to the input end of the LST decoder layer through a dimension-reducing module, and the output end of the last decoder layer is directly connected to the input end of the last LST decoder layer; the output end of the last LST encoder layer is connected to the input end of the LST decoder layer through a dimension-raising module, and is directly connected to the input ends of the remaining LST decoder layers; the output end of the last LST decoder layer is connected to the input end of the detection head module; S20, preliminary warm-up: fix the network parameters of the original DINO model and use historical tasks to train the parameters of the side network; S30, continuous learning incremental training: ER technology is used to continuously learn and train the new DINO model, and the forgetfulness priority sampling strategy is used to select historical task samples during the training process; S40, reasoning test: after training is completed, perform target detection tasks: fire detection, reflective clothing detection and outdoor advertising detection respectively. By evaluating the test set accuracy of the new DINO model for historical tasks, the degree of forgetting of the new DINO model for historical tasks is obtained. By evaluating the test set accuracy of new tasks, the generalization ability of the new DINO model for new tasks is obtained.

2. The method for optimizing a target detection model according to claim 1, characterized in that: In step S20, the specific method of the preliminary preheating is: the parameters of the newly added side network are trained by the training data of the historical tasks using the original DINO model, and the network parameters of the original DINO model are fixed; after updating the parameters of the side network by gradient descent, the trained side network can be well adapted to the historical tasks.

3. The method for optimizing a target detection model according to claim 2, characterized in that: In step S30, the specific implementation steps of adopting the forgetfulness priority sampling strategy to select historical task samples include: S301, before training, use the new DINO model to be trained to calculate the loss function for all historical task samples, and record the loss function value of each historical task sample , and assign an identical sampling weight coefficient , the initial value is set to 1, that is ; S302, in the first training iteration, each historical task sample is uniformly sampled, and the loss function is calculated for the sampled historical samples using the current new DINO model , the loss function value of the historical task samples that have not been sampled remains unchanged; S303, at the nth training iteration, each historical task sample is The probability sampling is used to calculate the loss function value of the sampled historical task samples using the current new DINO model , the loss function value of the historical task samples that have not been sampled remains unchanged; and the weight coefficient is used to update the historical task samples ,in: ; In the formula, Indicates positive temperature coefficient; S304, repeat S303 until the continuous learning training is completed.

4. The method for optimizing a target detection model according to claim 3, characterized in that: In step S40, the specific method of the reasoning test is: input the pictures in the test set of the historical task or the pictures in the test set of the new task as the target detection pictures into the new DINO model, the new DINO model will generate the prediction results corresponding to the target detection pictures, and perform mAP calculation on the prediction results and the target detection pictures to obtain the accuracy of the target detection pictures.

5. An optimization device for a target detection model, applying the optimization method for a target detection model according to any one of claims 1 to 4, characterized in that: The optimization device of the target detection model comprises: The model optimization module is used to add a side branch network to the original DINO model trained by historical tasks to obtain a new DINO model; The model training module is used to fix the network parameters of the original DINO model, use historical tasks to train the parameters of the side network, and use ER technology to continuously learn and train the new DINO model; The data processing module is used to select historical task samples using a forgetfulness-first sampling strategy during training; The test evaluation module is used to obtain the degree of forgetting of the new DINO model for historical tasks by evaluating the test set accuracy of the new DINO model for historical tasks, and to obtain the generalization ability of the new DINO model for new tasks by evaluating the test set accuracy of new tasks.

6. An electronic device, characterized in that: comprises at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more processors, so that the electronic device implements the method for optimizing the target detection model as described in any one of claims 1-4.

7. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the method for optimizing the target detection model described in any one of claims 1-4.