A perturbation enhancement method, system and device for open world object detection
By optimizing the classification and regression prototype branch model through prototype learning and random perturbation data, the problem that closed-world object detection algorithms cannot identify unknown categories is solved, and the accuracy and incremental scalability of open-world object detection are improved.
Patent Information
- Application Number
- CN202311000471.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-09
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2043-08-09
AI Technical Summary
Existing object detection algorithms are mainly designed for closed worlds, cannot effectively identify unknown categories, and have poor incremental scalability in detection capabilities.
By introducing prototype learning and random perturbation data, and using instance feature maps of uniform size to reduce the distribution range of known task categories, perturbation samples are generated and combined with new task samples for training. This optimizes the classification and regression prototype branch model and improves the ability to detect unknown task categories.
It achieves high-precision detection and incremental learning capabilities for unknown task categories, alleviates the network's forgetting of known task categories, and improves the accuracy and incremental scalability of open-world object detection.
Smart Images
Figure CN117152556B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of target detection, in particular to a disturbance enhancement method, system and device for open world target detection. BACKGROUND
[0002] Most of the current mainstream target detection algorithms are for closed world, that is, the target detection algorithm is mostly for known category task classification, and the detection model can perform well on the test set with the same distribution as the training set through multiple iterations of training on the given training set, but it cannot accurately detect unknown category task classification.
[0003] However, the known category target detection algorithm has the following shortcomings:
[0004] (1) Only known categories can be identified, which is quite different from the recognition ability of people in the real world;
[0005] (2) The incremental expansion of the recognition ability of the detection model is poor, and it cannot accurately and timely identify unknown category task classification. SUMMARY
[0006] The purpose of the embodiments of the present application is to provide a disturbance enhancement method, system and device for open world target detection, so as to improve the incremental expansion of the recognition ability of open world target detection and improve the accuracy of open world target detection.
[0007] To achieve the above purpose, the embodiments of the present application provide the following solutions:
[0008] A disturbance enhancement method for open world target detection comprises:
[0009] Obtaining training data; the training data comprises current task training data and original task training data;
[0010] Taking PASCAL VOC and MS-COCO data sets as training data;
[0011] According to the original task training data, a given distribution is fitted to obtain random disturbance data;
[0012] Element-level addition is performed on the random disturbance data and the current task training data to obtain disturbance samples of the current task training data;
[0013] According to the disturbance samples and original samples of the current task training data, feature extraction is performed to obtain an instance feature map f of uniform size; after global pooling is performed on the instance feature map f of uniform size, intermediate features Z are obtained.
[0014] The mean of the intermediate features Z is calculated to obtain a new prototype center Pnew ; update the new prototype center P new and the initial prototype center P by a momentum parameter η weighted update to obtain an updated prototype center P1;
[0015] Calculate the intermediate feature f mid of any one task in the original task training data, and obtain the corresponding prototype center distance;
[0016] According to the prototype center distance, obtain M task data samples closest to the updated prototype center P1 in the arbitrary task; take the M task data samples as a playback example; adjust the classification regression prototype branch model according to the playback example of all tasks in the original task training data, and obtain an optimized classification regression prototype branch model;
[0017] Input the intermediate feature Z into the optimized classification regression prototype branch model to obtain the known task categories and unknown task categories of the current task training data and the original task training data.
[0018] Optionally, the current task training data includes 1 base task and N incremental tasks.
[0019] The original task training data includes 1 base task and N incremental tasks; wherein N is a positive integer.
[0020] Optionally, fitting a given distribution according to the original task training data to obtain random perturbation data, specifically including:
[0021] Fitting a given distribution according to the intermediate features of the N incremental tasks in the original task training data to obtain random perturbation data; the random perturbation data is the data with the optimal intermediate feature fitting distribution.
[0022] Optionally, according to the perturbation samples and original samples of the current task training data, feature extraction is performed to obtain an instance feature map f of uniform size, specifically including:
[0023] Using a target detector and a feature extraction network to extract features in the perturbation samples and original samples of the current task training data to obtain a first feature map;
[0024] Input the first feature map into a region proposal network for candidate frame extraction to obtain a candidate frame;
[0025] Input the candidate frame and the first feature map into a RoI network to obtain the instance feature map f of uniform size.
[0026] Optionally, the updated classification regression prototype branch model includes a classification branch, a regression branch and a prototype branch.
[0027] The structure of the classification branch and the regression branch is the original Faster R-CNN structure; the prototype branch is composed of a fully connected layer.
[0028] Optionally, the new prototype center P new and the initial prototype center P are subjected to weighted update with a momentum parameter η, to obtain an updated prototype center P1, and specifically comprising:
[0029] .
[0030] Optionally, the intermediate feature f mid of any one task in the original task training data is calculated, to obtain a corresponding prototype center distance, and specifically comprising:
[0031] ;
[0032] wherein dist is the corresponding prototype center distance, represents the i-th initial prototype center, represents the i-th intermediate feature.
[0033] To achieve the above object, embodiments of the present application further provide the following schemes:
[0034] A perturbation enhancement system for open world object detection, comprising:
[0035] A data acquisition module is configured to acquire training data; the training data comprises current task training data and original task training data;
[0036] The PASCAL VOC and MS-COCO data sets are used as training data;
[0037] A fitting given distribution module is connected with the data acquisition module and is configured to fit a given distribution according to the original task training data, to obtain random perturbation data;
[0038] An element-level addition module is connected with the data acquisition module and the fitting given distribution module respectively, and is configured to perform element-level addition on the random perturbation data and the current task training data, to obtain perturbation samples of the current task training data;
[0039] A feature extraction module is connected with the element-level addition module and is configured to perform feature extraction on the perturbation samples and original samples of the current task training data, to obtain instance feature maps f of uniform size; after global pooling is performed on the instance feature maps f of uniform size, intermediate features Z are obtained;
[0040] The weighted updating module is connected with the feature extraction module and is configured to calculate a mean value of the intermediate feature Z to obtain a new prototype center P new ; and perform weighted updating with a momentum parameter of the new prototype center P new and the initial prototype center P to obtain an updated prototype center P1.
[0041] The prototype center distance calculation module is connected with the data acquisition module and is configured to calculate an intermediate feature f mid of any one task in the original task training data to obtain a corresponding prototype center distance.
[0042] The classification regression prototype branch model construction module is connected with the prototype center distance calculation module and the data acquisition module respectively and is configured to obtain M task data samples closest to the updated prototype center P1 in the any one task according to the prototype center distance; take the M task data samples as a replay example; and adjust a classification regression prototype branch model according to the replay example of all tasks in the original task training data to obtain an optimized classification regression prototype branch model.
[0043] The output module is connected with the feature extraction module, the element-level addition module and the fitting given distribution module respectively and is configured to input the intermediate feature Z into the optimized classification regression prototype branch model to obtain known task categories and unknown task categories of the current task training data and the original task training data.
[0044] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the perturbation enhancement method for open-world object detection when executing the computer program.
[0045] A non-transitory computer-readable storage medium has a computer program stored thereon, and the computer program is executed to implement the perturbation enhancement method for open-world object detection.
[0046] In the embodiment of the present application, the perturbation enhancement method for open-world object detection is based on prototype learning and random perturbation data, reduces the distribution range of known task categories through the size-unified instance feature map f, thereby facilitating the discrimination of unknown task categories in the detection stage, improving the discriminability of known task categories, and further strengthening the detection ability of unknown task categories. The method realizes the open-world idea, realizes high-precision detection of unknown task categories, and the ability of incremental learning of unknown task categories.
[0047] Meanwhile, the random disturbance data is obtained by fitting a given distribution through original task training data, so as to promote the incremental learning ability of the classification regression prototype branch model, avoid the class bias phenomenon of the network, relieve the forgetting of the original task training data while learning new task classes, and have good performance in detecting known task classes and unknown task classes. BRIEF DESCRIPTION OF DRAWINGS
[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0049] Figure 1 A flowchart of the disturbance enhancement method for open world target detection provided by the embodiments of the present application is shown.
[0050] Figure 2 A detailed structure diagram of the disturbance enhancement system for open world target detection provided by the embodiments of the present application is shown.
[0051] Figure 3 A diagram showing the introduction of a prototype branch on the Faster R-CNN head of the target detection model in a closed world provided by the embodiments of the present application is shown.
[0052] Figure 4 A diagram showing the generation of random disturbance as noise using the best fitting distribution provided by the embodiments of the present application is shown.
[0053] Symbol explanation:
[0054] Data acquisition module-1, fitting given distribution module-2, element-level addition module-3, feature extraction module-4, weighted update module-5, prototype center distance calculation module-6, classification regression prototype branch model construction module-7, output module-8. DETAILED DESCRIPTION
[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0056] The present application aims to provide a disturbance enhancement method and system for open world target detection, to solve the problems of poor incremental expansion of the existing open world target detection recognition ability and low accuracy of target detection.
[0057] In the embodiments of the present application, the closed world is converted into a more realistic open world, the ability to detect unknown categories is improved, and the ability of model incremental learning is improved by using statistical information of known samples. First, prototype learning is introduced, candidate box embedding feature aggregation and candidate box embedding feature space compression are proposed to reduce the distribution range of known class features in the embedding space, and the prediction results far from the prototype center are regarded as unknown classes. Then, the distribution information of old samples is used to perturb the current task samples, the old knowledge is retained and the new knowledge is learned, the representative samples of previous tasks are extracted to construct a playback example set to fine-tune the network, and finally an open world object detection algorithm with high performance is realized.
[0058] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments.
[0059] Figure 1 An exemplary flow of the above-mentioned perturbation enhancement method for open world object detection is shown. Each step will be described in detail below.
[0060] Step 1: obtaining training data; the training data includes: current task training data and original task training data;
[0061] The PASCAL VOC and MS-COCO data sets are used as training data.
[0062] Random perturbation data is obtained by fitting a given distribution according to the original task training data.
[0063] The current task training data includes: 1 base task and N incremental tasks.
[0064] The original task training data includes: 1 base task and N incremental tasks; wherein N is a positive integer.
[0065] Random perturbation data is obtained by fitting a given distribution according to the original task training data, specifically including:
[0066] Random perturbation data is obtained by fitting a given distribution according to the intermediate features of the N incremental tasks in the original task training data; the random perturbation data is the data with the optimal distribution fitting of the intermediate features.
[0067] In one example, the skilled person can flexibly involve the value of N, such as 3, 4, 5, etc., which will not be described here. In the following, taking 3 as an example, the current task training data or the original task training data is divided into 4 groups according to the major categories, as 4 training tasks, wherein the first task is the base task, and the subsequent three tasks are the incremental tasks. The number of classes and training samples contained in each task is shown in Table 1.
[0068] Table 1
[0069]
[0070] Step 2: Element-level addition is performed on the random perturbation data and the current task training data to obtain a perturbation sample of the current task training data; if the current task is a base task, skip;
[0071] In one example, PASCAL VOC and MS-COCO datasets are used as training data, and four training tasks are divided, including one base task and three incremental tasks, as shown in Table 1. First, the base task is trained, at this time, only the training data of the base task is included, and no perturbation sample is included. After the training is completed, the prototype center and the intermediate feature of the base task category are saved to select the playback example. Then, the incremental tasks are trained in turn, including the training data and the perturbation sample of the current task.
[0072] The generated random perturbation and the training data of the current task are element-level added to obtain a perturbation sample of the current task training data. The perturbation sample contains statistical information of the old category, which alleviates the catastrophic forgetting of the network when learning new tasks. The current task sample and the generated perturbation sample are sent into the classification regression prototype branch network model together. If the current task is a base task, only the training sample of the current task is input;
[0073] Step 3: Feature extraction is performed on the perturbation sample and the original sample of the current task training data to obtain an instance feature map f of uniform size; after global pooling is performed on the instance feature map f of uniform size, intermediate features Z are obtained; specifically including:
[0074] The features of the perturbation sample and the original sample of the current task training data are extracted by using a target detector and a feature extraction network to obtain a first feature map;
[0075] The first feature map is input into a region proposal network to extract a candidate frame to obtain a candidate frame;
[0076] The candidate frame and the first feature map are input into an RoI network to obtain the instance feature map f of uniform size.
[0077] In one example, the perturbation sample of the training data of the current task is input, and the playback example and the perturbation sample of the old task are not included;
[0078] A Faster R-CNN object detector is used, and the first feature map of the input sample is extracted through a ResNet-50 backbone network. The first feature map is then fed into a Region Proposal Network (RPN) to extract candidate boxes. Subsequently, the candidate boxes and the first feature map are compared... Figure 1 The feature maps of instances with uniform size are then fed into the RoI (Region of Interest). The feature map f is processed by global pooling to obtain intermediate features. ;
[0079] Step 4: Calculate the mean of the intermediate feature Z to obtain the new prototype center P. new The new prototype center P new A weighted update of the initial prototype center P with momentum parameter η is performed to obtain the updated prototype center P1; specifically including:
[0080] .
[0081] In one example, the prototype center is fine-tuned. The new prototype center P new The initial prototype center P is calculated by means of various embedded features (intermediate features Z), and it will be obtained through P and P. new The momentum parameter between them is The weighted update yields the updated prototype center P1;
[0082] Step 5: Calculate the intermediate feature f of any task in the original task training data. mid This yields the corresponding prototype center distance; specifically including:
[0083] ;
[0084] Where dist is the corresponding prototype center distance. This represents the i-th initial prototype center. This represents the i-th intermediate feature.
[0085] Step 6: Based on the prototype center distance, obtain the M task data samples that are closest to the updated prototype center P1 in any task; use the M task data samples as replay examples; adjust the classification and regression prototype branch model according to the replay examples of all tasks in the original task training data to obtain the optimized classification and regression prototype branch model.
[0086] The updated classification and regression prototype branch model includes: a classification branch, a regression branch, and a prototype branch;
[0087] The structure of the classification branch and the regression branch is the original Faster R-CNN structure; the prototype branch is composed of a fully connected layer.
[0088] In one example, the 50 samples closest to the prototype center are saved for each class as a replay example. The network is fine-tuned with the replay examples of all old tasks. If it is a base task, skip.
[0089] After completing the training of the current task, the optimized classification and regression prototype branch model can efficiently detect the current and previous task classes, and correctly mark unknown classes that have not been seen. Perform the above steps for all tasks until the last task training is completed.
[0090] Step 7: input the intermediate feature Z into the optimized classification and regression prototype branch model to obtain the known task classes and unknown task classes of the current task training data and the original task training data.
[0091] In one example, the classification branch and the regression branch in the optimized classification and regression prototype branch model use the structure of the original Faster R-CNN, and after obtaining logits through a fully connected layer, the softmax normalization exponential function is used to convert it into the corresponding probability value and boundary box offset value of each class. The prototype branch is composed of a fully connected layer, and the obtained Fpt(z) is used to calculate the prototype loss ;
[0092] Based on prototype learning, a prototype branch is introduced on the Faster R-CNN head under a closed world, as shown in Figure 3 The branch is essentially a fully connected layer, and the purpose is to improve the discriminability of known classes. The set of prototype centers of each class is defined as follows:
[0093] ;
[0094] where d represents the dimension of each prototype center, and N represents the total number of classes. Each prototype center is initialized as a one-hot encoding, and is manually fine-tuned through an exponential moving average during the training process. The embedding features of the candidate boxes proposed by Faster R-CNN are aggregated to improve the tightness of known classes in the embedding space, thereby better detecting unknown classes. The distance between the embedding features and the prototype center is defined as follows:
[0095] ;
[0096] The calculation process of the probability that the model predicts the label belongs to class k is as follows:
[0097] ;
[0098] where F denotes the function of R-CNN head, in order to make the embedding features closer to the prototype centers of the corresponding classes during training, the embedding feature aggregation is achieved by distance-based cross-entropy loss, the calculation process is as follows:
[0099] ;
[0100] Then the distribution of known classes in the feature space is compressed to reduce the open space risk. A constraint term open space risk loss with variable interval R is proposed , the calculation process is as follows:
[0101] ;
[0102] where R is a learnable parameter and is initialized to 0. Under the constraint of further assist enhance the embedding ability of the network.
[0103] Finally, the two losses are combined and weighted as the optimization target of the prototype branch:
[0104] ;
[0105] By using the embedding feature aggregation and embedding feature space compression of the prototype branch, the discriminability of the model for known classes is improved, thereby reducing the uncertainty of unknown class targets.
[0106] where, denotes the Euclidean distance, denotes the dot product, denotes the prototype center of the k-th class, denotes height, width, and channel number, denotes the intermediate feature of the candidate frame feature after global average pooling, denotes the prototype branch function.
[0107] Use old knowledge to generate adversarial noise, and synthesize adversarial samples with training samples of new tasks to promote model training. Specifically, when training the current task, input the data into the R-CNN head, and get the intermediate feature after global average pooling. During training, a fixed number of n intermediate features are stored for all classes of the current task. These intermediate features contain a lot of important statistical data of old classes.
[0108] Then use these old class data to fit the given 10 common distributions, and calculate the sum of squares of errors of the corresponding distributions. The smaller the sum of squares of errors, the better the fitting effect of the corresponding distribution. Use the best fitting distribution to generate random disturbance as noise , such asFigure 4 The training data of the new task is added to the old task training data element by element:
[0109]
[0110] Wherein, is the training data of task t.
[0111] The generated perturbation sample contains rich statistical information of the old class, which can provide sufficient old class knowledge to the network to avoid significant decline in detection performance of the old task class when learning the new task.
[0112] The obtained perturbation sample and the training sample of the new task are jointly trained to obtain the prototype center of each current class. After training, the 50 samples closest to each prototype center are saved as playback examples. After the training phase is completed, the model is fine-tuned using all old task (including the current task) playback examples to improve the stability of the model to the old task class.
[0113] In testing, not only known classes are identified, but also unknown classes are correctly identified. For test data, after being input into the optimized classification regression prototype branch model, a plurality of prediction box information can be obtained, each prediction box containing a classification layer score, a position and a prototype branch score. The detection result with a very low classification score will be directly filtered by a set threshold, and these boxes are regarded as background. And the prediction result with a higher classification score but far away from all prototype centers is regarded as an unknown class. The judgment condition is as follows:
[0114]
[0115] Wherein, represents the distance-based probability of the embedding feature of the prediction box on the prototype branch and the jth prototype center, is a threshold value for measuring whether the prediction box is an unknown class. Thus, the detection of unknown classes is realized.
[0116] The application discloses a perturbation enhancement method for open world target detection, which has the following outstanding features and advantages compared with existing technologies:
[0117] (1) The embodiment of the application innovatively approaches the open world idea of the real world, and proposes embedding feature aggregation and embedding feature space compression based on prototype learning to reasonably allocate feature space for known classes and unknown classes, thereby improving the discriminability of known classes and reducing the uncertainty of unknown class targets.
[0118] (2) A perturbation sample generation method based on old sample features is proposed. Perturbation data carrying old knowledge is generated by using the distribution information of old sample features, which is more meaningful for incremental learning in open worlds than general uniform distribution perturbation. The generated perturbation samples and new task samples are used together to train the network, further enhancing the network's detection capability.
[0119] In summary, in this embodiment of the invention, the perturbation enhancement method for open-world object detection is based on prototype learning and random perturbation data. It reduces the distribution range of known task categories by using a uniformly sized instance feature map f, thereby facilitating the differentiation of unknown task categories during the detection phase. This improves the discriminative power of known task categories and enhances the detection capability for unknown task categories. This method embodies the open-world concept, achieving high-precision detection of unknown task categories and the ability to incrementally learn unknown task categories.
[0120] Meanwhile, random perturbation data is obtained by fitting a given distribution to the original task training data, which promotes the incremental learning ability of the classification and regression prototype branch model, while avoiding the class bias phenomenon in the network. It can alleviate the forgetting of the original task training data while learning new task categories, and has good performance in detecting both known and unknown task categories.
[0121] Example 1:
[0122] Acquire training data; training data includes: current task training data and original task training data; divide the current task training data into 4 groups according to major categories, as 4 training tasks, as shown in Table 1, where task 1 is the base task and subsequent tasks 2-4 are incremental tasks.
[0123] The original task training data is divided into 4 groups according to major categories, which are used as 4 training tasks, such as tasks S1-S4. Task S1 is used as the base task, and subsequent tasks S2-S4 are used as incremental tasks.
[0124] Based on the data in task S1 from the original task training data, fit a given distribution to obtain the random perturbation data of task S1.
[0125] Add the random perturbation data of Task S1 and the data of Task 1 in the current task training data element-wise to obtain the perturbation sample of Task 1 in the current task training data.
[0126] Based on the perturbed samples of Task 1 and the original samples of Task 1 in the current task training data, feature extraction is performed to obtain a uniformly sized instance feature map f; after global pooling of the uniformly sized instance feature map f, the intermediate feature Z is obtained.
[0127] Calculate the mean of the intermediate feature Z to obtain the new prototype center P. new; the new prototype center P new and the initial prototype center P, to obtain an updated prototype center P1;
[0128] calculate the intermediate feature f mid , to obtain the prototype center distance of the corresponding task S1 random disturbance data.
[0129] According to the task S1 random disturbance data prototype center distance, the M task S1 data samples closest to the updated prototype center P1 in the task S1 are obtained; the M task S1 data samples are added to the current task training data task1 as a playback example, to obtain a playback example task1; the classification regression prototype branch model is adjusted according to the playback example task1, to obtain an optimized classification regression prototype branch model.
[0130] At this point, the classification regression prototype branch model for task1 data is optimized, and then the above process is repeated to optimize the classification regression prototype branch models corresponding to tasks 2-4 in turn. After traversing the four tasks, the final optimized classification regression prototype branch model is obtained.
[0131] Finally, the intermediate feature Z is input into the optimized classification regression prototype branch model, to obtain the known task categories and unknown task categories of the current task training data and the original task training data.
[0132] To achieve the above object, the embodiment of the present application further provides the following scheme:
[0133] Please refer to Figure 2 A perturbation enhancement system for open world object detection, comprising:
[0134] The data acquisition module 1 is used for acquiring training data; the training data comprises: current task training data and original task training data;
[0135] The PASCAL VOC and MS-COCO data sets are used as training data.
[0136] The fitting given distribution module 2 is connected with the data acquisition module 1, and is used for fitting a given distribution according to the original task training data, to obtain random disturbance data.
[0137] The element-level addition module 3 is connected with the data acquisition module 1 and the fitting given distribution module 2 respectively, and is used for performing element-level addition on the random disturbance data and the current task training data, to obtain perturbation samples of the current task training data.
[0138] The feature extraction module 4 is connected with the element-level addition module 3, and the feature extraction module 4 is used for performing feature extraction according to the perturbation sample and the original sample of the current task training data, so as to obtain an instance feature map f of uniform size; after global pooling is performed on the instance feature map f of uniform size, an intermediate feature Z is obtained;
[0139] The weighted updating module 5 is connected with the feature extraction module 4, and the weighted updating module 5 is used for calculating the mean value of the intermediate feature Z, so as to obtain a new prototype center P new ; the new prototype center P new and the initial prototype center P are subjected to weighted updating with a momentum parameter η, so as to obtain an updated prototype center P1;
[0140] The prototype center distance calculation module 6 is connected with the data acquisition module 1, and the prototype center distance calculation module 6 is used for calculating the intermediate feature f mid of any one task in the original task training data, so as to obtain a corresponding prototype center distance;
[0141] The classification regression prototype branch model construction module 7 is connected with the prototype center distance calculation module 6 and the data acquisition module 1 respectively, and the classification regression prototype branch model construction module 7 is used for obtaining M task data samples closest to the updated prototype center P1 in any one task according to the prototype center distance; the M task data samples are taken as playback examples; the classification regression prototype branch model is adjusted according to the playback examples of all tasks in the original task training data, so as to obtain an optimized classification regression prototype branch model;
[0142] The output module 8 is connected with the feature extraction module 4, the element-level addition module 3 and the fitting given distribution module 2 respectively, and the output module 8 is used for inputting the intermediate feature Z into the optimized classification regression prototype branch model, so as to obtain known task categories and unknown task categories of the current task training data and the original task training data.
[0143] Further, the application further provides an electronic device, which can comprise a processor, a communication interface, a memory and a communication bus. Wherein the processor, the communication interface and the memory complete mutual communication through the communication bus. The processor can call a computer program in the memory, so as to execute the computer program, and the processor realizes the perturbation enhancement method of open world target detection when executing the computer program.
[0144] In addition, the computer program in the memory described above is realized in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.
[0145] Further, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the open world target detection perturbation enhancement method.
[0146] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.
[0147] The principles and implementation manners of the embodiments of the present application are described by applying specific examples in this paper. The above embodiment description is only used to help understand the method and core idea of the embodiments of the present application; at the same time, for those skilled in the art, according to the idea of the embodiments of the present application, the specific implementation manner and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the embodiments of the present application.
Claims
1. A perturbation augmentation method for open-world object detection, the method comprising: The method comprises the following steps: acquiring training data; the training data comprises current task training data and original task training data; the training data is image data; fitting a given distribution according to the original task training data to obtain random perturbation data; performing element-level addition on the random perturbation data and the current task training data to obtain perturbation samples of the current task training data; performing feature extraction on the perturbation samples and original samples of the current task training data to obtain instance feature maps f of uniform size; after global pooling is performed on the instance feature maps f of uniform size, intermediate features Z are obtained; calculating the mean value of the intermediate feature Z to obtain a new prototype center P new ; performing a weighted update with a momentum parameter η on the new prototype center P new and the initial prototype center P to obtain an updated prototype center P1; calculating the intermediate feature f of any one task in the original task training data mid obtaining the corresponding prototype center distance; obtaining M task data samples closest to the updated prototype center P1 in the arbitrary task according to the prototype center distance; taking the M task data samples as replay examples; adjusting a classification regression prototype branch model according to the replay examples of all tasks in the original task training data to obtain an optimized classification regression prototype branch model; inputting the intermediate features Z into the optimized classification regression prototype branch model to obtain known task categories and unknown task categories of the current task training data and the original task training data.
2. The perturbation augmentation method for open-world object detection of claim 1, wherein, the current task training data comprises one base task and N incremental tasks; the original task training data comprises one base task and N incremental tasks; wherein N is a positive integer.
3. The perturbation augmentation method for open-world object detection of claim 2, wherein, fitting a given distribution according to the intermediate features of the N incremental tasks in the original task training data to obtain random perturbation data; the random perturbation data is data with the optimal fitting distribution of intermediate features. performing feature extraction on the perturbation samples and original samples of the current task training data to obtain instance feature maps f of uniform size, specifically comprising:
4. The perturbation augmentation method for open-world object detection of claim 1, wherein, extracting features in the perturbation samples and original samples of the current task training data by using a target detector and a feature extraction network to obtain first feature maps; inputting the first feature maps into a region proposal network to extract candidate frames to obtain candidate frames; inputting the candidate frames and the first feature maps into an RoI network to obtain the instance feature maps f of uniform size. the optimized classification regression prototype branch model comprises a classification branch, a regression branch and a prototype branch; 5. The perturbation augmentation method for open-world object detection of claim 1, wherein, the structure of the classification branch and the regression branch is the original Faster R-CNN structure; the prototype branch is composed of a full connection layer. The method comprises the following steps:
6. The perturbation augmentation method for open-world object detection of claim 1, wherein, The new prototype center P new and the initial prototype center P are updated by a momentum parameter η to obtain an updated prototype center P1, specifically including: 。 7. The perturbation augmentation method for open-world object detection of claim 6, wherein, calculating the intermediate feature f of any one task in the original task training data mid obtaining the corresponding prototype center distance, specifically comprising: ； where dist is the corresponding prototype center distance, denotes the i-th initial prototype center, denotes the i-th intermediate feature.
8. A perturbation augmentation system for open world object detection, the system comprising: a data acquisition module is configured to acquire training data; the training data comprises current task training data and original task training data; the training data is image data; a fitting given distribution module is connected with the data acquisition module and is configured to fit a given distribution according to the original task training data to obtain random perturbation data; an element-level addition module is connected with the data acquisition module and the fitting given distribution module respectively and is configured to perform element-level addition on the random perturbation data and the current task training data to obtain perturbation samples of the current task training data; The feature extraction module is connected with the element-level addition module, and is configured to perform feature extraction according to the perturbed sample and the original sample of the current task training data to obtain an instance feature map f of uniform size; and perform global pooling on the instance feature map f of uniform size to obtain intermediate features Z. The weighting updating module is connected with the feature extraction module and is configured to calculate a mean value of the intermediate feature Z to obtain a new prototype center P new The weighting updating module is connected with the feature extraction module and is configured to calculate a mean value of the intermediate feature Z to obtain a new prototype center P new and the initial prototype center P to obtain an updated prototype center P1. A prototype center distance calculation module is connected with the data acquisition module and configured to calculate the intermediate feature f of any one task in the original task training data mid , to obtain the corresponding prototype center distance. The classification regression prototype branch model construction module is connected with the prototype center distance calculation module and the data acquisition module respectively, and is configured to obtain M task data samples closest to the updated prototype center P1 in the arbitrary task according to the prototype center distance; take the M task data samples as a replay example; and adjust a classification regression prototype branch model according to the replay example of all tasks in the original task training data to obtain an optimized classification regression prototype branch model. The output module is connected with the feature extraction module, the element-level addition module and the fitting given distribution module respectively, and is configured to input the intermediate features Z into the optimized classification regression prototype branch model to obtain known task categories and unknown task categories of the current task training data and the original task training data.
9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the perturbation enhancement method for open world object detection according to any one of claims 1-7.
10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the perturbation enhancement method for open world object detection according to any one of claims 1-7.