A target detection method and device in a long-tail data scenario

Abstract

The present invention belongs to the technical field of object detection in deep learning, and specifically relates to an object detection method and device in a long-tail data scenario. The object detection method of the present invention includes: constructing a long-tail object detection data set; maintaining a feature cache module during model training to provide a data source basis for accurate upsampling at the object level; designing a distribution-aware dynamic upsampling strategy by using the positive correlation between the two-norm of the detector classification head weight vector and the sample size under the category; constructing a multi-class collaborative data augmentation module by using the similar semantic relationship between categories; designing a bidirectional adjustment loss function to adjust the gradient influence from both positive and negative directions; overall optimizing and training the model, and conducting effectiveness verification. The present invention can make up for the deficiencies and technical limitations of existing object detection methods in long-tail data scenarios when dealing with tail categories, and effectively improve the object detection accuracy, especially the detection accuracy of extremely challenging tail category objects.

Description

Technical Field

[0001] The present invention belongs to the technical field of object detection under deep learning, and specifically relates to an object detection method and device in a long-tail data scenario. Background Art

[0002] Benefiting from the rapid development of deep learning and the collection and construction of large-scale and balanced data sets, traditional object detection tasks have made great progress in recent years. However, in real application scenarios, the sample sizes available for different categories are extremely unbalanced, showing a long-tail distribution, that is, a small number of head categories contain a large number of samples, while a large number of tail categories only have a small number of samples. Therefore, an object detection method that can effectively handle object detection tasks in a long-tail data scenario will be beneficial to improving the accuracy of object detection tasks in real application scenarios. Currently, some work has focused on dealing with object detection tasks in a long-tail data scenario, which can be mainly divided into two categories, namely resampling-based methods and loss function reweighting-based methods. Resampling-based methods can be divided into methods of downsampling head categories and methods of upsampling tail categories. Their sampling frequencies are static and calculated based on prior knowledge of training data, which is extremely likely to cause performance damage to head categories or overfitting to tail categories. Loss function reweighting-based methods scale the gradients generated by the loss function through reweighting parameters. Existing methods mostly focus on weakening the negative gradient influence of head category samples on tail categories, while ignoring the optimization influence of gradients generated by tail category samples. At the same time, the existing two types of methods focus on solving the head-tail imbalance problem, treating the tail category set as a whole, while ignoring the differences in sample sizes and data properties among various categories within the tail category set, lacking refined and differentiated processing of each tail category, resulting in obvious performance and technical limitations. Therefore, there is an urgent need to design a technical solution that can overcome the defects of existing object detection methods in a long-tail data scenario to achieve accurate object detection in a long-tail data scenario. Summary of the Invention

[0003] The purpose of the present invention is to provide an object detection method and device in a long-tail data scenario, which can effectively improve the detection accuracy of a large number of tail category objects to improve the accuracy of object detection tasks in real application scenarios.

[0004] The object detection method in a long-tail data scenario provided by the present invention uses a variety of new technical means, including: feature caching, providing a data source basis for accurate object-level upsampling; designing an upsampling strategy based on the distribution perception of the two-norm of the weight vector of the detector classification head; using a multi-class collaborative data augmentation method with inter-class related semantic information for feature fusion; and bidirectional adjustment of the loss. Specifically as follows:

[0005] (1) Feature cache, which provides a data source basis for accurate target upsampling; specifically, it is maintained and updated during the training process of the model. Each category has a cache queue in the cache, which is used to cache the feature vectors of the targets corresponding to the category. The update of the cache queue follows the first-in, first-out principle;

[0006] (2) Perform upsampling; distribution-aware upsampling based on the second norm of the detector classification head weight vector; design the upsampling frequency according to the positive correlation between the second norm of the detector classification head weight vector and the sample size under each category, and perform dynamic upsampling on the tail categories with less sample size;

[0007] (3) Feature fusion; a multi-class collaborative data augmentation method that utilizes the inter-class relevant semantic information. Utilize the semantic similarity between categories to perform feature fusion on the samples under the most similar categories and the source samples to obtain the enhanced source samples for subsequent training processes;

[0008] (4) Bidirectional adjustment of loss; calculate the positive gradient promotion parameter and the negative gradient suppression parameter for each category using the classification head prediction probability distribution of the samples under each category, and adjust the gradient influence in both positive and negative directions to optimize the training of the detector.

[0009] Furthermore:

[0010] The feature cache described in (1) specifically includes:

[0011] Let the feature cache be composed of a cache queue set , where C is the total number of categories during the training process, which is used to cache the labeled target box feature-labeled target box pairs of C categories respectively during the training process; The length of each cache queue in is V, that is, is the labeled target box feature,

[0012] is the labeled target box, and the update of the queue follows the first-in, first-out principle.

[0013] Since the second norm of the detector classification head weight vector is positively correlated with the sample quantity under the category, that is, the second norm of the classification head weight vector can reflect the change of the sample data quantity under each category in real time during the training process; perform distribution-aware upsampling according to this situation; for category c k , its upsampling probability in one training iteration is:

[0014]

[0015] where ||w k || is the two-norm of the classification weight vector of category c in the classification header, and γ is a scaling hyperparameter. k

[0016] The multi-class collaborative data augmentation method using inter-class related semantic information described in (3) specifically includes three sub-processes: two-class collaborative attention, multi-class selection fusion, and source category information enhancement; input a source category feature and a set of related features First, through two-class collaborative attention, for each source feature-related feature pair calculate to obtain a single-class enhanced feature (as shown in formula (2)), and finally obtain a set of single-class enhanced features

[0017] Attn multi is the operation of the multi-head self-attention mechanism, and are collaborative attention features; are the query feature, key-value feature, and value feature derived from respectively; are the query feature, key-value feature, and value feature derived from respectively;

[0018] Then use multi-class selection fusion to selectively fuse to obtain a multi-class enhanced feature

[0019]

[0020] is the calculated weight vector, s = (s1, s2,..., sk);

[0021] Finally, use the source category information enhancement method to fuse the category prototype of the source category with the multi-class enhanced feature to obtain the final enhanced feature

[0022]

[0023] Conv 1×1 is a 1×1 convolution operation, and Concat is a channel concatenation operation.

[0024] The bidirectional adjustment loss described in (4) is specifically:

[0025] The basis for the two-way adjusted loss is the binary cross-entropy loss. The two-way adjusted loss is shown in Equation (5):

[0026]

[0027] Among them, is the positive gradient promotion coefficient for class c i , is the negative gradient suppression coefficient for class c i , is the one-hot label, p = [p1,..., p C T is the classification prediction score vector, and C is the total number of foreground classes.

[0028] The positive gradient promotion coefficient is calculated based on the vector r is updated by exponential moving average during the training iteration. r i reflects the degree to which samples belonging to class c i are likely to be misclassified:

[0029]

[0030] ||w i || is the L2 norm of the classification weight vector for class c i in the classification head; is the degree to which class c i is likely to be misclassified in the t-th training iteration, is the degree to which class c i was likely to be misclassified in the (t - 1)-th training iteration. η is the hyperparameter of the exponential moving average, and N is the total number of samples belonging to class c i in the t-th training iteration; is the degree of misclassification of the j-th sample belonging to class c i in the t-th training iteration; represents the predicted classification probability distribution z i (j) of the j-th sample belonging to class c t in the t-th training iteration, where the probability value of class c i is represents the predicted classification probability distribution z i of the j-th sample belonging to class c t (j) in the t-th training iteration, where the probability value of class c k is represents the total number of probability values in z t (j) that are greater than, or close to, . α and β are adjustment hyperparameters;​

[0031] Negative gradient inhibition coefficient is calculated as shown in formula (7):

[0032]

[0033] ||w i || is the two-norm of the classification weight vector for class c in the classification head; z i is the predicted value of class c in the predicted classification probability distribution z of sample x belonging to class c k ; z k is the predicted value of class c in the predicted classification probability distribution z of sample x belonging to class c k ; σ is the sigmoid operation; f(x) is a sigmoid-like activation function used to i limit to a fixed value range, where γ and μ are hyperparameter coefficients of the function f(x). k ; z i is the predicted value of class c in the predicted classification probability distribution z of sample x belonging to class c, σ is the sigmoid operation; f(x) is a sigmoid-like activation function used to limit to a fixed value range, where γ, μ are hyperparameter coefficients of the function f(x).

[0034] The present invention also provides a device corresponding to the object detection method in the above long-tail data scenario, specifically including:

[0035] A feature cache module, which includes cache queues corresponding to all classes during the training process and provides a data source for object-level data upsampling;

[0036] A distribution-aware upsampling module, which calculates the sampling frequency of each class based on the two-norm of the classification head weight vector and determines the classes and sampling samples to be upsampled in each training iteration;

[0037] A multi-class collaborative data augmentation module, which includes a two-class collaborative attention module, a multi-class selection and fusion module, and a source class information enhancement module, and fuses the inter-class similar semantic information and source class samples to achieve data augmentation;

[0038] A two-way adjustment loss module, which includes a positive gradient promotion coefficient calculation module and a negative gradient inhibition coefficient calculation module, and is used to adjust the gradient influence from both positive and negative directions to optimize the training process.

[0039] The present invention has at least the following beneficial effects:

[0040] The present invention designs an effective object detection method for long-tailed data scenarios, making up for the deficiencies and technical limitations of existing object detection methods in the refined processing of tail categories in long-tailed data scenarios. The present invention designs a feature caching module and a distribution-aware upsampling strategy to accurately and dynamically upsample the tail categories with insufficient sample sizes, effectively alleviating the detection inhibition of head categories on tail categories and preventing overfitting of tail categories. The present invention designs a multi-class collaborative data augmentation method to effectively and credibly augment the data of upsampled samples by utilizing the similar semantics between classes, increasing the data diversity of tail categories. At the same time, the present invention designs a bidirectional adjustment loss to dynamically adjust the gradient influence in both positive and negative directions, making the optimization direction favorable for tail categories. The present invention effectively improves the detection accuracy of tail category targets and the detection accuracy of all category targets in long-tailed data scenarios, and has high practical application value. At the same time, the present invention has good portability and generality and can be quickly migrated from one application scenario to a new application scenario.

[0041] Other advantages, objectives, and features of the present invention will be partially reflected by the following description and partially understood by those skilled in the art through the research and practice of the invention. Brief Description of the Drawings

[0042] Figure 1 is the framework diagram of the present invention.

[0043] Figure 2 is the schematic of feature caching and distribution-aware upsampling.

[0044] Figure 3 shows the multi-class collaborative data augmentation method.

[0045] Figure 4 shows the bidirectional adjustment loss. Detailed Description of the Embodiment

[0046] The following further elaborates on the present invention in conjunction with the drawings, enabling those skilled in the art to implement it with reference to the written description of the specification.

[0047] As Figure 1As shown in the figure, an embodiment of the present invention provides an object detection method in a long-tail data scenario, which includes four important components: a feature cache module that caches annotation box feature-annotation box pairs at the object level to provide a data source for upsampling; a distribution-aware upsampling strategy module that samples a number of object box feature-annotation box pairs from the feature cache module according to the designed upsampling rules for subsequent training processes; a multi-class collaborative data augmentation module that uses the similar semantics between classes to perform data augmentation on the object box features obtained by upsampling to increase the data diversity of the corresponding classes; and a bidirectional adjustment loss module that calculates the positive gradient promotion coefficient and negative gradient suppression coefficient for each class using the classification probability distribution of the samples to adjust the gradient influence and improve the optimization direction.

[0048] As Figure 2 shown, the feature cache is composed of a set of cache queues (C is the total number of classes during training), which is used to cache the annotation target box feature-annotation target box pairs of C classes respectively during the training process. Among them, the annotation target box feature is the feature after the annotation box area passes through RoI Align. The length of each cache queue in is V, that is is the annotation target box feature, is the annotation target box).

[0049] Assume that the image batch B t in the t-th training iteration contains N annotation target boxes. By extracting the features after RoI Align of these N annotation target boxes, N annotation target box feature-annotation target box pairs can be obtained, and then these N annotation target box feature-annotation target box pairs are respectively cached into the cache queues corresponding to the classes. If the cache queue is full, following the first-in-first-out principle, the earliest enqueued annotation target box feature-annotation target box pair is popped to complete the update of the cache queue.

[0050] In one training iteration, the distribution-aware upsampling strategy is based on the formula:

[0051]

[0052] to calculate the upsampling probabilities of C classes, and then select N classes that need to be upsampled according to the calculated upsampling probabilities. Then, M annotation target box feature-annotation target box pairs are randomly selected from the corresponding cache queues of the N classes that need to be upsampled in the feature cache module (that is, a total of M×N annotation target box feature-annotation target box pairs are sampled), and are used together with the training data in the current iteration for subsequent training processes.

[0053] For the target box features obtained by upsampling, a multi-class collaborative enhancement module is used to perform data enhancement on them. The working process of the multi-class collaborative enhancement module is as shown in Figure 3 . Specifically, for a source feature belonging to class first determine the K classes with the most similar semantics to class , and then randomly select one relevant feature from the corresponding cache queues of these K classes in to form a set of relevant features for the subsequent data enhancement process.

[0054] The semantic similarity between classes is measured by the cosine similarity between class prototypes. Given the class prototype set , by calculating the cosine similarity between pairwise class prototypes, a similarity matrix

[0055]

[0056] For class , find the K classes with the highest cosine similarity by searching the similarity matrix , and then select the relevant feature set

[0057] from the cache queue. Input the source class feature and the relevant feature set The multi-class collaborative data enhancement module first calculates two collaborative attention features for each source class feature - relevant feature pair through the cross-attention mechanism and Then add and fuse the two collaborative attention features to obtain a single-class enhanced feature

[0058]

[0059]

[0060]

[0061] Calculate K single-class enhanced features from the K source class feature - relevant feature pairs to form a single-class enhanced feature set Then, perform weighted fusion through the multi-class selection and fusion module to obtain a multi-class enhanced feature

[0062]

[0063] Based on the calculated weight vector:

[0064]

[0065] Among them, GAP is the global average pooling operation, Concat is the concatenation operation, and FC is the fully connected layer.

[0066] To strengthen the consistency between the enhanced samples and the real samples under the source category, the multi-class enhanced features Subsequently, it will pass through the source category information enhancement module, and through the prototype features of the source category Fusion is performed to obtain the final enhanced samples

[0067]

[0068] Perform multi-class collaborative data augmentation on the target box features in the M×N labeled target box feature-labeled target box pairs obtained by upsampling in each training iteration. The M×N enhanced target box feature-labeled target box pairs obtained will be used together with the original data in the current iteration for subsequent RoI feature extraction, classification, and regression processes.

[0069] Existing object detection methods in long-tailed data scenarios based on loss function reweighting simply alleviate the detection inhibition of the head category on the tail category by suppressing the negative gradient influence of the head category samples on the tail category, without considering the importance of the positive gradient influence generated by the tail category samples themselves on the learning and optimization of the tail category. Therefore, the embodiments of the present invention construct a bidirectional adjustment loss based on binary cross-entropy loss, and simultaneously promote the positive gradient and suppress the negative gradient to control the optimization to proceed in a direction beneficial to the tail category:

[0070]

[0071] Such as Figure 4 As shown, the bidirectional adjustment loss constructs positive gradient promotion parameters for each category And negative gradient suppression parameters And The construction of both depends on the classification probability distribution output by the classification head for each sample.

[0072] Since only a small number of samples are included in the tail category, it is prone to underlearning during the training process. During object classification, it is likely to be misclassified as a similar head category. The embodiments of the present invention maintain a vector Used to reflect the degree to which each category is likely to be misclassified (e.g., r i Reflects belonging to category c iThe degree to which samples are easily misclassified). r is updated by exponential moving average during training iterations:

[0073]

[0074]

[0075]

[0076] In the t-th training iteration, for the j-th target x belonging to class c detected in the current image batch B t in the detector classification head predicts the output classification probability distribution i For For Statistics except the background classification score among them and the prediction score of class c i After that, the number of scores close to is counted and then is scaled to obtain the misclassification degree of target x For all targets belonging to class c detected in B For B t in the detector classification head predicts the output classification probability distribution i in the detector classification head predicts the output classification probability distribution Update the degree r to which samples belonging to class c are easily misclassified i using the average value i .

[0077] is calculated based on r i and the second norm ||w i || of the classification head weight vector of class c i ||:

[0078]

[0079] Tail classes with a small number of samples are easily misclassified as other classes, with a large r i and a small ||w i ||, so it has a greater positive gradient promotion strength compared to head classes

[0080] In the long-tail data scenario, a large number of head class samples will have an excessive negative gradient impact on tail classes, suppressing the detection and classification of tail classes. In the embodiments of the present invention, while enhancing the positive gradient promotion of tail classes, a negative gradient suppression coefficient is designed to adjust the negative gradient impact of head classes on tail classes:

[0081]

[0082]

[0083] z k For the predicted classification probability distribution z of the sample x belonging to category c k in, the predicted value of class c k f(x) is a sigmoid-like activation function used to be restricted within a fixed value range. The classes with predicted probability values in z much smaller than z k can be better distinguished from class c k , so the negative gradient inhibition caused by the samples belonging to class c k will be weakened. Consistent with , also uses the L2 norm of the classification weight vector to mitigate the impact brought by the data volume gap.

[0084] This embodiment combines object-level precise data upsampling, data augmentation, and loss reweighting techniques, achieving a relatively comprehensive and refined processing of the tail classes in the long-tailed data scenario object detection task. Moreover, the important parameters in each technique are dynamically learned during the training process, without relying on the prior statistical knowledge of the dataset, and have high generality and portability.

[0085] The embodiment of the present application also provides an object detection device in the long-tailed data scenario, including a feature cache module, a distribution-aware upsampling module, a multi-class collaborative data augmentation module, and a bidirectional adjustment loss module:

[0086] The feature cache module includes cache queues corresponding to all classes during the training process, providing a data source for object-level data upsampling;

[0087] The distribution-aware upsampling module calculates the sampling frequency of each class based on the L2 norm of the classification head weight vector, and determines the classes and sampling samples that need to be upsampled in each training iteration;

[0088] The multi-class collaborative data augmentation module includes a two-class collaborative attention module, a multi-class selection and fusion module, and a source class information reinforcement module, which fuses the inter-class similar semantic information and source class samples to achieve data augmentation;

[0089] The bidirectional adjustment loss module includes a positive gradient promotion coefficient calculation module and a negative gradient inhibition coefficient calculation module, which are used to adjust the gradient influence from both positive and negative directions to optimize the training process.

[0090] The present application also provides a verification test to further prove the technical effect of the present application.

[0091] To verify the effectiveness of our method in the object detection task of the long-tail data scenario, we selected a benchmark dataset LVIS v1.0 for the object detection task of the long-tail data scenario and, based on LVIS v1.0, sampled and designed a more realistic and challenging dataset LVIS-X:

[0092] LVIS v1.0 is a common benchmark dataset for the object detection task of the long-tail data scenario, containing a total of 1203 categories. These categories are divided into 3 category groups according to the number of samples in the training set: G r (number of images under the category < 10), G c (number of images under the category in [11, 100]) and G f (number of images under the category > 100). Among them, G r accounts for 28% of the total number of categories.

[0093] LVIS-X is sampled from LVIS v1.0 using a Pareto distribution with α = 5 and contains the same 1203 categories as LVIS v1.0. The number of samples under each category has been greatly reduced compared to LVIS v1.0. These categories are divided into 3 category groups according to the number of samples in the training set: G r′ (number of labeled objects under the category ≤ 30), G c′ (number of labeled objects under the category in [31, 100]) and G f′ (number of labeled objects under the category > 100). Among them, G r′ accounts for 52% of the total number of categories.

[0094] To verify the superiority of this method, on LVIS v1.0 and LVIS-X, this embodiment is compared with the following existing object detection methods in several long-tailed data scenarios: RFS (extracted from "Agrim Gupta, Piotr Doll′ar, and Ross B. Girshick.: Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5356–5364, 2019."), EQL (extracted from "Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan.: Equalization loss for long-tailed object recognition. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 11659–11668, 2020."), EQL v2 (extracted from "Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li.: Equalization loss v2: A new gradient balance approach for long-tailed object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 1685–1694, 2021."), RIO (extracted from "Nadine Chang, Zhiding Yu, Yu-Xiong Wang, Animashree Anandkumar, Sanja Fidler, and Jose M. Alvarez: Image-level or object-level? a tale of two resampling strategies for long-tailed detection.InProceedings of the International Conference on Machine Learning,pages 1463–1472,2021.”),Seesaw Loss(摘自“Jiaqi Wang,Wenwei Zhang,Yuhang Zang,Yuhang Cao,Jiangmiao Pang,Tao Gong,Kai Chen,Ziwei Liu,Chen Change Loy,and Dahua Lin:Seesaw loss for long-tailed instance segmentation.In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition,pages 9695–9704,2021.”),C2AM(摘自“Tong Wang,Yousong Zhu,Yingying Chen,Chaoyang Zhao,Bin Yu,Jinqiao Wang,and Ming Tang:C2am loss:Chasing a better decision boundary forlong-tail object detection.In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition,pages 6970–6979,2022.”),AHRL(摘自“Banghuai Li:Adaptive hierarchical representation learning for long-tailedobject detection.In Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition,pages 2313–2322,2022.”), cRT (from “Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis: Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217, 2019.”), BAGS (from “Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng: Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 10988–10997, 2020.”), GOL (from “Kostas Alexandridis, Jiankang Deng, Anh Nguyen, and Shan Luo: Long-tailed instance segmentation using gumbel optimized loss. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.”). This embodiment uses the Mean Average Precision (mAP) on all classes and the Average Precision (AP) on each class group as evaluation metrics to measure the performance of each algorithm.

[0095] Tables 1 and 2 respectively give the result comparison on LVIS v1.0 and LVIS-X. As shown in Table 1, when the validation dataset is LVIS v1.0, among all the compared schemes, in this embodiment, when the backbone network is ResNet50-FPN, in the mAP of all classes and the AP of each class group (AP r , AP c , AP f are respectively in G r , G cand G f for the AP) achieved the best metrics. When the backbone network is ResNet101-FPN, the best results were obtained in terms of AP r , AP c , and sub-optimal results were achieved in terms of mAP and AP f . As shown in Table 2, when the validation dataset is LVIS-X, compared with other comparison schemes, in this embodiment, in the two scenarios where the backbone network is ResNet50-FPN and ResNet101-FPN, in terms of the overall mAP and the AP of each category group (AP r′ , AP c′ , AP f′ are the AP in G r′ , G c′ , and G f′ respectively) all achieved the best metrics. Especially under AP r′ , in this embodiment, when the backbone network is ResNet50-FPN, it is 3% higher than the sub-optimal result; when the backbone network is ResNet101-FPN, it is 6.6% higher than the sub-optimal result. The validation results on LVIS v1.0 and the more challenging LVIS-X comprehensively show that this embodiment has higher superiority compared with other existing methods, especially in dealing with the detection of a large number of tail category targets, which is of great significance for the target detection service in the long-tail data scenario in real life.

[0096] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the field, additional modifications can be easily made. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to the specific details and the illustrated and described examples here.

[0097] Table 1: Result comparison on LVIS v1.0

[0098]

[0099]

[0100] Table 2: Result comparison on LVIS-X

[0101]

Claims

1. A target detection method in a long-tailed data scenario, characterized in that, Use a variety of new technical means, including: feature caching, which provides a data source basis for accurate target-level upsampling; designing an upsampling strategy based on the distribution perception of the two-norm of the detector classification head weight vector; using a multi-class collaborative data augmentation method with inter-class related semantic information for feature fusion; bidirectional adjustment loss; specifically as follows: (1) Feature caching; it is maintained and updated during the training process of the model. Each category has a cache queue in the cache, which is used to cache the feature vectors of the corresponding category targets. The update of the cache queue follows the first-in-first-out principle; (2) Perform upsampling; distribution-aware upsampling based on the two-norm of the detector classification head weight vector; design the upsampling frequency according to the positive correlation between the two-norm of the detector classification head weight vector and the number of image samples under each category, and perform dynamic upsampling on the tail categories with fewer samples; (3) Feature fusion; use a multi-class collaborative data augmentation method with inter-class related semantic information, utilize the semantic similarity between categories, fuse the image samples and source samples under the most similar categories to obtain the enhanced source samples for subsequent training processes; (4) Bidirectional adjustment of loss; calculating the positive gradient promotion parameters for each category using the predicted probability distribution of the classification head of the image samples under each category and the negative gradient suppression parameters Adjust the gradient influence in both positive and negative directions to optimize the training of the detector; In step (4), specifically: The basis of the bidirectional adjustment loss is the binary cross-entropy loss. The bidirectional adjustment loss is shown in formula (5): Among them, is the positive gradient promotion coefficient for class c i , is the negative gradient inhibition coefficient for class c i , is the one - hot label, p = [p1,..., p C T is the classification prediction score vector, and C is the total number of foreground classes;​ Positive gradient promotion coefficient is calculated based on vectors maintained during the training process r is updated by exponential moving average during training iterations, r i where the reaction belongs to class c i the degree to which samples are likely to be misclassified: ||w i ||is the two-norm of the classification weight vector for class c in the classification header i ; is the degree to which class c is easily misclassified in the t-th training iteration i , is the degree to which class c is easily misclassified in the (t - 1)-th training iteration, η is the hyperparameter of the exponential moving average, and N is the total number of samples belonging to class c in the t-th training iteration i ; i is the degree of misclassification of the j-th sample belonging to class c in the t-th training iteration ; i represents the predicted classification probability distribution z (j) of the j-th sample belonging to class c in the t-th training iteration i where the probability value of class c t ; i represents the predicted classification probability distribution z (j) of the j-th sample belonging to class c in the t-th training iteration i where the probability value of class c t ; k represents the total number of probability values in z (j) that are greater than or equal to t , and α and β are adjustment hyperparameters ; Negative gradient inhibition coefficient is calculated as shown in formula (7): ||w i ||is the two-norm of the classification weight vector for class c in the classification header i ; z k is the predicted classification probability distribution z of the sample x belonging to class c k , and the predicted value of class c in z k is z i is the predicted classification probability distribution z of the sample x belonging to class c k , and the predicted value of class c in z i , σ is the sigmoid operation; f(x) is a sigmoid-like activation function used to confine to a fixed value range, where γ and μ are hyperparameter coefficients of the function f(x).

2. The target detection method in a long-tailed data scenario according to claim 1, characterized in that, The feature caching described in (1), specifically includes: Feature cache Consisted of a set of cache queues where C is the total number of categories during the training process, used to cache the labeled target box feature-labeled target box pairs of C categories respectively during the training process; The length of each cache queue in is V, that is is the labeled target box feature, is the labeled target box, and the update of the queue follows the first-in-first-out principle.

3. The target detection method in a long-tailed data scenario according to claim 2, characterized in that, The distribution-aware upsampling based on the two-norm of the detector classification head weight vector described in (2), specifically: Since the L2 norm of the detector classification head weight vector is positively correlated with the number of image samples under the category, that is, the L2 norm of the classification head weight vector can reflect the change of the image sample data volume under each category in real time during the training process; perform distribution-aware upsampling based on this situation; for category c k , its upsampling probability in one training iteration is as follows: where ||w k || is the two-norm of the classification weight vector for class c k in the classification head, and γ is a scaling hyperparameter.

4. The target detection method in a long-tailed data scenario according to claim 3, characterized in that, The multi-class collaborative data augmentation method using inter-class related semantic information described in (3) specifically includes three sub-processes: two-class collaborative attention, multi-class selection fusion, and source category information enhancement; input a source category feature F si and a set of related features First, through two-class collaborative attention for each source feature-related feature pair calculate to obtain a single-class enhanced feature As shown in formula (2), finally obtain a set of single-class enhanced features Attn multi For the operation of the multi-head self-attention mechanism, and are co-attention features; k si , v si are respectively the query feature, key-value feature, and value feature derived from ; are respectively the query feature, key-value feature, and value feature derived from ; Then use multi-class selection fusion to selectively fuse to obtain multi-class enhanced features is the calculated weight vector, s = (s1, s2,..., sk); Finally, use the source category information enhancement method to enhance the category prototype of the source category and the multi-class enhanced features to perform fusion, and obtain the final enhanced features Conv 1×1 is a 1×1 convolution operation, and Concat is a channel concatenation operation.

5. A target detection device in a long-tailed data scenario based on the target detection method according to any one of claims 1 to 4, characterized in that, Specifically includes: A feature caching module, which includes cache queues corresponding to all categories during the training process, and provides a data source for data upsampling at the target level; A distribution-aware upsampling module, which calculates the sampling frequency of each category based on the two-norm of the classification head weight vector, and determines the categories and sampling samples that need to be upsampled in each training iteration; A multi-class collaborative data augmentation module, which includes a two-class collaborative attention module, a multi-class selection fusion module, and a source category information enhancement module, and fuses the inter-class similar semantic information and source category samples to achieve data augmentation; A bidirectional adjustment loss module, which includes a positive gradient promotion coefficient calculation module and a negative gradient suppression coefficient calculation module, and is used to adjust the gradient influence from both positive and negative directions to optimize the training process.

Images