Unmanned aerial vehicle visual angle target detection method and system based on self-regularization point supervision

Abstract

The invention discloses an unmanned aerial vehicle visual angle target detection method and system based on self-regularization point supervision, and the method comprises the steps: obtaining a to-be-detected target image through an unmanned aerial vehicle, generating a feature vector of a candidate region through a region proposal mechanism, and enabling the to-be-detected target image to be associated with a target level point label in the image; the method comprises the following steps: constructing a PointSR target detection network model by adopting a self-regularization sampling mechanism based on a target detection network of weak supervised learning; and based on a Point SR target detection network model, carrying out point supervision target detection on the feature vector of the candidate region to obtain a point supervision target detection result under the view angle of the unmanned aerial vehicle. According to the method, a double-flow structure is adopted to classify the candidate regions, so that effective weak supervision target detection is realized. The unmanned aerial vehicle visual angle target detection method and system based on self-regularization point supervision can be widely applied to the technical field of image supervision target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image weakly supervised object detection, and particularly to a method and system for object detection from an unmanned aerial vehicle (UAV) perspective based on self-regularized point supervision. Background Art

[0002] There are several key problems in point-supervised object detection in related technologies, which limit the performance of dense small object detection from a UAV perspective. First, the static anchor box generation strategy uses predefined fixed aspect ratios, making it difficult to adapt to the diversity of object sizes and distributions in UAV images, resulting in a mismatch between the candidate boxes and the shapes of real objects. This problem is particularly significant in the early stage of model training, where the model fluctuates greatly due to insufficient anchor box coverage and has a slow convergence speed. Second, related methods generate negative samples by randomly sampling background regions, but a large number of samples are far from the target contour boundaries, contributing limitedly to model optimization. This inefficient sampling not only wastes computing resources but also reduces the model's ability to distinguish the edges of dense objects. In addition, related methods rely on fixed candidate boxes in the pseudo-box optimization stage and do not fully utilize high-information samples (such as negative samples with high uncertainty) for dynamic adjustment, resulting in inaccurate pseudo-box boundaries and prone to overlap or background miscoverage, especially in dense scenes. Finally, the lack of temporal information makes the anchor box distribution unable to evolve dynamically during the training process, further exacerbating the instability of pseudo-box generation. Summary of the Invention

[0003] To solve the above technical problems, the purpose of the present invention is to provide a method and system for object detection from a UAV perspective based on self-regularized point supervision, which uses a two-stream structure to classify candidate regions, thereby achieving effective point-supervised object detection in a UAV scenario.

[0004] The first technical solution adopted by the present invention is: a method for object detection from a UAV perspective based on self-regularized point supervision, including the following steps:

[0005] Obtain a target image to be detected by a UAV and generate a feature vector of candidate regions through a region proposal mechanism, where the target image to be detected is associated with object-level point labels in the image;

[0006] Based on a weakly supervised learning object detection network, use a self-regularized sampling mechanism to construct a PointSR object detection network model;

[0007] Based on the PointSR object detection network model, perform point-supervised object detection on the feature vector of the candidate region to obtain a point-supervised object detection result from a UAV perspective.

[0008] Further, the expression of the objective function for point-supervised object detection of the PointSR object detection network model is specifically as follows:

[0009]

[0010] In the above formula, X + represents the positive sample, X - represents the negative sample, min(·) represents the minimum operation, arg represents the average value calculation, θ represents the parameter of the model, and x + represents the region feature of the positive sample, and x - represents the region feature of the negative sample, M represents the number of targets in the image, C represents the total number of categories, i represents the i-th target, k represents the k-th category, L(·) represents the loss function, and f θ (·) represents the classifier based on MIL, represents the region feature of the i-th target on the k-th category, represents the true label of the i-th target on the k-th category, N represents the number of negative samples, j represents the j-th target, represents the region feature of the j-th negative sample, represents the true label of the j-th negative sample.

[0011] Furthermore, the object detection network of the weakly supervised learning adopts a two-stream structure, which is composed of a classification stream and a detection stream. The PointSR object detection network model includes a temporal integration encoder, a coarse-grained pseudo-box prediction module, and a pseudo-box optimization module. The temporal integration encoder, the coarse-grained pseudo-box prediction module, and the pseudo-box optimization module are connected in sequence. The coarse-grained pseudo-box prediction module includes an IS collector.

[0012] Furthermore, the step of performing point-supervised object detection on the feature vector of the candidate region based on the PointSR object detection network model to obtain the point-supervised object detection result from the drone perspective specifically includes:

[0013] Input the feature vector of the candidate region into the PointSR object detection network model;

[0014] Based on the temporal integration encoder of the PointSR object detection network model, perform dynamic anchor adjustment processing on the feature vector of the candidate region to construct an anchor prototype library;

[0015] Based on the coarse-grained pseudo-box prediction module of the PointSR object detection network model, perform coarse-grained pseudo-box prediction and negative sample information collection on the anchor prototype library to obtain coarse-grained pseudo-boxes and negative sample information with high loss values;

[0016] Based on the pseudo-box optimization module of the PointSR object detection network model, refine the coarse-grained pseudo-boxes according to the negative sample information with high loss values, and output the point-supervised object detection result from the drone perspective.

[0017] Furthermore, for the step of the temporal integration encoder based on the PointSR object detection network model to perform dynamic anchor adjustment processing on the feature vectors of the candidate regions and construct an anchor prototype library, it specifically includes:

[0018] Input the feature vectors of the candidate regions into the temporal integration encoder of the PointSR object detection network model to create feature candidate boxes;

[0019] Obtain the ratio information of the anchor width and height of the feature candidate boxes and perform temporal aggregation encoding processing to obtain the pseudo-box shapes for each category;

[0020] Store the pseudo-box shapes for each category as a category-aware prototype library to construct an anchor prototype library.

[0021] Furthermore, for the step of the coarse-grained pseudo-box prediction module based on the PointSR object detection network model to perform coarse-grained pseudo-box prediction and negative sample information collection on the anchor prototype library to obtain coarse-grained pseudo-boxes and negative sample information with high loss values, it specifically includes:

[0022] Input the anchor prototype library into the coarse-grained pseudo-box prediction module of the PointSR object detection network model;

[0023] Through a MIL classifier with a two-stream structure, perform coarse-grained pseudo-box prediction of the category scores at the image level on the anchor prototype library to construct coarse-grained pseudo-boxes;

[0024] Based on the IS collector, remove duplicates from the negative samples in the anchor prototype library through the non-maximum suppression algorithm to obtain the de-duplicated negative samples;

[0025] Obtain the loss values of the de-duplicated negative samples through the negative sample loss function, and sort the de-duplicated negative samples in descending order according to the loss values to obtain negative sample information with high loss values.

[0026] Furthermore, for the step of performing coarse-grained pseudo-box prediction of the category scores at the image level on the anchor prototype library through a MIL classifier with a two-stream structure to construct coarse-grained pseudo-boxes, it specifically includes:

[0027] Through a MIL classifier with a two-stream structure, perform matrix calculations on the anchor prototype library to obtain the category matrix and position prediction score matrix for each region;

[0028] Based on the softmax operator, perform element-wise multiplication on the category matrix and position prediction score matrix for each region to calculate the scores of the regions, and sum them up to obtain the category prediction scores at the image level;

[0029] Based on the class prediction scores at the image level, the top-k boxes with the highest scores in the same set are selected as the coarse-grained pseudo-boxes.

[0030] Furthermore, the expression of the loss function of the MIL classifier with a two-stream structure is specifically as follows:

[0031]

[0032] In the above formula, represents the loss function of the MIL classifier with a two-stream structure, y j,k represents the true label of the j-th target in the k-th class, represents the prediction score generated by the MIL-based predictor, M represents the number of positive samples, C represents the total number of classes, j represents the j-th target, k represents the k-th class, represents the prediction probability of the j-th positive sample in the k-th class.

[0033] Furthermore, the expression of the negative sample loss function is specifically as follows:

[0034]

[0035] In the above formula, represents the negative sample loss function, β and γ represent weight coefficients, represents the prediction score generated by the MIL-based predictor, N represents the number of negative samples, C represents the total number of classes, j represents the j-th target, k represents the k-th class.

[0036] The second technical solution adopted by the present invention is: a drone-view object detection system based on self-regularized point supervision, including:

[0037] The first module is used to obtain the target image to be detected by the drone and generate the feature vectors of the candidate regions through the region proposal mechanism, and the target image to be detected is associated with the target-level point labels in the image;

[0038] The second module is used to construct a PointSR object detection network model based on the object detection network of weak supervision learning and adopt a self-regularized sampling mechanism;

[0039] The third module is used to perform point supervision object detection on the feature vectors of the candidate regions based on the PointSR object detection network model to obtain the point supervision object detection results from the drone view.

[0040] The beneficial effects of the method and system of the present invention are as follows: By obtaining the feature vectors of candidate regions, and further based on the object detection network of weakly supervised learning, adopting a self-regularized sampling mechanism, generating high-quality pseudo-boxes through a self-regularized sampling strategy, adaptively adjusting the sampling space during model training, and finally based on the PointSR object detection network model, performing point-supervised object detection on the feature vectors of candidate regions, and classifying the candidate regions using a two-stream structure, thereby realizing effective weakly supervised object detection. Brief Description of the Drawings

[0041] Figure 1 is the flowchart of the steps of the method for drone perspective object detection based on self-regularized point supervision of the present invention;

[0042] Figure 2 is the structural block diagram of the system for drone perspective object detection based on self-regularized point supervision of the present invention;

[0043] Figure 3 is the structural schematic diagram of the PointSR object detection network model provided by a specific embodiment of the present invention;

[0044] Figure 4 is the schematic diagram of the comparison of the negative sample distributions of PointSR and Point-to-Box provided by a specific embodiment of the present invention;

[0045] Figure 5 is the schematic diagram of the visualization of the pseudo-boxes predicted by PointSR and SOTA methods provided by a specific embodiment of the present invention;

[0046] Figure 6 is the schematic diagram of the visualization of the detection results of the detector trained with the pseudo-boxes generated by PointSR and SOTA methods provided by a specific embodiment of the present invention;

[0047] Figure 7 is the loss and AP 50 curve of the entire training stage provided by a specific embodiment of the present invention;

[0048] Figure 8 is the AP 50 and IoU curve schematic diagrams at different sampling ratios δ provided by a specific embodiment of the present invention;

[0049] Figure 9 is the comparison schematic diagram of the pseudo-box optimization process provided by a specific embodiment of the present invention;

[0050] Figure 10 is the comparison schematic diagram between the rigid sampling strategy and the self-regularized sampling strategy provided by a specific embodiment of the present invention. Detailed Embodiments

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0052] First of all, it should be noted that ISOD (Image-level Supervised Object Detection) refers to a method of training a detector using image-level annotations instead of box-level annotations. WSDDNs (Weakly Supervised Deep Detection Networks) pioneered the integration of MIL (Multiple Instance Learning) into ISOD, and used a two-stream structure to classify candidate regions, thus achieving effective weakly supervised object detection. Inspired by this, the prior art designed a Proposal Cluster Learning (PCL) method, which uses candidate region clustering as supervision to infer the rough target location. However, the image-level annotations lack clear location information, resulting in low-quality candidate regions in potential target tools, such as background regions. This limitation makes it infeasible to directly apply ISOD to DroneDet (drone object detection).

[0053] PSOD (Point-Supervised Object Detection) utilizes point-level annotations, which provide more accurate object localization and classification compared to image-level annotations. At present, some scholars have proposed a model with center point annotations, introducing point supervision into object detection for the first time and replacing the traditional box-level supervision. Or a unified framework has been proposed, which can perform object detection under various annotation types. There has also been a design of a Point-to-Box framework based on multiple instance learning (MIL), which realizes object detection by classifying candidate boxes. In camouflaged object detection, a novel point-based learning paradigm is adopted, and an attention regulator is used to distinguish the objects hidden in the environment. However, these PSOD methods only focus on the objects in ground perspective images and perform poorly in complex aerial scenes due to the presence of a large number of dense small objects.

[0054] Existing aerial image PSOD methods can be divided into two categories: generative and discriminative. Generative methods, such as PLUG, use a short-path algorithm to generate pseudo-boxes with pixel-level labels. However, an obvious drawback is that the results are highly dependent on the accuracy of pixel-level mask generation. If there are outliers far from the target mask, the corresponding pseudo-labels will be severely affected.

[0055] In contrast, discriminative methods aim to improve the quality of pseudo-labels through advanced sampling strategies. PointOBB generates pseudo-labels by synergistically leveraging three different views to achieve coupled optimization of scale and angle. Point2Rbox combines point annotations and synthetic mode knowledge with bounding boxes to increase sample diversity and provide more supervision information for accurate pseudo-labels.

[0056] However, despite these advancements, existing methods are still limited by the fixed positive sample candidate proposals and the instability of the random distribution of negative samples. Specifically, in the traditional PSOD framework, the positive sample space remains unchanged during training, while the negative sample space is only slightly affected by rough pseudo-boxes. To address these limitations, embodiments of the present invention propose an effective and robust PSOD framework that generates high-quality pseudo-boxes through a self-regularized sampling strategy and adaptively adjusts the sampling space during model training.

[0057] Based on this, to address the limitations of static anchor boxes, the Temporal-Ensembling Encoder (TE encoder) constructs a category-aware prototype library by aggregating the aspect ratio information of historical pseudo-boxes. Specifically, after each training epoch, the aspect ratio prototypes of each target category are dynamically updated using exponential moving average (EMA), enabling the anchor boxes to gradually approximate the true target distribution. This mechanism significantly improves the matching degree between the anchor boxes and the data distribution and enhances training stability. For the problem of inefficient negative sampling, the scheme designs an Informative-Sample Collector (IS collector) that screens high-uncertainty negative samples by calculating the loss value of the MIL classifier and removes redundant samples using non-maximum suppression (NMS). This strategy focuses on the "difficult samples" near the decision boundary, forcing the model to learn more refined feature discrimination capabilities, thereby improving training efficiency. In the pseudo-box optimization phase, the self-regularized sampling mechanism suppresses low-confidence candidate boxes by dynamically adjusting the sampling space of positive and negative samples and combining informative negative samples. For example, in the Refinement phase, the center position of the coarse-grained pseudo-boxes is jittered and the aspect ratio is fine-tuned, while the high-loss negative samples are weighted and optimized. Finally, high-precision pseudo-boxes are retained through NMS. In addition, the continuous update of the TE encoder and the iterative optimization of the self-regularized sampling form a closed loop, enabling the model to adapt to annotation perturbations and data evolution.

[0058] Refer toFigure 1 , the present invention provides a method for drone perspective object detection based on self-regularized point supervision, and the method includes the following steps:

[0059] S100. Obtain the target image to be detected by the drone and generate the feature vector of the candidate region through the region proposal mechanism, and the target image to be detected is associated with the target-level point label in the image;

[0060] Specifically, use the convolutional neural network base network to extract the feature map of the training image. The base network can select the commonly used networks in the prior art, such as VGG16, resnet50, etc., and obtain the feature vector of each candidate region through region of interest pooling.

[0061] S200. Based on the object detection network of weak supervision learning, adopt the self-regularized sampling mechanism to construct the PointSR object detection network model;

[0062] Specifically, the object detection network of weak supervision learning adopts a two-stream structure, which consists of a classification stream and a detection stream. The PointSR (self-regularized point supervision) object detection network model includes a temporal ensemble encoder, a coarse-grained pseudo-box prediction module and a pseudo-box optimization module. The temporal ensemble encoder, the coarse-grained pseudo-box prediction module and the pseudo-box optimization module are connected in sequence, and the coarse-grained pseudo-box prediction module includes an IS collector.

[0063] In this embodiment, as Figure 3 shown, the architecture of PointSR in this embodiment consists of three parts: TE encoder (Temporal-Ensembling Encoder), coarse-grained pseudo-box prediction (Coarse Pseudo-box Prediction) and pseudo-box refinement (Pseudo-box Refinement). The entire training process includes a total of T training epochs, and each epoch consists of two stages: coarse-grained pseudo-box prediction and pseudo-box optimization. In the previous stage, coarse-grained pseudo-boxes are generated, and a set of negative samples are collected by sorting the loss values; in the latter stage, these negative samples and the positive samples in the previous stage are used to further optimize the pseudo-boxes. The TE encoder aggregates the temporal information in each epoch and is used to dynamically adjust the generation of candidate proposal boxes at the beginning of the coarse-grained pseudo-box prediction stage.

[0064] Furthermore, it should be noted that point-supervised object detection (PSOD) provides the minimum but additional unique location information for each object. The input image I contains M objects, and each object is associated with the point-level label y ∈ {0, 1} C associated. For the i-th object, its positive sample with the label ​ is generated by the static anchor strategy, while the negative samples with label are collected by random sampling on the background. The negative sampling space is defined as the difference between the image I and the predicted pseudo box regions (PB), denoted as diff(I, PB). Therefore, the optimization of existing PSOD can be expressed as: Among them, the positive sample x of each target

[0065]

[0066] remains unchanged, and the negative sampling space diff(I, PB) depends on PB. + Based on this, to solve the limitations of rigid sampling, the embodiments of the present invention propose a self-regularized sampling mechanism to dynamically adjust the sampling spaces of the positive sample X

[0067] and the negative sample X + using temporal and information constraints. According to the optimization expression of existing PSOD, the optimization of PSOD by PointSR of the present invention is expressed as: - In the above formula, X

[0068]

[0069] represents the positive sample, X + represents the negative sample, min(·) represents the minimum operation, arg represents the average calculation, θ represents the parameters of the model, including the weights of the fully connected layers of the classification stream and the detection stream, etc., x - represents the positive sample region feature, x + represents the negative sample region feature, M represents the number of targets in the image, C represents the total number of categories, i represents the i-th target, k represents the k-th category, L(·) represents the loss function (such as cross-entropy loss) to measure the difference between the predicted score and the true label y, f - represents the region feature of the i-th target on the k-th category, θ represents the true label of the i-th target on the k-th category, N represents the number of negative samples, j represents the j-th target, represents the region feature of the j-th negative sample, represents the true label of the j-th negative sample. represents the region feature of the j-th negative sample, represents the true label of the j-th negative sample.

[0070] In summary, the embodiments of the present invention, PointSR, adopt a self-regularized sampling mechanism to optimize the process of generating pseudo boxes from point-level annotations. As Figure 3As shown in the figure, the PointSR architecture of the embodiment of the present invention consists of three main components: a TE encoder, a coarse pseudo-box prediction, and a pseudo-box refinement. The TE encoder aggregates temporal information from the pseudo-labels predicted in each epoch and encodes it into an anchor prototype library. The coarse pseudo-box prediction uses the anchor prototype library for anchor adjustment and predicts the coarse pseudo-boxes using a MIL-based classifier. It also collects informative negative samples for refinement. Finally, the pseudo-box refinement module uses a MIL-based classifier to further refine the pseudo-boxes by suppressing low-confidence candidate positive samples.

[0071] S300. Based on the PointSR object detection network model, perform point-supervised object detection on the feature vectors of the candidate regions to obtain the point-supervised object detection results from the perspective of the unmanned aerial vehicle.

[0072] S310. Input the feature vectors of the candidate regions into the PointSR object detection network model.

[0073] S320. Based on the temporal integration encoder of the PointSR object detection network model, perform dynamic anchor adjustment processing on the feature vectors of the candidate regions to construct an anchor prototype library.

[0074] Specifically, input the feature vectors of the candidate regions into the temporal integration encoder of the PointSR object detection network model to create feature candidate boxes; obtain the ratio information of the anchor width and height of the feature candidate boxes and perform temporal aggregation encoding processing to obtain the pseudo-box shapes of each category; store the pseudo-box shapes of each category as a category-aware prototype library to construct an anchor prototype library.

[0075] In this embodiment, the first step in predicting the pseudo-boxes is to create candidate boxes. The method of the embodiment of the present invention predicts these candidate boxes in an anchor style, both statically and dynamically.

[0076] In the widely adopted static anchor generation, the width and height of the anchors are determined in advance by heuristic methods. Once the anchor shapes are set, their shapes remain unchanged throughout the training process. The static anchor strategy, following the anchor settings of Faster R-CNN, may not be optimal because it ignores the distribution of the underlying ground truth data and may lead to low-quality pseudo-boxes and slow convergence speed.

[0077] To address the limitations of static anchors, the embodiment of the present invention proposes a learning-based dynamic anchor adjustment module called "temporal integration encoder" (TE Encoder), as Figure 3As shown in (d) therein. The TE Encoder focuses on encoding the ratio of the width to the height of the anchor points, which is numerically more stable in the UAV perspective image than the object scale. At the end of each training iteration, the embodiments of the present invention can obtain high-quality pseudo-box predictions in the optimization phase as a reference for adjusting the anchor point shape.

[0078] Specifically, the TE Encoder aggregates the pseudo-box shapes of each category in each iteration and stores them as a category-aware prototype library. At round t, the c-th element P c ∈{0,1}, c∈{1,2,…,C} represents the ratio of the width to the height of the j-th object in the c-th category. P c is iteratively updated as follows:

[0079]

[0080] In the above formula, P' c represents the c-th element after iterative update, P c represents the c-th element, w tj , h tj represent the width and height of the j-th object.

[0081] S330. The coarse-grained pseudo-box prediction module based on the PointSR object detection network model performs coarse-grained pseudo-box prediction and negative sample information acquisition on the anchor point prototype library to obtain coarse-grained pseudo-boxes and negative sample information with high loss values.

[0082] Specifically, the anchor point prototype library is input into the coarse-grained pseudo-box prediction module of the PointSR object detection network model; through the MIL classifier with a two-stream structure, coarse-grained pseudo-box prediction of the category scores at the image level is performed on the anchor point prototype library to construct coarse-grained pseudo-boxes.

[0083] Among them, through the MIL classifier with a two-stream structure, matrix calculations are performed on the anchor point prototype library to obtain the category matrix and the position prediction score matrix of each region; based on the softmax operator, element-wise multiplication is performed on the category matrix and the position prediction score matrix of each region to calculate the scores of the regions, and summation is performed to obtain the category prediction scores at the image level; based on the category prediction scores at the image level, the top-k boxes with the highest scores in the same set are selected as the coarse-grained pseudo-boxes.

[0084] In this embodiment, PointSR of the present invention is constructed based on a weakly supervised learning object detection network (Weakly Supervised Deep Detection Networks, WSDDNs) for image supervised object detection (ISOD). The input data consists of a set of images, and each image is associated with an image-level label y ∈ {0, 1} C associated. Here, C represents the number of object categories. Candidate object regions R = {R1, R2, …, R |R|} are generated through a region proposal mechanism. The features of these regions are extracted through a fully connected layer.

[0085] WSDDNs adopt a two-stream structure, consisting of a classification stream and a detection stream. The extracted feature x is input into these two streams to generate two matrices x c and representing the class and location prediction scores for each region respectively. These scores are then processed through the corresponding softmax operators σ cla (·) and σ det (·), defined as:

[0086]

[0087] Then, the scores of the regions are calculated through element-wise multiplication as follows:

[0088] x R = σ cla (x c ) · σ det (x d )

[0089] Finally, the image-level class prediction score is obtained by summing the scores of all regions as follows:

[0090]

[0091] Given the sampling space X and label space Y of the proposal regions, the feature x of the proposal regions sampled from X in the image is input into the MIL-based classifier f θ (·) parameterized by θ to obtain the image-level class prediction score y ∈ Y. Therefore, ISOD can be expressed as:

[0092]

[0093] where L(·) is the loss function of the model, and k is the k-th class of the label. Note that the sampling space X is constant during training.

[0094] Furthermore, it should also be noted that the MIL loss Denoted as the cross-entropy loss of the positive samples, the expression of the loss function of the MIL classifier with a two-stream structure is specifically as follows:

[0095]

[0096] In the above formula, represents the loss function of the MIL classifier with a two-stream structure, and y j,k represents the true label of the j-th target on the k-th category. If the true category of the j-th target is the k-th category, then y j,k = 1; for other categories, y j,k = 0, represents the prediction score generated by the MIL-based predictor. M represents the number of positive samples, C represents the total number of categories, j represents the j-th target, and k represents the k-th category, represents the prediction probability of the j-th positive sample for the k-th category.

[0097] Therefore, in this embodiment, given the adjusted set of region proposals for each point label, a two-stream structure is adopted as the MIL classifier to find the best bounding box region in the set to represent the target. As shown in (b) of Figure 3 , the features of these proposals are extracted through the RoI layer and two fully connected layers. Secondly, these features are input into the two-stream structure, and the scores of each proposal in the same set are estimated according to the softmax operator formula. Then, the score of the set is calculated by summing up the scores of all proposals in the set. Finally, the top-k bounding boxes with the highest scores in the same set are selected as the rough pseudo bounding boxes.

[0098] Furthermore, based on the IS collector, the negative samples in the anchor prototype library are deduplicated through the non-maximum suppression algorithm to obtain the deduplicated negative samples; the loss values of the deduplicated negative samples are obtained through the negative sample loss function, and the deduplicated negative samples are sorted in descending order according to the loss values to obtain the negative sample information with high loss values.

[0099] In this embodiment, in order to capture informative negative samples with high uncertainty, the present invention embodiment proposes an IS collector, the purpose of which is to collect negative samples with high loss values. Specifically, the negative samples randomly sampled on the background and their corresponding losses estimated by the MIL-based classifier are input into the IS collector of the present invention embodiment for the refinement of the pseudo bounding boxes, as shown in Figure 3As shown in (e) of []. Considering the high computational cost of all negative samples, non-maximum suppression is first used for deduplication so that the embodiments of the present invention can remove redundant negative samples with high similarity and obtain representative samples. Then, the embodiments of the present invention sort the filtered negative samples in descending order of loss, so as to collect the samples with the largest amount of information for pseudo-box optimization.

[0100] Furthermore, it should be noted that the expression of the negative sample loss function is specifically as follows:

[0101]

[0102] In the above formula, represents the negative sample loss function, β and γ represent weight coefficients, represents the prediction score generated by the MIL-based predictor, N represents the number of negative samples, C represents the total number of categories, j represents the jth target, and k represents the kth category.

[0103] Among them, γ is set to 0.2, which is the same as in [], and β is the weight used to balance the contributions of different negative samples.

[0104] In summary, the embodiments of the present invention adjust the anchor points through the anchor point prototype library to obtain better coarse-grained pseudo-boxes, and at the same time collect a set of information-rich samples for the refinement of the pseudo-boxes. As Figure 4 shown, it is a comparison of the negative sample distributions of PointSR and the comparison method (Point-to-Box). Figure 4 The histogram (b) in [] shows the sample number distributions corresponding to different negative sample generation methods, where "distance" represents the Euclidean distance between the center of the negative sample and the center of the positive sample. The negative samples of PointSR (a) and Point-to-Box (c) are visually displayed in the figure respectively.

[0105] S340. The pseudo-box optimization module based on the PointSR object detection network model refines the coarse-grained pseudo-box according to the negative sample information with high loss values, and outputs the point supervision object detection result from the perspective of the drone.

[0106] In this embodiment, the samples with a large amount of information collected are used to suppress the candidate positive samples with low confidence, so as to optimize the pseudo-box. As Figure 1As shown in (c), the structure of the pseudo-box optimization model is similar to that of the coarse pseudo-box prediction model. Specifically, for positive sample generation, the embodiments of the present invention update the aspect ratio of the coarse pseudo-box and fine-tune its center position. At the same time, given the negative samples with a large amount of information in the coarse pseudo-box, after the RoI layer, all the features of the samples are input into the MIL classifier according to the calculation formula of the softmax operator. Finally, the relatively accurate Top-k candidate proposals are selected and merged into the finally predicted pseudo-box.

[0107] As Figure 4 shown in (b), many proposals generated by background random sampling are far from the target. Similarly, Figure 4 (c) shows some negative samples, indicating that they contribute little to object detection. For densely arranged objects in drone-view images, the edge information between objects and the relationship between objects and the surrounding environment are crucial for distinguishing each object. Random negative sampling in the background generates proposals only in a relatively stable sample space. This sampling method ensures the diversity of negative samples, but may include negative samples with low correlation, thus reducing the learning efficiency. In addition, this method uses the same weight for negative samples and cannot adaptively select useful and informative negative samples.

[0108] Figure 4 (a) shows that the negative samples generated by the information sampling strategy of the embodiments of the present invention are close enough to the decision boundary in the feature space. Using these negative samples for training helps the MIL classifier identify which candidate proposals should be given low scores and forces the model to learn finer distinctions because these negative samples contribute more significantly to the gradient. Therefore, high-quality pseudo-boxes can be obtained through information-rich negative sampling.

[0109] Finally, to verify the effectiveness of PointSR of the embodiments of the present invention for PSOD in DroneDet, the embodiments of the present invention conducted experiments on two benchmark datasets and compared the results of the embodiments of the present invention with four state-of-the-art (SOTA) methods. Next, the embodiments of the present invention first introduce the experimental datasets and implementation details, and then present the detailed experimental results.

[0110] As shown in Table 1, the per-class detection accuracies (AP 50 ), mean AP 50 and mloU of the detectors trained with the pseudo-boxes generated by PointSR of the embodiments of the present invention and SOTA methods on VisDrone are compared. "Faster-RCNN" represents the accuracy obtained by the detector trained with real box-level annotations and serves as the upper limit of performance.

[0111] Table 1 Per-class detection accuracies (AP 50) Average Precision (AP) 50 Data table for comparison of and mloU

[0112]

[0113]

[0114] In the experiments of the embodiments of the present invention, two benchmark datasets captured by drones were used: DroneVehicle and VisDrone.

[0115] For DroneVehicle, the DroneVehicle dataset contains 56,878 images, which are box-level annotated on five vehicle categories (i.e., cars, trucks, buses, vans, and freight vehicles), covering various scenarios captured under different lighting conditions from day to night, including urban roads, residential areas, and parking lots.

[0116] For VisDrone: The VisDrone dataset consists of 10,209 images with a resolution of 2000x1500 pixels and contains 542,000 instances, covering 10 common object categories in traffic scenarios, namely pedestrians, crowds, bicycles, cars, vans, trucks, tricycles, covered tricycles, buses, and motorcycles. These images were captured from various urban scenarios, presenting significant challenges due to the presence of a large number of small objects, different perspectives, and severe occlusions. Due to hardware memory limitations, in the embodiments of the present invention, according to the PLUG method, the original images were cropped into 512x512 pixel patches with an overlap of 128 pixels for training and inference.

[0117] To accurately reproduce the biases inherent in manual annotation, the embodiments of the present invention adopt a Rectified Gaussian Distribution (RG) with a central ellipse constraint, assuming that the point distribution of manual annotation approximately follows the RG distribution. Therefore, in the experiments of the embodiments of the present invention, the DroneVehicle and VisDrone datasets with point-level annotations were used to evaluate the method of the embodiments of the present invention, which are respectively referred to as "DroneVehicle (point supervision)" and "VisDrone (point supervision)".

[0118] In the embodiments of the present invention, PointSR proposed in the embodiments of the present invention is implemented on the NVIDIA RTX 4090 GPU based on the MMDetection toolbox. The Stochastic Gradient Descent (SGD) algorithm is used for optimization, and a 3× training schedule is adopted, that is, three times the number of training epochs, with each training epoch being 12 epochs. The learning rate is set to 0.02 and decays by a factor of 0.1 at the 24th and 33rd epochs respectively. All comparison models are configured with a ResNet-50 backbone network pre-trained on ImageNet.

[0119] The embodiments of the present invention evaluate the method of the embodiments of the present invention from three aspects: the quality of predicted pseudo-boxes, detection performance, and the robustness of the model. First, the mean Intersection over Union (mIoU) between the generated pseudo-boxes and the ground truth boxes is used to evaluate the quality of the predicted pseudo-boxes, where the closer the mIoU is to 1, the higher the accuracy. Second, the Average Precision (AP50) is used to evaluate the detection performance of the model. Third, the variance of AP under different point label distributions is used to evaluate the robustness of the model, and the lower the variance, the higher the robustness.

[0120] As Figure 5 shown, it is the visualization of the pseudo-boxes predicted by PointSR and the SOTA method. Among them, the ground truth boxes are shown in green.

[0121] For further comparison with the SOTA method, the embodiments of the present invention compare the performance of detectors trained using the pseudo-boxes generated by PointSR of the embodiments of the present invention and four state-of-the-art point-supervised drone detection methods (PLUG, PointOBB, Point2Rbox, and Point-to-Box). In these experiments, first, the pseudo-boxes are generated by PointSR of the embodiments of the present invention and the state-of-the-art methods. Then, these pseudo-boxes are used to train the classical detector Faster-RCNN to evaluate its detection performance and robustness. In addition, a Faster-RCNN is also trained using the ground truth box-level annotations, and its detection performance is regarded as the upper limit.

[0122] As shown in Table 2, the comparison of the Average Precision (AP50) and variance of detectors trained using the pseudo-boxes generated by PointSR of the embodiments of the present invention and the state-of-the-art methods under different point label distributions. PointSR always shows a lower variance on the two datasets, indicating higher stability and robustness to annotation perturbations.

[0123] Table 2 Comparison data table of Average Precision (AP50) and variance under different point label distributions

[0124]

[0125] As shown in Table 3, the detection accuracy, average AP, 50 and mloU of the detectors trained with the pseudo-boxes generated by PointSR of the embodiments of the present invention and SOTA methods on the Drone Vehicle dataset for different target categories are compared.

[0126] Table 3 Comparison data table of the detection accuracy, average value and mloU of the detectors trained with pseudo-boxes for different target categories

[0127]

[0128]

[0129] Figure 6 Visualization of the detection results of the detectors trained with the pseudo-boxes generated by PointSR and SOTA methods is shown. The true bounding boxes are shown in green, the correct detections are shown in yellow, and the misclassified or missed targets are shown in red.

[0130] According to Table 1 and Table 3, it can be seen that the detectors trained with the pseudo-boxes generated by PointSR of the embodiments of the present invention perform excellently in detection performance and are significantly better than the detectors trained with the pseudo-boxes generated by SOTA methods on both datasets. Specifically, compared with Point-to-Box, the model of the embodiments of the present invention improves AP50 by 7.2% on the Drone Vehicle (point supervision) dataset and by 3.3% on the VisDrone (point supervision) dataset. In addition, it is worth mentioning that Figure 6 in some cases, the detection results of the models trained with point-level annotations in the embodiments of the present invention are better than those of Faster-RCNN trained with box-level annotations.

[0131] The embodiments of the present invention use the mIoU (mean intersection over union) between the generated pseudo-boxes and the true boxes to evaluate the quality of the predicted pseudo-boxes, where the closer the mIoU is to 1, the higher the accuracy. Table 1 and Table 3 also show that PointSR has better mIoU than the SOTA (state-of-the-art) method on both datasets. Specifically, compared with Point-to-Box, the model of the embodiments of the present invention improves mIoU by 9.5% on DroneVehicle (point supervision) and by 2.9% on VisDrone (point supervision).

[0132] Figure 5The visualization results in show that the pseudo-boxes predicted in the embodiments of the present invention can accurately locate objects in challenging UAV perspective images and distinguish dense objects in complex scenes by utilizing the accurate position information of point annotations. In contrast, Point-to-Box often generates loose pseudo-boxes due to its limited sampling strategy in MIL (multiple instance learning) training, while PLUG occasionally predicts overly large boxes due to outliers in the generation process. These results confirm that the pseudo-boxes generated by the model in the embodiments of the present invention through point-level annotations have high quality.

[0133] To evaluate the robustness of the model, the embodiments of the present invention use the variance of AP50 on different point distributions. Specifically, the embodiments of the present invention assume that the manually annotated points follow a uniform distribution x ∼ U(-a,a), a = a×L, a = {0, 0.1, 0.3, 0.5}, and L is the maximum value of the bounding box width and height. U 0×L ,U 0.1×L ,U 0.3×L and U 0.5×L respectively represent the manually annotated points x ∼ U(-a×L, a×L) with offsets of 0, 10%, 30%, and 50%. As shown in Table 2, the model in the embodiments of the present invention achieves an AP50 variance of 0.171 on the UAV vehicle dataset. In contrast, the variance of Point-to-Box is 0.935, indicating that the model in the embodiments of the present invention is more robust to annotation perturbations.

[0134] The embodiments of the present invention conducted ablation experiments on the UAV vehicle (point supervision) dataset, using Point-to-Box as the baseline to verify the effectiveness of the key components in PointSR: the TE encoder and the high-quality sample collector.

[0135] As shown in Table 4, the impact of the TE encoder and the high-quality sample collector (IS) on the baseline Point-to-Box is evaluated on the UAV vehicle dataset.

[0136] Table 4 Evaluation Data Table

[0137] / TEE MS AP50 mloU [%] Baseline × × 35.26 86.0 Baseline + TEE √ × 36.18 86.7 Baseline + IS × √ 40.87 88.1 Baseline + TEE + IS √ √ 42.53 88.9

[0138] As Figure 7 shown, the loss and AP50 curves during the entire training phase. The blue curve represents the baseline training, while the orange curve shows the baseline training equipped with the TE encoder of the embodiments of the present invention. As Figure 8 shown are the AP50 and IoU curves at different sampling ratios δ.

[0139] Specifically, the embodiments of the present invention conducted a quantitative evaluation to verify the effectiveness of the TE encoder in PointSR of the embodiments of the present invention. The baseline equipped with the TE encoder is referred to as "baseline + TEE". Table 4 shows that compared with the original baseline model, the module proposed in the embodiments of the present invention increased AP50 by 0.92% and mIoU by 0.7%. In addition, the TE encoder promoted stable and efficient training, as shown in Figure 7 (a) of [reference], where the orange curve (baseline w TEE) drops faster and more smoothly, especially in the early stage of training, indicating that the anchor point learning model of the embodiments of the present invention achieved stable training and faster convergence. In addition, Figure 7 (b) of [reference] further illustrates that "baseline + TEE" obtained better detection performance. These experimental results prove the effectiveness of the TE encoder of the embodiments of the present invention.

[0140] The embodiments of the present invention evaluated the effectiveness of the high-quality sample collector in PointSR. Table 4 shows that the baseline with the high-quality sample collector module of the embodiments of the present invention, referred to as "baseline + IS", increased AP50 by 5.26% and mIoU by 2.2%. These experimental results confirm the effectiveness of the IS collector of the embodiments of the present invention. In addition, the embodiments of the present invention also analyzed the influence of the ratio δ (referred to as "sampling rate") of the selected negative samples to the entire negative samples on the detection performance. Figure 8 Visualization shows that the change of δ significantly affects the detection performance of PointSR of the embodiments of the present invention in terms of AP50 and mIoU, indicating its key role in the optimization results.

[0141] In summary, as shown in Figure 9 for the comparison of the pseudo-box optimization process, Figure 9 (a) in [reference] indicates that the rigid sampling strategy in Point-to-Box leads to low efficiency in the training process of low-quality pseudo-box labels. Figure 9 (b) in [reference] shows that the self-regularized sampling in PointSR of the embodiments of the present invention improves the stability and efficiency of the training process, thus obtaining high-quality pseudo-box labels; as shown in Figure 10 shown in Figure 10 (a) in [reference] indicates that Point-to-Box adopts a rigid sampling strategy, using fixed positive sample proposals and randomly sampled negative background regions to train a multi-instance learning (MIL)-based predictor. Figure 10 (b) in [reference] indicates that PointSR (embodiments of the present invention) adopts a self-regularized sampling strategy, incorporating time and information constraints into the pseudo-box generation process for sampling positive and negative samples.

[0142] Referring to Figure 2 , the drone perspective object detection system based on self-regularized point supervision includes:

[0143] The first module 201 is configured to obtain a target image to be detected by a drone and generate a feature vector of a candidate region through a region proposal mechanism, where the target image to be detected is associated with a target-level point label in the image;

[0144] The second module 202 is configured to construct a PointSR object detection network model by using a self-regularized sampling mechanism based on a weakly supervised learning object detection network;

[0145] The third module 203 is configured to perform point supervision object detection on the feature vector of the candidate region based on the PointSR object detection network model to obtain a point supervision object detection result from the perspective of the drone.

[0146] The content in the above method embodiments is applicable to the system embodiments of the present invention. The functions specifically implemented by the system embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0147] The above has specifically described the preferred embodiments of the present invention. However, the present invention is not limited to the described embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. An object detection method from the perspective of an unmanned aerial vehicle based on self-regularized point supervision, characterized in that, It includes the following steps: Obtain the target image to be detected by a drone and generate the feature vectors of candidate regions through a region proposal mechanism, where the target image to be detected is associated with the target-level point labels in the image; Based on the object detection network of weakly supervised learning, adopt a self-regularized sampling mechanism to construct a PointSR object detection network model; Based on the PointSR object detection network model, perform point-supervised object detection on the feature vectors of the candidate regions to obtain the point-supervised object detection results from the perspective of the drone.

2. The method for detecting an object from a UAV perspective based on self-regularized point supervision according to claim 1, wherein The expression of the objective function for the point-supervised object detection of the PointSR object detection network model is specifically as follows: In the above formula, X + represents the positive sample, X - represents the negative sample, min(·) represents the minimum operation, arg represents the average calculation, θ represents the parameters of the model, x + represents the region feature of the positive sample, x - represents the region feature of the negative sample, M represents the number of targets in the image, C represents the total number of categories, i represents the i-th target, k represents the k-th category, L(·) represents the loss function, f θ (·) represents the MIL-based classifier, represents the region feature of the i-th target on the k-th category, represents the true label of the i-th target on the k-th category, N represents the number of negative samples, j represents the j-th target, represents the region feature of the j-th negative sample, represents the true label of the j-th negative sample.

3. The method for detecting an object from a UAV perspective based on self-regularized point supervision according to claim 2, wherein The object detection network of weakly supervised learning adopts a two-stream structure, consisting of a classification stream and a detection stream. The PointSR object detection network model includes a temporal integration encoder, a coarse-grained pseudo-box prediction module, and a pseudo-box optimization module. The temporal integration encoder, the coarse-grained pseudo-box prediction module, and the pseudo-box optimization module are connected in sequence, and the coarse-grained pseudo-box prediction module includes an IS collector.

4. The method for detecting an object from a UAV perspective based on self-regularized point supervision according to claim 3, wherein The step of performing point-supervised object detection on the feature vectors of the candidate regions based on the PointSR object detection network model to obtain the point-supervised object detection results from the perspective of the drone specifically includes: Input the feature vectors of the candidate regions into the PointSR object detection network model; Based on the temporal integration encoder of the PointSR object detection network model, perform dynamic anchor adjustment processing on the feature vectors of the candidate regions to construct an anchor prototype library; Based on the coarse-grained pseudo-box prediction module of the PointSR object detection network model, perform coarse-grained pseudo-box prediction and negative sample information collection on the anchor prototype library to obtain coarse-grained pseudo-boxes and negative sample information with high loss values; Based on the pseudo-box optimization module of the PointSR object detection network model, refine the coarse-grained pseudo-boxes according to the negative sample information with high loss values and output the point-supervised object detection results from the perspective of the drone.

5. The method for drone perspective object detection based on self-regularized point supervision according to claim 4, wherein The step of performing dynamic anchor adjustment processing on the feature vectors of the candidate regions based on the temporal integration encoder of the PointSR object detection network model to construct an anchor prototype library specifically includes: Input the feature vectors of the candidate regions into the temporal integration encoder of the PointSR object detection network model to create feature candidate boxes; Obtain the ratio information of the anchor width and height of the feature candidate boxes and perform temporal aggregation encoding processing to obtain the pseudo-box shapes of each category; Store the pseudo-box shapes of each category as a category-aware prototype library to construct an anchor prototype library.

6. The method for detecting an object from an unmanned aerial vehicle perspective based on self-regularized point supervision according to claim 5, wherein The step of performing coarse-grained pseudo-box prediction and negative sample information collection on the anchor prototype library based on the coarse-grained pseudo-box prediction module of the PointSR object detection network model to obtain coarse-grained pseudo-boxes and negative sample information with high loss values specifically includes: Input the anchor prototype library into the coarse-grained pseudo-box prediction module of the PointSR object detection network model; Through a MIL classifier with a two-stream structure, perform coarse-grained pseudo-box prediction of the image-level category scores on the anchor prototype library to construct coarse-grained pseudo-boxes; Based on the IS collector, the negative samples in the anchor prototype library are de-duplicated through the non-maximum suppression algorithm to obtain the de-duplicated negative samples; The loss value of the de-duplicated negative samples is obtained through the negative sample loss function, and the de-duplicated negative samples are sorted in descending order according to the loss value to obtain the negative sample information with high loss values.

7. The method for detecting an object from an unmanned aerial vehicle perspective based on self-regularized point supervision according to claim 6, wherein The step of constructing the coarse-grained pseudo-box by predicting the coarse-grained class scores of the image level of the anchor prototype library through the MIL classifier with a two-stream structure specifically includes: Through the MIL classifier with a two-stream structure, matrix calculations are performed on the anchor prototype library to obtain the class matrix and the position prediction score matrix of each region; Based on the softmax operator, element-wise multiplication is performed on the class matrix and the position prediction score matrix of each region to calculate the score of the region, and summation is performed to obtain the class prediction score at the image level; Based on the class prediction score at the image level, the top-k boxes with the highest scores in the same set are selected as the coarse-grained pseudo-boxes.

8. The method for detecting an object from a UAV perspective based on self-regularized point supervision according to claim 7, wherein The expression of the loss function of the MIL classifier with a two-stream structure is specifically as follows: In the above formula, represents the loss function of the MIL classifier with a two-stream structure, y j,k represents the true label of the j-th target in the k-th category, represents the prediction score generated by the MIL-based predictor, M represents the number of positive samples, C represents the total number of categories, j represents the jth target, k represents the kth category, Represents the predicted probability of the kth class of the jth positive sample.

9. The method for detecting an object from a UAV perspective based on self-regularized point supervision according to claim 8, wherein The expression of the negative sample loss function is specifically as follows: In the above formula, represents the negative sample loss function, β and γ represent weight coefficients, represents the prediction score generated by the MIL-based predictor, N represents the number of negative samples, C represents the total number of categories, j represents the j-th target, and k represents the k-th category.

10. The UAV perspective target detection system based on self-regularized point supervision is characterized in that It includes the following modules: The first module is used to obtain the target image to be detected by the drone and generate the feature vector of the candidate region through the region proposal mechanism, and the target image to be detected is associated with the target-level point label in the image; The second module is used to construct the PointSR target detection network model by adopting the self-regularized sampling mechanism based on the target detection network of weakly supervised learning; The third module is used to perform point supervision target detection on the feature vector of the candidate region based on the PointSR target detection network model to obtain the point supervision target detection result from the drone perspective.

Images