Boundary frame refining directional target detection method based on pseudo frame guidance

Abstract

The invention discloses a bounding box refinement directional target detection method based on pseudo box guidance, which comprises the following steps of: 1, inputting an image into a backbone network and a pyramid network, and extracting a multi-scale feature map; 2, dividing a real bounding box to a corresponding feature level according to the size of the real bounding box, and learning representative points through deformable convolution to generate a self-adaptive point set; and 3, performing evaluation and dynamic allocation on the generated representative points by using an evaluation module (APAA), and inputting a pseudo box guide bounding box refinement module (PGBR) for further optimization. And 4, extracting features by using an RRoI feature extractor, encoding the features, predicting object category probability and refining a prediction bounding box. According to the method, the bounding box refining module is added, so that the bounding box positioning effect can be better realized; and meanwhile, different bounding box representation modes are used, so that the bounding box can be effectively positioned and identified, and the problem of discontinuous boundaries in traditional bounding box representation is solved.

Description

Technical Field

[0001] The present invention belongs to the field of long-term target detection and recognition, and in particular relates to a bounding box refinement directional target detection method based on pseudo-frame guidance. Background Art

[0002] As a basic technology in the field of computer vision, object detection has undergone a huge transformation from traditional algorithms to deep learning algorithms. Before the emergence of deep learning technology, object detection mainly relied on manually designed features and traditional machine learning algorithms, such as Haar features and HOG features. These methods achieve the effect of object detection by capturing target texture information by calculating the number of pixels or through methods such as gradient histograms. However, these methods have limited detection performance and generalization capabilities in complex scenes.

[0003] With the rapid development of deep learning technology, convolutional neural networks have begun to demonstrate their powerful feature extraction capabilities in the field of target detection. Algorithms based on convolutional neural networks first generate a large number of candidate regions that may contain targets, and then perform target detection and classification on these regions. These methods can be divided into two-stage models and one-stage models according to their stages. The R-CNN proposed by Girshick R. et al. [Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 580-587.] is a two-stage model. This type of method first needs to extract candidate regions from the entire image, and then process each region separately; while the one-stage model, such as the YOLO algorithm proposed by Redmon et al. [Redmon J. You only look once: Unified, real-time object detection [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.], directly performs end-to-end operations, directly detects targets from images, and omits operations related to region extraction. In contrast, the one-stage model usually takes less time than the two-stage model, but the accuracy of the two-stage model is relatively much higher than that of the one-stage model.

[0004] However, although these general target detection methods have achieved remarkable results in many application scenarios, they often ignore the target's directional information, which is an important defect in some specific applications. In order to solve the problem of missing target directional information, directional target detection technology came into being. Directional target detection not only needs to identify the category and position of the target, but also needs to accurately predict the direction of the target. At the same time, the refinement of the bounding box, the accuracy of target positioning, and the ability to handle complex scenes in directional target detection are still urgent problems to be solved in this field. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method for directional target detection based on bounding box refinement guided by pseudo-frame. This method inherits the high efficiency of general target detection and the accuracy of directional target detection. By introducing representative point learning and bounding box refinement strategies, it aims to achieve accurate detection and positioning of directional targets, thereby providing more reliable technical support for remote sensing image detection, drone monitoring and other fields.

[0006] Technical solution: A method for directional target detection based on pseudo-frame guided bounding box refinement of the present invention comprises the following steps:

[0007] Step 1: Input the read image into the ResNet backbone network to extract features, and then use the pyramid network FPN to upsample the image multiple times to obtain the multi-scale feature map of the image. For each scale of feature map extracted, first divide all the real bounding boxes into corresponding scale levels according to the scale of the real bounding box to obtain feature maps of different levels;

[0008] Step 2: For the feature maps of different levels obtained in step 1, the representative points are learned using deformable convolution, and their semantic positioning is used to generate adaptive point sets by refining the center point of the object, thereby learning the target position information and generating preliminary representative points;

[0009] Step 3: For the preliminary representative points obtained in step 2, accurate adjustments are obtained through deformable convolution and minimization of loss functions; after learning is completed, the corresponding predicted bounding boxes are generated according to the obtained representative points according to the strategy, and input into the pseudo-box guided bounding box refinement module PGBR for learning; at the same time, the representative points are refined through the evaluation module APAA, and the top k solutions are dynamically obtained from them to allocate samples of oriented representative points, and the predicted bounding boxes are generated and input into the pseudo-box guided bounding box refinement module PGBR;

[0010] Step 4: Use the bounding box representation method to describe the predicted bounding box, and extract its features in the input RRoI feature extractor. For the extracted RRoI features, use two consecutive fully connected layers and ReLU layers to encode the RRoI features. Finally, use two fully connected layers to predict the class probability of the object and refine the predicted bounding box.

[0011] Furthermore, step 1 specifically includes the following steps:

[0012] Step 101: For an input image, extract features through the trained ResNet50 backbone network;

[0013] Step 102: For the features extracted in step 101, the feature map output by the ResNet backbone network is input into the pyramid network FPN. A feature pyramid P is constructed by performing additional convolution operations on the low-level feature maps and upsampling and element-level addition of the high-level feature maps. The process is as follows:

[0014] P x =FPN upsample (Fig),x=2,…,7

[0015] Among them, P x They represent the feature maps obtained by upsampling the feature pyramid, and Fig represents the input image;

[0016] Step 103: For each scale feature map output by the FPN, assign it to a corresponding feature map level according to the scale of the true bounding box, ensuring that each true bounding box can correspond to its matching feature map level.

[0017] Furthermore, step 2 specifically includes the following steps:

[0018] Step 201: For the feature maps of different levels extracted in step 1, a portion of point sets S are extracted from the feature maps according to their sizes. p , which is then learned by subsequent models;

[0019] Step 202: Use deformable convolution to predict the point set S p Perform convolution operation to obtain the feature points for preliminary learning;

[0020] Step 203: The initially learned feature points are converted into bounding box representations through a directional conversion function G and input into a subsequent bounding box refinement module PGBR for adjustment. The directional conversion function G is generated by the point set S p, use the MinAeraRect or ConvexHull method to find the rotated rectangle with the smallest area from the point set on the oriented object to obtain the final predicted bounding box.

[0021] Furthermore, step 3 specifically includes the following steps:

[0022] Step 301: Use deformable convolution again to predict the point set S p Adjustments are made to obtain the refined feature points, and then converted into the predicted bounding box representation OB through the directional transformation function G p ;

[0023] Step 302: The obtained points are quality screened and dynamically allocated through the evaluation allocation module APAA, and the loss function L is designed from four aspects to adjust the learning of representative points. The loss function includes: classification loss L cls , positioning loss L loc , Direction loss L ori and the center point position loss L cp ;

[0024] Step 303: The predicted bounding box obtained by the assessment allocation module APAA is input into the subsequent bounding box refinement module PGBR for adjustment.

[0025] Furthermore, in step 302, the loss function L is specifically as follows:

[0026] L=L cls +μ 1 L loc +μ 2 L ori +μ 3 L cp

[0027] Here, μ 1 , μ 2 , μ 3 is a parameter used to balance the weights;

[0028] Classification loss L cls It is used to measure the accuracy of target classification and is calculated using Focal loss. The specific formula is as follows:

[0029] L cls =Focal Loss(OB p ,OB GT )

[0030] Among them, OB p Represents the predicted bounding box, OB GT represents the corresponding ground-truth bounding box;

[0031] Positioning loss L loc It is used to measure the deviation between the predicted bounding box and the true bounding box. It is calculated using the SmoothL1 loss function as follows:

[0032] L loc =Smooth L 1 (OB p ,OBG T )

[0033] Positioning loss L loc It is a measure of the distance between spatial positions. In order to enable the model to accurately locate the direction, the direction loss L is used ori To assist positioning, the direction loss L ori It is calculated by the Chamfer Distance of the point set. The formula for the chamfer distance is as follows:

[0034]

[0035] Among them, x represents the corner point of the predicted bounding box, and y represents the corner point of the corresponding real bounding box;

[0036] Center point position loss L cp It is to ensure the position of the center point, which is measured by calculating the Smooth L1 loss of the distance between the center point of the predicted bounding box and the center point of the true bounding box. The formula is as follows:

[0037]

[0038] in, represents the center point of the predicted bounding box, Represents the center point of the ground-truth bounding box.

[0039] Furthermore, step 4 specifically includes the following steps:

[0040] Step 401, using (x, y, r, Δα, Δβ) to represent the predicted bounding box, and learning the obtained predicted bounding box through an RRoI feature extraction layer to obtain relevant RRoI features;

[0041] Step 402: For the obtained RRoI features, two consecutive fully connected layers and ReLU layers are used to encode the RRoI features;

[0042] Step 403: After the encoding is completed, the specific classification and bounding box are predicted through two fully connected layers. The predicted bounding box finally obtained is expressed as:

[0043] P={x,y,r,Δα,Δβ}

[0044] Among them, {x, y} is the position of the center point of the bounding box representing the point prediction, r is the radius of the circle with the midpoint as the center, and {Δα, Δβ} are the offsets of the four corner points to the four positive points up, down, left and right, respectively.

[0045] The present invention also discloses a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0046] The present invention also discloses a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the present invention are implemented.

[0047] The present invention also discloses a computer program product, comprising a computer program / instruction, which implements the steps of the method of the present invention when executed by a processor.

[0048] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0049] 1. The present invention can achieve accurate positioning of the bounding box by adding a pseudo-frame-guided bounding box correction module, solve the problem that the point set is insensitive to direction, and achieve a better bounding box positioning effect.

[0050] 2. The present invention uses different bounding box representation methods, which not only solves the boundary discontinuity problem caused by angles in traditional bounding box representation, but also effectively reduces the amount and complexity of calculations, improves the accuracy of calculations, and facilitates the calculation of subsequent loss functions.

[0051] 3. For each scale of the feature map output by FPN, assign it to the corresponding feature map level according to the scale of the true bounding box, ensuring that each true bounding box can correspond to its most matching feature map level, so that the target can be more accurately located and identified in the subsequent target detection task.

[0052] 4. Through deformable convolution, the characteristics of the sampling points can be flexibly adjusted, and the model can learn the semantic information in the image well.

[0053] 5. Through a RRoI feature extraction layer, relevant RRoI features are obtained. The RRoI feature extractor can better capture the characteristics of the object, making the subsequent classification and bounding box prediction more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a model structure diagram of the present invention;

[0055] Figure 2 This is a schematic diagram of the APAA module of the present invention;

[0056] Figure 3 is a schematic diagram of a boundary box representation of the present invention;

[0057] Figure 4 It is a prediction image obtained after a given remote sensing image is detected by the target detection method of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0059] The present invention provides a collaborative saliency detection method based on sparse attention and edge guidance, the flow chart is as follows: Figure 1 As shown, the following steps are included:

[0060] Step 1: Input the read image into the ResNet backbone network to extract features, and then use the pyramid network (Feature Pyramid Network, FPN) to upsample the image multiple times to obtain the multi-scale feature map of the image. For each scale of feature map extracted, first divide all real bounding boxes into corresponding scale levels according to the scale of the ground-truth;

[0061] Step 101, first, for an input image, extract features through the trained ResNet50 backbone network;

[0062] Step 102: For the features extracted in step 101, the feature map output by the ResNet backbone network is input into the Feature Pyramid Network (FPN). A feature pyramid P is constructed by performing additional convolution operations on the low-level feature maps and upsampling and element-level addition of the high-level feature maps. The process is as follows:

[0063] P x =FPN upsample (Fig),x=2,…,7

[0064] Among them, P x They represent the feature maps obtained by upsampling the feature pyramid, and Fig represents the input image;

[0065] Step 103, for each scale feature map output by FPN, assign it to the corresponding feature map level according to the scale of the true bounding box, ensuring that each true bounding box can correspond to its most matching feature map level, so that the target can be more accurately located and identified in the subsequent target detection task.

[0066] Step 2: For the features of different levels extracted in step 1, Deformable Convolution is used to learn representative points, and its more accurate semantic positioning is used to generate adaptive point sets by refining from the center point of the object, thereby learning the target location information. After generating preliminary representative points, the model will generate predicted bounding boxes based on these representative points according to a certain strategy, and put them into the subsequent bounding box refinement stage for learning;

[0067] Step 201: First, for the feature maps of different levels extracted in step 1, a part of the point set S is extracted from the feature maps according to the size of the feature maps. p , so that subsequent models can learn from it;

[0068] Step 202: For the feature maps of different levels extracted in step 201, first use deformable convolution to predict the point set S p Perform convolution operations to obtain the initial learning feature points. The deformable convolution can flexibly adjust the characteristics of the sampling points, and the model can learn the semantic information in the image well;

[0069] Step 203: The initially learned feature points are converted into bounding box representations through a directional conversion function G and input into a subsequent bounding box refinement module PGBR for adjustment. The directional conversion function G is generated by the point set S p , using the MinAeraRect and ConvexHull methods, with the help of convex hull and other tools, the final predicted bounding box is obtained by finding the rotated rectangle with the smallest area from the point set on the oriented object.

[0070] Step 3: For the preliminary representative points learned in step 2, input them into the refinement stage similar to the initialization stage, and further obtain more accurate adjustments through deformable convolution and minimization of loss function; after learning is completed, the model will also generate corresponding predicted bounding boxes according to a certain strategy based on the obtained representative points, and input them into the subsequent bounding box refinement module for learning. In addition, the model will also refine these points through the evaluation module APAA, and dynamically obtain the top k solutions to allocate samples of directional representative points, and generate corresponding predicted bounding boxes and input them into the bounding box refinement module, as shown below: Figure 2 shown.

[0071] Step 301: For the feature point set obtained in step 202, deformable convolution is used again to adjust it, thereby obtaining refined feature points, and converting them into a bounding box representation OB through the directional transformation function G used in step 203. p ;

[0072] Step 302, the obtained points are quality screened and dynamically allocated through the evaluation allocation module APAA, and the loss function L is designed from four aspects to adjust the learning of representative points. The loss function consists of four parts: classification loss L cls , positioning loss L loc , Direction loss L ori And the center point position loss L cp The details are as follows:

[0073] L=L cls +μ 1 L loc +μ 2 L ori +μ 3 L cp

[0074] Here, μ 1 , μ 2 , μ 3 is a parameter used to balance the weights. Classification loss L cls Focal loss is used to measure the accuracy of target classification. The specific formula is as follows:

[0075] L cls =Focal Loss(OB p ,OB GT )

[0076] Positioning loss L loc It is used to measure the deviation between the predicted bounding box and the true bounding box. It uses the Smooth L1 loss function for calculation, as follows:

[0077] L loc =Smooth L 1 (OB p ,OB GT )

[0078] However, the positioning loss L loc It is just a measure of the distance in spatial position. In order to enable the model to accurately locate the direction, we use the direction loss L ori To assist positioning. Specifically, the direction loss L ori It is calculated by the chamfer distance of the point set. The chamfer distance formula is as follows:

[0079]

[0080] Center point position loss L cpThe center point is ensured to facilitate the determination of the center point in the subsequent prediction of the bounding box positioning, so as to improve the stability of the overall detection. It is measured by calculating the Smooth L1 loss of the distance between the center point of the predicted bounding box and the center point of the true bounding box. The formula is as follows:

[0081]

[0082] Step 303, inputting the predicted bounding box obtained by the APAA evaluation allocation module in step 302 into a subsequent bounding box refinement module for adjustment;

[0083] Step 4: For the predicted bounding boxes obtained in steps 2 and 3, first extract the features of the input RRoI feature extractor, and then encode the RRoI features using two consecutive fully connected layers and ReLU layers given the RRoI features. The last two fully connected layers are responsible for predicting the class probability of the corresponding object and refining the predicted bounding box.

[0084] Step 401, the predicted bounding box obtained by the conversion in step 203 and step 303 is represented by (x, y, r, Δα, Δβ). The obtained predicted bounding box is learned through an RRoI feature extraction layer to obtain relevant RRoI features. The RRoI feature extractor can better capture the characteristics of the object, making the subsequent classification and bounding box prediction more accurate;

[0085] Step 402: for the RRoI feature obtained in step 401, two consecutive fully connected layers and ReLU layers are used to encode the RRoI feature;

[0086] Step 403, after the encoding is completed, the specific classification and bounding box are predicted through two fully connected layers, and the predicted bounding box finally obtained can be expressed as:

[0087] P={x,y,r,Δα,Δβ}

[0088] The specific meaning is Figure 3 As shown:

[0089] Among them, {x, y} is the position of the center point of the bounding box representing the point prediction, r is the radius of the circle with the midpoint as the center, and {Δα, Δβ} are the offsets of the four corner points to the four positive points up, down, left and right, respectively.

[0090] Figure 4 The figure shows the prediction image obtained after the given remote sensing image is detected by the target detection method of the present invention. As shown in the figure, the model correctly identifies the aircraft in the image, marks its type, and marks their corresponding positions with red bounding boxes.

Claims

1. A bounding box refinement directional target detection method based on pseudo-box guidance, characterized in that: The steps include: Step 1: Input the read image into the ResNet backbone network to extract features, and then use the pyramid network FPN to upsample the image multiple times to obtain the multi-scale feature map of the image. For each scale of feature map extracted, first divide all the real bounding boxes into corresponding scale levels according to the scale of the real bounding box to obtain feature maps of different levels; Step 2: For the feature maps of different levels obtained in step 1, input them into the adaptive point set generation module to learn the representative points using deformable convolution, and use their semantic positioning to generate adaptive point sets by refining the center point of the object, thereby learning the target position information and generating preliminary representative points; Step 3: For the preliminary representative points obtained in step 2, accurate adjustments are obtained through deformable convolution and minimization of loss functions; after learning is completed, the corresponding predicted bounding boxes are generated according to the obtained representative points according to the strategy, and input into the pseudo-box guided bounding box refinement module PGBR for learning; at the same time, the representative points are refined through the evaluation module APAA, and the top k solutions are dynamically obtained from them to allocate samples of oriented representative points, and the predicted bounding boxes are generated and input into the pseudo-box guided bounding box refinement module PGBR; Step 4: Use the bounding box representation method to describe the predicted bounding box, and extract its features in the input RRoI feature extractor. For the extracted RRoI features, use two consecutive fully connected layers and ReLU layers to encode the RRoI features. Finally, use two fully connected layers to predict the class probability of the object and refine the predicted bounding box.

2. The method for directional target detection based on pseudo-frame guided bounding box refinement according to claim 1, characterized in that: Step 1 specifically includes the following steps: Step 101: For an input image, extract features through the trained ResNet50 backbone network; Step 102: For the features extracted in step 101, the feature map output by the ResNet backbone network is input into the pyramid network FPN. A feature pyramid P is constructed by performing additional convolution operations on the low-level feature maps and upsampling and element-level addition of the high-level feature maps. The process is as follows: P x =FPN upsample (Fig),x=2,…,7 Among them, P x They represent the feature maps obtained by upsampling the feature pyramid, and Fig represents the input image; Step 103: For each scale feature map output by the FPN, assign it to a corresponding feature map level according to the scale of the true bounding box, ensuring that each true bounding box can correspond to its matching feature map level.

3. The method for directional target detection based on pseudo-frame guided bounding box refinement according to claim 1, characterized in that: Step 2 specifically includes the following steps: Step 201: For the feature maps of different levels extracted in step 1, a portion of point sets S are extracted from the feature maps according to their sizes. p , which is then learned by subsequent models; Step 202: Use deformable convolution to predict the point set S p Perform convolution operation to obtain the feature points for preliminary learning; Step 203: The initially learned feature points are converted into bounding box representations through a directional conversion function G and input into a subsequent bounding box refinement module PGBR for adjustment. The directional conversion function G is generated by the point set S p , use the MinAeraRect or ConvexHull method to find the rotated rectangle with the smallest area from the point set on the oriented object to obtain the final predicted bounding box.

4. The method for directional target detection based on pseudo-frame guided bounding box refinement according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step 301: Use deformable convolution again to predict the point set S p Adjustments are made to obtain the refined feature points, and then converted into the predicted bounding box representation OB through the directional transformation function G p ; Step 302: The obtained points are quality screened and dynamically allocated through the evaluation allocation module APAA, and the loss function L is designed from four aspects to adjust the learning of representative points. The loss function includes: classification loss L cls , positioning loss L loc , Direction loss L ori and the center point position loss L cp ; Step 303: The predicted bounding box obtained by the assessment allocation module APAA is input into the subsequent bounding box refinement module PGBR for adjustment.

5. The method for directional target detection based on pseudo-frame guided bounding box refinement according to claim 4, characterized in that: In step 302, the loss function L is specifically as follows: L=L cls +μ1L loc +μ2L ori +μ3L cp Here, μ1, μ2, and μ3 are parameters used to balance the weights; Classification loss L cls It is used to measure the accuracy of target classification and is calculated using Focal loss. The specific formula is as follows: L cls =Focal Loss(OB p ,OB GT ) Among them, OB p Represents the predicted bounding box, OB GT represents the corresponding ground-truth bounding box; Positioning loss L loc It is used to measure the deviation between the predicted bounding box and the true bounding box. It is calculated using the Smooth L1 loss function as follows: L loc =Smooth L1(OB p ,OB GT ) Positioning loss L loc It is a measure of the distance between spatial positions. In order to enable the model to accurately locate the direction, the direction loss L is used. ori To assist positioning, the direction loss L ori It is calculated by the Chamfer Distance of the point set. The formula for the chamfer distance is as follows: Among them, x represents the corner point of the predicted bounding box, and y represents the corner point of the corresponding real bounding box; Center point position loss L cp It is to ensure the position of the center point, which is measured by calculating the Smooth L1 loss of the distance between the center point of the predicted bounding box and the center point of the true bounding box. The formula is as follows: in, represents the center point of the predicted bounding box, Represents the center point of the ground-truth bounding box.

6. The method for directional target detection based on pseudo-frame guided bounding box refinement according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 401, using (x, y, r, Δα, Δβ) to represent the bounding box, the obtained bounding box is learned through an RRoI feature extraction layer to obtain relevant RRoI features; Step 402: For the obtained RRoI features, two consecutive fully connected layers and ReLU layers are used to encode the RRoI features; Step 403: After the encoding is completed, the specific classification and bounding box are predicted through two fully connected layers. The predicted bounding box finally obtained is expressed as: P = {x, y, r, Δα, Δβ} Among them, {x, y} is the position of the center point of the bounding box representing the point prediction, r represents the radius of the circle with the midpoint as the center, and {Δα, Δβ} represent the offsets of the four corner points to the four positive points up, down, left and right, respectively.

7. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

8. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images