A high-precision remote sensing target detection method and system based on detr

Abstract

The application provides a high-precision remote sensing target detection method and system based on DETR, relates to the technical field of computer vision, and through acquisition of an image including at least one target, a target detection network based on a DETR model is provided, point-axis representation including a group of point sets and axes independent of shapes is performed on the target in the image, point set learning is supervised through maximum projection loss, and axis representation learning is supervised through cross-axis loss, prediction bounding boxes of the target contained in the image and prediction category information of the target are obtained through a trained model, and flexibility and accuracy of target detection are realized.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a high-precision remote sensing target detection method and system based on DETR. Background Technology

[0002] In the field of remote sensing target detection, rotated target detection follows a grid-guided localization mechanism framework, proposing orientation-sensitive keypoint heatmaps and performing unordered regression on them. Point-set-based target detection utilizes the Jarvis algorithm to transform the point set into a convex hull, then calculates the loss using real-labeled data. Specifically, these methods determine the object's bounding box by calculating the minimum area rectangle enclosing these contour points. However, while these point-set-based methods excel in accurately capturing target locations, their performance is significantly affected by the quality and quantity of contour points. Furthermore, they neglect the target's primary orientation. This limitation is particularly pronounced for irregularly shaped targets, potentially reducing the flexibility and accuracy of target detection. Summary of the Invention

[0003] In view of this, embodiments of this application provide a high-precision remote sensing target detection method and system based on DETR, in order to overcome the above problems or at least partially solve the above problems.

[0004] The first aspect of this application provides a high-precision remote sensing target detection method based on DETR, the method comprising:

[0005] Obtain a training set, train the DETR model using the training set to obtain the object detection network, supervise point set learning using maximum projection loss, and supervise axis representation learning using cross-axis loss;

[0006] An image is subjected to target detection using a target detection network to obtain a target detection result. The image includes at least one target. The target detection result includes a point-axis representation of the target. The point-axis representation includes a set of points and a shape-independent axis. The set of points includes K predicted points, where the Kth predicted point is the predicted center point of the target. The shape-independent axis includes a first axis corresponding to four angles. The first axis corresponding to the four angles is obtained by discretizing the four angles and encoding them with Gaussian smoothness. The shape-independent axis is used to represent the main directionality of the target.

[0007] The target detection network processes the point set and the shape-independent axis to obtain the predicted bounding box of the target and the predicted category information of the target.

[0008] Optionally, the shape-independent axis includes the first axis corresponding to the four angles; the step of processing the point set and the shape-independent axis through the target detection network to obtain the predicted bounding box of the target includes:

[0009] A coordinate system is established with the Kth prediction point as the origin and the first axis corresponding to the four angles as the coordinate axes. The K prediction points included in the point set are projected onto the first axis corresponding to the four angles respectively.

[0010] Obtain the distances between the K predicted points included in the point set and the projection points of each point projected onto the first axis corresponding to the four angles;

[0011] The distances are filtered to obtain four target distances, which are the maximum distances generated when the first axes corresponding to the four angles are projected respectively.

[0012] Based on the Kth prediction point and the four target distances, a predicted bounding box for the target is generated.

[0013] Optionally, the object detection network includes: a feature extraction module, an encoder, an object-to-point query transformation module, a point detection decoder, and a prediction head. The pre-trained object detection network's processing of the image includes:

[0014] The feature extraction module extracts features from the image to obtain multiple feature patches.

[0015] The encoder encodes multiple feature map patches to generate multiple target queries;

[0016] Multiple target queries are filtered, and the N target queries are transformed by the target-to-point query conversion module to obtain N sets of point queries;

[0017] The point detection decoder decodes the multiple target queries and the N sets of point queries to obtain the decoded N sets of point queries;

[0018] The decoded N sets of point queries are processed by the first MLP layer in the prediction head to obtain the point set of N targets;

[0019] The decoded N sets of point queries are processed by the second MLP layer in the prediction head, the processing results are spliced, and the spliced ​​results are feature-projected by the MLP layer in the prediction head to obtain the shape-independent axes of the N targets.

[0020] The fourth MLP layer in the prediction head processes the decoded N sets of point queries, concatenates the processing results, and then the fifth MLP layer in the prediction head performs feature projection on the concatenated results to obtain the predicted category information of the N targets.

[0021] Optionally, a training set is obtained, and the DETR model is trained using the training set to obtain the object detection network:

[0022] Obtain a training set, which includes multiple training images. Each training image includes multiple training targets with attached annotation information, including the ground bounding boxes of the training targets and the ground class information of the training targets.

[0023] Extend the center point of the true bounding box to the edge of the true bounding box to determine the radial vector of the training target, and determine the axis of the training target through the radial vector, wherein the axis of the training target includes a second axis corresponding to four angles;

[0024] The training image is input into the target detection network to be trained to obtain the training target detection result, wherein the training target detection result includes the point axis representation of the training target and the prediction type information of the training target, and the point axis representation of the training target includes the prediction point set of the training target and the prediction axis of the training target;

[0025] Supervised point set learning using maximum projection loss includes: calculating point loss based on the predicted point set, the radial vector, and the center point of the ground bounding box;

[0026] The axis representation learning is supervised by cross-axis loss, including: calculating axis loss based on the second axis corresponding to the four angles and the predicted axis;

[0027] The category loss is calculated based on the predicted type information of the training target and the true category information;

[0028] The point-axis loss is obtained based on the point loss and the axis loss. The target detection network to be trained is updated based on the point-axis loss and the category loss until the target detection network training is completed.

[0029] Optionally, point set learning is supervised by maximum projection loss, and axis representation learning is supervised by cross-axis loss. The corresponding formulas for calculating point-axis losses are as follows:

[0030]

[0031] Wherein, N is the number of training targets, λ1 is the point loss coefficient, λ2 is the axis loss coefficient, and so on. For the point loss, the For the shaft loss, the For the predicted point set, the For the radial vector, the C i The center point of the true bounding box, the For the prediction axis, the The second axis corresponds to the four angles.

[0032] Optionally, the step of decoding the plurality of target queries and the N sets of point queries through the point detection decoder to obtain the decoded N sets of point queries includes:

[0033] The point detection decoder performs self-attention calculation on each group of point queries in the N groups of point queries, so that the point queries included in each group of point queries can interact with each other.

[0034] The point detection decoder performs self-attention calculation between the center point queries of each pair of point queries in the N sets of point queries, so that the center point queries included in each pair of point queries can interact.

[0035] The point detection decoder performs cross-attention calculation on the multiple target queries.

[0036] Optionally, a second aspect of this application provides a high-precision remote sensing target detection system based on DETR, the system comprising:

[0037] The training module is used to acquire the training set, train the DETR model using the training set to obtain the object detection network, supervise point set learning through maximum projection loss, and supervise axis representation learning through cross-axis loss.

[0038] An object detection module is used to perform object detection on an image through an object detection network to obtain object detection results. The image includes at least one object. The object detection results include a point-axis representation of the object. The point-axis representation includes a set of points and a shape-independent axis. The set of points includes K predicted points, where the Kth predicted point is the predicted center point of the object. The shape-independent axis includes a first axis corresponding to four angles. The first axis corresponding to the four angles is obtained by discretizing the four angles and encoding them with Gaussian smoothness. The shape-independent axis is used to represent the main directionality of the object.

[0039] The processing module is used to process the point set and the shape-independent axis through the target detection network to obtain the predicted bounding box of the target and the predicted category information of the target.

[0040] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the first aspect.

[0041] A fourth aspect of this application provides a computer-readable medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0042] A fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0043] The beneficial effects of this application are:

[0044] This application provides a high-precision remote sensing target detection method and system based on DETR. By acquiring a training set, a DETR model is trained using the training set to obtain a target detection network. Maximum projection loss is used to supervise point set learning, and cross-axis loss is used to supervise axis representation learning. The target detection network is used to detect targets in an image, obtaining a target detection result. The image includes at least one target. The target detection result includes a point-axis representation of the target. The point-axis representation includes a set of points and shape-independent axes. The point set includes K predicted points, where the Kth predicted point is the predicted center point of the target. The shape-independent axes include a first axis corresponding to four angles, obtained by discretizing the four angles and encoding them with Gaussian smoothing. The shape-independent axes represent the main orientation of the target. The target detection network processes the point set and the shape-independent axes to obtain the predicted bounding box of the target and the predicted category information of the target.

[0045] The DETR model can be trained using a training set to obtain an object detection network. Point set learning is supervised by maximum projection loss, and axis representation learning is supervised by cross-axis loss. Furthermore, the object detection network is used to detect objects in the image, including a set of points and point-axis representations that are independent of shape. The object detection results are then processed to obtain the predicted bounding boxes and predicted class information of the objects in the image, thus achieving both flexibility and accuracy in object detection. Attached Figure Description

[0046] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0047] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram illustrating a bounding box-oriented target representation method in related technologies;

[0049] Figure 2 This is a schematic diagram illustrating a point set-based target representation method in related technologies;

[0050] Figure 3 This is a flowchart of a high-precision remote sensing target detection method based on DETR provided in an embodiment of this application;

[0051] Figure 4 This is an architecture diagram of a high-precision remote sensing target detection method based on DETR provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram showing the point axis representation of a high-precision remote sensing target detection method based on DETR provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the real target detection result and the training target detection result of the training image in a high-precision remote sensing target detection method based on DETR provided in an embodiment of this application;

[0054] Figure 7 This is a schematic diagram of the framework of a high-precision remote sensing target detection system based on DETR provided in an embodiment of this application. Detailed Implementation

[0055] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] Compared to natural images, remote sensing images have characteristics such as complex backgrounds, fewer target pixels, and large scale variations. Based on these characteristics, if the target detection methods for natural images are directly applied to target detection in remote sensing images, the final detection results will be unsatisfactory.

[0058] In related technologies, remote sensing target detection methods are divided into two types based on the characteristics of target representation: bounding box-oriented methods and point set-based methods.

[0059] Figure 1 This is a schematic diagram illustrating a bounding box-oriented object detection method in related technologies, such as... Figure 1 As shown, bounding box-oriented methods typically represent targets in each image as (x, y, w, h, θ), where x and y represent the center point of the target, w and h represent the width and height of the target, and θ represents the target's orientation angle. However, bounding box-oriented methods often face the problem of "boundary interruption," which refers to the sudden switching of the target's angle θ due to the bounded periodicity of the orientation.

[0060] Specifically, when the lengths of the long and short sides of the bounding box are similar, the angle θ may abruptly switch between θ and θ+90° (for θ<0) or θ-90° (for θ>0). Furthermore, under certain angle definitions, the width w and length h may also abruptly interchange. These limitations make bounding box-oriented methods unstable and inefficient when dealing with target detection scenarios where the bounding box is square or nearly circular, resulting in insufficient accuracy for bounding box-oriented target representation methods.

[0061] Figure 2 This is a schematic diagram illustrating a point-set-based target detection method in related technologies. Point-set-based target representation methods determine the target's bounding box by calculating the minimum area rectangle surrounding the target's contour points. However, while these point-set-based methods excel at accurately capturing target locations, their performance is significantly affected by the quality and quantity of contour points. Furthermore, they neglect the target's primary orientation. This limitation is particularly pronounced for irregularly shaped targets, which may reduce the method's effectiveness, such as... Figure 2 As shown, the target (i.e. Figure 2 The direction of the aircraft (in the image) is not accurately indicated.

[0062] To overcome the limitations of the aforementioned methods, we introduce a high-precision remote sensing target detection method based on DETR. This method represents targets using point-axis representation (point set and axis), making it suitable for remote sensing target detection. Furthermore, this method avoids direct direction regression, instead employing a more flexible and geometrically intuitive representation based on point set and axis. We combine this target representation method with the DETR framework to form a high-precision remote sensing target detection method based on DETR.

[0063] Figure 3 This is a flowchart illustrating a high-precision remote sensing target detection method based on DETR, according to an embodiment of this application. Figure 4 This is an architecture diagram of a high-precision remote sensing target detection method based on DETR provided in an embodiment of this application, as shown below. Figure 3 and Figure 4 As shown, this embodiment provides a high-precision remote sensing target detection method based on DETR, the method comprising:

[0064] In step S101, a training set is obtained, and the DETR model is trained using the training set to obtain the object detection network. The point set learning is supervised by the maximum projection loss, and the axis representation learning is supervised by the cross-axis loss.

[0065] In step S102, target detection is performed on the image using a target detection network to obtain a target detection result. The image includes at least one target. The target detection result includes a point-axis representation of the target. The point-axis representation includes a set of points and a shape-independent axis. The set of points includes K predicted points, where the Kth predicted point is the predicted center point of the target. The shape-independent axis includes a first axis corresponding to four angles. The first axis corresponding to the four angles is obtained by discretizing the four angles and encoding them with Gaussian smoothness. The shape-independent axis is used to represent the main directionality of the target.

[0066] In step S103, the target detection network processes the point set and the shape-independent axis to obtain the predicted bounding box of the target and the predicted category information of the target.

[0067] First, in step S101, a training set is obtained, and the DETR model is trained using the training set to obtain the object detection network. The point set learning is supervised by the maximum projection loss, and the axis representation learning is supervised by the cross-axis loss.

[0068] In this embodiment, a training set containing training data, including image and response annotation information, is obtained. The obtained training set is used to train the DETR model, which is an object detection model based on the Transformer framework. Finally, an object detection network is trained, which can be used to detect and recognize target objects in images. During the training process, maximum projection loss is used to supervise the model to learn the position information of objects in the image so as to accurately detect objects. At the same time, cross-axis loss is also used to supervise the model to learn axis (category) representation so as to accurately classify the detected objects.

[0069] Further, in step S102, target detection is performed on the image through a target detection network to obtain a target detection result. The image includes at least one target. The target detection result includes a point-axis representation of the target. The point-axis representation includes a set of points and a shape-independent axis. The set of points includes K predicted points, and the Kth predicted point is the predicted center point of the target. The shape-independent axis includes a first axis corresponding to four angles. The first axis corresponding to the four angles is obtained by discretizing the four angles and encoding them with Gaussian smoothness. The shape-independent axis is used to represent the main directionality of the target.

[0070] In this embodiment, an image containing at least one target is input into a trained target detection network to obtain target detection results. The target detection network is trained based on the DETR model and outputs target detection results for each target contained in the image. Each target detection result includes a point-axis representation for the target. The point-axis representation includes a set of points and a shape-independent axis. The set of points includes K predicted points, each of which is a point related to the target, such as a point falling on the target boundary or contour, a point falling inside the target, or a boundary point or key point on the target boundary or contour. Key points are points where the angle between adjacent line segments contained in the target boundary changes significantly. The Kth predicted point is the predicted center point of the target, which is the geometric center point of the target. The shape-independent axis includes a first axis corresponding to four angles. The first axis corresponding to four angles is encoded by discretizing the four angles and applying Gaussian smoothing to form a four-peak label, representing the main directionality of the target.

[0071] In this embodiment, we propose a point-axis representation, where each oriented target i is represented by a set of points. and an axis independent of shape A combined description. The point set contains K points, where the Kth point is designated as the center point of the target.

[0072] Figure 5This is a schematic diagram of the point axis representation in a high-precision remote sensing target detection method based on DETR provided in an embodiment of this application, as shown below. Figure 5 As shown, the point-axis representation has the following characteristics:

[0073] 1. By using circular labels, we ensure the consistency of angles at the beginning and end of the defined range, thereby avoiding the following boundary problems: when the target's orientation is 360°, it can be regarded as 0°. If the regression is 1°, intuitively, 1° should have a smaller loss than 360°, but the numerical difference is relatively large. However, this application uses circular labels, which connect the beginning and end (360° is 0°), making the difference between 360° and 1° smaller, and equivalent to the loss of 359° compared to 1°.

[0074] 2. The definition of the axis is independent of the definitions of length and width, thereby avoiding ambiguity in the definition of the boundary and alleviating the following boundary problem: when dealing with target detection scenarios where the bounding box of the target is square or nearly circular, it becomes unstable and ineffective, resulting in insufficient accuracy of the bounding box-oriented target representation method.

[0075] Point-axis representation encapsulates positional details through point sets and conveys direction through axes, providing a precise depiction of oriented objects. Next, we will delve into the loss constraints associated with point-axis representation.

[0076] Finally, in step S103, the point set and the shape-independent axis are processed to obtain the predicted bounding box of the target and the predicted category information of the target. In this embodiment, the predicted bounding box of the target is obtained by combining the point set and the shape-independent axis in the output target detection result, and the predicted category information of each target in the image is predicted, whereby the predicted category information is the type of the target.

[0077] In this embodiment, a DETR model is trained using a training set to obtain an object detection network. Point set learning is supervised by maximum projection loss, and axis representation learning is supervised by cross-axis loss. The object detection network is used to detect objects in an image containing at least one object. The image is input into the pre-trained object detection network to obtain object detection results. These results include the point-axis representation of the object, which comprises a set of points and shape-independent axes. The point set includes K predicted points, with the Kth predicted point being the predicted center point of the object. The shape-independent axes include a first axis corresponding to four angles, obtained by discretizing the four angles and encoding them with Gaussian smoothing. The shape-independent axes represent the main directionality of the object. The object detection network is trained on the DETR model and processes the point set and the shape-independent axes to obtain the predicted bounding box and predicted category information of the object.

[0078] It can use a pre-trained object detection network based on the DETR model to detect objects in an image, which includes a set of points and point-axis representations of shape-independent axes. The object detection results can then be processed to obtain the predicted bounding boxes and predicted category information of the objects in the image, thus achieving both flexibility and accuracy in object detection.

[0079] Optionally, in the embodiments of this application, the DETR model can be replaced with the framework of other detection models.

[0080] Optionally, the shape-independent axis includes the first axis corresponding to the four angles; step S103 includes:

[0081] A coordinate system is established with the Kth prediction point as the origin and the first axis corresponding to the four angles as the coordinate axes. The K prediction points included in the point set are projected onto the first axis corresponding to the four angles respectively.

[0082] Obtain the distances between the K predicted points included in the point set and the projection points of each point projected onto the first axis corresponding to the four angles;

[0083] The distances are filtered to obtain four target distances, which are the maximum distances generated when the first axes corresponding to the four angles are projected respectively.

[0084] Based on the Kth prediction point and the four target distances, a predicted bounding box for the target is generated.

[0085] Specifically, in this embodiment of the application, the shape-independent axis includes the first axis corresponding to the four angles. Step S103 specifically includes: establishing a coordinate system with the Kth predicted point in the point set as the origin and the first axis corresponding to the four angles included in the shape-independent axis as the coordinate axes; projecting the K predicted points in the point set onto the first axis corresponding to the four angles respectively, that is, each predicted point will correspond to two projection points; obtaining the distance between the K predicted points in the point set and their respective projection points on the first axis corresponding to the four angles; each of the four axes included on the first axis corresponding to the four angles will have a distance from each predicted point; filtering the multiple distances obtained from the above projection to obtain four target distances; the target distance is the maximum distance generated when the first axis corresponding to the four angles is projected, that is, the distance generated by the point farthest from the axis will be taken as the target distance for each axis; generating the predicted bounding box of the target based on the Kth predicted point in the target's point set and the four target distances obtained.

[0086] Optionally, the object detection network includes: a feature extraction module, an encoder, an object-to-point query transformation module, a point detection decoder, and a prediction head. The pre-trained object detection network's processing of the image includes:

[0087] The feature extraction module extracts features from the image to obtain multiple feature patches.

[0088] The encoder encodes multiple feature map patches to generate multiple target queries;

[0089] Multiple target queries are filtered, and the N target queries are transformed by the target-to-point query conversion module to obtain N sets of point queries;

[0090] The point detection decoder decodes the multiple target queries and the N sets of point queries to obtain the decoded N sets of point queries;

[0091] The decoded N sets of point queries are processed by the first MLP layer in the prediction head to obtain the point set of N targets;

[0092] The decoded N sets of point queries are processed by the second MLP layer in the prediction head, the processing results are spliced, and the spliced ​​results are feature-projected by the MLP layer in the prediction head to obtain the shape-independent axes of the N targets.

[0093] The fourth MLP layer in the prediction head processes the decoded N sets of point queries, concatenates the processing results, and then the fifth MLP layer in the prediction head performs feature projection on the concatenated results to obtain the predicted category information of the N targets.

[0094] Specifically, in the embodiments of this application, such as Figure 4 As shown, the pre-trained object detection network includes: a feature extraction module, an encoder, an object-to-point query transformation module, a point detection decoder, and a prediction head. The image processing procedure using the pre-trained object detection network includes:

[0095] The feature extraction module extracts features from the image input to the object detection network. Given an image with three color channels (3×H0×W0), the module first extracts multi-scale feature patches. These patches are then fed into the encoder, which uses global context information to enhance the information in these multi-scale feature patches. This process encodes multiple feature patches, generating multiple target queries, which are then filtered (i.e.,...). Figure 4 The process involves selecting N target queries based on their effectiveness. These N queries are then input into a target-to-point query conversion module. This module transforms the target queries into point queries. Specifically, it includes a conversion function that converts the target queries into initial condition point queries. For the i-th target query... Its reference point is Convert it to a K-point query These point queries are based on target queries. generate.

[0096] It is worth emphasizing that for the query of the Kth point... It is defined as a center point query. Like most DETR (Detection Transformer) frameworks, we decompose the target query into location embedding and content embedding. For center point queries... Use MLP layers to predict relative to a reference point. The center point, and generate its position embedding.

[0097]

[0098] In this embodiment, each target corresponds to a target query, each target query corresponds to a set of point queries, a point set corresponds to a set of point queries, each point in the point set corresponds to a point query, and the center point corresponds to a center point query. After obtaining N sets of point queries corresponding to N target queries, the point detection decoder decodes the multiple target queries and the N sets of point queries to obtain the decoded N sets of point queries. Further, the decoded N sets of point queries are processed by the first MLP layer in the prediction head to obtain a point set of N targets. The second MLP layer in the prediction head processes the decoded N sets of point queries and concatenates the processing results. The MLP layer in the prediction head performs feature projection on the concatenated results to obtain N target axes independent of shape. The fourth MLP layer in the prediction head processes the decoded N sets of point queries and concatenates the processing results. The fifth MLP layer in the prediction head performs feature projection on the concatenated results to obtain the predicted category information of the N targets.

[0099] The following is the output of the prediction head:

[0100]

[0101] MLP1 is the first MLP layer, MLP2 is the second MLP layer, MLP3 is the third MLP layer, MLP4 is the fourth MLP layer, and MLP5 is the fifth MLP layer. The point set representing the target. For point query, Indicates an axis that is independent of shape. This indicates the predicted category information for the target.

[0102] Optionally, obtaining a training set and using the training set to train a DETR model to obtain an object detection network includes:

[0103] Obtain a training set, which includes multiple training images. Each training image includes multiple training targets with attached annotation information, including the ground bounding boxes of the training targets and the ground class information of the training targets.

[0104] Extend the center point of the true bounding box to the edge of the true bounding box to determine the radial vector of the training target, and determine the axis of the training target through the radial vector, wherein the axis of the training target includes a second axis corresponding to four angles;

[0105] The training image is input into the target detection network to be trained to obtain the training target detection result, wherein the training target detection result includes the point axis representation of the training target and the prediction type information of the training target, and the point axis representation of the training target includes the prediction point set of the training target and the prediction axis of the training target;

[0106] Supervised point set learning using maximum projection loss includes: calculating point loss based on the predicted point set, the radial vector, and the center point of the ground truth bounding box; calculating axis loss based on the second axis corresponding to the four angles and the predicted axis; and calculating category loss based on the predicted type information of the training target and the ground truth category information.

[0107] Supervising axis representation learning through cross-axis loss includes: obtaining point-axis loss based on the point loss and the axis loss; updating the target detection network to be trained based on the point-axis loss and the category loss until the target detection network is trained.

[0108] Figure 6 This is a schematic diagram of the real target detection result and the training target detection result of the training image in a high-precision remote sensing target detection method based on DETR provided in an embodiment of this application.

[0109] Specifically, in the embodiments of this application, a series of training sets are first collected. The training sets include multiple training images, and the training images include multiple training targets with attached annotation information. The annotation information is obtained by pre-annotating each training target included in the training images. The annotation information includes the true bounding box of each training target and the true category information of each training target. The true bounding box includes the coordinate information of the four vertices.

[0110] like Figure 6 As shown, constructing real object detection results for the training objective includes: converting the coordinate information of the four vertices of the real bounding box into the center point C of the real bounding box. i The coordinates of the center point C of the actual bounding box. i Extending to the edge of the true bounding box, we obtain the radial vector of the training target. (like Figure 6 , including V i 1 V i 2 V i 3 V i 4 And determine the axis of the training target through the radial vector. Among them, the axis of the training target This includes four second axes corresponding to different angles, each second axis corresponding to an angle in one direction. The training target detection network is then used to construct training target detection results. This includes: inputting the training image into the training target detection network to obtain training target detection results for the training targets included in the training image. The training target detection results include the point-axis representation of the training targets and the predicted type information of the training targets. The point-axis representation of the training targets includes the predicted point set of the training targets. The prediction axis of the training target

[0111] Furthermore, a point loss is calculated based on the predicted point set, radial vector, and center point of the ground truth bounding box. This point loss measures the accuracy of the predicted point set. An axis loss is calculated based on the second axis corresponding to the four angles and the predicted axis. This axis loss measures the accuracy of the predicted axis. A category loss is calculated based on the predicted type information and the ground truth category information of the training target. The results of the point loss and axis loss constitute the point-axis loss. The parameters of the target detection network to be trained are updated using the point-axis loss and category loss. Constraints are then enforced on the predictions of the target detection network to be trained using the maximum projection loss and cross-axis loss. After updating the parameters of the target detection network to be trained, the above training steps are repeated until the target detection network to be trained is completed.

[0112] Optionally, the point set learning is supervised by the maximum projection loss, and the axis representation learning is supervised by the cross-axis loss. The corresponding formula for calculating the point-axis loss is as follows:

[0113]

[0114] Wherein, N is the number of training targets, λ1 is the point loss coefficient, λ2 is the axis loss coefficient, and so on. For the point loss, the For the shaft loss, the For the predicted point set, the v i For the radial vector, the C i The center point of the true bounding box, the For the prediction axis, the The second axis corresponds to the four angles.

[0115] Specifically, in the embodiments of this application, point axis loss The calculation formula is:

[0116]

[0117] At the aforementioned point axis loss The calculation formula includes two parts: point loss. and shaft loss Point loss and shaft loss These are all for a single training objective; point loss. and shaft loss By summing the coefficients of each training target, the point-axis loss for a single training target is obtained. Then, the average point-axis loss for each training target included in the training image is calculated to obtain the point-axis loss obtained from training with the training images.

[0118] Optionally, the step of decoding the plurality of target queries and the N sets of point queries through the point detection decoder to obtain the decoded N sets of point queries includes:

[0119] The point detection decoder performs self-attention calculation on each group of point queries in the N groups of point queries, so that the point queries included in each group of point queries can interact with each other.

[0120] The point detection decoder performs self-attention calculation between the center point queries of each pair of point queries in the N sets of point queries, so that the center point queries included in each pair of point queries can interact.

[0121] The point detection decoder performs cross-attention calculation on the multiple target queries.

[0122] Specifically, in the DETR model, traditional decoders play a crucial role in facilitating information interaction between target queries. Decoders primarily include self-attention to promote interaction between target queries and cross-attention to promote interaction between target queries and image features. In this embodiment, the decoder is designed to obtain a point detection decoder applied to point queries. This point detection decoder includes not only attention calculations between target queries but also attention calculations between point queries within a group and between groups.

[0123] For N target queries corresponding to N sets of point queries, a parameter-shared self-attention layer is applied within each set to capture the relationship between point queries. The point detection decoder performs self-attention calculations between point queries within each set of N point queries output by the target-to-point query conversion module. This means that each point query in each set participates in the self-attention calculation, enabling interaction between the point queries included in each set. Furthermore, for every two sets of point queries in the N sets, self-attention calculations are performed between the center point queries of each set. The center point of each set is extracted to form a new set, and the center point of each set performs self-attention calculations with the center points of other sets, enabling interaction between the center point queries included in each pair of sets to capture the relationship between target queries. After performing group self-attention (intra-group self-attention + inter-group self-attention) calculations, cross-attention calculations are performed between target queries. This includes further processing of all set point queries through a deformable cross-attention module and an FFN (Feed Forward Network) module.

[0124] Figure 7 This is a schematic diagram of the framework of a high-precision remote sensing target detection system based on DETR according to an embodiment of this application, as shown below. Figure 7 As shown in the figure, this embodiment provides a high-precision remote sensing target detection system based on DETR, the system comprising:

[0125] Training module 11 is used to acquire a training set, train the DETR model using the training set to obtain the object detection network, supervise point set learning through maximum projection loss, and supervise axis representation learning through cross axis loss.

[0126] The target detection module 12 is used to perform target detection on an image through a target detection network to obtain a target detection result. The image includes at least one target. The target detection result includes a point-axis representation of the target. The point-axis representation includes a set of points and a shape-independent axis. The set of points includes K predicted points, where the Kth predicted point is the predicted center point of the target. The shape-independent axis includes a first axis corresponding to four angles. The first axis corresponding to the four angles is obtained by discretizing the four angles and encoding them with Gaussian smoothness. The shape-independent axis is used to represent the main directionality of the target.

[0127] The processing module 13 is used to process the point set and the shape-independent axis through the target detection network to obtain the predicted bounding box of the target and the predicted category information of the target.

[0128] Optionally, the shape-independent axis includes the first axis corresponding to the four angles; the processing module 13 includes:

[0129] The projection module is used to establish a coordinate system with the Kth prediction point as the origin and the first axis corresponding to the four angles as the coordinate axes, and to project the K prediction points included in the point set onto the first axis corresponding to the four angles respectively.

[0130] The distance acquisition module is used to acquire the distance between the K predicted points included in the point set and the projection points of each point projected onto the first axis corresponding to the four angles.

[0131] A filtering module is used to filter the distances to obtain four target distances, wherein the target distances are the maximum distances generated when the first axes corresponding to the four angles are projected respectively;

[0132] The predicted bounding box generation module is used to generate the predicted bounding box of the target based on the Kth predicted point and the four target distances.

[0133] Optionally, the pre-trained object detection network includes: a feature extraction module, an encoder, an object-to-point query transformation module, a point detection decoder, and a prediction head; the system further includes:

[0134] The feature extraction module is used to extract features from the image to obtain multiple feature patches;

[0135] The encoding module is used to encode multiple feature map patches through the encoder to generate multiple target queries;

[0136] The target query filtering module is used to filter multiple target queries. The target-to-point query conversion module converts the filtered N target queries to obtain N sets of point queries.

[0137] The decoding module is used to decode the plurality of target queries and the N sets of point queries through the point detection decoder to obtain the decoded N sets of point queries;

[0138] The point query processing module is used to process the decoded N sets of point queries through the first MLP layer in the prediction head to obtain the point set of N targets;

[0139] The shape-independent axis acquisition module is used to process the decoded N sets of point queries through the second MLP layer in the prediction head, stitch the processing results together, and perform feature projection on the stitched results through the MLP layer in the prediction head to obtain the shape-independent axes of the N targets.

[0140] The prediction category information acquisition module is used to process the decoded N sets of point queries through the fourth MLP layer in the prediction head, concatenate the processing results, and perform feature projection on the concatenated results through the fifth MLP layer in the prediction head to obtain the prediction category information of the N targets.

[0141] Optionally, the training module 11 includes:

[0142] A training set acquisition module is used to acquire a training set, which includes multiple training images. The training images include multiple training targets with attached annotation information. The annotation information includes the ground bounding boxes of the training targets and the ground category information of the training targets.

[0143] The determination module is used to extend the center point of the real bounding box to the edge of the real bounding box to determine the radial vector of the training target, and determine the axis of the training target through the radial vector, wherein the axis of the training target includes a second axis corresponding to four angles;

[0144] A training unit is used to input the training image into the target detection network to be trained to obtain the training target detection result, wherein the training target detection result includes the point axis representation of the training target and the prediction type information of the training target, and the point axis representation of the training target includes the prediction point set of the training target and the prediction axis of the training target;

[0145] The loss calculation module is used to calculate point loss based on the predicted point set, the radial vector and the center point of the true bounding box, calculate axis loss based on the second axis corresponding to the four angles and the predicted axis, and calculate category loss based on the prediction type information of the training target and the true category information.

[0146] An update module is used to obtain a point-axis loss based on the point loss and the axis loss, and update the target detection network to be trained based on the point-axis loss and the category loss until the target detection network training is completed.

[0147] Optionally, the formula for calculating the point axis loss is:

[0148]

[0149] Wherein, N is the number of training targets, λ1 is the point loss coefficient, λ2 is the axis loss coefficient, and so on. For the point loss, the For the shaft loss, the For the predicted point set, the For the radial vector, the C iThe center point of the true bounding box, the For the prediction axis, the The second axis corresponds to the four angles.

[0150] Optionally, the shape-independent axis acquisition module includes:

[0151] The intra-group self-attention calculation module is used to perform self-attention calculation between point queries within each of the N groups of point queries through the point detection decoder, so that the point queries included in each group of point queries can interact with each other.

[0152] The inter-group self-attention calculation module is used to perform self-attention calculation between the center point queries of each pair of point queries in the N groups of point queries through the point detection decoder, so that the center point queries included in each pair of point queries can interact.

[0153] The cross-attention calculation module is used to perform cross-attention calculation on the multiple target queries through the point detection decoder.

[0154] Based on the same inventive concept, another embodiment of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the high-precision remote sensing target detection method based on DETR as described in any of the above embodiments.

[0155] Based on the same inventive concept, another embodiment of this application also provides a computer program product, including a computer program that is executed by a processor as described in any of the above embodiments, the high-precision remote sensing target detection method based on DETR.

[0156] Based on the same inventive concept, another embodiment of this application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, it implements the high-precision remote sensing target detection method based on DETR as described in any of the above embodiments.

[0157] As the system is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment.

[0158] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0164] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0165] The above provides a detailed description of a high-precision remote sensing target detection method and system based on DETR provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A high-precision remote sensing target detection method based on DETR, characterized in that, The method includes: Obtain a training set, train the DETR model using the training set to obtain the object detection network, supervise point set learning using maximum projection loss, and supervise axis representation learning using cross-axis loss; An image is subjected to target detection using a target detection network to obtain a target detection result. The image includes at least one target. The target detection result includes a point-axis representation of the target. The point-axis representation includes a set of points and a shape-independent axis. The set of points includes K predicted points, where the Kth predicted point is the predicted center point of the target. The shape-independent axis includes a first axis corresponding to four angles. The first axis corresponding to the four angles is obtained by discretizing the four angles and encoding them with Gaussian smoothness. The shape-independent axis is used to represent the main directionality of the target. The target detection network processes the point set and the shape-independent axis to obtain the predicted bounding box of the target and the predicted category information of the target. The target detection network includes: a feature extraction module, an encoder, a target-to-point query transformation module, a point detection decoder, and a prediction head. The pre-trained target detection network processes the image as follows: The feature extraction module extracts features from the image to obtain multiple feature patches. The encoder encodes multiple feature map patches to generate multiple target queries; Multiple target queries are filtered, and the N target queries are transformed by the target-to-point query conversion module to obtain N sets of point queries; The point detection decoder decodes the multiple target queries and the N sets of point queries to obtain the decoded N sets of point queries; The decoded N sets of point queries are processed by the first MLP layer in the prediction head to obtain the point set of N targets; The decoded N sets of point queries are processed by the second MLP layer in the prediction head, the processing results are spliced, and the spliced ​​results are feature-projected by the MLP layer in the prediction head to obtain the shape-independent axes of the N targets. The N sets of point queries after decoding are processed by the fourth MLP layer in the prediction head, and the processing results are concatenated. The concatenated results are then subjected to feature projection by the fifth MLP layer in the prediction head to obtain the predicted category information of the N targets. The process of obtaining a training set and using the training set to train a DETR model to obtain an object detection network includes: Obtain a training set, which includes multiple training images. Each training image includes multiple training targets with attached annotation information, including the ground bounding boxes of the training targets and the ground class information of the training targets. Extend the center point of the true bounding box to the edge of the true bounding box to determine the radial vector of the training target, and determine the axis of the training target through the radial vector, wherein the axis of the training target includes a second axis corresponding to four angles; The training image is input into the target detection network to be trained to obtain the training target detection result, wherein the training target detection result includes the point axis representation of the training target and the prediction type information of the training target, and the point axis representation of the training target includes the prediction point set of the training target and the prediction axis of the training target; Supervised point set learning using maximum projection loss includes: calculating point loss based on the predicted point set, the radial vector, and the center point of the ground bounding box; The axis representation learning is supervised by cross-axis loss, including: calculating axis loss based on the second axis corresponding to the four angles and the predicted axis; The category loss is calculated based on the predicted type information of the training target and the true category information; The point-axis loss is obtained based on the point loss and the axis loss. The target detection network to be trained is updated based on the point-axis loss and the category loss until the target detection network training is completed.

2. The DETR-based high-precision remote sensing target detection method according to claim 1, characterized in that, The shape-independent axis includes the first axis corresponding to the four angles; The step of processing the point set and the shape-independent axes through the target detection network to obtain the predicted bounding box of the target includes: A coordinate system is established with the Kth prediction point as the origin and the first axis corresponding to the four angles as the coordinate axes. The K prediction points included in the point set are projected onto the first axis corresponding to the four angles respectively. Obtain the distances between the K predicted points included in the point set and the projection points of each point projected onto the first axis corresponding to the four angles; The distances are filtered to obtain four target distances, which are the maximum distances generated when the first axes corresponding to the four angles are projected respectively. Based on the Kth prediction point and the four target distances, a predicted bounding box for the target is generated.

3. The DETR-based high-precision remote sensing target detection method according to claim 2, characterized in that, Point set learning is supervised by maximum projection loss, and axis representation learning is supervised by cross-axis loss. The corresponding formulas for calculating the point-axis loss are as follows: Among them, the The number of training targets, the The point loss coefficient, the The shaft loss coefficient, the For the point loss, the For the shaft loss, the For the predicted point set, the For the radial vector, the The center point of the true bounding box, the For the prediction axis, the The second axis corresponds to the four angles.

4. The DETR-based high-precision remote sensing target detection method according to claim 1, characterized in that, The step of decoding the multiple target queries and the N sets of point queries using the point detection decoder to obtain the decoded N sets of point queries includes: The point detection decoder performs self-attention calculation on each group of point queries in the N groups of point queries, so that the point queries included in each group of point queries can interact with each other. The point detection decoder performs self-attention calculation between the center point queries of each pair of point queries in the N sets of point queries, so that the center point queries included in each pair of point queries can interact. The point detection decoder performs cross-attention calculation on the multiple target queries.

5. A high-precision remote sensing target detection system based on DETR, characterized in that, The system includes: The training module is used to acquire the training set, train the DETR model using the training set to obtain the object detection network, supervise point set learning through maximum projection loss, and supervise axis representation learning through cross-axis loss. An object detection module is used to perform object detection on an image through an object detection network to obtain object detection results. The image includes at least one object. The object detection results include a point-axis representation of the object. The point-axis representation includes a set of points and a shape-independent axis. The set of points includes K predicted points, where the Kth predicted point is the predicted center point of the object. The shape-independent axis includes a first axis corresponding to four angles. The first axis corresponding to the four angles is obtained by discretizing the four angles and encoding them with Gaussian smoothness. The shape-independent axis is used to represent the main directionality of the object. The processing module is used to process the point set and the shape-independent axis through the target detection network to obtain the predicted bounding box of the target and the predicted category information of the target; The target detection network includes: a feature extraction module, an encoder, a target-to-point query conversion module, a point detection decoder, and a prediction head. The system also includes: The feature extraction module is used to extract features from the image to obtain multiple feature patches; The encoding module is used to encode multiple feature map patches through the encoder to generate multiple target queries; The target query filtering module is used to filter multiple target queries. The target-to-point query conversion module converts the filtered N target queries to obtain N sets of point queries. The decoding module is used to decode the plurality of target queries and the N sets of point queries through the point detection decoder to obtain the decoded N sets of point queries; The point query processing module is used to process the decoded N sets of point queries through the first MLP layer in the prediction head to obtain the point set of N targets; The shape-independent axis acquisition module is used to process the decoded N sets of point queries through the second MLP layer in the prediction head, stitch the processing results together, and perform feature projection on the stitched results through the MLP layer in the prediction head to obtain the shape-independent axes of the N targets. The prediction category information acquisition module is used to process the decoded N sets of point queries through the fourth MLP layer in the prediction head, concatenate the processing results, and perform feature projection on the concatenated results through the fifth MLP layer in the prediction head to obtain the prediction category information of the N targets. The training module includes: A training set acquisition module is used to acquire a training set, which includes multiple training images. The training images include multiple training targets with attached annotation information. The annotation information includes the ground bounding boxes of the training targets and the ground category information of the training targets. The determination module is used to extend the center point of the real bounding box to the edge of the real bounding box to determine the radial vector of the training target, and determine the axis of the training target through the radial vector, wherein the axis of the training target includes a second axis corresponding to four angles; A training unit is used to input the training image into the target detection network to be trained to obtain the training target detection result, wherein the training target detection result includes the point axis representation of the training target and the prediction type information of the training target, and the point axis representation of the training target includes the prediction point set of the training target and the prediction axis of the training target; The loss calculation module is used to calculate point loss based on the predicted point set, the radial vector and the center point of the true bounding box, calculate axis loss based on the second axis corresponding to the four angles and the predicted axis, and calculate category loss based on the prediction type information of the training target and the true category information. An update module is used to obtain a point-axis loss based on the point loss and the axis loss, and update the target detection network to be trained based on the point-axis loss and the category loss until the target detection network training is completed.

6. An electronic device, comprising: It includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the high-precision remote sensing target detection method based on DETR as described in any one of claims 1-4.

7. A computer readable medium characterized by It stores a computer program, wherein the computer program, when executed by a processor, implements the high-precision remote sensing target detection method based on DETR as described in any one of claims 1-4.

8. A computer program product comprising a computer program, characterized in that, When executed by a processor, the computer program implements the high-precision remote sensing target detection method based on DETR as described in any one of claims 1-4.

Images