Small target detection method and device for remote sensing image, equipment and storage medium

Abstract

The invention provides a small target detection method and device for a remote sensing image, equipment and a storage medium, and relates to the field of remote sensing image detection. The method for detecting the small target of the remote sensing image comprises the following steps: constructing a target detection model; training the target detection model; respectively obtaining a coarse detection result and a fine detection result of the to-be-detected image through a trained target detection model; and combining the coarse detection result and the fine detection result to complete the detection of the small target in the to-be-detected image. According to the embodiment of the invention, the detection performance of the detection model on dense small targets can be improved.

Description

Technical Field

[0001] This application relates to the field of remote sensing image detection, and more specifically, to a method, apparatus, device, and storage medium for detecting small targets in remote sensing images. Background Technology

[0002] Currently, deep learning technology has become the foundation for widespread applications in scenarios such as autonomous driving and robotics. However, the detection of dense small targets in remote sensing images from drones and satellites still faces challenges. Compared to targets at normal scale, the detection performance of small targets is significantly lower, mainly due to two reasons: 1) The scaling operation of the original image by the neural network model leads to the loss of features of small targets; 2) The downsampling operation in the neural network model weakens the features of small targets, making them more susceptible to background noise.

[0003] To improve the detection performance of small targets and dense regions, methods such as increasing the size of the input image or reducing the downsampling rate of the CNN are generally used to maintain high-resolution feature maps. However, simply increasing the resolution of the feature maps will result in huge computational costs.

[0004] Furthermore, to fully utilize the high-resolution features of remote sensing images and reduce information loss, a method can be adopted that crops the input image into uniform blocks (patches) and then performs target detection on the high-resolution patches through upsampling. However, targets in remote sensing images are usually sparse and uneven, and a large amount of irrelevant background information leads to a lot of redundant computation in image interpretation tasks. Therefore, although this method helps improve accuracy, the cropping operation introduces additional time overhead and is not suitable for scenarios with high real-time requirements.

[0005] How to detect small targets and densely populated areas of small targets while maintaining high detection accuracy and controlling computational costs remains a challenging problem. Summary of the Invention

[0006] According to one aspect of this application, a method for detecting small targets in a remote sensing image is provided, comprising: constructing a target detection model; training the target detection model; obtaining coarse detection results and fine detection results of the image to be detected through the trained target detection model; and merging the coarse detection results and fine detection results to complete the detection of small targets in the image to be detected.

[0007] According to some embodiments, constructing an object detection model includes: obtaining a preset basic detection model, which includes a backbone network, a classification head network, and a location regression head network; setting parameters for the preset basic detection model to output feature maps through the backbone network; constructing a query head network, which is parallel to the classification head network and the location regression head network; and obtaining an object detection model based on the preset basic detection model and the query head network.

[0008] According to some embodiments, the backbone network has multiple layers, each of which can output a feature map. The higher the number of layers in the backbone network, the lower the resolution of the feature map output by that layer. Training the object detection model includes: training a preset base detection model on low-resolution feature maps; and training the preset base detection model and query head network on high-resolution feature maps.

[0009] According to some embodiments, training an object detection model includes: obtaining annotation information of small objects and densely occluded small objects on a feature map; based on the annotation information, obtaining the mask and the pixel value of the mask corresponding to the feature map to determine the region in the feature map to perform sparse convolution calculation; and calculating the loss of the object detection model through a loss function based on the pixel value of the mask to complete the training of the object detection model.

[0010] According to some embodiments, obtaining annotation information of small targets and densely occluded small targets on a feature map includes: calculating the target scale of small targets on the feature map to filter small targets; calculating the intersection-union ratio between each pair of target boxes of all targets on the feature map to filter densely occluded small targets; and generating annotation information of the filtered small targets and densely occluded small targets for use in training the query head network.

[0011] According to some embodiments, based on annotation information, obtaining the mask corresponding to the feature map and the pixel value of the mask includes: obtaining the mask corresponding to the feature map; obtaining a target set composed of small targets and densely occluded small targets based on annotation information; calculating the minimum distance between each location point on the feature map and the center point of the target in the target set and the target scale corresponding to the minimum distance; and determining the pixel value of the mask based on the minimum distance and the target scale corresponding to the minimum distance.

[0012] According to some embodiments, a coarse detection result and a fine detection result of the image to be detected are obtained through a trained object detection model, including: obtaining the feature map corresponding to the image to be detected through the backbone network of the object detection model; obtaining the coarse detection result based on the low-resolution feature map through the classification head network and the position regression head network of the object detection model; performing sparse convolution calculation on the high-resolution feature map through the query head network of the object detection model; and obtaining the fine detection result on the high-resolution feature map through the classification head network and the position regression head network based on the result of the sparse convolution calculation.

[0013] According to some embodiments, sparse convolution computation is performed on a high-resolution feature map by a query head network of an object detection model, including: obtaining the pixel values ​​of the mask corresponding to the low-resolution feature map through the query head network based on the low-resolution feature map; querying the position points on the low-resolution feature map that satisfy preset rules based on the pixel values ​​of the mask corresponding to the low-resolution feature map; mapping the position points to the corresponding positions in the high-resolution feature map to obtain the point set of the corresponding positions in the high-resolution feature map; obtaining a sparse tensor set based on the point set of the corresponding positions; and performing sparse convolution computation based on the sparse tensor set.

[0014] According to some embodiments, merging coarse detection results and fine detection results to complete the detection of small targets in the image to be detected includes: merging coarse detection results and fine detection results to filter candidate boxes of small targets; and performing suppression filtering on the candidate boxes of small targets obtained after filtering.

[0015] According to one aspect of this application, a device for detecting small targets in remote sensing images is provided, comprising: a model building module for building a target detection model; a model training module for training the target detection model; a prediction module for obtaining coarse detection results and fine detection results of the image to be detected through the trained target detection model; and a data processing module for merging the coarse detection results and fine detection results to complete the detection of small targets in the image to be detected.

[0016] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the method as described above.

[0017] According to one aspect of this application, a computer-readable storage medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0018] According to the embodiments of this application, the technical solution of this application roughly locates the target position on the low-resolution feature map and performs fine detection on the dense small targets on the high-resolution feature map. At the same time, it introduces a query head network and sparse convolution to save computation, thereby achieving fast inference and high-precision target detection, saving computing resources, improving query efficiency, and having wide applicability and accuracy.

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application.

[0021] Figure 1 This paper illustrates a method for detecting small targets in a remote sensing image according to an example embodiment of this application.

[0022] Figure 2 A flowchart illustrating the determination of the sparse convolution computation region according to an example embodiment of this application is shown.

[0023] Figure 3 A block diagram of a device for detecting small targets in a remotely sensed image according to an example embodiment of this application is shown.

[0024] Figure 4 A block diagram of an electronic device according to an example embodiment of this application is shown. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0026] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.

[0027] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0028] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0029] This application provides a method, apparatus, device, and storage medium for small target detection in remote sensing images, which can realize rapid inference of the detection model and high-precision target detection.

[0030] The following will describe in detail, with reference to the accompanying drawings, a method, apparatus, device, and storage medium for detecting small targets in remote sensing images according to embodiments of this application.

[0031] This application uses the following terms:

[0032] FPN (Feature Pyramid Network): It achieves feature fusion and enhancement by constructing a top-down feature pyramid structure, aiming to extract features from different scales, thereby better detecting targets of different sizes.

[0033] NMS (Non-Maximum Suppression): A commonly used technique in object detection to select the best box from multiple overlapping candidate boxes (or detected object boxes).

[0034] IoU (Intersection over Union): In object detection, this metric is used to evaluate the performance of detection algorithms, representing the degree of overlap between the predicted bounding box and the ground truth bounding box. The IoU formula is to divide the intersection of the predicted and ground truth bounding boxes by the total area of ​​their combined area (the union).

[0035] Mask: A concept in computer vision and image processing, typically a matrix or image of the same size as the original image, used to indicate or identify specific pixels or regions in the original image. Masks can be binary (i.e., containing only 0 or 1 values), where a pixel value of 1 indicates that the corresponding pixel in the original image belongs to a specific region or target, while a pixel value of 0 indicates that it does not belong to that region or target.

[0036] Figure 1 This paper illustrates a method for detecting small targets in a remote sensing image according to an example embodiment of this application.

[0037] like Figure 1 As shown, in step S100, the detection device constructs a target detection model.

[0038] For example, in step S100, the detection device constructs a target detection model based on a preset basic detection model.

[0039] In response to the operation command, the detection device first acquires the preset basic detection model.

[0040] According to some embodiments, the preset base detection model can be a RetinaNet object detection model, which includes a backbone network with an FPN layer structure that can output multi-scale feature maps, as well as a classification head network and a location regression head network.

[0041] In response to the operation command, the detection device sets the model parameters in the configuration file of the preset basic detection model so that the feature maps output by each layer in the backbone network have different resolutions.

[0042] For example, assuming the input image size of the preset base detection model is H*W and the number of image channels is C, then the feature map output by the multi-layer backbone network with FPN layer structure... The feature map size H'*W' output from layer l satisfies Among them, operators The value indicates rounding down, and l represents the number of layers in the backbone network. The larger l is, the higher the number of layers, and the lower the resolution of the output feature map.

[0043] For example, the backbone network outputs feature maps of multiple layers, numbered P2 to P7. The resolution of feature map P2 is equivalent to 1 / 4 of the input image resolution, P3 to 1 / 8, P4 to 1 / 16, and so on. The higher the number of layers in the backbone network, the lower the resolution of its output feature maps.

[0044] According to some embodiments, feature maps with a resolution less than or equal to 1 / 32 of the input image resolution can be regarded as low-resolution feature maps, and feature maps with a resolution greater than 1 / 32 of the input image resolution can be regarded as high-resolution feature maps.

[0045] In response to the operation command, the detection device constructs a query header network to query the target location.

[0046] According to some embodiments, the query head network may consist of four 3x3 convolutional layers, with one 3x3 convolutional layer used for target location prediction. Each feature map output by the backbone network corresponds to a specific layer in the query head network.

[0047] According to some embodiments, the query for the target location can be performed in a cascaded manner, that is, in the low-resolution P output of the backbone network... l Layer feature maps predict target location at high resolution P l-1 Sparse convolution is performed at the corresponding positions of the layer feature maps.

[0048] In response to the operation command, the detection device adds a query header network to the structure of the preset basic detection model to form a target detection model.

[0049] According to some embodiments, the query head network is parallel to the classification head network and the location regression head network, that is, the input data of the query head network is the same as the input data of the classification head network and the location regression head network.

[0050] In step S200, the detection device trains the target detection model.

[0051] For example, in step S200, the detection device trains the target detection model on feature maps of different resolutions output by the backbone network of the preset basic detection model.

[0052] In response to the operation command, the detection device inputs the training image into the backbone network of the preset basic detection model to obtain multi-layer feature maps of different resolutions output by the backbone network.

[0053] According to some embodiments, the detection device trains a pre-defined base detection model (including a classification head network and a location regression head network) on the low-resolution feature map of the high-level output of the backbone network.

[0054] According to some embodiments, the detection device trains a pre-defined base detection model (including a classification head network and a location regression head network) and a query head network on the high-resolution feature maps of the low-level outputs of the backbone network.

[0055] The detection device filters all targets on the feature map to obtain small targets and dense areas of small targets with occlusion.

[0056] According to some embodiments, the detection device filters small targets by calculating the target scale of the small target on the feature map, and filters densely occluded small targets by calculating the cross-union ratio between each pair of the target boxes of all targets on the feature map.

[0057] Based on the selected small targets and densely occluded small targets, the detection device generates corresponding annotation information for use in training the head network.

[0058] The detection device acquires the mask corresponding to the feature map and determines the pixel value of the mask based on the annotation information of small targets and densely occluded small targets, so as to determine the region in the feature map to perform sparse convolution calculation.

[0059] Based on the pixel values ​​of the mask, the detection device calculates the losses of the classification head network, the location regression head network, and the query head network in the target detection model using different loss functions, and obtains the final loss of the target detection model.

[0060] According to some embodiments, the loss function for the classification head network and query head network can be the Focal Loss function, and the loss function for the location regression head network can be a specific loss function. Furthermore, the loss function for each layer of the feature map output by the object detection model can be expressed by the following formula:

[0061]

[0062] in, and The loss functions corresponding to the classification head network, the location regression head network, and the query head network are C, respectively. l R l and V l The output score for the corresponding branch. This is the annotation truth information, where The pixel values ​​are those of the generated mask; layers without a query header network will not have them. loss.

[0063] The detection device sums the losses of all layers of the feature map output by the target detection model to obtain the final loss F of the target detection model. all It can be expressed by the following formula.

[0064] F all =a l F l (2)

[0065] Among them, a l These are the weight coefficients for each layer, used to balance the weight of each layer's loss in the total loss during training.

[0066] According to some embodiments, after the target detection model is trained, the detection device saves the parameters corresponding to the target detection model, including the loss weight parameters of the target detection model.

[0067] In step S300, the detection device obtains coarse detection results and fine detection results of the image to be detected through the trained target detection model.

[0068] For example, in step S300, the detection device inputs the image to be detected into the trained target detection model and obtains the coarse detection result and the fine detection result of the image to be detected, respectively.

[0069] The detection device inputs the image to be detected into a trained target detection model, and outputs feature maps corresponding to multiple layers of the image to be detected at different resolutions through the backbone network of the target detection model.

[0070] On low-resolution feature maps, the detection device obtains target category and coarse location information on the feature map through the classification head network and location regression head network of the target detection model to generate coarse detection results.

[0071] On the high-resolution feature map, based on the query head network of the object detection model, the detection device determines the corresponding position for sparse convolution calculation in the high-resolution feature map.

[0072] According to some embodiments, the detection device will detect low-resolution feature maps P l Input the query header network to obtain the corresponding mask V through the query header network. l The pixel values ​​are used to query the low-resolution feature map P. l The position points that meet the preset rules.

[0073] According to some embodiments, the preset rule can be set to a low-resolution feature map P. l The predicted scores are the positions where the pixel value of the mask is greater than a preset first threshold σ. Positions with a predicted score greater than the first threshold σ have a pixel value of 1, while positions with a predicted score less than the first threshold σ have a pixel value of 0.

[0074] According to some embodiments, the detection device will detect low-resolution feature maps P l Mapping points that satisfy preset rules to a high-resolution feature map P l-1 Above, to obtain a high-resolution feature map P l-1 The corresponding point set is used to generate a sparse tensor set.

[0075] According to some embodiments, the low-resolution feature map P l and high-resolution feature map P l-1 Hl-1 =2H l W l-1 =2W l The relationship, in which H l 、W l For P l The width and height of the layer, H l-1 、W l-1 For P l-1 Width and height of the layer.

[0076] According to some embodiments, the low-resolution feature map P l Position points that meet the preset rules Mapped to a high-resolution feature map P l-1 Location query point set k p l-1 The following relationship is satisfied:

[0077]

[0078] Furthermore, the detection device uses high-resolution feature maps P l-1 Location query point set k p l-1 Query and obtain the feature values ​​at the corresponding positions to construct a sparse tensor set.

[0079] The detection device performs sparse convolution calculations based on the sparse tensor sets corresponding to the corresponding positions in the high-resolution feature map.

[0080] According to some embodiments, the detection device initializes the sparse convolution kernel parameters using a preset sparse convolution tool library. The parameters used for initialization are the parameters of the already trained classification head network and location regression head network.

[0081] Based on the sparse convolution calculation results, the detection device obtains the prediction results of the high-resolution feature map, which serves as the accurate detection result.

[0082] In step S400, the detection device combines the coarse detection results and the fine detection results to complete the detection of small targets in the image to be detected.

[0083] For example, in step S400, the detection device merges the coarse detection result and the fine detection result of the image to be detected obtained by the target detection model to complete the detection of small targets in the image to be detected.

[0084] After obtaining the coarse and fine detection results of the image to be detected through the trained target detection model, the detection device merges the coarse and fine detection results to filter out candidate boxes of small targets with high confidence.

[0085] The detection device performs suppression filtering on the candidate boxes of small targets obtained after merging the coarse detection results and the fine detection results, so as to suppress duplicate candidate boxes and complete the detection and recognition of small targets in the image to be detected.

[0086] According to some embodiments, NMS can be used to filter candidate boxes.

[0087] According to the embodiments of this application, the technical solution of this application saves the computational load of the target detection model by introducing a query header network and sparse convolution calculation, thereby achieving fast inference and high-precision target detection and improving query efficiency.

[0088] Figure 2 A flowchart illustrating the determination of the sparse convolution computation region according to an example embodiment of this application is shown.

[0089] like Figure 2 As shown, step S200 also includes steps S210, S220 and S230.

[0090] In step S210, the detection device filters out small targets on the feature map and obtains the corresponding annotation information.

[0091] The detection device acquires feature maps of different resolutions across multiple layers through the backbone network of a pre-set basic detection model.

[0092] The detection device filters all targets on the feature map to obtain small targets and dense areas of small targets with occlusion.

[0093] According to some embodiments, the detection device first acquires feature map P. l The anchor boxes are pre-defined. The anchor boxes of the feature maps output from different layers have different scales and aspect ratios.

[0094] The detection device determines small targets on the feature map based on the relationship between the target scale of each target on the feature map and the anchor frame scale of the feature map, and obtains the corresponding annotation information.

[0095] According to some embodiments, the feature map P can be... l upper target scale box scale Less than or equal to feature map P l Corresponding anchor frame dimensions AS l The goal is defined as the small goal O small box scale ≤AS l Small goal O small The target scale can be calculated using the following formula.

[0096]

[0097] Where (x1, y1) is the small objective Osmall The top-left corner coordinates of the bounding box, (x2, y2), represent the small target O. small The coordinates of the bottom right corner of the bounding box.

[0098] Based on the selected small targets, the detection device generates corresponding annotation information.

[0099] According to some embodiments, the format of the annotation information for small targets can be as follows: in, The coordinates of the center point of the target bounding box. The width and height of the target bounding box.

[0100] In step S220, the detection device filters out densely occluded small targets on the feature map and obtains the corresponding annotation information.

[0101] Based on the annotation information of the selected small targets, the detection device identifies densely occluded small targets.

[0102] According to some embodiments, based on the annotation information of the target bounding box of the small target, the detection device first calculates the feature map P. l The intersection-union ratio (IoU) between each pair of bounding boxes of all targets is calculated, and the IoU of the overlapping regions between each pair of bounding boxes is obtained. ij , to represent the IoU value between the i-th bounding box and the j-th bounding box, where i,j∈(1,2,…N), and N is the number of labeled bounding boxes.

[0103] According to some embodiments, the overlapping area between any two target boxes can be greater than a preset second threshold T. d The target bounding box is defined as the target bounding box of the candidate densely occluded small target.

[0104] According to some embodiments, the detection device determines a set of candidate densely occluded small target bounding boxes based on the maximum number of overlap connections between target bounding boxes. Here, the number of overlap connections represents the number of times a target bounding box overlaps with all other target bounding boxes.

[0105] Then, the detection device calculates the ratio of the area of ​​each target box in the target bounding box of the candidate densely occluded small targets to the total area of ​​the feature map. If the ratio of the area of ​​the smallest target box in the group is greater than a preset third threshold T, the detection device will proceed. r If the target area of ​​the smallest target box in the group is less than the preset third threshold T, the detection device will not treat the target corresponding to that group as a densely occluded small target. r If the detection device determines that the target corresponding to the target box of the group satisfies the characteristic of dense occlusion of small targets, then the detection device will determine that the target satisfies the characteristic of dense occlusion of small targets.

[0106] The detection device removes bounding boxes that do not meet the characteristics of densely occluded small targets from the candidate bounding boxes of densely occluded small targets until there are no overlapping bounding boxes, in order to obtain all densely occluded small targets O. density .

[0107] Based on the selected densely occluded small targets, the detection device generates corresponding annotation information.

[0108] In step S230, based on the annotation information of small targets and densely occluded small targets, the detection device obtains the mask and the pixel value of the mask corresponding to the feature map in order to determine the region in the feature map to perform sparse convolution calculation.

[0109] The detection device inputs the feature map into the query head network and obtains the mask corresponding to the feature map output by the query head network.

[0110] Based on the annotation information of the selected small targets and densely occluded small targets, the detection device obtains a target set composed of small targets and densely occluded small targets.

[0111] According to some embodiments, the detection device calculates the feature map P. l Each position point (x, y) on the target set is related to the center point of the target set. Minimum distance D between l [x][y] can be expressed by the following formula.

[0112]

[0113] Assume s l (x,y) represents the feature map P. l The target scale corresponding to the minimum distance between each position point (x, y) on the target and the center point of the target in the target set is D. l For the target scale corresponding to [x][y], the detection device can determine the mask V using the following formula. l pixel values,

[0114]

[0115] According to some embodiments, when the pixel value of the mask is 1, the corresponding location point is in a region where a small target exists, and sparse convolution calculation is required to obtain the precise location of the small target. When the pixel value of the mask is 0, the corresponding location point is not in a region where a small target exists, and sparse convolution calculation is not required.

[0116] Figure 3 A block diagram of a device for detecting small targets in a remotely sensed image according to an example embodiment of this application is shown.

[0117] like Figure 3As shown, the detection device 100 includes a model building module 110, a model training module 120, a prediction module 130, and a data processing module 140.

[0118] The model building module 110 obtains the preset basic detection model.

[0119] The model building module 110 sets model parameters in the configuration file of the preset basic detection model so that the feature maps output by each layer in the backbone network have different resolutions.

[0120] The model building module 110 constructs a query header network to perform queries on target locations.

[0121] The model building module 110 adds a query header network to the structure of the preset basic detection model to form a target detection model.

[0122] The model training module 120 inputs the training images into the backbone network of the preset base detection model to obtain multi-layer feature maps of different resolutions output by the backbone network.

[0123] The model training module 120 filters all targets on the feature map to obtain dense regions of small targets and small targets with occlusion.

[0124] Based on the selected small targets and densely occluded small targets, the model training module 120 generates corresponding annotation information for use in training the head network.

[0125] The model training module 120 obtains the mask corresponding to the feature map and determines the pixel value of the mask based on the annotation information of small targets and densely occluded small targets, so as to determine the region in the feature map to perform sparse convolution calculation.

[0126] Based on the pixel values ​​of the mask, the model training module 120 calculates the losses of the classification head network, the location regression head network, and the query head network in the object detection model using different loss functions, and obtains the final loss of the object detection model.

[0127] The prediction module 130 inputs the image to be detected into the trained target detection model and outputs feature maps corresponding to the image to be detected at different resolutions through the backbone network of the target detection model.

[0128] On the low-resolution feature map, the prediction module 130 obtains the target category and coarse location information on the feature map through the classification head network and the location regression head network of the target detection model to generate coarse detection results.

[0129] On the high-resolution feature map, the prediction module 130, based on the query head network of the object detection model, determines the corresponding position for sparse convolution calculation in the high-resolution feature map.

[0130] The prediction module 130 performs sparse convolution calculations based on the sparse tensor set corresponding to the corresponding position in the high-resolution feature map.

[0131] Based on the sparse convolution calculation results, the prediction module 130 obtains the prediction results of the high-resolution feature map as the fine detection result.

[0132] After obtaining the coarse and fine detection results of the image to be detected through the trained target detection model, the data processing module 140 merges the coarse and fine detection results to filter out candidate boxes of small targets with high confidence.

[0133] The data processing module 140 performs suppression filtering on the candidate boxes of small targets obtained after merging the coarse detection results and fine detection results, so as to suppress duplicate candidate boxes and complete the detection and recognition of small targets in the image to be detected.

[0134] Figure 4 A block diagram of an electronic device according to an example embodiment of this application is shown.

[0135] like Figure 4 As shown, the electronic device 600 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0136] like Figure 4 As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program code, which can be executed by the processing unit 610, causing the processing unit 610 to perform the methods described in this specification according to the various exemplary embodiments of this application. For example, the processing unit 610 can perform, for example... Figure 1 The method shown.

[0137] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0138] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0139] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0140] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0141] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0142] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0144] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0145] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0146] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0147] The embodiments of this application have been described in detail above. These descriptions are solely for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, its specific implementation methods, and its application scope, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting small targets in remote sensing images, characterized in that: include: Build an object detection model; Training the target detection model; Using the trained target detection model, the coarse detection result and the fine detection result of the image to be detected are obtained respectively; The coarse detection result and the fine detection result are combined to complete the detection of the small target in the image to be detected.

2. The method according to claim 1, characterized in that Build a target detection model, including: Obtaining a preset basic detection model, wherein the preset basic detection model includes a backbone network, a classification head network, and a position regression head network; Setting parameters of the preset basic detection model to output a feature map through the backbone network; Constructing a query head network, wherein the query head network is parallel to the classification head network and the position regression head network; Based on the preset basic detection model and the query head network, the target detection model is obtained.

3. The method according to claim 2, characterized in that The backbone network has multiple layers, each layer can output a feature map, and the higher the number of layers of the backbone network, the lower the resolution of the feature map output by the layer; Training the target detection model includes: Training the preset basic detection model on the low-resolution feature map; The preset basic detection model and the query head network are trained on high-resolution feature maps.

4. The method according to claim 2, characterized in that Training the target detection model includes: Obtaining labeling information of small objects and densely occluded small objects on the feature map; Based on the annotation information, obtaining a mask corresponding to the feature map and a pixel value of the mask to determine an area in the feature map where sparse convolution calculation is performed; According to the pixel values ​​of the mask, the loss of the target detection model is calculated through a loss function to complete the training of the target detection model.

5. The method according to claim 4, characterized in that Obtaining labeling information of small objects and densely occluded small objects on the feature map, including: Calculating the target scale of the small target on the feature map to screen the small target; Calculating the intersection-and-union ratios between the object frames of all objects on the feature map to filter out the densely occluded small objects; Generate annotation information of the filtered small objects and densely occluded small objects for training the query head network.

6. The method according to claim 4, characterized in that Based on the annotation information, obtaining a mask corresponding to the feature map and a pixel value of the mask includes: Obtaining a mask corresponding to the feature map; Based on the annotation information, obtaining a target set consisting of the small targets and the densely occluded small targets; Calculating the minimum distance between each position point on the feature map and the center point of the target in the target set and the target scale corresponding to the minimum distance; A pixel value of the mask is determined based on the minimum distance and a target scale corresponding to the minimum distance.

7. The method according to claim 1, characterized in that Through the trained target detection model, the coarse detection results and fine detection results of the image to be detected are obtained respectively, including: Obtaining a feature map corresponding to the image to be detected through the backbone network of the target detection model; Based on the low-resolution feature map, obtaining the coarse detection result through the classification head network and the position regression head network of the target detection model; Performing sparse convolution calculations on high-resolution feature maps through a query head network of the target detection model; Based on the result of the sparse convolution calculation, the precise detection result is obtained on the high-resolution feature map through the classification head network and the position regression head network.

8. The method according to claim 7, characterized in that The query head network of the target detection model performs sparse convolution calculations on the high-resolution feature map, including: Based on the low-resolution feature map, obtaining pixel values ​​of the mask corresponding to the low-resolution feature map through the query head network; According to the pixel value of the mask corresponding to the low-resolution feature map, query the position point that meets the preset rule on the low-resolution feature map; Mapping the position points to corresponding positions of the high-resolution feature map to obtain a point set at corresponding positions of the high-resolution feature map; Acquire a sparse tensor set according to the point set at the corresponding position; The sparse convolution calculation is performed based on the sparse tensor set.

9. The method according to claim 1, characterized in that: Combining the coarse detection result and the fine detection result to complete the detection of the small target in the image to be detected, including: Combining the coarse detection result and the fine detection result to screen a candidate frame of the small object; The candidate boxes of the small targets obtained after screening are subjected to suppression filtering.

10. A small target detection device for remote sensing images, characterized in that: include: Model building module, used to build target detection model; A model training module, used for training the target detection model; The prediction module is used to obtain the coarse detection result and the fine detection result of the image to be detected respectively through the trained target detection model; The data processing module is used to merge the coarse detection result and the fine detection result to complete the detection of the small target in the image to be detected.

11. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images