A multi-stage remote sensing image target detection method, apparatus, equipment and medium

Abstract

This application relates to a multi-stage remote sensing image target detection method, apparatus, computer device, and computer-readable storage medium. The multi-stage remote sensing image target detection method includes: inputting a remote sensing image into a pre-trained first target detection network, outputting multiple prediction boxes and corresponding prediction information; obtaining a first target detection result when the confidence level of the prediction box is greater than a second threshold; and, when the confidence level of the prediction box is between the first and second thresholds, cropping, rotating, and scaling the remote sensing image based on the position information of the prediction boxes, then inputting the image into a pre-trained second target detection network to obtain a second target detection result; and obtaining a final target detection result based on the first and second target detection results. This solves the problem of low accuracy in remote sensing image target detection results and improves the accuracy of remote sensing image target detection results.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to a multi-stage remote sensing image target detection method, apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] Object detection is one of the most important and challenging branches of computer vision. It has been widely applied in people's lives, such as in security surveillance and autonomous driving. The task of object detection is to locate instances of a specific type of semantic object.

[0003] Remote sensing image target detection is used in high-tech military confrontations to obtain timely and accurate battlefield information, identify strategic strike targets, and provide precise qualitative and location information. It also plays a crucial role in civilian fields such as resource exploration, environmental monitoring, and urban planning. By performing target identification on remote sensing images obtained from satellites, aircraft, or spacecraft, information such as terrain, equipment, and troop deployments in the photographed area can be obtained.

[0004] Unlike general ground images, remote sensing images are taken from above, and because the attitude and altitude of satellites and spacecraft change during flight, the acquired targets have multiple angles and scales. For example, ships may be facing any direction (up, down, left, right), and the resolution may be 1m per pixel or 5m per pixel. Current ground target detection algorithms are mostly designed for single angles, such as pedestrian and vehicle detection from surveillance cameras, where the targets are upright people and vehicles at similar angles, and both pedestrians and vehicles are perpendicular to the horizontal axis. Furthermore, current target regression boxes are mostly horizontal rectangles. Applying horizontal rectangle prediction to target detection in remote sensing images leads to significant variations in the features within the target box at different angles.

[0005] Due to its real-time performance, the YOLO algorithm has been widely used in industry. YOLO is a typical single-stage object detection algorithm. Its principle is to use multi-scale (usually three-scale) feature layers to detect objects of different sizes. Each feature point is responsible for detecting objects whose center point is at its location, and each feature point will output a corresponding prediction result.

[0006] For object detection algorithms, confidence thresholding is necessary during model post-processing. Because the distribution of deep learning datasets is not perfectly uniform, the confidence distribution of object detection results varies across different scenarios in practical applications. For example, the overall confidence of noisy images is low, and the confidence distribution of individual object detection results is relatively concentrated. High-resolution images, on the other hand, can detect objects very well, resulting in high confidence for results with objects and low confidence for results without objects, leading to a more dispersed confidence distribution. To maintain high accuracy, a relatively high confidence threshold is often set, which results in very low recall. Furthermore, due to the multi-angle and multi-scale characteristics of objects in remote sensing images, the data distribution is wide, making it difficult for a single model to handle both large and small targets.

[0007] There is currently no effective solution to the problem of low accuracy in target detection results of remote sensing images in related technologies. Summary of the Invention

[0008] Therefore, it is necessary to provide a multi-stage remote sensing image target detection method, apparatus, computer equipment, and computer-readable storage medium to address the aforementioned technical problems.

[0009] In a first aspect, embodiments of this application provide a multi-stage remote sensing image target detection method, the method comprising:

[0010] The remote sensing image is input into a pre-trained first object detection network, which outputs multiple prediction boxes and corresponding prediction information; the prediction information includes confidence level, classification probability, and the location information of the prediction box.

[0011] If the confidence level corresponding to the prediction box is greater than the second threshold, the first target detection result is obtained;

[0012] When the confidence level corresponding to the predicted bounding box is between the first threshold and the second threshold, the remote sensing image is cropped according to the position information of the predicted bounding box to obtain at least one first remote sensing image. After rotating and scaling each first remote sensing image, it is input into a pre-trained second target detection network to obtain a second target detection result; wherein, the first threshold is less than the second threshold.

[0013] Based on the first target detection result and the second target detection result, the final target detection result is obtained.

[0014] In one embodiment, the location information includes the length and width of the prediction box, the angle between the long side of the prediction box and the positive x-axis in a Cartesian coordinate system, and the center coordinates of the prediction box; wherein the Cartesian coordinate system is established with the top left corner of the remote sensing image as the origin, with the horizontal direction to the right as the positive x-axis and the vertical direction downward as the positive y-axis.

[0015] In one embodiment, the rotation and scaling processing of each of the first remote sensing images includes:

[0016] Each of the first remote sensing images is rotated so that the angle between the long side of each first remote sensing image and the x-axis is zero, and each of the first remote sensing images is scaled according to the same preset size.

[0017] In one embodiment, the step of rotating and scaling each of the first remote sensing images and then inputting them into a pre-trained second target detection network to obtain the second target detection result includes:

[0018] Using the scaling ratio of each of the first remote sensing images as the pixel value and the preset size as the size, a corresponding scaling information map is generated;

[0019] The scaled first remote sensing image and the corresponding scaled information map are stitched together to obtain at least one second remote sensing image.

[0020] At least one of the second remote sensing images is input into a pre-trained second target detection network to obtain the second target detection result.

[0021] In one embodiment, inputting at least one of the second remote sensing images into a pre-trained second target detection network to obtain a second target detection result includes:

[0022] At least one of the second remote sensing images is input into a pre-trained second target detection network, which outputs at least one prediction box and corresponding prediction information.

[0023] If the confidence level corresponding to the prediction box is greater than the third threshold, the second target detection result is obtained.

[0024] In one embodiment, the process of inputting the remote sensing image into a pre-trained first object detection network includes:

[0025] Obtain the first training sample, which is a remote sensing image that has been augmented by rotation, cropping, splicing, brightness contrast changes, blurring, and scaling.

[0026] The first training sample is input into a convolutional neural network, a feature extraction network is used to extract feature maps, and a pyramid network structure is used to fuse multi-scale features to obtain the final feature map. Based on the feature map, the detection result is obtained.

[0027] Calculate the loss function based on the detection results and the actual results;

[0028] The parameters of the convolutional neural network are updated based on the loss function to obtain the pre-trained first object detection network.

[0029] In one embodiment, the process of rotating and scaling each of the first remote sensing images before inputting them into the pre-trained second object detection network includes:

[0030] A second training sample is obtained, which includes remote sensing images with and without targets after cropping, and the cropped remote sensing images are rotated and scaled.

[0031] The second training sample is input into a convolutional neural network to extract feature maps, and the detection result is obtained based on the feature maps;

[0032] Calculate the loss function based on the detection results and the actual results;

[0033] The parameters of the convolutional neural network are updated based on the loss function to obtain the pre-trained second object detection network.

[0034] In one embodiment, the process of rotating and scaling the remote sensing image includes:

[0035] The remote sensing image is perturbed, and the perturbation includes at least one of center position perturbation, length and width perturbation, and angle perturbation.

[0036] Secondly, embodiments of this application also provide a multi-stage remote sensing image target detection device, the device comprising:

[0037] The input module is used to input remote sensing images into a pre-trained first target detection network and output multiple prediction boxes and corresponding prediction information; the prediction information includes confidence, classification probability, and the location information of the prediction boxes.

[0038] The first target detection module is used to obtain a first target detection result when the confidence level corresponding to the prediction box is greater than a second threshold.

[0039] The second target detection module is used to crop the remote sensing image according to the position information of the predicted box when the confidence level of the predicted box is between the first threshold and the second threshold, to obtain at least one first remote sensing image, and to input the first remote sensing image into a pre-trained second target detection network after rotating and scaling each first remote sensing image to obtain a second target detection result; wherein the first threshold is less than the second threshold.

[0040] The acquisition module is used to obtain the final target detection result based on the first target detection result and the second target detection result.

[0041] Thirdly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the method described in the first aspect above.

[0042] The aforementioned multi-stage remote sensing image target detection method, apparatus, computer equipment, and computer-readable storage medium input a remote sensing image into a pre-trained first target detection network, outputting multiple prediction boxes and corresponding prediction information. The prediction information includes confidence level, classification probability, and position information of the prediction boxes. If the confidence level of a prediction box is greater than a second threshold, a first target detection result is obtained. If the confidence level of a prediction box is between the first and second thresholds, the remote sensing image is cropped based on the position information of the prediction boxes to obtain at least one first remote sensing image. Each first remote sensing image is then rotated and scaled before being input into a pre-trained second target detection network to obtain a second target detection result. The first threshold is less than the second threshold. Based on the first and second target detection results, a final target detection result is obtained. This solves the problem of low accuracy in remote sensing image target detection results and improves the accuracy of remote sensing image target detection results.

[0043] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0045] Figure 1 This is a hardware structure block diagram of a terminal for a multi-stage remote sensing image target detection method in one embodiment;

[0046] Figure 2 This is a flowchart illustrating a multi-stage remote sensing image target detection method in one embodiment;

[0047] Figure 3 This is a schematic diagram illustrating the location information of a prediction box in one embodiment;

[0048] Figure 4 This is a schematic diagram illustrating the rotation and scaling of a first remote sensing image in one embodiment;

[0049] Figure 5 This is a schematic diagram of the process for obtaining the detection result of the second target in one embodiment;

[0050] Figure 6 This is a flowchart illustrating the specific steps of S503 in one embodiment;

[0051] Figure 7 This is a schematic diagram of the process of obtaining a pre-trained first object detection network in one embodiment;

[0052] Figure 8 This is a schematic diagram of the training process of the first object detection network in one embodiment;

[0053] Figure 9 This is a schematic diagram of the process for obtaining a pre-trained second object detection network in one embodiment;

[0054] Figure 10 This is a schematic diagram of the training process of the second object detection network in one embodiment;

[0055] Figure 11 This is a schematic diagram illustrating the perturbation of a remote sensing image containing a target object in one embodiment;

[0056] Figure 12 This is a flowchart illustrating a preferred embodiment of a multi-stage remote sensing image target detection method;

[0057] Figure 13 This is a structural block diagram of a multi-stage remote sensing image target detection device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0059] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of this application.

[0060] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0061] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0062] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the multi-stage remote sensing image target detection method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0063] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multi-stage remote sensing image target detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0064] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0065] This application provides a multi-stage remote sensing image target detection method, such as... Figure 2 As shown, the method includes the following steps:

[0066] Step S201: Input the remote sensing image into the pre-trained first target detection network, and output multiple prediction boxes and corresponding prediction information; the prediction information includes confidence, classification probability and position information of the prediction boxes;

[0067] Specifically, embodiments of this application can employ any object detection algorithm to construct a pre-trained first object detection network, including but not limited to the YOLOv5 algorithm. The first object detection network has a pyramid structure and can simultaneously detect objects at multiple scales. This embodiment uses the YOLOv5 algorithm as an example, performing object detection at three scales. Specifically, the remote sensing image is input into the pre-trained first object detection network, which outputs multiple prediction boxes and prediction information corresponding to each box. The output prediction boxes encompass three different scales. The prediction information includes confidence level, classification probability, and the location information of the prediction box, and can be represented as [x...]. c,y c ,w,h,conf,θ,c0,…,c n-1 ], where conf is the confidence level, [c0,…,c n-1 The classification probability for each category, [x] c ,y c [w,h,θ] represents the location information of the predicted bounding box.

[0068] like Figure 3 As shown, the location information of one of the predicted bounding boxes is displayed [x] c ,y c Specifically, a rectangular coordinate system is established with the top-left corner of the remote sensing image as the origin, the horizontal direction to the right as the positive x-axis, and the vertical direction downwards as the positive y-axis. w and h are the length and width of the prediction box, respectively, and θ is the angle between the longer side of the prediction box and the positive x-axis in the rectangular coordinate system. c and y c θ and π represent the center coordinates of the predicted bounding box, respectively. Considering the periodicity of angles, θ∈[0,π).

[0069] Furthermore, the multiple predicted bounding boxes are preprocessed. Specifically, a first threshold σ1 is set, predicted bounding boxes with confidence conf < σ1 are deleted, and predicted bounding boxes with confidence conf ≥ σ1 are retained. Non-Maximum Suppression (NMS) is then applied to the remaining predicted bounding boxes to obtain N predicted bounding boxes. Subsequent object detection operations are performed on these N predicted bounding boxes.

[0070] Step S202: If the confidence level corresponding to the predicted box is greater than the second threshold, obtain the first target detection result;

[0071] Specifically, a second threshold σ2 (σ2>σ1) is set, and N' predicted bounding boxes (N'≤N) with conf≥σ2 are selected based on this second threshold σ2. These N' predicted bounding boxes and their corresponding prediction information constitute the first target detection result.

[0072] Step S203: If the confidence level corresponding to the predicted bounding box is between the first threshold and the second threshold, the remote sensing image is cropped according to the position information of the predicted bounding box to obtain at least one first remote sensing image. After rotating and scaling each first remote sensing image, it is input into a pre-trained second target detection network to obtain a second target detection result; wherein, the first threshold is less than the second threshold.

[0073] Step S204: Based on the first target detection result and the second target detection result, the final target detection result is obtained.

[0074] Specifically, the sum of the first target detection result and the second target detection result is the final target detection result.

[0075] Steps S201 to S204 above involve inputting a remote sensing image into a pre-trained first target detection network, which outputs multiple prediction boxes and corresponding prediction information. The prediction information includes confidence level, classification probability, and the location information of the prediction boxes. If the confidence level of a prediction box is greater than a second threshold, a first target detection result is obtained. If the confidence level of a prediction box is between the first and second thresholds, the remote sensing image is cropped based on the location information of the prediction boxes to obtain at least one first remote sensing image. Each first remote sensing image is then rotated and scaled before being input into a pre-trained second target detection network to obtain a second target detection result. The first threshold is less than the second threshold. Based on the first and second target detection results, a final target detection result is obtained. This addresses the low accuracy problem of remote sensing image target detection models in related technologies. This application improves the accuracy of target detection results through a multi-stage target detection network.

[0076] In one embodiment, the rotation and scaling processing of each of the first remote sensing images includes:

[0077] Each of the first remote sensing images is rotated so that the angle between the long side of each first remote sensing image and the x-axis is zero, and each of the first remote sensing images is scaled according to the same preset size.

[0078] Specifically, such as Figure 4 The diagram illustrates rotating one of the first remote sensing images so that the angle between the long side of the first remote sensing image and the x-axis is zero, and then scaling the first remote sensing image to a preset size (size×size), where size×size represents the length and width of the first remote sensing image after scaling. For example, in this embodiment, size×size is set to 384x384.

[0079] In one embodiment, such as Figure 5 As shown, the process of rotating and scaling each of the first remote sensing images and then inputting them into a pre-trained second target detection network to obtain the second target detection result includes the following steps:

[0080] Step S501: Using the scaling ratio of each of the first remote sensing images as the pixel value and the preset size as the size, generate a corresponding scaling information map;

[0081] Specifically, the scaling ratio of each first remote sensing image is calculated, namely the scaling ratio of the longer side (ratio_w) and the scaling ratio of the shorter side (ratio_h), where... A scaled infographic of size size × size is generated by using ratio_w and ratio_h as alternating pixel values.

[0082] Step S502: The scaled first remote sensing image and the corresponding scaled information map are stitched together to obtain at least one second remote sensing image.

[0083] Specifically, the first remote sensing image after scaling and the corresponding scaled information map are both size×size. The first remote sensing image after scaling and the corresponding scaled information map are stitched together. Specifically, the concat function is used to fuse features, thereby increasing the number of channels in the first remote sensing image after scaling, and obtaining the corresponding second remote sensing image.

[0084] Step S503: Input at least one of the second remote sensing images into a pre-trained second target detection network to obtain the second target detection result.

[0085] In this embodiment, considering that the scaled first remote sensing image becomes blurred, resulting in varying degrees of blurriness among different input first remote sensing images, a scaling factor feature is added to the remote sensing image reflectance feature by combining the scaling information map corresponding to the first remote sensing image, thereby improving the accuracy of the second target detection result.

[0086] In one embodiment, such as Figure 6 As shown, the step of inputting at least one of the second remote sensing images into a pre-trained second target detection network to obtain the second target detection result includes the following steps:

[0087] Step S601: Input at least one of the second remote sensing images into a pre-trained second target detection network, and output at least one prediction box and corresponding prediction information;

[0088] Specifically, when one of the second remote sensing images is input into a pre-trained second object detection network, multiple predicted bounding boxes are obtained. The predicted bounding box with the highest confidence and its corresponding prediction information are then taken as the output. Therefore, if M second remote sensing images are obtained in step S502, these M images are input into the pre-trained second object detection network, and M predicted bounding boxes and their corresponding prediction information are output.

[0089] Step S602: If the confidence level corresponding to the predicted box is greater than the third threshold, the second target detection result is obtained.

[0090] Specifically, the confidence level of the output prediction box is represented by conf'. A third threshold σ3 is set, and prediction boxes with conf' < σ3 are deleted, while those with conf' ≥ σ3 are retained. The retained prediction boxes and their corresponding prediction information constitute the second target detection result.

[0091] In one embodiment, such as Figure 7 As shown, the steps before inputting the remote sensing image into the pre-trained first object detection network include:

[0092] Step S701: Obtain the first training sample, which is a remote sensing image that has been augmented by rotation, cropping, splicing, brightness contrast changes, blurring, and scaling.

[0093] Specifically, the first training sample is a remote sensing image with a rotated target and annotations, which is augmented by methods such as rotation, cropping, stitching, brightness contrast changes, blurring, and scaling (enlarging or shrinking).

[0094] Step S702: Input the first training sample into a convolutional neural network, extract feature maps using a feature extraction network, further fuse multi-scale features using a pyramid network structure to obtain the final feature map, and obtain the detection result based on the feature map;

[0095] Specifically, such as Figure 8 As shown, this embodiment uses the YOLOv5 algorithm as an example. The first training sample is input into the convolutional neural network. Feature map F1 is obtained through convolutional blocks. Feature map F2 is obtained after downsampling through convolution with a stride of 2 and convolutional blocks. This process is repeated multiple times to obtain feature maps F3, F4, and F5 (the network can be deepened or pruned as needed). Low-level feature C5 is upsampled and combined with the features of the previous level. After passing through convolutional blocks, feature C4 is obtained (repeated multiple times). Object detection bounding box prediction is performed on the features (C3 / C4 / C5) at each resolution. H3 detects small targets, H4 detects medium-sized targets, and H5 detects large targets. The detection results are then obtained.

[0096] Step S703: Calculate the loss function based on the detection results and the actual results;

[0097] Step S704: Update the parameters of the convolutional neural network based on the loss function to obtain the pre-trained first object detection network. This network can simultaneously detect objects at multiple scales.

[0098] In one embodiment, such as Figure 9 As shown, the steps before inputting the first remote sensing images into the pre-trained second target detection network after rotation and scaling include:

[0099] Step S801: Obtain the second training sample, which includes remote sensing images with targets after cropping and remote sensing images without targets, and perform rotation and scaling processing on the cropped remote sensing images.

[0100] Specifically, the second training samples include positive samples and negative samples. Positive samples are remote sensing images with targets after cropping, and negative samples are remote sensing images without targets after cropping. The cropped remote sensing images are rotated and scaled so that the angle between the long side of the cropped remote sensing image and the x-axis is zero, and they are scaled to the same preset size.

[0101] Step S802: Input the second training sample into the convolutional neural network to extract feature maps, and obtain detection results based on the feature maps;

[0102] Specifically, such as Figure 10 As shown, taking the YOLOv5 algorithm as an example, the first training sample is input into the convolutional neural network, and the feature map F1 is obtained through the convolutional block. The feature map F2 after downsampling is obtained through the convolution and convolutional block with a stride of 2. The above steps are repeated to obtain the feature map F3. F3 is passed through the convolutional block to obtain C3. The target detection box is predicted on the feature C3 to obtain the target detection result.

[0103] Step S803: Calculate the loss function based on the detection results and the actual results;

[0104] Step S804: Update the parameters of the convolutional neural network based on the loss function to obtain the pre-trained second object detection network.

[0105] Specifically, the pre-trained second object detection network is a single-scale object prediction network, which only predicts the object detection result of the feature map at one scale.

[0106] In one embodiment, the process before cropping the remote sensing image containing the target includes: perturbing the remote sensing image containing the target, and cropping the perturbed remote sensing image containing the target; wherein the perturbation includes at least one of center position perturbation, length and width perturbation, and angle perturbation.

[0107] In this embodiment, to better fit the situations that might occur in real-world application scenarios, the cropping is not strictly based on the actual location of the target object. Instead, the center position, length, width, and angle of the remote sensing image are perturbed to a certain extent before cropping. These three perturbations can be randomly combined and selected. For example... Figure 11 As shown in the figure, the solid box represents the actual location of the remote sensing image before the disturbance, and the dashed box represents the location of the remote sensing image after the disturbance and cropping.

[0108] The present embodiment will now be described and illustrated through preferred embodiments.

[0109] Figure 12 This is a preferred flowchart of the multi-stage remote sensing image target detection method in this embodiment, as follows: Figure 12 As shown, this multi-stage remote sensing image target detection method includes the following steps:

[0110] Step S901: Acquire remote sensing images;

[0111] Step S902: Input the remote sensing image into the trained first target detection network and output multiple prediction boxes and corresponding prediction information;

[0112] Step S903: Determine whether the confidence level conf of the predicted bounding box is greater than or equal to the first threshold σ1. If yes, proceed to step S904.

[0113] Step S904: Perform non-maximum suppression on the predicted bounding box;

[0114] Step S905: Determine whether the confidence level conf of the predicted bounding box is greater than or equal to the second threshold σ2. If yes, proceed to S906; otherwise, proceed to S907.

[0115] Step S906, obtain target detection result 1;

[0116] Step S907: Based on the location information of the predicted bounding box, the remote sensing image is cropped, rotated, and scaled.

[0117] Step S908: Input the remote sensing image processed in step S907 into the trained second target detection network and output the corresponding prediction box and prediction information.

[0118] Step S909: Determine whether the confidence of the predicted bounding box conf' is greater than or equal to the third threshold σ3. If yes, proceed to S910.

[0119] Step S910, obtain target detection result 2;

[0120] Step S911: Based on target detection result 1 and target detection result 2, obtain the target detection result.

[0121] This application also provides a multi-stage remote sensing image target detection device, such as... Figure 13 As shown, the device includes:

[0122] The input module 91 is used to input remote sensing images into a pre-trained first target detection network and output multiple prediction boxes and corresponding prediction information; the prediction information includes confidence, classification probability and position information of the prediction boxes.

[0123] The first target detection module 92 is used to obtain a first target detection result when the confidence level corresponding to the prediction box is greater than the second threshold.

[0124] The second target detection module 93 is used to, when the confidence level corresponding to the predicted box is between a first threshold and a second threshold, crop the remote sensing image according to the position information of the predicted box to obtain at least one first remote sensing image, and input the first remote sensing image into a pre-trained second target detection network after rotating and scaling each first remote sensing image to obtain a second target detection result; wherein, the first threshold is less than the second threshold.

[0125] The module 94 is used to obtain the final target detection result based on the first target detection result and the second target detection result.

[0126] In one embodiment, the location information includes the length and width of the prediction box, the angle between the long side of the prediction box and the positive x-axis in a Cartesian coordinate system, and the center coordinates of the prediction box; wherein the Cartesian coordinate system is established with the top left corner of the remote sensing image as the origin, with the horizontal direction to the right as the positive x-axis and the vertical direction downward as the positive y-axis.

[0127] In one embodiment, the second target detection module 93 is further configured to:

[0128] Each of the first remote sensing images is rotated so that the angle between the long side of each first remote sensing image and the x-axis is zero, and each of the first remote sensing images is scaled according to the same preset size.

[0129] In one embodiment, the second target detection module 93 is further configured to:

[0130] Using the scaling ratio of each of the first remote sensing images as the pixel value and the preset size as the size, a corresponding scaling information map is generated;

[0131] The scaled first remote sensing image and the corresponding scaled information map are stitched together to obtain at least one second remote sensing image.

[0132] At least one of the second remote sensing images is input into a pre-trained second target detection network to obtain the second target detection result.

[0133] In one embodiment, the second target detection module 93 is further configured to:

[0134] At least one of the second remote sensing images is input into a pre-trained second target detection network, which outputs at least one prediction box and corresponding prediction information.

[0135] If the confidence level corresponding to the prediction box is greater than the third threshold, the second target detection result is obtained.

[0136] In one embodiment, the input module 91 is further configured to:

[0137] Obtain the first training sample, which is a remote sensing image that has been augmented by rotation, cropping, splicing, brightness contrast changes, blurring, and scaling.

[0138] The first training sample is input into a convolutional neural network, a feature extraction network is used to extract feature maps, and a pyramid network structure is used to fuse multi-scale features to obtain the final feature map. Based on the feature map, the detection result is obtained.

[0139] Calculate the loss function based on the detection results and the actual results;

[0140] The parameters of the convolutional neural network are updated based on the loss function to obtain the pre-trained first object detection network.

[0141] In one embodiment, the second target detection module 93 is further configured to:

[0142] A second training sample is obtained, which includes remote sensing images with and without targets after cropping, and the cropped remote sensing images are rotated and scaled.

[0143] The second training sample is input into a convolutional neural network to extract feature maps, and the detection result is obtained based on the feature maps;

[0144] Calculate the loss function based on the detection results and the actual results;

[0145] The parameters of the convolutional neural network are updated based on the loss function to obtain the pre-trained second object detection network.

[0146] In one embodiment, the second target detection module 93 is further configured to:

[0147] The remote sensing image containing the target object is perturbed, and the perturbed remote sensing image containing the target object is cropped; wherein, the perturbation includes at least one of center position perturbation, length and width perturbation, and angle perturbation.

[0148] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0149] This application also provides a computer-readable storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps as described in any of the above embodiments of multi-stage remote sensing image target detection.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-stage remote sensing image target detection method, characterized in that, The method includes: The remote sensing image is input into a pre-trained first object detection network, which outputs multiple prediction boxes and corresponding prediction information; the prediction information includes confidence level, classification probability, and the location information of the prediction box. If the confidence level corresponding to the prediction box is greater than the second threshold, the first target detection result is obtained; When the confidence level corresponding to the predicted bounding box is between a first threshold and a second threshold, the remote sensing image is cropped according to the position information of the predicted bounding box to obtain at least one first remote sensing image. Each first remote sensing image is then rotated and scaled before being input into a pre-trained second target detection network to obtain a second target detection result. The first threshold is less than the second threshold. The process of rotating and scaling each first remote sensing image before inputting it into the pre-trained second target detection network to obtain the second target detection result includes: generating a corresponding scaling information map using the scaling ratio of each first remote sensing image as the pixel value and a preset size; stitching the scaled first remote sensing image and the corresponding scaling information map together to obtain at least one second remote sensing image; and inputting the at least one second remote sensing image into the pre-trained second target detection network to obtain the second target detection result. Based on the first target detection result and the second target detection result, the final target detection result is obtained.

2. The method according to claim 1, characterized in that, The location information includes the length and width of the prediction box, the angle between the long side of the prediction box and the positive x-axis in the Cartesian coordinate system, and the center coordinates of the prediction box; wherein, the Cartesian coordinate system is established with the upper left corner of the remote sensing image as the origin, with the horizontal to the right as the positive x-axis and the vertical downward as the positive y-axis.

3. The method according to claim 1, characterized in that, The rotation and scaling processing of each of the first remote sensing images includes: Each of the first remote sensing images is rotated so that the angle between the long side of each first remote sensing image and the x-axis is zero, and each of the first remote sensing images is scaled according to the same preset size.

4. The method according to claim 1, characterized in that, The step of inputting at least one of the second remote sensing images into a pre-trained second target detection network to obtain a second target detection result includes: At least one of the second remote sensing images is input into a pre-trained second target detection network, which outputs at least one prediction box and corresponding prediction information. If the confidence level corresponding to the prediction box is greater than the third threshold, the second target detection result is obtained.

5. The method according to claim 1, characterized in that, The process of inputting the remote sensing image into a pre-trained first object detection network includes: Obtain the first training sample, which is a remote sensing image that has been augmented by rotation, cropping, splicing, brightness contrast changes, blurring, and scaling. The first training sample is input into a convolutional neural network, a feature extraction network is used to extract feature maps, and a pyramid network structure is used to fuse multi-scale features to obtain the final feature map. Based on the feature map, the detection result is obtained. Calculate the loss function based on the detection results and the actual results; The parameters of the convolutional neural network are updated based on the loss function to obtain the pre-trained first object detection network.

6. The method according to claim 1, characterized in that, Before inputting the first remote sensing images into the pre-trained second object detection network after rotation and scaling, the process includes: A second training sample is obtained, which includes remote sensing images with and without targets after cropping, and the cropped remote sensing images are rotated and scaled. The second training sample is input into a convolutional neural network to extract feature maps, and the detection result is obtained based on the feature maps; Calculate the loss function based on the detection results and the actual results; The parameters of the convolutional neural network are updated based on the loss function to obtain the pre-trained second object detection network.

7. The method according to claim 6, characterized in that, The remote sensing image containing the target object after cropping includes: The remote sensing image containing the target object is perturbed, and the perturbed remote sensing image containing the target object is cropped; wherein, the perturbation includes at least one of center position perturbation, length and width perturbation, and angle perturbation.

8. A multi-stage remote sensing image target detection device, characterized in that, The device includes: The input module is used to input remote sensing images into a pre-trained first target detection network and output multiple prediction boxes and corresponding prediction information; the prediction information includes confidence, classification probability, and the location information of the prediction boxes. The first target detection module is used to obtain a first target detection result when the confidence level corresponding to the prediction box is greater than a second threshold. The second target detection module is configured to, when the confidence level corresponding to the predicted bounding box is between a first threshold and a second threshold, crop the remote sensing image based on the position information of the predicted bounding box to obtain at least one first remote sensing image, and input the first remote sensing images after rotation and scaling into a pre-trained second target detection network to obtain a second target detection result; wherein the first threshold is less than the second threshold; wherein, the step of inputting the first remote sensing images after rotation and scaling into the pre-trained second target detection network to obtain the second target detection result includes: generating a corresponding scaling information map with the scaling ratio of each first remote sensing image as the pixel value and a preset size as the size; stitching the scaled first remote sensing image and the corresponding scaling information map to obtain at least one second remote sensing image; and inputting the at least one second remote sensing image into the pre-trained second target detection network to obtain the second target detection result; The acquisition module is used to obtain the final target detection result based on the first target detection result and the second target detection result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images