Large-format remote sensing image target detection method based on situation-fine perception and related equipment

Abstract

The application discloses a large-format remote sensing image target detection method based on situation-fine perception and related equipment, and belongs to the technical field of computer vision. The method comprises the following steps: acquiring a remote sensing image and cutting it into remote sensing image blocks, and extracting a multi-scale feature map of the remote sensing image blocks by using a pre-trained backbone network; based on the multi-scale feature map, cutting the remote sensing image blocks are input into a situation awareness classifier to set a classification label for the regions of the remote sensing image blocks; target detection is performed on the remote sensing image blocks according to the classification label to obtain a plurality of image block target detection results; and the plurality of image block target detection results are fused and post-processed to obtain a complete target detection result. The application can solve the problem that invalid information cannot be effectively filtered when target detection is performed on a large-format high-resolution image in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a method and related equipment for target detection in large-format remote sensing images based on situational-fine perception. Background Technology

[0002] With the development of artificial intelligence technology, large-format, high-resolution image target detection technology is gradually being widely used in resource-constrained environments or equipment, such as spaceborne and airborne systems, playing an increasingly important role in military reconnaissance, detection and early warning, precision guidance, emergency rescue, and intelligent transportation. Achieving large-format, high-resolution image target detection in environments or equipment with limited storage and computing resources is unaffected by network bandwidth, significantly reducing latency while improving data security and privacy.

[0003] It is worth noting that there are still several key challenges in target detection of large-format high-resolution images: (1) With the rapid development of technologies such as UAVs and satellites, the acquisition of large-format high-resolution images has become a reality and has been widely used in various fields. However, in practical applications, the rapid and accurate detection of target areas from these large-format images remains an urgent problem to be solved. (2) Large-format high-resolution images contain a large amount of irrelevant information, and a large amount of computing power, storage power, and bandwidth are used for processing irrelevant data, which greatly increases the burden on the hardware system. In most applications, the data processed by the deep model is mostly low-value areas, and the effective information accounts for only a small part of the total data.

[0004] Therefore, when processing large-format, high-resolution remote sensing images, existing technologies often consume a lot of time in processing invalid information, which seriously affects the detection speed. Summary of the Invention

[0005] The purpose of this invention is to provide a target detection method and related equipment for large-format remote sensing images based on situational awareness and fine-grained perception, so as to solve the problem that existing technologies cannot effectively filter invalid information when performing target detection on large-format high-resolution images.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Firstly, a target detection method for large-format remote sensing images based on situational awareness and fine-grained perception includes the following steps:

[0008] Remote sensing images are acquired and segmented into remote sensing image patches. A pre-trained backbone network is used to extract multi-scale feature maps of the remote sensing image patches.

[0009] Based on the multi-scale feature map, the cut remote sensing image patch is input into the situational awareness classifier to set classification labels for the regions of the remote sensing image patch;

[0010] Target detection is performed on the remote sensing image patch based on the classification label to obtain target detection results for several image patches;

[0011] The target detection results of the several image blocks are fused and post-processed to obtain a complete target detection result.

[0012] In some implementations, the remote sensing image patch and the classification label are used as the input and output of the semantic-aware graph network, respectively, to train a situational awareness classifier.

[0013] In some implementations, the remote sensing image patch is processed sequentially through the ASAM module, DGFUM module, SDM module and Softmax function in the semantic awareness graph network to form a probability distribution on the semantic label of the remote sensing image patch.

[0014] In some implementations, the step of inputting the segmented remote sensing image patch into a situational awareness classifier based on the multi-scale feature map to set classification labels for the regions of the remote sensing image patch specifically includes:

[0015] Based on the multi-scale feature map, the situational awareness classifier determines the target area in the remote sensing image patch;

[0016] When the target area is 0, the category label of the corresponding area is set to 0;

[0017] When 0 < target area ≤ 70*70 pixels, the category label of the corresponding area is set to 1;

[0018] When the target area is greater than 70*70 pixels, the category label for the corresponding area is set to 2.

[0019] In some implementations, the step of performing target detection on the remote sensing image patch based on the classification label specifically includes:

[0020] When the classification label is 0, no target detection is performed;

[0021] When the classification label is 2, the corresponding region is input into Fast-R-CNN for object detection;

[0022] When the classification label is 1, the corresponding region is input into the RoI Transformer for target detection.

[0023] In some implementations, the pre-trained backbone network is ResNet-50.

[0024] Secondly, a large-format remote sensing image target detection system based on situational awareness and fine-grained perception includes:

[0025] The preprocessing module is used to acquire remote sensing images and cut them into remote sensing image patches, and to extract multi-scale feature maps of the remote sensing image patches using a pre-trained backbone network.

[0026] The situational awareness module is used to input the segmented remote sensing image blocks into the situational awareness classifier based on the multi-scale feature map and set classification labels for the regions of the remote sensing image blocks.

[0027] The fine perception module is used to perform target detection on the remote sensing image patch according to the classification label, and obtain target detection results for several image patches;

[0028] The image fusion module is used to fuse and post-process the target detection results of the several image blocks to obtain a complete target detection result.

[0029] Thirdly, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable in the processor, wherein the processor, when executing the computer program, implements the steps of the large-format remote sensing image target detection method based on situational awareness and fine perception.

[0030] Fourthly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the large-format remote sensing image target detection method based on situational awareness and fine-grained perception.

[0031] Fifthly, a computer program product comprising a computer program that, when executed by a processor, implements the steps of the large-format remote sensing image target detection method based on situational awareness and fine perception.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] This invention provides a target detection method for large-format remote sensing images based on situational awareness and fine-grained perception. First, a situational awareness classifier is used to set classification labels for remote sensing image block regions. Then, target detection is performed on the remote sensing image blocks according to the classification labels, which solves the problem that existing technologies cannot filter invalid information during target detection.

[0034] Furthermore, this invention first uses a situational awareness classifier to determine the target area in a remote sensing image patch, and then sets different classification labels based on the target area. When the classification label is 0, no target detection is performed; when the classification label is 2, the corresponding image patch is input into Faster-R-CNN for target detection; when the classification label is 1, the corresponding image patch is input into RoI Transformer for target detection. In summary, situational awareness is performed first, followed by fine-grained awareness. Fine-grained awareness uses different detectors to process image patches at different levels based on the situational awareness results. For large targets in sparse scenes, the fast and accurate characteristics of Faster-R-CNN are fully utilized for detection; for small targets in dense scenes, the more accurate characteristics of RoI Transformer are utilized for detection. Therefore, this invention effectively filters out invalid information when processing large-format, high-resolution images, greatly reducing the time spent on target detection of invalid information in existing technologies and improving target detection speed. In addition, this invention uses dedicated networks for detection of targets of different sizes, which can significantly improve overall accuracy and avoid the problem of poor detection results for small targets. Attached Figure Description

[0035] Figure 1 The images shown are target detection results of large-format high-frequency remote sensing images according to an embodiment of the present invention, where (a) and (b) are target detection results obtained by this embodiment for two different large-format high-frequency remote sensing images, respectively.

[0036] Figure 2 This is a general block diagram of the large-format remote sensing image target detection method based on situational awareness and fine perception proposed in this embodiment of the invention;

[0037] Figure 3 A flowchart illustrating a large-format remote sensing image target detection method based on situational awareness and fine-grained perception provided in an embodiment of the present invention;

[0038] Figure 4 This is a structural diagram of a large-format remote sensing image target detection system based on situational awareness and fine perception, provided in an embodiment of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. The content described herein is for explanation rather than limitation of the present invention.

[0040] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification and claims of this invention are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, systems, products, or devices.

[0041] like Figure 2 and Figure 3 As shown, this embodiment provides a target detection method for large-format remote sensing images based on situational awareness and fine-grained perception, including the following steps:

[0042] S1, preprocessing is performed on a large-format, high-resolution remote sensing image dataset;

[0043] S1.1: For the large-format remote sensing images in the DOTA dataset, a sliding window method is used to cut the original remote sensing images into 1024*1024 remote sensing image blocks;

[0044] S1.2: Extract multi-scale feature maps of image patches using a pre-trained backbone network ResNet-50.

[0045] S2, remote sensing image patches and classification labels are used as the input and output of the semantic-aware graph network (SAGN) respectively, and trained to obtain a situational awareness classifier. The situational awareness classifier can effectively determine whether there is a target and the corresponding region category in the remote sensing image patch, and set classification labels for the region of the remote sensing image patch.

[0046] The semantic-aware graph network specifically includes:

[0047] The Adaptive Semantic Analysis Module (ASAM) uses a 5×5 convolutional layer to segment remotely sensed image patches into multiple semantic regions. The number of output channels corresponds to the number of categories N. During the transformation from input F to F', the maximum value of each feature point along its channel direction is identified, serving as an identifier for that semantic region. These identifiers enable indexing of semantic regions within F'. Here, F represents the input data, and F' represents the output data.

[0048] The Dynamic Graph Feature Update Module (DGFUM) segments remotely sensed image patches into N semantic regions, each represented by its central feature. These regions are treated as nodes in a graph, and the relationships between nodes are quantified by evaluating their similarity using a trainable adjacency matrix. Within the Graph Convolutional Network (GCN) framework, this approach refines visual features by integrating contextual information between semantic regions.

[0049] The Scenario Decision Module (SDM) uses refined visual features and related semantic regions to classify remote sensing image patches into final semantic labels, and applies 1×1 convolutional layers to process these features, with the number of output channels equal to the number of categories.

[0050] Finally, the output of the SDM module is normalized using the Softmax function to ensure that a probability distribution is formed on the semantic labels.

[0051] To incorporate global information, a three-class classification task is designed in the situational awareness classifier. This task generates corresponding classification labels by determining the size of targets present in the segmented remote sensing image patches, which guides the selection of a suitable detector. Specifically:

[0052] When there is no target (i.e., the target area is 0), the corresponding region's category label is set to 0; when 0 < target area ≤ 70*70 pixels, the corresponding region's category label is set to 1; when the target area > 70*70 pixels, the corresponding region's category label is set to 2. If an image contains multiple targets, the number of targets of each category in the image is counted separately, and the area of ​​the target with the largest number of targets is taken as the final result.

[0053] S3 involves inputting remote sensing image patches into different detectors based on classification labels for target detection, achieving accurate target localization and recognition. In the fine-grained perception part, the bounding box coordinates, category, and detection difficulty of the segmented remote sensing image patches are used as training labels.

[0054] Faster R-CNN is derived from Fast R-CNN by introducing a Region Proposal Network (RPN). The RPN can generate Regions of Interest (RoIs) of various sizes, thus exhibiting good adaptability to targets of different sizes. However, Faster R-CNN may suffer from missed detections when dealing with extremely small targets. Nevertheless, compared to other two-stage detection networks, Faster R-CNN performs better in terms of speed. RoITransformer performs better in handling irregularly shaped targets, and because it better preserves the information about relationships between targets, it surpasses Faster R-CNN in handling target density. However, RoITransformer is relatively more complex and computationally more demanding, inevitably resulting in slower speed compared to Faster R-CNN.

[0055] To address this, this embodiment proposes a Situational-Fine Awareness Network (SFANet). For regions without targets, they are directly filtered out without subsequent target detection, reducing computational overhead. For regions with targets, based on the classification labels in S2, the corresponding regions are input into different networks for target detection, obtaining the location and category information of the targets in each image block, which serves as the target detection results for several image blocks.

[0056] Specifically, when the classification label is 0, no object detection is performed; when the classification label is 2, the corresponding image patch is input into Faster-R-CNN for object detection; when the classification label is 1, the corresponding image patch is input into RoI Transformer for object detection.

[0057] S4: Finally, based on the overlap relationship between image blocks, the target detection results of several image blocks are merged and post-processed to obtain the complete target detection result.

[0058] Based on the above, this embodiment conducts experiments on the DOTA-v1.0 dataset:

[0059] The DOTA-v1.0 dataset contains 2806 aerial images, each with a pixel size ranging from 800*800 to 4000*4000, containing objects of different scales, orientations, and shapes. These DOTA images were annotated by experts using 15 common object categories, including: airplanes, ships, tanks, baseball fields, tennis courts, basketball courts, runways, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields, and basketball courts. The fully annotated DOTA images contain 188,282 instances, each labeled with an arbitrary quadrilateral.

[0060] 1. Experimental Details

[0061] Our model was implemented using the PyTorch framework and trained on a single A5000 GPU. The pre-trained backbone networks were tested using ResNet-50. The entire model was trained using a stochastic gradient descent optimizer with a momentum of 0.9 and weight decay of 1e-4. The initial learning rate was set to 0.005. The batch size was set to 1, and the number of training epochs was set to 12. Mean Average Precision (mAP) and FPS (Frames Per Second) were used as metrics throughout the experiments.

[0062] 2. Experimental Results of Data Preprocessing

[0063] For remote sensing images with extremely high resolution in the DOTA-v1.0 dataset, a sliding window method is used to cut large-format images into image blocks of size 1024×1024.

[0064] 3. Experimental results of a single detector head on the DOTA-v1.0 dataset

[0065] As shown in Table 1, the target detection method for large-format remote sensing images based on situational awareness and fine perception adopted in this embodiment is compared with that of Faster-R-CNN and ROI Transformer alone on the DOTA-v1.0 dataset. The method proposed in this embodiment achieves better results in both mAP and FPS.

[0066] Table 1. mAP and FPS of a single detector head on the DOTA-v1.0 dataset

[0067]

[0068] 4. Experimental results of dual-head detection on the DOTA-v1.0 dataset

[0069] Table 2 Comparison of mAP and FPS with single-head detection on the DOTA v-1.0 dataset

[0070]

[0071] As shown in Table 2, the target detection method for large-format remote sensing images based on situational awareness proposed in this embodiment was compared with the situational awareness-only method on the DOTA v-1.0 dataset. The proposed method has higher mAP than Faster R-CNN and faster FPS than RoI Transformer.

[0072] 5. Experimental Results

[0073] like Figure 1 As shown, the method in this embodiment can effectively monitor and identify target objects in remote sensing images, demonstrating the high efficiency and accuracy of the proposed method in target recognition tasks and effectively filtering out invalid information.

[0074] like Figure 4 As shown, this embodiment also provides a large-format remote sensing image target detection system based on situational awareness and fine-grained perception, including:

[0075] The preprocessing module is used to acquire remote sensing images and cut them into remote sensing image patches, and to extract multi-scale feature maps of the remote sensing image patches using a pre-trained backbone network.

[0076] The situational awareness module is used to input the segmented remote sensing image blocks into the situational awareness classifier based on the multi-scale feature map and set classification labels for the regions of the remote sensing image blocks.

[0077] The situation awareness module is specifically used to train a situation awareness classifier by taking the remote sensing image patch and the classification label as the input and output of the semantic awareness graph network, respectively. The remote sensing image patch is processed in the semantic awareness graph network by the ASAM module, DGFUM module, SDM module and Softmax function in sequence to form a probability distribution on the semantic label of the remote sensing image patch.

[0078] When the target area is 0, the situational awareness classifier sets the classification label of the corresponding area to 0;

[0079] When 0 < target area ≤ 70*70 pixels, the situational awareness classifier sets the classification label of the corresponding region to 1;

[0080] When the target area is greater than 70*70 pixels, the situational awareness classifier sets the classification label of the corresponding area to 2.

[0081] The fine perception module is used to perform target detection on the remote sensing image patch according to the classification label, and obtain target detection results for several image patches;

[0082] The fine perception module is specifically used to perform no target detection when the classification label is 0; to input the corresponding region into Faster-R-CNN for target detection when the classification label is 2; and to input the corresponding region into RoI Transformer for target detection when the classification label is 1.

[0083] The image fusion module is used to fuse and post-process the target detection results of the several image blocks to obtain a complete target detection result.

[0084] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0085] This embodiment also provides a computer device, which includes a processor and a memory. The memory is used to store a computer program (in this embodiment, the computer program includes computational components and iterative components, capable of model calculation and model updating). The computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to realize the corresponding method flow or corresponding function. The processor described in this embodiment can be used for the operation of a large-format remote sensing image target detection method based on situational awareness and fine perception.

[0086] This embodiment also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the large-format remote sensing image target detection method based on situational awareness in the above embodiment.

[0087] This embodiment also provides a computer program product, which includes a computer program that, when executed by a processor, implements the corresponding steps of the large-format remote sensing image target detection method based on situational awareness in the above embodiment.

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A target detection method for large-format remote sensing images based on situational awareness and fine-grained perception, characterized in that, Includes the following steps: Remote sensing images are acquired and segmented into remote sensing image patches. A pre-trained backbone network is used to extract multi-scale feature maps of the remote sensing image patches. Based on the multi-scale feature map, the cut remote sensing image patch is input into the situational awareness classifier to set classification labels for the regions of the remote sensing image patch; The remote sensing image patch and classification label are used as the input and output of the semantic perception graph network, respectively, to train a situational awareness classifier. Target detection is performed on the remote sensing image patch based on the classification label to obtain target detection results for several image patches; The target detection results of the several image blocks are fused and post-processed to obtain a complete target detection result; The remote sensing image patch is processed sequentially through the ASAM module, DGFUM module, SDM module and Softmax function in the semantic perception graph network to form a probability distribution on the semantic label of the remote sensing image patch; The step of inputting the segmented remote sensing image patch into the situational awareness classifier based on the multi-scale feature map and setting classification labels for the regions of the remote sensing image patch specifically includes: Based on the multi-scale feature map, the situational awareness classifier determines the target area in the remote sensing image patch; When the target area is 0, the category label of the corresponding area is set to 0; When 0 < target area ≤ 70*70 pixels, the category label of the corresponding area is set to 1; When the target area is greater than 70*70 pixels, the category label for the corresponding area is set to 2.

2. The method for target detection in large-format remote sensing images based on situational awareness and fine-grained perception as described in claim 1, characterized in that, The step of performing target detection on the remote sensing image patch based on the classification label specifically includes: When the classification label is 0, no target detection is performed; When the classification label is 2, the corresponding region is input into Faster-R-CNN for object detection; When the classification label is 1, the corresponding region is input into the RoI Transformer for target detection.

3. The method for target detection in large-format remote sensing images based on situational awareness and fine-grained perception according to claim 1, characterized in that, The pre-trained backbone network is ResNet-50.

4. A large-format remote sensing image target detection system based on situational awareness and fine-grained perception, characterized in that, include: The preprocessing module is used to acquire remote sensing images and cut them into remote sensing image patches, and to extract multi-scale feature maps of the remote sensing image patches using a pre-trained backbone network. The situation awareness module is used to input the segmented remote sensing image patch into the situation awareness classifier based on the multi-scale feature map and set classification labels for the regions of the remote sensing image patch; the remote sensing image patch and the classification labels are used as the input and output of the semantic perception graph network, respectively, to train and obtain the situation awareness classifier; The fine perception module is used to perform target detection on the remote sensing image patch according to the classification label, and obtain target detection results for several image patches; The image fusion module is used to fuse and post-process the target detection results of the several image blocks to obtain a complete target detection result. The remote sensing image patch is processed sequentially through the ASAM module, DGFUM module, SDM module and Softmax function in the semantic perception graph network to form a probability distribution on the semantic label of the remote sensing image patch; The step of inputting the segmented remote sensing image patch into the situational awareness classifier based on the multi-scale feature map and setting classification labels for the regions of the remote sensing image patch specifically includes: Based on the multi-scale feature map, the situational awareness classifier determines the target area in the remote sensing image patch; When the target area is 0, the category label of the corresponding area is set to 0; When 0 < target area ≤ 70*70 pixels, the category label of the corresponding area is set to 1; When the target area is greater than 70*70 pixels, the category label for the corresponding area is set to 2.

5. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable in the processor. When the processor executes the computer program, it implements the steps of the large-format remote sensing image target detection method based on situational awareness as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the large-format remote sensing image target detection method based on situational awareness as described in any one of claims 1 to 3.

7. A computer program product, said computer product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the large-format remote sensing image target detection method based on situational awareness as described in any one of claims 1 to 3.

Images