Large-format remote sensing image target detection method based on situation-fine perception and related equipment

Abstract

The invention discloses a large-breadth remote sensing image target detection method and related equipment based on situation-fine perception, and belongs to the technical field of computer vision, the method comprises the following steps: obtaining a remote sensing image, cutting the remote sensing image into remote sensing image blocks, and extracting a multi-scale feature map of the remote sensing image blocks by using a pre-trained backbone network; based on the multi-scale feature map, inputting the cut remote sensing image blocks into a situation awareness classifier, and setting classification labels for areas of the remote sensing image blocks; performing target detection on the remote sensing image blocks according to the classification labels to obtain a plurality of image block target detection results; and fusing and post-processing the target detection results of the plurality of image blocks to obtain a complete target detection result. According to the invention, the problem that invalid information cannot be effectively filtered when target detection is carried out on a large-format high-resolution image in the prior art can be solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a large-scale remote sensing image target detection method based on situation-fine perception and related equipment. Background Art

[0002] With the development of artificial intelligence technology, large-format high-resolution image target detection technology has gradually been widely used in satellite-borne, airborne and other resource-limited environments or equipment, and plays an increasingly important role in military reconnaissance, detection and early warning, precision guidance, emergency rescue, intelligent transportation, etc. Large-format high-resolution image target detection in environments or equipment with limited storage, computing and other resources can be realized without being affected by network bandwidth, greatly reducing latency, and improving data security and privacy.

[0003] It is worth noting that there are still several key challenges in large-format high-resolution image target detection: (1) With the rapid development of technologies such as drones and satellites, the acquisition of large-format high-resolution images has become a reality and has been widely used in various fields. However, in practical applications, quickly and accurately detecting target areas from these large-format images is still an urgent problem that needs to be solved. (2) Large-format high-resolution images contain a large amount of irrelevant information, and a large amount of computing power, storage power, and bandwidth are used to process irrelevant data, which greatly increases the burden on the hardware system. In most applications, the vast majority of data processed by deep models are low-value areas, and effective information accounts for only a small part of the total data.

[0004] Therefore, when existing technologies process large-format and high-resolution remote sensing images, target detection often consumes a lot of time in processing invalid information, which seriously affects the detection speed. Summary of the invention

[0005] The purpose of the present invention is to provide a large-format remote sensing image target detection method and related equipment based on situation-fine perception, so as to solve the problem that the prior art cannot effectively filter out invalid information when performing target detection on large-format high-resolution images.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, a large-format remote sensing image target detection method based on situation-fine perception comprises the following steps: Acquire a remote sensing image and cut it into remote sensing image blocks, and use a pre-trained backbone network to extract a multi-scale feature map of the remote sensing image blocks; Based on the multi-scale feature map, inputting the cut remote sensing image block into a situation awareness classifier to set a classification label for the area of ​​the remote sensing image block; Performing target detection on the remote sensing image block according to the classification label to obtain target detection results of several image blocks; The target detection results of the several image blocks are fused and post-processed to obtain a complete target detection result.

[0007] In some implementations, the remote sensing image blocks and classification labels are used as input and output of a semantic perception graph network, respectively, and trained to obtain a situation awareness classifier.

[0008] In some embodiments, the remote sensing image block is processed in the semantic perception graph network in sequence through an ASAM module, a DGFUM module, an SDM module, and a Softmax function to form a probability distribution on the semantic label of the remote sensing image block.

[0009] In some embodiments, the step of inputting the cut remote sensing image block into a situation awareness classifier to set a classification label for a region of the remote sensing image block based on the multi-scale feature map specifically includes: Based on the multi-scale feature map, a situation awareness classifier determines the target area in the remote sensing image block; When the target area is 0, the classification label of the corresponding area is set to 0; When 0<target area≤70*70 pixels, the classification label of the corresponding area is set to 1; When the target area is greater than 70*70 pixels, the classification label of the corresponding area is set to 2.

[0010] In some implementations, the step of performing target detection on the remote sensing image block according to the classification label specifically includes: When the classification label is 0, no target detection is performed; When the classification label is 2, the corresponding area is input into Fast-R-CNN for target detection; When the classification label is 1, the corresponding region is input into the RoI Transformer for object detection.

[0011] In some embodiments, the pre-trained backbone network is ResNet-50.

[0012] In the second aspect, a large-format remote sensing image target detection system based on situation-fine perception includes: A preprocessing module, used to acquire remote sensing images and cut them into remote sensing image blocks, and use a pre-trained backbone network to extract multi-scale feature maps of the remote sensing image blocks; A situation awareness module, configured to input the cut remote sensing image block into a situation awareness classifier to set a classification label for a region of the remote sensing image block based on the multi-scale feature map; A fine perception module, used to perform target detection on the remote sensing image block according to the classification label to obtain target detection results of several image blocks; The image fusion module is used to fuse and post-process the target detection results of the plurality of image blocks to obtain a complete target detection result.

[0013] In a third aspect, an electronic device comprises a memory, a processor, and a computer program stored in the memory and executable in the processor, wherein the processor implements the steps of a method for large-format remote sensing image target detection based on situation-fine perception when executing the computer program.

[0014] In a fourth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the large-format remote sensing image target detection method based on situation-fine perception are implemented.

[0015] In a fifth aspect, a computer program product is provided, the computer product comprising a computer program, and the computer program, when executed by a processor, implements the steps of the method for large-format remote sensing image target detection based on situation-fine perception.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a large-format remote sensing image target detection method based on situation-fine perception. First, a classification label is set for a remote sensing image block area through a situation awareness classifier, and then target detection is performed on the remote sensing image block according to the classification label, thereby solving the problem that the prior art cannot filter out invalid information during target detection.

[0017] Further, the present invention first determines the target area in the remote sensing image block by a situation awareness classifier, and then sets different classification labels according to the target area. When the classification label is 0, no target detection is performed. When the classification label is 2, the corresponding image block is input into Faster-R-CNN for target detection; when the classification label is 1, the corresponding image block is input into RoI Transformer for target detection. In general, situation awareness is performed first, and then fine perception is performed. Fine perception uses different detectors to process image blocks of different levels according to the results of situation awareness. For large targets in sparse scenes, the fast and accurate characteristics of Faster-R-CNN are fully utilized for detection; for small targets in dense scenes, RoI Transformer is used for more accurate detection. Therefore, when processing large-format high-resolution images, the present invention effectively filters invalid information, greatly reduces the time spent by the prior art for target detection of invalid information, and improves the target detection speed. In addition, the present invention uses a dedicated network for detection of targets of different sizes, which can significantly improve the overall accuracy and avoid the problem of poor detection effect of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 1 is a diagram showing target detection results of a large-format high-frequency remote sensing image according to an embodiment of the present invention, wherein (a) and (b) are target detection results of two different large-format high-frequency remote sensing images obtained by this embodiment; Figure 2 The overall block diagram of the large-scale remote sensing image target detection method based on situation-fine perception proposed in an embodiment of the present invention; Figure 3 A flow chart of a large-format remote sensing image target detection method based on situation-fine perception provided by an embodiment of the present invention; Figure 4 A structural diagram of a large-format remote sensing image target detection system based on situation-fine perception provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings, and the described content is intended to explain the present invention rather than to limit it.

[0020] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, systems, products or apparatus.

[0021] like Figure 2 and Figure 3 As shown, this embodiment provides a large-format remote sensing image target detection method based on situation-fine perception, comprising the following steps: S1, preprocessing under large-format high-resolution remote sensing image dataset; S1.1: For the large-format remote sensing images in the DOTA dataset, a sliding window method is used for cutting processing, and the original remote sensing images are cut into 1024*1024 remote sensing image blocks; S1.2: Use the pre-trained backbone network ResNet-50 to extract multi-scale feature maps of image patches.

[0022] S2, using the remote sensing image block and the classification label as the input and output of the Semantic-aware Graph Network (SAGN), and training to obtain a situation awareness classifier, so that the situation awareness classifier can effectively determine whether the remote sensing image block contains the target and the category of the corresponding area, and set the classification label for the area of ​​the remote sensing image block; The semantic perception graph network specifically includes: The Adaptive Semantic Analysis Module (ASAM) applies a 5×5 convolutional layer to segment remote sensing image blocks into multiple semantic regions. The number of output channels corresponds to the number of categories N. During the conversion from input F to F', the maximum value of each feature point in its channel direction is identified as the identifier of the semantic region. These identifiers make it possible to index the semantic region in F'. F represents the input data and F' represents the output data.

[0023] The Dynamic Graph Feature Update Module (DGFUM) divides the remote sensing image patch into N semantic regions, each represented by its central feature. These regions are regarded as nodes in the graph, and the relationship between them is quantified by evaluating the similarity between nodes using a trainable adjacency matrix. In the framework of the Graph Convolutional Network (GCN), this method refines the visual features by integrating the contextual information between semantic regions.

[0024] The Scenario Decision Module (SDM) utilizes the refined visual features and associated semantic regions to classify the final semantic labels for remote sensing image patches and applies a 1×1 convolutional layer to process these features, with the number of output channels equal to the number of categories.

[0025] Finally, the output of the SDM module is normalized by the Softmax function to ensure the formation of a probability distribution on the semantic label.

[0026] In order to introduce global information, a three-classification task is designed in the situation awareness classifier. The corresponding classification labels are generated by judging the size of the target in the cut remote sensing image block to guide the selection of a suitable detector. Specifically: When there is no target, that is, the target area is 0, the classification label of the corresponding area is set to 0; when 0<target area≤70*70 pixels, the classification label of the corresponding area is set to 1; when the target area>70*70 pixels, the classification label of the corresponding area is set to 2. If there are multiple targets in an image, the number of each type of target in the image is counted separately, and the area of ​​the target with the largest number is the final result.

[0027] S3, according to the classification labels, the remote sensing image blocks are input into different detectors for target detection to achieve accurate target positioning and recognition. In the fine perception part, the coordinates of the labeled boxes, categories and detection difficulty of the cut remote sensing image blocks are used as training labels.

[0028] Faster-R-CNN is obtained by introducing Region Proposal Network (RPN) on the basis of Fast-R-CNN. RPN can generate Region of Interest (RoI) of various sizes, so it has good adaptability to objects of different sizes. However, Faster-R-CNN may have problems with missed detection when dealing with extremely small objects. Nevertheless, compared with other two-stage detection networks, Faster R-CNN performs better in speed. RoITransformer performs better in dealing with irregularly shaped objects, and because it can better retain the relationship information between objects, it is better than Faster R-CNN in dealing with object density. However, RoITransformer is relatively complex and has higher computational complexity, so it is bound to lag behind Faster R-CNN in speed.

[0029] To this end, this embodiment proposes a situational-fine awareness network (SFANet). For areas without targets, they are directly filtered out and no subsequent target detection is performed to reduce computational overhead. For areas with targets, the corresponding areas are input into different networks for target detection according to the classification labels in S2, and the location information and category information of the target in each image block are obtained as the target detection results of several image blocks.

[0030] Specifically, when the classification label is 0, no target detection is performed; when the classification label is 2, the corresponding image block is input into Faster-R-CNN for target detection; when the classification label is 1, the corresponding image block is input into RoI Transformer for target detection.

[0031] S4: Finally, according to the overlapping relationship between the image blocks, the target detection results of several image blocks are merged and post-processed to obtain the complete target detection result.

[0032] Based on the above content, this embodiment conducts experiments on the DOTA-v1.0 dataset: DOTA-v1.0 dataset: The dataset contains 2806 aerial images, each with a pixel size ranging from 800*800 to 4000*4000, containing objects of different scales, directions and shapes. These DOTA images are annotated by experts using 15 common target categories, including: airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, ground runways, ports, bridges, large vehicles, small vehicles, helicopters, roundabouts, football fields and basketball courts. The fully annotated DOTA images contain 188282 instances, each of which is marked by an arbitrary quadrilateral.

[0033] 1. Experimental details We implemented our models using the PyTorch framework and trained all models on a single A5000 GPU card. The pre-trained backbone networks were tested using ResNet-50 respectively. The entire model was trained by a stochastic gradient descent optimizer with a momentum of 0.9 and a weight decay of 1e-4. The initial learning rate of the network was set to 0.005. The batch size was set to 1 and the number of training epochs was set to 12. The average precision (mAP) and FPS (Frames Per Second) were used as the metrics for the entire experiment.

[0034] 2. Data preprocessing experimental results For the extremely high-resolution remote sensing images in the DOTA-v1.0 dataset, a sliding window method is used to cut the large-format images into image blocks of size 1024×1024.

[0035] 3. Experimental results of a single detection head on the DOTA-v1.0 dataset As shown in Table 1, the large-format remote sensing image target detection method based on situation-fine perception adopted in this embodiment is compared with Faster-R-CNN and ROI Transformer alone on the DOTA-v1.0 dataset. The method proposed in this embodiment achieves better results in both mAP and FPS.

[0036] Table 1 mAP and FPS of a single detection head on the DOTA-v1.0 dataset

[0037] 4. Experimental results of dual detection heads on DOTA-v1.0 dataset Table 2 Comparison of mAP and FPS between dual detection heads and single detection head on DOTA v-1.0 dataset

[0038] As shown in Table 2, the large-format remote sensing image target detection method based on situation-fine perception proposed in this embodiment is compared with the situation-awareness-only method on the DOTA v-1.0 dataset. The method is higher than Faster R-CNN in mAP and faster than RoI Transformer in FPS.

[0039] 5. Experimental results like Figure 1 As shown, the method of this embodiment can effectively monitor and identify the target object in the remote sensing image, which proves the efficiency and accuracy of the method proposed in this embodiment in the target recognition task and can effectively filter out invalid information.

[0040] like Figure 4 As shown, this embodiment also provides a large-format remote sensing image target detection system based on situation-fine perception, including: A preprocessing module, used to acquire remote sensing images and cut them into remote sensing image blocks, and use a pre-trained backbone network to extract multi-scale feature maps of the remote sensing image blocks; A situation awareness module, configured to input the cut remote sensing image block into a situation awareness classifier to set a classification label for a region of the remote sensing image block based on the multi-scale feature map; The situation awareness module is specifically used to use the remote sensing image block and the classification label as the input and output of the semantic perception graph network respectively, and train to obtain a situation awareness classifier. The remote sensing image block is processed in the semantic perception graph network in sequence through the ASAM module, the DGFUM module, the SDM module and the Softmax function to form a probability distribution on the semantic label of the remote sensing image block; When the target area is 0, the situation awareness classifier sets the classification label of the corresponding area to 0; When 0<target area≤70*70 pixels, the situation awareness classifier sets the classification label of the corresponding area to 1; When the target area is greater than 70*70 pixels, the situation awareness classifier sets the classification label of the corresponding area to 2.

[0041] A fine perception module, used to perform target detection on the remote sensing image block according to the classification label to obtain target detection results of several image blocks; The fine perception module is specifically used for not performing target detection when the classification label is 0; when the classification label is 2, inputting the corresponding area into Faster-R-CNN for target detection; when the classification label is 1, inputting the corresponding area into RoI Transformer for target detection.

[0042] The image fusion module is used to fuse and post-process the target detection results of the plurality of image blocks to obtain a complete target detection result.

[0043] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0044] In this embodiment, a computer device is also provided, which includes a processor and a memory, the memory is used to store a computer program (in this embodiment, the computer program includes a computing component and an iterative component, which can perform model calculation and model update), the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the large-scale remote sensing image target detection method based on situation-fine perception.

[0045] This embodiment also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the large-format remote sensing image target detection method based on situation-fine perception in the above embodiment.

[0046] This embodiment also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the corresponding steps of the large-format remote sensing image target detection method based on situation-fine perception in the above embodiment are implemented.

[0047] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0048] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0049] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A large-scale remote sensing image target detection method based on situation-fine perception, characterized in that: The following steps are involved: Acquire a remote sensing image and cut it into remote sensing image blocks, and use a pre-trained backbone network to extract a multi-scale feature map of the remote sensing image blocks; Based on the multi-scale feature map, inputting the cut remote sensing image block into a situation awareness classifier to set a classification label for the area of ​​the remote sensing image block; Performing target detection on the remote sensing image block according to the classification label to obtain target detection results of several image blocks; The target detection results of the several image blocks are fused and post-processed to obtain a complete target detection result.

2. The large-format remote sensing image target detection method based on situation-fine perception according to claim 1 is characterized in that: The remote sensing image blocks and classification labels are used as input and output of a semantic perception graph network respectively, and training is performed to obtain a situation awareness classifier.

3. The large-format remote sensing image target detection method based on situation-fine perception according to claim 2 is characterized in that: The remote sensing image block is processed in the semantic perception graph network in sequence through an ASAM module, a DGFUM module, an SDM module and a Softmax function to form a probability distribution on the semantic label of the remote sensing image block.

4. The large-format remote sensing image target detection method based on situation-fine perception according to claim 3 is characterized in that: The step of inputting the cut remote sensing image block into a situation awareness classifier to set a classification label for the region of the remote sensing image block based on the multi-scale feature map specifically includes: Based on the multi-scale feature map, a situation awareness classifier determines the target area in the remote sensing image block; When the target area is 0, the classification label of the corresponding area is set to 0; When 0<target area≤70*70 pixels, the classification label of the corresponding area is set to 1; When the target area is greater than 70*70 pixels, the classification label of the corresponding area is set to 2.

5. The large-format remote sensing image target detection method based on situation-fine perception according to claim 4 is characterized in that: The step of performing target detection on the remote sensing image block according to the classification label specifically includes: When the classification label is 0, no target detection is performed; When the classification label is 2, the corresponding area is input into Faster-R-CNN for target detection; When the classification label is 1, the corresponding region is input into the RoI Transformer for object detection.

6. The large-format remote sensing image target detection method based on situation-fine perception according to claim 1 is characterized in that: The pre-trained backbone network is ResNet-50.

7. A large-format remote sensing image target detection system based on situation-fine perception, characterized in that: include: A preprocessing module, used to acquire remote sensing images and cut them into remote sensing image blocks, and use a pre-trained backbone network to extract multi-scale feature maps of the remote sensing image blocks; A situation awareness module, configured to input the cut remote sensing image block into a situation awareness classifier to set a classification label for a region of the remote sensing image block based on the multi-scale feature map; A fine perception module, used to perform target detection on the remote sensing image block according to the classification label to obtain target detection results of several image blocks; The image fusion module is used to fuse and post-process the target detection results of the plurality of image blocks to obtain a complete target detection result.

8. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable in the processor, wherein when the processor executes the computer program, the steps of a large-format remote sensing image target detection method based on situation-fine perception as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a large-format remote sensing image target detection method based on situation-fine perception as described in any one of claims 1 to 6 are implemented.

10. A computer program product, the computer product comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of a large-format remote sensing image target detection method based on situation-fine perception as described in any one of claims 1 to 6 are implemented.

Images