Urban remote sensing image target detection method and device

Abstract

The application discloses a kind of urban remote sensing image target detection method and device, comprising: obtaining urban remote sensing image, pre-processing image obtains subgraph;Subgraph is input mixed attention backbone network and carries out feature extraction, obtains feature map;Double detection network is constructed, and the deviation between the predicted rotating boundary frame and the true value is obtained by using smooth-z loss function, the loss value is obtained by iterative optimization, and new boundary frame is obtained;The optimal boundary frame is retained, and the final detection result is output.The method and device of the application can be aimed at satellite or airborne urban remote sensing image, realize the accurate detection of urban remote sensing target with different directions under the bird's-eye view, and have universality.

Description

Technical Field

[0001] This invention relates to the field of target detection technology in computer vision, and in particular to a method and apparatus for target detection in urban remote sensing images. Background Technology

[0002] Target detection in remote sensing images, especially when facing targets from different directions, remains a significant challenge in this field. Existing target detection technologies can be broadly categorized into traditional machine learning-based methods and deep learning-based methods. Traditional machine learning methods search for and classify targets within a given image using a sliding window, typically requiring manual feature design. Deep learning-based methods generally involve four steps: image feature extraction, image feature fusion, target classification and regression, and backpropagation.

[0003] For traditional remote sensing image target detection tasks, most algorithms generally use the sliding window method to obtain candidate regions and then classify and identify targets of interest based on these regions. However, this method requires manual feature design beforehand, and the designed features sometimes fail to effectively extract image feature information. Furthermore, the presence of targets of varying scales and angles in remote sensing images further hinders the application of traditional target detection methods. With the continuous development of deep learning technology, more and more deep learning algorithms are being applied to the remote sensing field. The unique big data transfer learning method of deep learning has further improved remote sensing information extraction technology. Not only are the underlying traditional features of remote sensing images, such as texture and shape, being utilized to a great extent, but the extraction of semantic features from remote sensing images also makes the classification and identification of remote sensing targets faster and more accurate, greatly improving the accuracy of remote sensing image target detection. However, in recent years, most remote sensing image target detection algorithms still use the general horizontal bounding box detection method. While this can solve the problem of multi-scale and multi-target remote sensing images to some extent, it still cannot effectively address the problem of multi-angle targets in remote sensing images, especially the background redundancy interference caused by objects and the inaccurate bounding box localization. When dealing with closely spaced objects, the inherent limitations of the general horizontal bounding box lead to significant overlap between multiple bounding boxes, introducing excessive background and other useless information, which greatly affects the detection performance of remote sensing images. Rotated bounding box detection methods, however, can make the bounding boxes fit the target edges more closely by rotating them at appropriate angles, suppressing interference from useless information. Therefore, research on remote sensing image target detection algorithms based on rotated bounding boxes is an important research direction in the field of remote sensing. Summary of the Invention

[0004] This invention provides a method and apparatus for detecting targets in urban remote sensing images, in order to solve the technical problem that existing technologies have difficulty achieving good detection results for urban remote sensing targets with varying angles.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] In a first aspect, embodiments of this application propose a method for target detection in urban remote sensing images, comprising:

[0007] Acquire urban remote sensing images and preprocess the images to obtain sub-images;

[0008] The subgraph is input into a hybrid attention backbone network for feature extraction to obtain a feature map.

[0009] A dual detection network is constructed to process the feature map and obtain the predicted rotated bounding box;

[0010] The deviation between the predicted rotated bounding box and the ground truth is obtained by using the smooth-z loss function, and the new bounding box is obtained by iteratively optimizing the loss value.

[0011] Preserve the optimal bounding box and output the final detection result.

[0012] Furthermore, the urban remote sensing images are visible light images captured by satellite or airborne sensors;

[0013] The preprocessing involves cutting the original image into multiple smaller images and inputting them into the network's attention backbone. Through image stitching and post-processing, the prediction results of the sub-images are then merged into a larger image.

[0014] Furthermore, the feature extraction includes:

[0015] Urban remote sensing images The input is fed into a deep convolutional neural network model, which extracts features from the global and local information of the target in the image through hybrid self-attention, and finally outputs an integrated feature map. .

[0016] Furthermore, a detection decoupling network is constructed within the dual detection network. This network predicts the target's category information and location / angle information separately by splitting the classification and regression operations, including:

[0017] For the input feature map After classification, the target's category information C is obtained; after regression, the target's position and angle information are obtained. Where x and y are the x-coordinate and y-coordinate of the center point of the bounding box, respectively, and h and w are the length and width of the bounding box, respectively. This represents the rotation angle of the bounding box.

[0018] Furthermore, an angle correction network is constructed within the dual detection network, and the corrected angle information of the target is obtained through angle regression, resulting in the corrected predicted rotated bounding box, including:

[0019] feature map Similarly, regression operations are performed on the input angle correction network to obtain the correction angle information. , ,and Find the L1 norm to obtain the bias. ,like If it is greater than the preset threshold x, then Give The purpose is to obtain the corrected rotation angle, and vice versa. Then, the obtained position and angle information are fused to finally output the predicted rotation bounding box.

[0020] Furthermore, a smooth-z loss function is proposed to obtain the deviation between the predicted rotated bounding box and the ground truth. By iteratively optimizing the loss value, a new bounding box is obtained, including:

[0021] Based on the obtained predicted rotated bounding boxes, an initial loss value between the predicted rotated bounding boxes and the ground truth is obtained using a loss function; then, the urban remote sensing image is processed. Re-extract feature points, iterate a preset number of times, and obtain The set of loss values ​​under real-label supervision { }, select As loss value The lowest loss result is used to update the position and angle information; where, This represents the loss value obtained in the Nth iteration. The result is given by N, which represents the preset number of iterations, and min, which represents the number of iterations to minimize the loss value. Finally, the loss function converges to obtain the new bounding box.

[0022] Furthermore, the process of preserving the optimal bounding box and outputting the final detection result includes:

[0023] A new list of bounding boxes is generated from all the obtained bounding boxes. The bounding boxes are then sorted using a formula to obtain the coordinates and confidence scores of the optimal bounding box.

[0024] Secondly, embodiments of this application also propose an urban remote sensing image target detection device, used to implement the method described in any embodiment of the first aspect of this application, including: an acquisition module, a detection module, and a selection module.

[0025] The acquisition module is used to acquire urban remote sensing images and preprocess the images to obtain sub-images.

[0026] The detection module includes a hybrid attention unit, a dual detection network unit, and an optimization unit. The hybrid attention unit extracts features from the image to obtain a feature map. The dual detection network unit processes the feature map to obtain a predicted rotated bounding box. The optimization unit uses a smooth-z loss function to obtain the deviation between the predicted rotated bounding box and the ground truth, and iteratively optimizes the loss value to obtain a new bounding box.

[0027] The selection module is used to retain the optimal bounding box and output the final detection result.

[0028] This application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the first aspect of this application.

[0029] This application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any embodiment of the first aspect of this application.

[0030] The beneficial effects of the technical solution provided by this invention include at least the following:

[0031] This invention provides a method and apparatus for target detection in urban remote sensing images. First, image sub-images are obtained through preprocessing. Then, a hybrid attention backbone network is used to extract target features. Next, a dual detection network is used to perform classification, regression, and angle correction on the feature maps. The smooth-z loss function is used to optimize the deviation between predicted and ground truth values. Finally, optimal selection of bounding boxes is performed, achieving accurate target detection in urban remote sensing images. The detection method provided by this invention can accurately detect rotating targets in urban remote sensing images. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating a method for detecting targets in urban remote sensing images provided in an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the hybrid attention network structure for acquiring feature maps provided in an embodiment of the present invention;

[0035] Figure 3This is a schematic diagram of the dual detection network structure for obtaining the predicted rotated bounding box provided in an embodiment of the present invention;

[0036] Figure 4 This is a flowchart illustrating the method for optimizing and updating the smooth-z loss function to obtain a new bounding box, provided in an embodiment of the present invention.

[0037] Figure 5 This is a flowchart illustrating the optimal bounding box filtering method provided in an embodiment of the present invention;

[0038] Figure 6 This is an embodiment of the urban remote sensing image target detection device of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0040] Embodiments of the present invention provide a method for target detection in urban remote sensing images, such as... Figure 1 As shown, the method includes the following steps:

[0041] Step 110: Acquire urban remote sensing images and preprocess the images to obtain sub-images;

[0042] Step 120: Input the subgraph into the hybrid attention backbone network for feature extraction to obtain the feature map;

[0043] Step 130: Construct a dual detection network to process the feature map and obtain the predicted rotated bounding box;

[0044] Step 140: Use the smooth-z loss function to obtain the deviation between the predicted rotated bounding box and the true value, and obtain the new bounding box by iteratively optimizing the loss value;

[0045] Step 150: Retain the optimal bounding box and output the final detection result.

[0046] To address the problem of target detection in remote sensing images, this embodiment provides a novel method for target detection in urban remote sensing images, which can be implemented using electronic devices. The method involves inputting urban remote sensing images into a computer, extracting image features based on hybrid attention, using a dual detection network to regress and correct the angle information of the feature map position angle, calculating the rotation box deviation using the smooth-z loss function, filtering bounding boxes using an optimal bounding box selection method, and outputting the final rotation detection box.

[0047] Furthermore, such as Figure 2 As shown, step 120 involves inputting the subgraph into a hybrid attention backbone network for feature extraction to obtain a feature map. This specifically includes:

[0048] A hybrid attention mechanism is used to extract features from the input urban remote sensing sub-map.

[0049] The main process of the hybrid attention mechanism is to compress the input feature map through convolution. The compressed feature map is then input into a multi-head self-attention module to extract spatial features of key image regions. Next, an upsampling operation expands the feature map to its original size, and it is concatenated with the input feature map. The fused feature map undergoes the same operation again to obtain local features. The formula for self-attention is as follows:

[0050]

[0051] Where Q represents the query vector, K represents the key vector, and V represents the weights. Represents a vector The dimension of the matrix is ​​T, where T represents the transpose of the matrix.

[0052] Simultaneously, the feature maps are subjected to max pooling and average pooling respectively, and the pooled feature maps are then summed by channel addition. Convolution and activation operations are then performed on the summed feature maps to obtain the global image features. The two new feature maps are then concatenated to form a single feature map, which is the hybrid attention processing procedure.

[0053] Furthermore, such as Figure 3 As shown, step 130 involves constructing a dual detection network to process the feature map and obtain the predicted rotated bounding box. Specifically, this includes:

[0054] A detection decoupling network is constructed in the dual detection network to predict the target's category information and position angle information by splitting the classification and regression operations.

[0055] The feature map is classified by treating the target's label category as a discrete value, thus treating the target category as a classification problem. The network detection head uses a classifier, and the classifier outputs a corresponding number of values ​​based on the number of target categories to be predicted, each representing a prediction score for a positive sample belonging to a particular category. Assuming the urban remote sensing image training set is... ,in, This represents the output feature vector of the detection head; This represents the true value of the sample, where the value is the pre-defined label indicating the true class; n is the number of samples in the training set. After forward propagation of the network, for a single training sample, the output of the classifier can be represented as...

[0056]

[0057] Where T denotes the transpose of the matrix. These are the weight parameters connecting the neurons in the detection head to the i-th output neuron of the softmax classifier; It is a probability vector, where the sum of each term is 1. Each term represents the probability value of the sample belonging to the corresponding category, and the category with the highest probability is taken as the classification result.

[0058] The feature map is regressed as follows: First, the translation, size, and angle changes of the target box are modeled. Then, L2 norm regularization is applied for least-squares linear regression to avoid overfitting data with high-dimensional features. Finally, the refined target box is output. The purpose of bounding box regression is to learn a mapping relationship between the correct ground-truth box and the region proposal box. Where P represents the region candidate box. These represent the horizontal and vertical coordinates of the center of the image rectangle, the length and width of the rectangle, and the rotation angle of the rectangle, respectively. Here, G represents the correct annotation box. Five learnable functions are used. , , , , Converting the candidate region box P into the correct bounding box G: In reality, due to the existence of errors, the converted bounding box... It generally does not completely coincide with the correct annotation box G. (P to...) The transformation includes translation and scaling of the bounding box. , The corresponding bounding box translation transformation is as follows: , , , The corresponding bounding box translation transformation is as follows:

[0059] , ,

[0060] The corresponding angle transformations are as follows: .

[0061] An angle correction network is constructed in the dual detection network, and the corrected angle information of the target is obtained through angle regression operation to obtain the corrected predicted rotating bounding box.

[0062] The specific method is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Similarly, the correction angle information is obtained by performing a regression operation on the correction network. , , Calculate the L1 norm of each value with the true value to obtain their respective deviations. If the deviation is minimized, then... Give The purpose is to obtain the corrected rotation angle, and conversely, to keep it unchanged. The angle correction transformation can be expressed as: Then use angle correction transformation replace The corrected angle can then be obtained. .

[0063] Furthermore, such as Figure 4 As shown, step 140 involves using the smooth-z loss function to obtain the deviation between the predicted rotated bounding box and the ground truth. The loss value is then iteratively optimized to obtain a new bounding box. Specifically, this includes:

[0064] The deviation between the predicted rotated bounding box and the ground truth is obtained using the smooth-z loss function. The new bounding box is obtained by iteratively optimizing the loss value.

[0065] The design process of the smooth-z loss function is as follows:

[0066] When calculating the bounding box during training, considering the weak correlation between the general detector head and horizontal bounding box classification and regression, the classification score and regression localization cannot be effectively linked, resulting in unreliable results. Therefore, a matching degree is used in anchor box assignment to measure the correlation between the two, promoting further convergence of the regression loss. The regression parameters are as follows:

[0067]

[0068]

[0069]

[0070]

[0071] in These represent the center coordinates, width, height, and angle of the actual bounding box, respectively. These represent the center coordinates, width, height, and angle of the anchor frame, respectively. These represent the center coordinates, width, height, and angle of the predicted bounding box, respectively. Represents the anchor box and the actual value deviation, Similarly; Represents the predicted bounding box and anchor box deviation, Similarly.

[0072] Based on the above, in this embodiment, the optimization iteration execution process is as follows:

[0073] (1) First calculate the state and activation value of each layer until the last layer;

[0074] (2) Calculate the error of each layer. The error calculation process proceeds from the last layer forward.

[0075] (3) Calculate the gradient of the connection weights for each neuron;

[0076] (4) Update the parameters according to the gradient descent rule.

[0077] Iterate through the above steps until the stopping criterion is met.

[0078] Furthermore, such as Figure 5 As shown, to implement step 150, an optimal bounding box filtering method is proposed to retain the optimal bounding box and output the final detection result, specifically including:

[0079] First, a new list of bounding boxes is generated for each target. Then, the coordinates and confidence score of the optimal bounding box are calculated using a formula. The confidence score of the optimal bounding box is set to the average confidence score of all the boxes that form it. The coordinates of the optimal bounding box are the weighted sum of the coordinates of the boxes that make it up, where the weights are the confidence scores of the respective boxes. The calculation formula is as follows:

[0080]

[0081] Where C is the confidence level of the optimal bounding box. Let A be the confidence score of the i-th detection box in the list, and let A be the optimal selection coefficient. The coordinates of the merged boxes in the updated list are given by N, where N represents the number of bounding boxes. Therefore, boxes with higher confidence contribute more to the merged box coordinates than those with lower confidence.

[0082] To implement the method of any embodiment of the first aspect of this application, embodiments of this application also propose an urban remote sensing image target detection device, such as... Figure 6 As shown, it includes:

[0083] The acquisition module 610 acquires urban remote sensing images and preprocesses the images to obtain sub-images, as in step 110.

[0084] The detection module 600 further includes a hybrid attention unit 620, a dual detection network unit 630, and an optimization unit 640. The hybrid attention unit is used to extract features from the image and obtain a feature map, as in step 120.

[0085] The dual detection network unit obtains the predicted rotated bounding box by processing the feature map, as in step 130.

[0086] The optimization unit obtains the deviation between the predicted rotated bounding box and the true value through the smooth-z loss function, and obtains a new bounding box by iteratively optimizing the loss value, as in step 140.

[0087] Select module 650 to retain the optimal bounding box and output the final detection result, as in step 160.

[0088] Furthermore, embodiments of further optimizations of each module of the device in this application for implementing steps 110-150 are shown in Figures 2-6 and related descriptions, and will not be repeated here.

[0089] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0091] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] It should also be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0094] Finally, it should be noted that the above description represents a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for target detection in urban remote sensing images, characterized in that, include: Acquire urban remote sensing images and preprocess the images to obtain sub-images; The subgraph is input into a hybrid attention backbone network for feature extraction to obtain a feature map. A dual detection network is constructed to process the feature maps and obtain the predicted rotated bounding boxes, including: Construct a detection decoupling network in a dual detection network, and predict the target's category information and position angle information separately by splitting the classification and regression operations; An angle correction network is constructed within the dual detection network. Simultaneously, angle regression is used to obtain the corrected angle information of the target, resulting in the corrected predicted rotation bounding box. The feature map is then used to... Similarly, regression operations are performed on the input angle correction network to obtain the correction angle information. , ,and Find the L1 norm to obtain the bias. ,like If it is greater than the preset threshold x, then Give The purpose is to obtain the corrected rotation angle, and vice versa. Then, the obtained position and angle information are fused to finally output the predicted rotation bounding box. The smooth-z loss function is used to obtain the deviation between the predicted rotated bounding box and the ground truth value. The loss value is then iteratively optimized to obtain a new bounding box. Specifically, based on the obtained predicted rotated bounding box, the initial loss value between the predicted rotated bounding box and the ground truth value is obtained using the loss function. Then, the urban remote sensing image is processed... Re-extract feature points, iterate a preset number of times, and obtain The set of loss values ​​under real-label supervision { }, select As loss value The lowest loss result is used to update the position and angle information; where, This represents the loss value obtained in the Nth iteration. The result is that N represents the preset number of iterations, and min represents the number of iterations to minimize the loss value; finally, the loss function converges to obtain the new bounding box. Preserve the optimal bounding box and output the final detection result.

2. The method for target detection in urban remote sensing images as described in claim 1, characterized in that, The urban remote sensing images are visible light images captured by satellite or airborne sensors; The preprocessing involves cutting the original image into multiple smaller images and inputting them into the network's attention backbone. Through image stitching and post-processing, the prediction results of the sub-images are then merged into a larger image.

3. The method for target detection in urban remote sensing images as described in claim 2, characterized in that, The feature extraction process includes: Urban remote sensing images The input is fed into a deep convolutional neural network model, which extracts features from the global and local information of the target in the image through hybrid self-attention, and finally outputs an integrated feature map. .

4. The method for target detection in urban remote sensing images as described in claim 1, characterized in that, The detection decoupling network in the constructed dual detection network predicts the target's category information and position angle information separately by splitting the classification and regression operations, including: For the input feature map After classification, the target's category information C is obtained; after regression, the target's position and angle information are obtained. Where x and y are the x-coordinate and y-coordinate of the center point of the bounding box, respectively, and h and w are the length and width of the bounding box, respectively. This represents the rotation angle of the bounding box.

5. The method for target detection in urban remote sensing images as described in claim 1, characterized in that, The optimal bounding box is retained, and the final detection result is output, including: A new list of bounding boxes is generated from all the obtained bounding boxes. The bounding boxes are then sorted using a formula to obtain the coordinates and confidence scores of the optimal bounding box.

6. A target detection device for urban remote sensing images, used to implement the method described in any one of claims 1 to 5, characterized in that, include: The acquisition module is used to acquire urban remote sensing images and preprocess the images to obtain sub-images; The detection module includes a hybrid attention unit, a dual detection network unit, and an optimization unit; The hybrid attention unit is used to extract features from the image and obtain a feature map; The dual detection network unit obtains the predicted rotated bounding box by processing the feature map; The optimization unit obtains the deviation between the predicted rotated bounding box and the true value through the smooth-z loss function, and obtains a new bounding box by iteratively optimizing the loss value. The selection module is used to retain the optimal bounding box and output the final detection result.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1 to 5.

Images