A method for remote sensing image target detection based on rotation region generation network

By designing a rotating area generation network, using the rotating candidate area and multi-task loss function, the problem of poor rotation object detection in remote sensing images is solved, and high-precision effect in multi-directional object detection is achieved.

CN112395975BActive Publication Date: 2024-11-29NANJING HONGTU ARTIFICIAL INTELLIGENCE TECH RES INST CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202011284348.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-11-17
Publication Date
2024-11-29
Estimated Expiration
2040-11-17

AI Technical Summary

Technical Problem

It is difficult to extract the feature description of rotational target detection in remote sensing images, and the prior art is poor in the detection of multidirectional targets.

Method used

Design a remote sensing image object detection method based on the rotating area generation network, including prediction of rotation candidate areas, rotating ROI pooling and post-processing module of oblique frame NMS, and conduct network training through five-tuple rotating bounding box marking and multi-task loss function , generate target candidate area with rotation angle information.

Benefits of technology

The accuracy of multi-directional object detection in remote sensing images is significantly improved, especially in different fine-grained detection tasks on the HRSC2016 data set, verifying the feasibility and effectiveness of the rotary object detection method.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112395975B_ABST
    Figure CN112395975B_ABST
Patent Text Reader

Abstract

The present invention discloses a remote sensing image target detection method based on a rotation region generation network. The process of rotating target detection using this method includes the following steps: obtaining a remote sensing image data set, selecting a training set and a test set according to a certain ratio, and marking the training set with a five-tuple rotation bounding box; building a deep learning network framework, designing a network loss function for network training; using the training set images to train the network framework; using the trained model to test the test set to obtain a target detection result with rotation angle information. This method can effectively solve the problem of extracting features with rotation invariance to describe different orientations of the target, while ensuring efficiency, improving the robustness of the network in detecting the rotating target, and verifying the effectiveness of the detection network on a public data set. This method can be widely used in target detection tasks with multi-directional targets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of pattern recognition and image processing, and in particular to a remote sensing image target detection method based on a rotation region generation network. Background Art

[0002] Another important issue in the application of remote sensing image target detection is the detection of rotating targets. In ordinary images, targets are all horizontal. Due to the effect of gravity, the targets to be detected generally have certain directional prior knowledge, such as trees grow vertically, and the wheels of cars are generally facing down and not upside down. However, due to the perspective, the perspective in remote sensing images is basically looking down from a high altitude, and the direction of the target is uncertain, which can present different directions. Therefore, how to extract features with rotation invariance to describe the different orientations of the target becomes an important issue. In general target detection tasks, the anchor frame set by {8×8, 16×16, 32×32}×{1∶1, 1∶2, 2∶1} is generally used to cover most common targets in natural scenes. However, for target detection tasks in some special scenes, such as text line detection and remote sensing target detection, the robustness of this anchor frame setting aligned with the coordinate axis is poor. Summary of the invention

[0003] Purpose of the invention: In view of the above problems, the present invention proposes a remote sensing image target detection method based on a rotation region generation network, which solves the problem of detecting multi-directional targets in remote sensing images by designing a rotation region generation network that includes prediction of rotated target candidate regions, rotation ROI pooling, and oblique frame NMS post-processing modules.

[0004] Technical solution: To achieve the purpose of the present invention, the present invention proposes a remote sensing image target detection method based on a rotation region generation network, which specifically includes the following steps:

[0005] Step 1: Obtain a remote sensing image dataset, select a training set and a test set according to a certain ratio, and mark the training set with a five-tuple rotation bounding box;

[0006] Step 2: Build a deep learning network framework and design a network loss function for network training;

[0007] Step 3: Use the training set images to train the network framework;

[0008] Step 4: Use the trained model to test the test set to obtain the target detection results with rotation angle information.

[0009] Furthermore, in step 1, a remote sensing image data set is obtained, a training set and a test set are selected according to a certain ratio, and the training set is marked with a five-tuple rotation bounding box, as follows:

[0010] Obtain a remote sensing image dataset and divide it into a training set and a test set according to a certain ratio. For the training set, use a five-tuple (x, y, w, h, θ) to encode the position information of the rotated bounding box in the original image for each target to be detected on each image, that is, the true bounding box. Among them, (x, y) represents the coordinates of the center point of the true bounding box of the target in the original image, (w, h) represents the width and height of the true bounding box of the target, θ represents the angle between the true bounding box of the target and the positive direction of the horizontal x-axis of the original image, and θ is specified to be in the interval [0, π). Record each true bounding box containing five-tuple information and its corresponding target category information in each picture in the training set into a text file to generate a labeling file to complete the labeling of the training set. No labeling is required for the test set.

[0011] Furthermore, in step 2, a deep learning network framework is built and a network loss function is designed for network training, and the method is as follows:

[0012] Build a deep learning network framework and select the ResNet network as the backbone network to convolve the images in the training set to generate feature maps.

[0013] A rotation candidate region generation network is added after the ResNet network to predict and distinguish the foreground and background on the feature map. The input of the rotation candidate region generation network is the feature map, and the output is the target candidate region with rotation angle information predicted on the feature map, that is, the prediction box. The target candidate region includes five parameters (x, y, w, h, θ). The rotation candidate region generation network adds the θ angle parameter on the basis of the traditional RPN network, that is, when generating the prediction box parameters, the original (x, y, w, h) 4 parameters are increased to (x, y, w, h, θ) 5 parameters.

[0014] The rotation candidate region generation network adopts a multi-task loss function, where the classification loss is:

[0015] L cls (p,u) = -log pu

[0016] Where u is defined as the sample category label of the rotated bounding box. When u = 1, it means that the rotated bounding box contains the target, that is, the foreground; when u = 0, it means that the rotated bounding box does not contain the target, that is, the background, and does not participate in the calculation; p is the classification probability of the prediction box calculated by the Softmax function, that is, the probability that the prediction box generated by the rotated candidate region generation network is the foreground.

[0017] The regression loss is:

[0018]

[0019] Where v = (v x, v y , v w , v h , v θ ) represents the five prediction parameters of the prediction box of the candidate region, v * =(v x * , v y * , v w * , v h * , v θ * ) represents the five parameters of the true value. The function is defined as:

[0020]

[0021] In order to achieve scale invariance, it is also necessary to * Perform scale normalization. Assume that the current anchor box parameters are (x a ,y a , w a ,h a ,θ a ), then v and v * After scale normalization, we get:

[0022]

[0023]

[0024]

[0025]

[0026] In the formula, x, x a , x * Represent the horizontal coordinates of the predicted bounding box, anchor box, and true bounding box, respectively. The same is true for y, w, h, and θ; Among them, k∈Z makes Same reason.

[0027] After the rotation candidate region generation network, a rotation ROI pooling layer is added. The input of the rotation ROI pooling layer is the feature map and the target candidate region output by the rotation candidate region generation network. The output is a fixed-size feature vector generated from the feature map according to the candidate region. Assume that the hyperparameter of the rotation ROI pooling layer is W r and H r , that is, each rotated target candidate region of size w×h generated by the rotated candidate region generation network is divided into W r ×H rsub-regions, each of which has a size of w s ×h s , where w s =w / W r ,h s =h / H r Each segmented sub-region has the same rotation angle as the entire candidate region, and the sub-region is rotated at the same angle as the candidate region within the sub-region. s ]×[h s ], the feature values ​​of the sampling points are calculated by bilinear interpolation, and then the sampling points in each sub-region are max-pooled to obtain a feature map of fixed size.

[0028] Finally, a fully connected layer is added after the rotation ROI pooling layer. According to the generated feature vector, the target position and category information corresponding to the input image are generated through the parallel regression layer and classification layer to obtain the final result.

[0029] The entire network model adopts a multi-task loss function, and its formula is as follows:

[0030] L(p,u,v * , v)=L cls (p,u)+λL loc (v * , v)

[0031] Where λ represents the penalty coefficient of regression loss, and the definitions of other symbols are the same as before.

[0032] For classification loss, the Softmax loss function is used; for regression loss, only the loss when u=1 is calculated, and the loss is smoothed in L 1 The IoU penalty coefficient is added to the loss function. IoU represents the intersection-over-union ratio of two boxes. 1 is defined as:

[0033]

[0034] Furthermore, in step 3, the network framework is trained using the training set images in the following manner:

[0035] Use the labeled training set and labeled files to train the built model. When setting the training, when the total loss L(p, u, v * , v) is less than the preset value, the iterative training is stopped.

[0036] Furthermore, in step 4, the trained model is used to test the test set to obtain target detection results with rotation angle information, and the method is as follows: the test set image is input into the network to obtain the target detection results with rotation angle information.

[0037] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0038] This paper proposes a remote sensing image target detection algorithm based on a rotating region generation network. Compared with other networks, such as the SRBBS+ oblique frame NMS method and the RR-CNN network, the RRDet network has significantly improved the detection accuracy of most categories of targets, and has relatively good detection results in different fine-grained detection tasks on the HRSC2016 dataset, which proves that the proposed network has a strong detection capability for multi-category rotating ship targets, and verifies the feasibility and effectiveness of the rotating target detection method designed in this paper. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a specific flow chart of the method of the present invention;

[0040] Figure 2 It is the overall network structure diagram;

[0041] Figure 3 It is the anchor frame design drawing;

[0042] Figure 4 It is a schematic diagram of rotation ROI pooling. DETAILED DESCRIPTION

[0043] The details of the present invention are further described below with reference to the accompanying drawings and examples.

[0044] The present invention proposes a remote sensing image target detection method based on a rotation region generation network. Figure 1 As shown, the specific steps include:

[0045] Step 1: Obtain a remote sensing image dataset, select a training set and a test set according to a certain ratio, and mark the training set with a five-tuple rotation bounding box, as follows:

[0046] Obtain a remote sensing image dataset and divide it into a training set and a test set according to a certain ratio. For the training set, use a five-tuple (x, y, w, h, θ) to encode the position information of the rotated bounding box in the original image for each target to be detected on each image, that is, the true bounding box. Among them, (x, y) represents the coordinates of the center point of the true bounding box of the target in the original image, (w, h) represents the width and height of the true bounding box of the target, θ represents the angle between the true bounding box of the target and the positive direction of the horizontal x-axis of the original image, and θ is specified to be in the interval [0, π). Record each true bounding box containing five-tuple information and its corresponding target category information in each picture in the training set into a text file to generate a labeling file to complete the labeling of the training set. No labeling is required for the test set.

[0047] Step 2: Build a deep learning network framework and design a network loss function for network training, as follows:

[0048] refer to Figure 2 , build a deep learning network framework, select the ResNet network as the backbone network, and use it to convolve the images in the training set to generate feature maps.

[0049] A rotation candidate region generation network is added after the ResNet network to predict and distinguish the foreground and background on the feature map. The input of the rotation candidate region generation network is the feature map, and the output is the target candidate region with rotation angle information predicted on the feature map, that is, the prediction box. The target candidate region includes five parameters (x, y, w, h, θ). The rotation candidate region generation network adds the θ angle parameter on the basis of the traditional RPN network, that is, when generating the prediction box parameters, the original (x, y, w, h) 4 parameters are increased to (x, y, w, h, θ) 5 parameters.

[0050] The rotation candidate region generation network adopts a multi-task loss function, where the classification loss is:

[0051] L cls (p,u) = -log pu

[0052] Where u is defined as the sample category label of the rotated bounding box. When u = 1, it means that the rotated bounding box contains the target, that is, the foreground; when u = 0, it means that the rotated bounding box does not contain the target, that is, the background, and does not participate in the calculation; p is the classification probability of the prediction box calculated by the Softmax function, that is, the probability that the prediction box generated by the rotated candidate region generation network is the foreground.

[0053] The regression loss is:

[0054]

[0055] Where v = (v x , vy , v w , v h , v θ ) represents the five prediction parameters of the prediction box of the candidate region, v * =(v x * , v y * , v w * , v h * , v θ * ) represents the five parameters of the true value. The function is defined as:

[0056]

[0057] In order to achieve scale invariance, it is also necessary to * Perform scale normalization. Assume that the current anchor box parameters are (x a ,y a , w a ,h a ,θ a ), then v and v * After scale normalization, we get:

[0058]

[0059]

[0060]

[0061]

[0062] In the formula, x, x a , x * Represent the horizontal coordinates of the predicted bounding box, anchor box, and true bounding box, respectively. The same is true for y, w, h, and θ; Among them, k∈Z makes Same reason.

[0063] After the rotation candidate region generation network, a rotation ROI pooling layer is added. The input of the rotation ROI pooling layer is the feature map and the target candidate region output by the rotation candidate region generation network. The output is a fixed-size feature vector generated from the feature map according to the candidate region. Assume that the hyperparameter of the rotation ROI pooling layer is W r and H r , that is, each rotated target candidate region of size w×h generated by the rotated candidate region generation network is divided into W r ×H rsub-regions, each of which has a size of w s ×h s , where w s =w / W r ,h s =h / H r Each segmented sub-region has the same rotation angle as the entire candidate region, and the sub-region is rotated at the same angle as the candidate region within the sub-region. s ]×[h s ], the feature values ​​of the sampling points are calculated by bilinear interpolation, and then the sampling points in each sub-region are max-pooled to obtain a feature map of fixed size.

[0064] Finally, a fully connected layer is added after the rotation ROI pooling layer. According to the generated feature vector, the target position and category information corresponding to the input image are generated through the parallel regression layer and classification layer to obtain the final result.

[0065] The entire network model adopts a multi-task loss function, and its formula is as follows:

[0066] L(p,u,v * , v)=L cls (p,u)+λL loc (y * , v)

[0067] Where λ represents the penalty coefficient of regression loss, and the definitions of other symbols are the same as before.

[0068] For classification loss, the Softmax loss function is used; for regression loss, only the loss when u=1 is calculated, and the loss is smoothed in L 1 The IoU penalty coefficient is added to the loss function. IoU represents the intersection-over-union ratio of two boxes. 1 is defined as:

[0069]

[0070] Step 3: Use the training set images to train the network framework as follows:

[0071] Use the labeled training set and labeled files to train the built model. When setting the training, when the total loss L(p, u, v * , v) is less than the preset value, the iterative training is stopped.

[0072] Step 4: Use the trained model to test the test set to obtain the target detection results with rotation angle information, as follows: Input the test set images into the network to obtain the target detection results with rotation angle information.

[0073] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for remote sensing image target detection based on a rotation region generation network. It is characterized in that The method comprises the following steps: Step 1: Obtain a remote sensing image dataset, select a training set and a test set according to a certain ratio, and mark the training set with a five-tuple rotation bounding box; Step 2: Build a deep learning network framework, design a network loss function for network training, build a deep learning network framework, design a network loss function for network training, and use the training set images to train the network framework. The method is as follows: Build a deep learning network framework and input the images in the training set into the ResNet network to generate feature maps; A rotation candidate region generation network is added after the ResNet network to predict and distinguish the foreground and background on the feature map. The input of the rotation candidate region generation network is the feature map, and the output is the target candidate region with rotation angle information predicted on the feature map, that is, the prediction box. The target candidate region includes five parameters (x, y, w, h, θ). The rotation candidate region generation network adds the θ angle parameter on the basis of the traditional RPN network, that is, when generating the prediction box parameters, the original (x, y, w, h) 4 parameters are increased to (x, y, w, h, θ) 5 parameters; The rotation candidate region generation network adopts a multi-task loss function, where the classification loss is: Lcls(p,u)=-log pu Among them, u is defined as the sample category label of the rotated bounding box. When u = 1, it means that the rotated bounding box contains the target, that is, the foreground; When u = 0, it means that the rotated bounding box does not contain the target, that is, the background, and does not participate in the calculation; p is the classification probability of the prediction box calculated by the Softmax function, that is, the probability that the prediction box generated by the rotated candidate region generation network is the foreground; The regression loss is: Where v = (v x , v y , v w , v h , v θ ) represents the five prediction parameters of the prediction box of the candidate region, v * =(v x * , v y * , v w * , v h * , v θ * ) represents 5 parameters of true value, among which, The function is defined as: In order to achieve scale invariance, it is also necessary to * Perform scale normalization, assuming that the current anchor box parameter is (x a ,y a , w a ,h a ,θ a ), then v and v* are scaled and normalized to obtain: Where x, x a , x * Represent the horizontal coordinates of the predicted bounding box, anchor box, and true bounding box, respectively. The same is true for y, w, h, and θ; where k∈Z such that Similarly; A rotation ROI pooling layer is added after the rotation candidate region generation network. The input of the rotation ROI pooling layer is the feature map and the target candidate region output by the rotation candidate region generation network. The output is a feature vector of a fixed size generated from the feature map according to the candidate region. Assuming that the hyper parameters of the rotation ROI pooling layer are Wr and Hr, that is, each rotation target candidate region of size w×h generated by the rotation candidate region generation network is divided into W r ×H r sub-regions, each of which has a size of w s ×h s , where w s =w / W r ,h s =h / H r Each segmented sub-region has the same rotation angle as the entire candidate region, and the sub-region is rotated at the same angle as the candidate region within the sub-region. s ]×[h s ], the feature values ​​of the sampling points are calculated by bilinear interpolation, and then the sampling points in each sub-region are max-pooled to obtain a feature map of fixed size; Finally, a fully connected layer is added after the rotation ROI pooling layer. According to the generated feature vector, the target position and category information corresponding to the input image are generated through the parallel regression layer and classification layer to obtain the final result. The entire network model adopts a multi-task loss function, and its formula is as follows: L(p,u,v*,v)=Lcls(p,u)+Lloc(v*,v) In the formula, λ represents the penalty coefficient of regression loss; For classification loss, the Softmax loss function is used; for regression loss, only the loss when u=1 is calculated, and the IoU penalty coefficient is added on the basis of the smooth L1 loss function. IoU represents the intersection-over-union ratio of two boxes. The regression loss function IoU-smooth L1 including the IoU penalty coefficient is defined as: Step 3: Use the training set images to train the network framework; Step 4: Use the trained model to test the test set to obtain the target detection results with rotation angle information.

2. According to claim 1, a remote sensing image target detection method based on a rotation region generation network, It is characterized in that The step 1 is to obtain a remote sensing image data set, select a training set and a test set according to a certain ratio, and use a five-tuple rotated bounding box to mark the training set. The method is as follows: obtain a remote sensing image data set, divide it into a training set and a test set according to a certain ratio, for the training set, use a five-tuple (x, y, w, h, θ) to encode each target to be detected on each image to determine its position information in the original image, that is, the real bounding box; wherein (x, y) represents the coordinates of the center point of the real bounding box of the target in the original image, (w, h) represents the width and height of the real bounding box of the target, θ represents the angle between the real bounding box of the target and the positive direction of the horizontal x-axis of the original image, and it is stipulated that θ belongs to the interval [0, π), each real bounding box containing five-tuple information of each picture in the training set and its corresponding target category information are recorded in a text file to generate a labeling file, and the labeling of the training set is completed; the images in the test set are processed according to the above method, but the test set images do not need to be labeled.

3. According to claim 2, a remote sensing image target detection method based on a rotation region generation network, It is characterized in that In step 3, the network framework is trained using the training set images. The method is as follows: the built model is trained using the labeled training set and the labeled file. When the total loss L(p, u, v * , v) is less than the preset value, the iterative training is stopped.

4. The method for remote sensing image target detection based on a rotation region generation network according to claim 3, It is characterized in that In step 4, the trained model is used to test the test set to obtain the target detection result with rotation angle information, and the method is as follows: the test set image is input into the network to obtain the target detection result with rotation angle information.

Citation Information

Patent Citations

  • Mask-based remote sensing image rotating target detection method

    CN110874593A

  • Local picture rapid detection method based on image content identification

    CN111061904A

  • Remote sensing image target detection method based on new frame regression loss function

    CN111091105A

  • Remote sensing ship identification method based on dense feature fusion and pixel-level attention

    CN111563473A

  • Vehicle-mounted video target detection method based on deep learning

    WO2020181685A1