A lightweight multi-scale remote sensing image rotating target detection method and system

Abstract

The application provides a lightweight multi-scale remote sensing image rotating target detection method and system, aiming at the problem that the target and background in the remote sensing image are more complex, based on a YOLOv4 model, shallow feature information is added in a feature fusion network; in order to alleviate the problem of feature loss caused by too many network layers, a weighted bidirectional feature pyramid is used as the feature fusion network; in order to improve the receptive field of the feature, a multi-scale hollow convolution module is added after the feature fusion, so as to improve the detection precision; the remote sensing image rotating target detection, the long side representation method represents the angle parameter, the annular smoothing label of the Gaussian function is used to convert the angle parameter from the regression problem to the classification problem, and the loss problem caused by the angle periodicity is reduced; a depth separable convolution and a global attention module are used to design a lightweight module, and the traditional convolution is replaced in the detection model, so that the model is lightweight.

Description

TECHNICAL FIELD

[0001] The present application relates to a lightweight multi-scale remote sensing image rotating target detection method and system, belonging to the field of remote sensing image application, and suitable for remote sensing image target extraction. BACKGROUND

[0002] The rapid development and use of remote sensing satellites and unmanned aerial vehicles greatly improve the acquisition speed of remote sensing images, and a large amount of remote sensing images provide rich information for disaster control, traffic planning, ecological environment monitoring, and topographic survey. How to efficiently obtain important information from a large amount of images has become a hot research topic. In recent years, target detection based on deep learning has achieved great success in traditional image fields, but the target detection algorithm suitable for natural images does not necessarily perform well on remote sensing images. Therefore, the detection algorithm for natural images cannot be directly applied to remote sensing image target detection, and the reasons are as follows: (1) Compared with natural images, remote sensing images are more complex, with large image size and relatively small target size, i.e., small targets in large remote sensing images, and dense arrangement; (2) Remote sensing images are taken from a bird's eye view, and targets are distributed in any direction; (3) Most target detection models have a large number of parameters and are complex, requiring high hardware requirements in practical applications. SUMMARY

[0003] The present application solves the technical problem of overcoming the deficiencies of the prior art in terms of large remote sensing image size, large target size difference, and most small targets, and provides a lightweight multi-scale remote sensing image rotating target detection method, which can significantly reduce the parameter quantity and computational quantity of the model while realizing more expressive diagonal box detection of target positions in remote sensing images.

[0004] In a first aspect, a lightweight multi-scale remote sensing image rotating target detection method is provided, characterized by the following steps:

[0005] Step 1: Fusing a depth separable convolution and an attention mechanism into a lightweight network module;

[0006] Step 2: Improving the YOLOv4 network, i.e., using the lightweight network module of step (1) to obtain a lightweight remote sensing image target detection YOLO-SFF model based on shallow feature fusion, training according to the model to obtain a remote sensing image target detection result based on a horizontal box;

[0007] Step 3: Based on the remote sensing image target detection result based on a horizontal box obtained in step (2), designing a remote sensing image rotating target detection method to obtain a remote sensing image target detection result based on a diagonal box through the method.

[0008] Further, the lightweight network module in step 1 is implemented as follows:

[0009] Step (11): Deep separable convolution is to decompose the standard convolution into depth convolution and pointwise convolution, which significantly reduces the amount of calculation, but reduces the feature extraction capability. The channel attention mechanism module can learn the importance of different channel features. The attention mechanism module is added after the depth convolution and before the pointwise convolution, obtaining a lightweight network module, which reduces the parameter amount and calculation amount while improving the feature extraction capability.

[0010] Step (12): In the residual module of the backbone network of the detection model, the 3x3 standard convolution uses a lightweight module. In the feature fusion module, the 3x3 convolution uses a lightweight module.

[0011] Further, in step 2, the implementation steps of the lightweight remote sensing image target detection YOLO-SFF model based on shallow feature fusion are as follows:

[0012] Step (21): On the basis of the original output of the backbone network of the YOLOv4 model, a shallow feature is added to the three kinds of down-sampling features, i.e. on the basis of the 8 times down-sampling feature, the 16 times down-sampling feature, and the 32 times down-sampling feature, a 4 times down-sampling feature is designed to output, and a total of four scale features are obtained;

[0013] Step (22): According to the four scale features obtained in step (21), a bidirectional feature pyramid with learnable weights is used, wherein the bidirectional feature pyramid is increased with horizontal connection of the same scale, and the four scale features from bottom to top are fused with the four scale features from top to bottom. After fusion, the 8 times down-sampling feature, the 16 times down-sampling feature and the 32 times down-sampling feature are output, the shallow feature fusion is completed, and three scale fused features are obtained;

[0014] Step (23): The three scale fused features obtained in step (22) are used for multi-scale empty convolution, the target receptive field is expanded, and the input of the multi-scale empty convolution and the result of the multi-scale empty convolution are added by using self-learning weights. The features after addition are output to the detection head, and the remote sensing image target detection YOLO-SFF model based on shallow feature fusion is obtained;

[0015] Step (24): In the backbone network and feature fusion network of the YOLO-SFF model, the lightweight network module of the deep separable convolution and the global attention mechanism module fusion is processed twice by the lightweight network module of the backbone network and the feature fusion network, and the lightweight YOLO-SFF model for remote sensing image rotating target detection based on shallow feature fusion is obtained. Finally, the remote sensing image target detection result based on horizontal frame is obtained.

[0016] Further, the remote sensing image rotating target detection method in step 3 includes:

[0017] Step (31): Convert the eight-parameter (x1, y1), (x2, y2), (x3, y3), (x4, y4) representation of the target box in the remote sensing image into the five-parameter (x min ,y min ), (x max ,y max ), θ representation, wherein the target oblique box is rotated to a horizontal box using the long side representation, the rotation angle θ of the oblique box and the position coordinates (x min ,y min ), (x max ,y max ) of the upper left point and the lower right point of the horizontal box are obtained;

[0018] Step (32): Convert the angle true value θ into a one-dimensional array T of length 180 using the circular smoothing label of the Gaussian function; wherein the value of the array T at the angle θ is 1, and as the angle changes from θ to the left and to the right, the value of the array T becomes smaller and smaller until it becomes 0;

[0019] Step (33): Increase the output of the angle information in the detection head of the lightweight remote sensing image target detection YOLO-SFF model based on shallow feature fusion, increase the 180-dimensional output, correspond to the one-dimensional array of the true value angle length 180, represent the 0-180° output of the angle;

[0020] Step (34): Add the cross-entropy loss function of the angle information in the loss function, so that the angle information converges in the training, and obtain the loss function of the angle information;

[0021] Step (35): Use the cross ratio of the oblique box to remove the overlapping detection boxes, and obtain the remote sensing image target detection result based on the oblique box.

[0022] In a second aspect, the present application provides a lightweight multi-scale remote sensing image rotated target detection system, comprising: a lightweight module, a horizontal box remote sensing target detection module, and an oblique box remote sensing target detection module:

[0023] Lightweight module: fuse global attention mechanism after depth convolution of depth separable convolution and before point-by-point convolution to obtain a lightweight network module;

[0024] Horizontal box target detection module: extract features from a remote sensing image through multiple lightweight modules, output four different scales of features: 4 times down-sampling features, 8 times down-sampling features, 16 times down-sampling features, and 32 times down-sampling features; fuse the four scales of features output from the backbone network using a lightweight feature fusion module to fuse the shallow features and the deep features with each other, output three scales of features: 8 times down-sampling features, 16 times down-sampling features, and 32 times down-sampling features, and finally obtain a remote sensing image target detection result based on the horizontal box based on the three scales of features;

[0025] Rotating target detection module: a slanted frame detection module is added behind the feature fusion module, a circular smooth label of a Gaussian function is used to convert the angle information into a one-dimensional array with a length of 180, the model can output the angle classification information of the target, and finally the remote sensing image target detection result based on the slanted frame is obtained.

[0026] Compared with the prior art, the advantages of the present application are:

[0027] (1) In terms of remote sensing image detection accuracy, the traditional detection method does not aim at the characteristics of small remote sensing image targets and large image target scale difference, and the present application proposes an improved target detection method based on YOLOv4. In order to enhance the small target feature information, shallow feature information is added to the feature fusion network. In order to alleviate the problem of feature information loss caused by too many network layers, weighted bidirectional features are used as the feature fusion network, and in order to improve the feature receptive field, a multi-scale hollow convolution module is added after the feature fusion. This method effectively solves the characteristics of large image size and small target in remote sensing image detection, and the detection accuracy reaches 56.21%, which is 11.83% higher than the original YOLOv4 network.

[0028] (2) In terms of rotating target detection, the traditional method calculates the angle of the target by regression, but the angle information is a periodic information, and the traditional regression method will encounter boundary discontinuity in the detection of targets in any direction, the present application successfully converts the angle information into a classification problem and applies it to a single-stage remote sensing image detection model; realize the slanted frame detection of remote sensing image target, the effect is better in the detection of densely arranged and inclined objects, and the slanted frame can better surround the target area and better identify the position of the target.

[0029] (3) In terms of model lightweight, the traditional method mainly designs the lightweight of the backbone network of the detection model, and the present application uses depth separable convolution and global attention module for lightweight design in the backbone network and feature fusion network, which can loss a small amount of precision, and the detection network reduces the calculation amount by 55% and the parameter amount by 57% under the condition of losing 6% precision, greatly reducing the parameter amount and calculation amount of the model. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The flowchart of the lightweight multi-scale remote sensing image rotating target detection method and system of the present application;

[0031] Figure 2 The depth separable convolution diagram in the present application;

[0032] Figure 3 The lightweight module of the depth separable convolution and attention mechanism fusion in the present application;

[0033] Figure 4 For the multi-scale receptive field network diagram in step 2 in the application;

[0034] Figure 5 For the structure diagram of the improved model algorithm in step 2 in the application;

[0035] Figure 6 The conversion diagram of the eight-parameter method and the five-parameter method of the skew frame in the application;

[0036] Figure 7 For the comparison diagram of the skew frame and the horizontal frame detection results. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the protection scope of the application.

[0038] As shown in the figure, Figure 1 The method for light-weight multi-scale remote sensing image rotating target detection of the application includes the following main steps:

[0039] Step one: light-weight module

[0040] The standard convolution is replaced with a depth separable convolution, the spatial convolution and the channel convolution are separated, the SE attention mechanism is added after the depth convolution and before the point convolution, the global attention is added to different channels, the feature extraction capability of the network is improved, finally the depth separable convolution and the SE attention mechanism are fused to form a light-weight network module.

[0041] Figure 2 And Figure 3 The light-weight module structure is shown.

[0042] As shown in the figure, Figure 2 The depth separable convolution is divided into a depth convolution and a point-by-point convolution, the depth convolution is in spatial feature fusion, the point-by-point convolution is in channel feature fusion, the 3-channel image is subjected to a depth convolution with a depth of 3 to obtain a 3-channel result, and then subjected to 4 point-by-point convolutions to obtain a 4-channel convolution result.

[0043] As shown in the figure, Figure 3 It is a residual network that fuses the depth separable convolution and the attention mechanism module, first subjected to a 1x1 convolution, then subjected to the depth separable convolution and the attention mechanism module, and finally subjected to a 1x1 point-by-point convolution, which can effectively reduce the parameter amount;

[0044] Step two: horizontal frame remote sensing target detection

[0045] First, the lightweight model is used in the residual module of the backbone network CSPDarkNet53 of the YOLOv4 model, and four scale feature maps are output, i.e. 4 times down-sampling features, 8 times down-sampling features, 16 times down-sampling features and 32 times down-sampling features, and the sizes are 152x152, 76x76, 38x38 and 19x19 respectively.

[0046] Then, the BiFPN feature fusion network of the lightweight model is used, in which the lateral connection of the same scale is added in the bidirectional feature pyramid, and the four scale features are fused from bottom to top and from top to bottom, and three scale features are output, i.e. 8 times down-sampling features, 16 times down-sampling features and 32 times down-sampling features.

[0047] Then, before the three scale features after fusion are input into the detection head, the RFB multi-scale receptive field network is added, the target receptive field is expanded, and the input and the result of the multi-scale dilated convolution are added by using the self-learning weight. The features are output to the detection head, the detection head uses the detection head of YOLOv4 to predict the bounding box, first calculates the coordinates, width and height in the prediction box, secondly filters the prediction boxes with confidence less than 0.5 according to the confidence threshold, and finally uses non-maximum suppression to determine the final prediction box (the center position, length and width of the target and the confidence), to obtain the remote sensing image target detection result based on the horizontal box. The calculation formula of the bounding box is:

[0048] b x =σ(t x )+c x

[0049] b y =σ(t y )+c y (1)

[0050]

[0051]

[0052] In the formula, p w , p h are the width and height of the set prior box, c x , c y are the coordinates of the upper left position of the target in the grid relative to the upper left corner of the image, σ(t x ), σ(t y ) are the distance between the upper left corner of the grid where the target center point is located and the center point of the prediction box, b x , b y are the horizontal and vertical coordinates of the prediction box center point, t w , th These represent the offsets of the predicted bounding box relative to the width and height of the prior bounding box, respectively. w b h σ and σ' represent the width and height of the prediction box, respectively; σ() is the Sigmoid function, which limits the left offset to within (0,1), ensuring that the grid only predicts objects whose center falls within this region.

[0053] Figure 4 and Figure 5 The network implementation process of the YOLO-SFF remote sensing image target detection model of the present invention is demonstrated.

[0054] like Figure 4 As shown, the input layer is subjected to dilated convolutions with kernel sizes of 1×1 and 3×3 and dilation scales of 1, 3, and 5, and then concatenated. After being directly connected to the input layer and then residually connected, it passes through the activation layer. α and β are the weights learned by the network during residual connection training, where rate is the size of the dilated convolution kernel.

[0055] like Figure 5 As shown, the YOLO-SFF network is the remote sensing image target detection model of this invention. First, a lightweight CSPDarknet53 is used as the backbone network to extract features at four scales. Then, a bidirectional feature pyramid network is used to fuse the features at the four scales from bottom to top and from top to bottom. Finally, the detection results are output after multi-scale dilated convolution and detection head. Resblock_Body is the basic module that makes up CSPDarknet53; DownSampling is downsampling, UpSampling is upsampling, and Add is addition.

[0056] Step 2: Rotating Target Detection

[0057] First, the eight-parameter (x1,y1), (x2,y2), (x3,y3), (x4,y4) representation of the target bounding box in the remote sensing image is transformed into a five-parameter (x1,y1), (x2,y2), (x3,y3), (x4,y4) representation. min ,y min ), (x max ,y max The θ notation, where the long side notation is used to rotate the target diagonal frame to a horizontal frame, yields the rotation angle θ of the diagonal frame and the position coordinates (x, y) of the top-left and bottom-right points of the horizontal frame. min ,y min ), (x max ,y max The formula for calculating the rotation of a rectangle is:

[0058]

[0059] In the formula, x0, y0 is the rotated horizontal and vertical coordinates; x, y is the original horizontal and vertical coordinates; θ is the counterclockwise rotation angle; xc, yc is the center point coordinate of the rectangle.

[0060] Then, the conversion of the angle true value θ into a one-dimensional array T of length 180 is performed using the circular smoothing label of the Gaussian function; wherein the value of the array T at the angle θ is 1, and as the angle changes from θ to left and right, the value of the array T becomes smaller and smaller until it becomes 0. The conversion formula for converting the angle true value into a one-dimensional array of length 180 degrees using the circular smoothing label from numbers is:

[0061]

[0062] In the formula, g(x) is a window function, and the radius θ of the window function represents the angle of the current slanting frame, and here a Gaussian function with a standard deviation of 2 is used as the window function of the CSL; θ is the angle value, and r is the angle range.

[0063] Then, the angle loss function is added to the model loss function, so that the angle information is constantly convergent in the training, and the angle loss function is:

[0064]

[0065] In the formula, KxK is the network formed after the input image is divided by the model; i, j is the coordinate of the network; M is the number of predicted frames generated by each grid; is the grid containing the target; is the angle of the real frame; a i is the angle of the predicted frame.

[0066] Then, after the network outputs the detection result, the slanting frame and the cross ratio are combined to remove the overlapping frame using the non-maximum suppression algorithm, and the slanting frame detection result of the remote sensing image target is obtained.

[0067] Figure 6 and Figure 7 The five-parameter slanting frame representation method and the horizontal frame and slanting frame detection effect are shown.

[0068] As Figure 6 shown, the eight-parameter method needs four points (x1, y1), (x2, y2), (x3, y3), (x4, y4) to represent the position, and the black frame is a horizontal frame, which only needs two points (x min , y min ), (x max , y max ) to represent, and the horizontal frame is rotated by θ degrees around the center point to the position of the slanting frame, then (x min , y min ), (x max , y maxThe θ-5 parameter rule can represent a slanted frame.

[0069] like Figure 7 As shown, the left image shows ship detection in a remote sensing image with a slanted frame, and the right image shows ship detection in a remote sensing image with a horizontal frame. The slanted frame detection can effectively separate the target from the background and accurately represent the target's boundary, effectively reducing the area difference between the target's bounding box and the target's true boundary. In areas with dense targets, the horizontal frame indicates severe target overlap, and the slanted frame can effectively solve this problem.

[0070] The above description is merely an embodiment of a lightweight, multi-scale remote sensing image rotation target detection method according to the present invention. The present invention is not limited to the above embodiments. This specification is for illustrative purposes only and does not limit the scope of the claims. It will be apparent to those skilled in the art that many substitutions, improvements, and variations are possible. All technical solutions formed using equivalent substitutions or equivalent transformations fall within the protection scope claimed by the present invention.

Claims

1. A method for lightweight multi-scale remote sensing image rotation target detection, characterized in that, Comprising the following steps: Step 1: the depth separable convolution and attention mechanism are fused into a lightweight network module; Step 2: the YOLOv4 network is improved, that is, the lightweight network module of step (1) is used to obtain a lightweight remote sensing image target detection YOLO-SFF model based on shallow feature fusion, and the model is trained to obtain a remote sensing image target detection result based on a horizontal box; Step 3: based on the remote sensing image target detection result based on the horizontal box obtained in step (2), a remote sensing image rotating target detection method is designed, and the remote sensing image target detection result based on the inclined box is obtained through the method; The lightweight network module in step 1 is implemented as follows: Step (11): the depth separable convolution is to decompose the standard convolution into depth convolution and pointwise convolution, which significantly reduces the calculation amount, but reduces the feature extraction capability, and the channel attention mechanism module can learn the importance of different channel features, and the attention mechanism module is added after the depth convolution and before the pointwise convolution to obtain the lightweight network module, which reduces the parameter amount and calculation amount while improving the feature extraction capability; Step (12): in the residual module of the backbone network of the detection model, the 3*3 standard convolution is replaced with the lightweight module, and in the feature fusion module, the 3*3 convolution is replaced with the lightweight module; In step 2, the implementation steps of the lightweight remote sensing image target detection YOLO-SFF model based on shallow feature fusion are as follows: Step (21): based on the three kinds of down-sampling features originally output by the backbone network of the YOLOv4 model, a shallow feature is added, that is, based on the 8 times down-sampling feature, the 16 times down-sampling feature and the 32 times down-sampling feature, a 4 times down-sampling feature is designed to output, and a total of four scale features are obtained; Step (22): according to the four scale features obtained in step (21), a bidirectional feature pyramid with learnable weights is used, wherein a horizontal connection of the same scale is added in the bidirectional feature pyramid, and the four scale features from bottom to top and the four scale features from top to bottom are fused, and after fusion, the 8 times down-sampling feature, the 16 times down-sampling feature and the 32 times down-sampling feature are output, the shallow feature fusion is completed, and three scale fused features are obtained; Step (23): the three scale fused features obtained in step (22) are used for multi-scale empty convolution, the target receptive field is expanded, and the input of the multi-scale empty convolution and the result of the multi-scale empty convolution are added by using self-learning weights, and the added features are output to the detection head to obtain the remote sensing image target detection YOLO-SFF model based on shallow feature fusion; Step (24): the lightweight network module of the depth separable convolution and the global attention mechanism module is used in the backbone network and the feature fusion network of the YOLO-SFF model, and the lightweight YOLO-SFF model based on shallow feature fusion remote sensing image rotating target detection is obtained through twice lightweight processing of the backbone network and the feature fusion network, and finally the remote sensing image target detection result based on the horizontal box is obtained; The remote sensing image rotating target detection method in step 3 comprises: Step (31): convert the eight parameters of the target box in the remote sensing image into five parameters of the target box in the image representation into five parameters representation, where the target oblique box is rotated to a horizontal box using the long side representation, to obtain the rotation angle of the oblique box and the position coordinates of the upper left point and the lower right point of the horizontal box ; Step (32): Use the circular smoothing label of the Gaussian function to obtain the true angle value is transformed into a one-dimensional array T of length 180; where at angle ... The value of array T is 1, as the angle changes from... As the array T is moved to the left or right, its value decreases until it becomes 0. Step (33): Add the output of angle information in the detection head of the lightweight remote sensing image target detection YOLO-SFF model based on shallow feature fusion, increase 180-dimensional output, corresponding to the one-dimensional array of true value angle length 180, representing the output of 0-180° of angle; Step (34): Add the cross-entropy loss function of angle information in the loss function, so that the angle information is constantly convergent in training, and obtain the loss function of angle information; Step (35): Use the cross ratio of the inclined frame to eliminate the overlapping detection frame, and obtain the remote sensing image target detection result based on the inclined frame.

2. A target detection system for implementing the lightweight multi-scale remote sensing image rotation target detection method of claim 1, characterized in that Including: Lightweight module, horizontal frame remote sensing target detection module, rotating remote sensing target detection module: Lightweight module: fuse global attention mechanism after depth convolution of depth separable convolution and before pointwise convolution to obtain lightweight network module; Horizontal frame target detection module: the remote sensing image is subjected to feature extraction through multiple lightweight modules, and four different scale features are output: 4 times down-sampling feature, 8 times down-sampling feature, 16 times down-sampling feature and 32 times down-sampling feature; the four scale features output from the backbone network are subjected to feature fusion using a lightweight feature fusion module, the shallow features and the deep features are fused with each other, and three scale features are output: 8 times down-sampling feature, 16 times down-sampling feature and 32 times down-sampling feature; finally, the remote sensing image target detection result based on the horizontal frame is obtained based on the three scale features; Rotating target detection module: add a rotating detection module after the feature fusion module, use a circular smooth label of a Gaussian function to convert the angle information into a one-dimensional array with a length of 180, the model can output the angle classification information of the target, and finally obtain the remote sensing image target detection result based on the inclined frame.

Images