A model and method for target detection in optical satellite remote sensing images based on multi-scale contextual features

Abstract

The present invention relates to a target detection method, and specifically to an optical satellite remote sensing image target detection model and method based on multi-scale context features. In order to solve the shortcomings of the prior art that small targets are difficult to detect due to low resolution and small pixel size, the optical satellite remote sensing image target detection model based on multi-scale context features includes a backbone network, a multi-scale context feature fusion unit and a detection head. The multi-scale context feature fusion unit includes a multi-scale feature processing module and a context feature fusion module. The context feature fusion module is used to extract local features and context features, splice the local features and context features, and output them after superimposing different weights. The optical satellite remote sensing image target detection method based on multi-scale context features establishes and trains the above target detection model, and then inputs the satellite image to be detected into the trained target detection model to obtain the detection result.

Description

Technical Field

[0001] The present invention relates to a target detection method, and in particular to an optical satellite remote sensing image target detection model and method based on multi-scale context features. Background Art

[0002] Object detection technology is mainly used to identify and locate objects in a given image. Identifying objects specifically refers to identifying the category of the object, while locating objects refers to giving a bounding box of the object. The bounding box contains the entire object and minimizes the background area as much as possible, and is usually a rectangle.

[0003] The object detection model based on convolutional neural network has the accuracy and generalization performance that surpasses traditional computer vision algorithms, and has become the mainstream algorithm in the field of object detection models. At present, the object detection model based on convolutional neural network is mainly divided into two algorithm frameworks: one is the single-stage object detection model represented by YOLO, and the other is the two-stage object detection model represented by Faster R-CNN. The difference between the single-stage object detection model and the two-stage object detection model is that the single-stage object detection model does not need to scan the candidate area where the object may exist first, but directly estimates the category and position of the object in the image. This algorithm has good timeliness and has important application value in the field of object detection.

[0004] The detection objects of the target detection model are usually optical satellite remote sensing images, which are images taken by optical remote sensing sensors carried on satellites. They have the characteristics of long shooting distance, wide image coverage, and low spatial resolution. The spatial resolutions of civil and commercial optical satellite remote sensing images (hereinafter referred to as "satellite images") mainly include 0.3 meters, 0.5 meters and 1 meter. In these satellite images, it is relatively easy to detect large targets such as dams, bridges, airports or ports, while it is relatively difficult to detect small targets such as vehicles, ships and aircraft. The difficulties are mainly reflected in the following two aspects: 1) The resolution of the target is low, the loss of appearance features is large, and the features contained in the appearance of the target are not significantly discriminative; 2) The pixel size of the target is relatively small, and the size of the bounding box is also small, which requires a high positioning accuracy of the target detection model. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art that small targets have low resolution and small pixel size, which makes target detection difficult, and to provide an optical satellite remote sensing image target detection model and method based on multi-scale contextual features.

[0006] Invention concept:

[0007] In satellite images, the size of targets varies, and they are usually regularly associated with contextual features due to the scene and location they are in. In the field of target detection, "contextual features" refer to the surrounding scene information of the target. For example, ships traveling in the ocean are associated with their wakes, vehicles on city streets are usually associated with the roads they are on, and airplanes at airports are usually associated with runways or apron scenes. Therefore, combining the surrounding scene information can better identify small targets. In summary, using contextual features can help detect small targets in satellite images that lack prominent features in appearance and improve the accuracy of target detection models. Based on the above ideas, the present invention proposes a method for detecting targets in optical satellite remote sensing images based on multi-scale context features, obtains features of different scales, uses the correlation between context features and targets to assist the target detection model in identifying small targets in satellite images, and improves the YOLOv8 target detection model with relatively high accuracy and timeliness. In the multi-scale feature fusion stage, local features and context features are extracted respectively by ordinary convolution and dilated convolution, and corresponding post-processing is performed to adjust the weights of local features and context features, so as to effectively fuse context features in multi-scale feature maps, improve the accuracy of the target detection model, alleviate the impact of the problem that the appearance of small targets lacks prominent features on target detection, and improve the recognition and positioning accuracy of small targets in satellite images. The YOLOv8 target detection model is based on a convolutional neural network and requires a large amount of labeled data to train the neural network. Therefore, the present invention is suitable for application scenarios where labeled data can be obtained. The specific implementation steps of the present invention include: 1) obtaining a target data set with labeled files; 2) building a target detection model; 3) training the target detection model; 4) inputting a picture to be detected and obtaining a target detection result.

[0008] To achieve the above objectives, the technical solutions provided by the present invention are as follows:

[0009] An optical satellite remote sensing image target detection model based on multi-scale context features is special in that it includes a backbone network, a multi-scale context feature fusion unit and a detection head; the backbone network includes a plurality of convolutional layers connected in series, the input end of the first convolutional layer is used to input a satellite image to be detected, the satellite image to be detected is processed by the previous convolutional layer and output to the next convolutional layer for processing, and passes through a plurality of convolutional layers in sequence, each convolutional layer outputs feature maps of different levels respectively; the multi-scale context feature fusion unit includes a multi-scale feature processing module and a context feature fusion module; the multi-scale feature processing module is used to extract features from feature maps of different levels, and upsample the feature map output by the last convolutional layer, and output the corresponding multi-scale feature maps. The invention discloses a method for detecting the multi-scale features of a convolutional layer. The method comprises the following steps: extracting local features and context features, upsampling and fusion of features of feature maps output by other required convolutional layers, and outputting corresponding multi-scale features respectively. The multi-scale features are then processed by a context feature fusion module and feature fusion to obtain fusion features. The context feature fusion module comprises a context feature extraction module, a feature splicing module and a weighted processing module. The context feature extraction module is used to extract local features and context features. The feature splicing module is used to splice local features with context features. The weighted processing module is used to superimpose different weights on local features and context features to obtain weighted features and output them. The detection head recognizes multi-scale features and / or fusion features respectively, and outputs the prediction vector of the marked image and the target category and the prediction value of the bounding box.

[0010] Furthermore, the context feature extraction module includes a common convolution kernel and a dilated convolution kernel, the common convolution kernel is used to extract local features, and the dilated convolution kernel is used to extract context features.

[0011] Furthermore, a normalization and average pooling module is included between the feature splicing module and the weighted processing module, which is used to normalize and average pool the spliced ​​local features and context features to achieve dimensionality reduction processing.

[0012] Furthermore, the weighted processing module is set as a fully connected layer, and the corresponding weights of local features of various targets at different scales and their context features are obtained through learning in the fully connected layer, and different weights are superimposed on the local features and their context features respectively.

[0013] Furthermore, the common convolution kernel and the dilated convolution kernel are both 3×3 convolution kernels, and the dilation rate of the dilated convolution kernel is 2.

[0014] Furthermore, the multiple convolutional layers connected in series have the same structure, denoted as Ci, where i∈{1,2,3,4,5};

[0015] The multi-scale feature processing module upsamples the features obtained by the convolution layer C5 as the multi-scale feature P5, upsamples the features obtained by the convolution layer C4 and fuses them with the multi-scale feature P5 to obtain the multi-scale feature P4, and upsamples the features obtained by the convolution layer C3 and fuses them with the multi-scale feature P4 to obtain the multi-scale feature P3; the multi-scale feature P3 is processed by the context feature fusion module to obtain a weighted feature, which is fused with the multi-scale feature P4 to obtain a fused feature N4, and the fused feature N4 is fused with the multi-scale feature P5 after being processed by the context feature fusion module to obtain a fused feature N5;

[0016] The detection head identifies the multi-scale feature P3, the fusion feature N4 and the fusion feature N5, and outputs a marked image and a prediction vector of the target category and a prediction value of the bounding box.

[0017] At the same time, the present invention also provides an optical satellite remote sensing image target detection method based on multi-scale context features, which is special in that it includes the following steps:

[0018] Step 1: construct the above-mentioned optical satellite remote sensing image target detection model based on multi-scale context features;

[0019] Step 2: Use the target data set containing the target category to be detected to train the target detection model established in step 1 to obtain corresponding parameters, and input them into the target detection model to obtain a trained target detection model;

[0020] The target data set includes multiple satellite images and corresponding annotation files, and the satellite images contain target categories to be detected;

[0021] Step 3: Input the satellite image to be detected into the trained target detection model, and output the labeled image as well as the prediction vector of the target category and the prediction value of the bounding box.

[0022] Furthermore, step 2 is specifically as follows:

[0023] The target detection model established in step 1 is trained using the target data set for 200 to 500 rounds. During the training process, gradient back propagation is used to optimize the parameters of the target detection model. The parameters corresponding to the minimum value of the loss function during the training process are saved as training results and input into the target detection model.

[0024] Furthermore, in step 2, the target detection model established in step 1 is trained for 300 rounds using the target data set of the target category to be detected.

[0025] Beneficial effects of the present invention:

[0026] 1. The optical satellite remote sensing image target detection model based on multi-scale contextual features of the present invention fully considers the correlation between the target and its contextual features in the satellite image, improves the YOLOv8 target detection model, adds a contextual feature fusion module (CFB, Context Fusion Block) to the original feature fusion module, uses contextual features to assist target detection, and superimposes different weights on local features and contextual features respectively, so as to improve the recognition ability of small targets with unclear appearance features.

[0027] 2. In the optical satellite remote sensing image target detection model based on multi-scale contextual features of the present invention, ordinary convolution kernels and dilated convolution kernels are set in the context feature extraction module. Larger area information is obtained by dilated convolution kernels, and context features of the target surrounding area are extracted. The extraction method is simple and efficient.

[0028] 3. In the optical satellite remote sensing image target detection method based on multi-scale contextual features of the present invention, an optical satellite remote sensing image target detection model based on multi-scale contextual features is first established, and corresponding parameters are obtained through training. Then, the satellite image to be detected is input for recognition, which can accurately identify targets of various scales and improve the target recognition ability and positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the structure of the target detection model according to an embodiment of the present invention;

[0030] Figure 2 is a workflow diagram of a context feature fusion module according to an embodiment of the present invention;

[0031] Figure 3 is a flow chart of an embodiment of the present invention;

[0032] Figure 4 is a satellite image output by target detection in an embodiment of the present invention, wherein a, b, c, and d are labeled images of different targets respectively. DETAILED DESCRIPTION

[0033] The technical principle of the optical satellite remote sensing image target detection method based on multi-scale context features of the present invention is as follows: firstly, satellite images with labeled files are obtained as original data sets for target detection, satellite images with labeled files are screened according to target categories of interest, and then the labeled file formats of the screened satellite images with labeled files are converted into YOLO format to obtain target data sets, and at the same time, a target detection model is established, and relevant program files are configured, and the screened target data sets are input into the established target detection model for training and optimization, and then the satellite images to be detected are input into the trained target detection model to obtain corresponding target detection results, and the target detection results include a labeled image, a prediction vector of the target category and a prediction value of a bounding box.

[0034] like Figure 3 As shown, the present invention specifically comprises the following steps:

[0035] 1) Obtain the original data set, filter the target categories required in the original data set as needed, obtain the target data set, and adjust the annotation file format of the data set;

[0036] The original dataset consists of multiple satellite images with annotation files, which include multiple target categories, such as larger dams, bridges, airports and ports, and smaller vehicles, ships and airplanes. According to the focus of the actual specific task, the target category to be detected is selected, and other unnecessary target categories are removed to obtain the target dataset.

[0037] In this embodiment, the original data set is the large-scale satellite image target detection data publicly released by the organizer of the xView Challenge. This embodiment selects vehicles, ships and aircraft with high analysis value as the target categories to be detected, and selects pictures containing these target categories to be detected to construct the target data set. After processing, a total of 9296 800×800 pixel satellite images are obtained, including 445,473 ground targets. Target data sets generally use bounding boxes to represent the position and size of targets. Specifically, a bounding box with four sides tangent to the target contour is used to represent the position and size of the target, and the four sides of the bounding box are parallel to the four sides of the image. The bounding box can be represented by its center point coordinates ( x , y ) and its width w and high h It is represented as a quaternion array ( x , y , w , h). In addition, the target categories in the target data set can be written in the form of a list: ['airplane', 'car', 'ship'], and the target categories are indexed and numbered, and the target categories are represented in the form of a list as [0,1,2]. Based on the above, the annotation file of the target data set is constructed. The specific method is to create a one-to-one corresponding txt file with the same name as each image in the target data set as the annotation file. Each txt file records the categories and bounding boxes of all targets that appear in the corresponding image. The annotation information of each target occupies one line in the txt file, which includes the index number of the target category and the quaternion array of the bounding box ( x , y , w , h ), a total of 5 elements, separated by spaces.

[0038] 2) Build a target detection model;

[0039] The target detection model of the present invention is a neural network model improved based on the YOLOv8 model. Figure 1 In order to effectively fuse multi-scale local features and context features, the feature pyramid structure for implementing multi-scale feature fusion in the YOLOv8 model is improved. The target detection model of the present invention includes a backbone network, a multi-scale context feature fusion unit, and a detection head for outputting target detection results. The target detection results include a labeled image, a prediction vector of the target category, and a prediction value of the bounding box. The backbone network is the same as the convolutional neural network used by the YOLOv8 model, and is composed of 5 convolutional layers Ci with the same structure connected in series, where i is the number of convolutional layers, i∈{1,2,3,4,5}, and each convolutional layer includes a set of convolution kernels, such as Figure 1 As shown in the figure, the input image is sequentially convolved through the series of convolutional layers to obtain a feature map, whose size is reduced while the number of channels is enlarged. Among them, the size of the feature map processed by the convolutional layers C1-C3 is relatively large, and many detailed features of the input image are retained, that is, shallow features are retained. Each pixel of the feature map is used as a feature point, and the pixel scale of the original image corresponding to each feature point is small. Therefore, the feature map processed by the convolutional layers C1-C3 is called a shallow feature map. As the network deepens, the size of the feature map processed by the convolutional layers C4 and C5 is relatively small, and is called a deep feature map. Each feature point corresponds to a larger range of input images, and only deep features are retained. Therefore, the deep feature map is considered to be an expression of the high-level semantic features of the image.

[0040] For target detection tasks, the key features required by different target categories may be deep features or shallow features. Generally speaking, smaller targets rely on shallow image features, while large targets rely on deep image features. Therefore, relevant scholars have proposed a feature pyramid structure for multi-scale feature fusion, so that the detection head can obtain feature map information at multiple levels, thereby realizing the detection of targets of various sizes. The feature pyramid structure of the YOLOv8 model uses a general convolutional layer to reduce the size of the shallow feature map. The feature map of the i-th layer f i The size is reduced through a common convolutional layer and then combined with the i+1 The layer feature maps are fused to obtain the fusion feature ( f i | f i+1 ), where the general convolution layer uses a 3×3 convolution kernel, and the corresponding receptive field is very small, which can only extract local features, but cannot effectively extract the context features of the target. However, the spatial resolution of satellite remote sensing images is low, and the commonly used spatial resolution is 0.3m~1m. Small ships and vehicle targets appear as small mosaics lacking appearance features in satellite remote sensing images, which brings great difficulties to high-precision target detection. Context features refer to the surrounding scene information of the target. For example, the surrounding scene of a small ship target is usually a water scene, and the surrounding scene of a vehicle target is usually a road scene. They are highly correlated, so it is feasible to use context features to identify these targets.

[0041] Therefore, the multi-scale context feature fusion unit of the present invention adds a context feature fusion module (CFB, Context Fusion Block) on the basis of the feature pyramid structure of the YOLOv8 model. The specific improvement details are as follows:

[0042] The multi-scale context feature fusion unit includes a multi-scale feature processing module and CFB, such as Figure 1As shown. The input image is processed by convolution layers C1-C5 to obtain features at different levels. In this embodiment, since the information obtained by C1 and C2 is limited, no processing is performed. The features obtained by convolution layers C3-C5 are input into the multi-scale feature processing module. The feature scale is expanded through upsampling operation, and then the features at different levels are fused to obtain multi-scale features P3-P5. Specifically, the features obtained by convolution layer C5 are directly used as multi-scale features P5 after upsampling, the features obtained by convolution layer C4 are fused with multi-scale features P5 after upsampling to obtain multi-scale features P4, and the features obtained by convolution layer C3 are fused with multi-scale features P4 after upsampling to obtain multi-scale features P3. In order to further extract features, the multi-scale feature P3 is processed by CFB and then fused with the multi-scale feature P4 to obtain a fused feature N4 with richer information. The fused feature N4 is processed by CFB and then fused with the multi-scale feature P5 to obtain a fused feature N5 with richer information.

[0043] The structure of CFB is as follows Figure 2 As shown in Figure 1, CFB includes a 3×3 ordinary convolution kernel and a 3×3 dilated convolution kernel with a dilation rate of 2. The ordinary convolution kernel is used to extract local features in multi-scale features or fused features. f local , by expanding the convolution kernel to obtain larger area information, extracting context features of the area around the target f context , and then the local features and context features are spliced ​​together through feature splicing, recorded as ( f local | f context ), the object detection model has different reliance on context features at different receptive field scales, so after dimensionality reduction processing through normalization and average pooling modules, ( f local | f context ) is input to the fully connected layer, and the fully connected layer obtains the weights of local features and context features through training and learning, and superimposes the weights on them respectively to obtain the weighted features ((1+ w local ) f local , (1+ w context ) f context ),in, w local and w contextare the weights of local features and contextual features, respectively. The 3×3 expanded convolution kernel set in CFB has a larger receptive field than the 3×3 ordinary convolution kernel, and can be used to extract contextual features. In the present invention, CFB is used to replace all convolution layers in the original feature pyramid structure that fuse shallow feature maps to deep feature maps. Therefore, CFB is added to the feature map fusion at different levels of the feature pyramid structure of the present invention to improve the feature extraction and representation of small targets that rely on contextual features, and to use contextual features to improve the recognition ability of target detection models for small targets that are difficult to identify, thereby improving the accuracy of target detection. Taking into account that different categories of targets in satellite images have different degrees of dependence on contextual features, categories with clear and easily identifiable appearance features are more dependent on local features, such as large passenger aircraft, which have better recognition effects only through local features, while targets with relatively scarce appearance features need to rely on contextual features.

[0044] The detection head receives multi-scale features P3, fusion features N4 and fusion features N5, identifies features of different scales in the input image, outputs a labeled image, and transforms the feature map into a 1D feature vector output through a multi-layer fully connected layer structure. The 1D feature vector contains the prediction vector of the target category and the prediction value of the bounding box ( x , y , w , h ).

[0045] 3) Use the target dataset to train the target detection model;

[0046] The target data set processed in step 1) is input into the target detection model constructed in step 2) for training for 200 to 500 rounds. During the training process, the target detection model automatically optimizes the neural network parameters of the target detection model using gradient back propagation, so that the neural network gradually adapts to the data distribution of satellite images, and saves the parameters of the neural network corresponding to the minimum value of the loss function during the training process as the training result, which is input into the target detection model to obtain the trained target detection model. Through the feedback of the loss function of network training, the fully connected layer in CFB can obtain different weights for local features and context features to improve the adaptability and versatility of the target detection model to different categories of targets.

[0047] Among them, the loss function L detect is the general loss for target detection and is obtained by:

[0048] L detect = L cls + L reg + L conf

[0049] Among them, L clsRefers to the loss function of the target detection result for the prediction of the target category, L reg Refers to the loss function of the target detection result for the positioning of the bounding box, that is, the intersection over union (IoU) of the predicted bounding box and the true bounding box, L conf Refers to the loss function of the confidence of target detection, L reg and L conf It is actually the cross entropy loss between the predicted result and the true value.

[0050] In this embodiment, 300 rounds of training are performed.

[0051] 4) Input the image to be tested into the trained target detection model to obtain the target detection result;

[0052] Through the above steps 1) to 3), a target detection model with high accuracy for small target detection in satellite images is obtained, the satellite image to be tested is input into the target detection model, a labeled image is output, and prediction vectors and prediction values ​​of bounding boxes of target categories of various scales in the image are output.

[0053] This embodiment conducts simulation experiments based on the hardware platform of Nvidia RTX3090 GPU and Intel Xeon Platinum 8260 CPU, the operating system is Linux Ubuntu 18.04, and PyCharm software and PyTorch 2.2.1 deep learning framework are used. According to the above steps, the target detection model for satellite image target detection task is obtained, and the satellite image to be tested is input into the target detection model to obtain the corresponding target detection results, such as Figure 4 The figure shows the visualization effect of some detection results of this embodiment, and it can be seen that the target detection model of the present invention has a high detection accuracy for small targets on the ground in satellite images.

Claims

1. An optical satellite remote sensing image target detection model based on multi-scale context features, characterized by: It includes a backbone network, a multi-scale context feature fusion unit and a detection head; The backbone network includes a plurality of convolutional layers connected in series, wherein the input end of the first convolutional layer is used to input the satellite image to be detected, and the satellite image to be detected is processed by the previous convolutional layer and then output to the next convolutional layer for processing, and passes through a plurality of convolutional layers in sequence, and each convolutional layer outputs feature maps of different levels respectively; the structures of the plurality of convolutional layers connected in series are the same, which are denoted as Ci, wherein i∈{1,2,3,4,5}; The multi-scale context feature fusion unit includes a multi-scale feature processing module and a context feature fusion module; the multi-scale feature processing module upsamples the feature obtained by the convolution layer C5 as the multi-scale feature P5, upsamples the feature obtained by the convolution layer C4 and fuses it with the multi-scale feature P5 to obtain the multi-scale feature P4, and upsamples the feature obtained by the convolution layer C3 and fuses it with the multi-scale feature P4 to obtain the multi-scale feature P3; the multi-scale feature P3 is processed by the context feature fusion module to obtain a weighted feature, which is fused with the multi-scale feature P4 to obtain a fused feature N4, and the fused feature N4 is processed by the context feature fusion module and fused with the multi-scale feature P5 to obtain a fused feature N5; The context feature fusion module includes a context feature extraction module, a feature splicing module and a weighted processing module; the context feature extraction module includes a common convolution kernel and a dilated convolution kernel, the common convolution kernel is used to extract local features, and the dilated convolution kernel is used to extract context features; the feature splicing module is used to splice local features with context features; the weighted processing module is set as a fully connected layer, which is used to learn and obtain the corresponding weights of local features of various targets at different scales and their context features, and to superimpose different weights on the local features and context features to obtain weighted features and output them; a normalization and average pooling module is also included between the feature splicing module and the weighted processing module, which is used to normalize and average pool the spliced ​​local features and context features to achieve dimensionality reduction processing; The detection head recognizes the multi-scale feature P3, the fusion feature N4 and the fusion feature N5 respectively, and outputs the prediction vector of the marked image and the prediction value of the bounding box of the target category.

2. The optical satellite remote sensing image target detection model based on multi-scale context features according to claim 1, characterized in that: The common convolution kernel and the dilated convolution kernel are both 3×3 convolution kernels, and the dilation rate of the dilated convolution kernel is 2.

3. A method for target detection in optical satellite remote sensing images based on multi-scale context features, characterized in that: The following steps are involved: Step 1: construct the optical satellite remote sensing image target detection model based on multi-scale context features as described in claim 1 or 2; Step 2: Use the target data set containing the target category to be detected to train the target detection model established in step 1 to obtain corresponding parameters, and input them into the target detection model to obtain a trained target detection model; The target data set includes multiple satellite images and corresponding annotation files, and the satellite images contain target categories to be detected; Step 3: Input the satellite image to be detected into the trained target detection model, and output the labeled image as well as the prediction vector of the target category and the prediction value of the bounding box.

4. According to claim 3, a method for detecting targets in optical satellite remote sensing images based on multi-scale context features, characterized in that: Step 2 is as follows: The target detection model established in step 1 is trained using the target data set for 200 to 500 rounds. During the training process, gradient back propagation is used to optimize the parameters of the target detection model. The parameters corresponding to the minimum value of the loss function during the training process are saved as training results and input into the target detection model.

5. The method for detecting targets in optical satellite remote sensing images based on multi-scale context features according to claim 4, characterized in that: In step 2, the target detection model established in step 1 is trained for 300 rounds using the target data set of the target category to be detected.

Images