Earth background clutter identification method based on cascaded combination network

Abstract

The application discloses a kind of earth background small target identification method based on cascade combination network, mainly solve the problem that existing technology is not easy to realize and small target area rough segmentation and small target identification under the observation of dynamic and complex earth background.It is scheme: construct, enhance, mark, convert and divide earth background small target visible light picture dataset;Based on existing semantic segmentation model, by introducing attention mechanism, multi-scale small target feature fusion structure, residual network constructs earth background small target segmentation model, and its training is realized to the rough segmentation of background and small target area;Existing small target detection model is trained to obtain small target identification model under the earth background, and the model is used to complete the re-identification of the picture after rough segmentation and preprocessing.The application improves the rough segmentation and identification precision between complex, variable earth background and small target of air-to-ground observation, and can be used for remote sensing image, traffic monitoring, automatic driving multiple scenes.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, and specifically relates to a method for recognizing small targets against a background of land, which can be used in a variety of scenarios such as military aviation, remote sensing images, land and building surveying and planning, traffic monitoring, and autonomous driving. Background Technology

[0002] In recent years, with the technological development in the field of target detection, small target recognition has gradually become a hot topic and a challenge. Recognizing small targets against a terrain background has become one of the most challenging tasks in target detection. Semantic segmentation technology is a key technology for achieving coarse segmentation of the region between the terrain background and small targets, while target recognition models are an effective way to further identify targets based on the segmentation results, ultimately improving the recognition effect of small targets against a terrain background. This invention has significant research value and increasingly widespread application value in diverse future scenarios such as military aviation, remote sensing imagery, land and building surveying and planning, traffic monitoring, and autonomous driving.

[0003] The challenges of identifying small targets against complex and varied terrain backgrounds include low resolution, dimness, difficulty in localization and visibility, and indistinguishability from the background. Existing technologies primarily employ object detection models or semantic segmentation for target recognition. Object detection models use bounding boxes to label and predict small targets in images, but they often only indicate the approximate location of the target. Semantic segmentation classifies targets at the pixel level, specifying the label of a particular pixel in the image, such as identifying a target (e.g., the pixel of a car). However, for even smaller targets, segmentation coverage is low, and edge segmentation performance has room for improvement. Therefore, for identifying small targets of the same type but different shapes, or different buildings, using a single semantic segmentation or object recognition model cannot achieve accurate segmentation and recognition of small targets against a terrain background.

[0004] Patent application CN202210296396.3 discloses a method for extracting buildings from remote sensing images based on an improved DeepLabV3+. It proposes a dense ASPP module, where the ASPP structure in the segmentation model uses different dilated convolution rates of 6, 12, and 18, combined in a cascaded manner. The input of each dilated convolution layer is concatenated with the outputs of all preceding parallel dilated convolution layers as the input to the next dilated convolution layer. However, this dilation rate fails to capture more effective small target features at various scales, is not conducive to segmenting deep, low-resolution small target feature maps, and fails to efficiently fuse shallow and deep features.

[0005] Patent application CN202210030581.8 discloses a semantic segmentation method for remote sensing images based on channel-spatial attention and DeepLabV3plus. It uses deep features extracted from the Xception backbone network model as input to a channel attention convolutional structure, and the product of the output and the deep features is used as input to a spatial attention convolutional structure. The product of the output of the spatial attention convolutional module and the deep features is used as the output of the channel-spatial attention convolutional module. This method only processes some deep, small target features extracted by the backbone network, neglecting shallow features within the backbone network. High-resolution feature maps with large features have smaller receptive fields and contain rich location information, making them more suitable for detecting small objects.

[0006] Paper CN202210677113.X discloses an improved semantic segmentation method based on DeepLabv3+. In its spatial pyramid module, it uses a 1x1 convolution, three dilated convolutions with dilation rates of 6, 12, and 18, and an ImagePooling (global average pooling) module. The three dilated convolutions are used to capture receptive field information and feature information at different scales, while global average pooling and the 1x1 convolution are used for feature extraction. However, as the model depth increases, especially in scenarios where small objects occupy a small proportion of the overall image, the ability of just three dilated convolutions to extract features of smaller-resolution, deeper, and higher-level semantic small objects remains limited.

[0007] Patent application CN202210704308.9 discloses a small target recognition algorithm that divides an image into key and non-key areas, grids these areas into 3x3 units, and uses neurons to read the feature values ​​of each unit and compare them with a database. This allows the system to determine whether each target is sufficiently clear for capture based on its volume data. However, this single-image segmentation method and gridded image recognition algorithm may overlook some edge information of small targets in complex and varied backgrounds and scenes where the gray values ​​or colors of small targets are similar. Furthermore, it cannot effectively distinguish between overlapping features of the same and different targets, ultimately affecting the effectiveness of background and small target recognition. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of the prior art by proposing a method for identifying small targets in the ground background based on a cascaded combined network. This method solves the problem of coarse segmentation of the ground background and target area through fusion detection, and enables the identification of small targets in complex and variable ground backgrounds during air-to-ground observation.

[0009] The technical solutions for achieving the objectives of this invention include the following:

[0010] (1) Construct a dataset of small ground background images using a semantic segmentation model and a YOLOv5-s small object detection network model:

[0011] 1a) Preprocessing of a dataset of visible light images of small targets against a background of the earth, including image enhancement and software annotation;

[0012] 1b) Divide the preprocessed dataset into a training set and a validation set for the semantic segmentation model in an 8:2 ratio;

[0013] 1c) Divide the preprocessed dataset into training, validation and test sets for the YOLOv5-S small object detection network model in an 8:1:1 ratio;

[0014] (2) Improve the existing Deeplabv3plus semantic segmentation model and construct a small target segmentation model for the ground background:

[0015] 2a) In the backbone network of the semantic segmentation network model, one shallow high-resolution feature layer and two deep low-resolution feature layers are processed by an attention mechanism to obtain a deeper feature layer.

[0016] 2b) Based on the small target feature fusion ASPP feature fusion structure in the semantic segmentation model, a new small target feature fusion branch is added, and the dilation rate parameter of its internal dilated convolution is adjusted to 4, 8, 12, and 16 to form a multi-scale small target feature fusion ASPP-6 structure, which is used to strengthen the first deep low-resolution feature layer after fusion.

[0017] 2c) Use the backbone network in the semantic segmentation network model to extract shallow high-resolution feature layers suitable for small object segmentation, use 1x1 convolution to adjust the number of channels, and stack the shallow high-resolution feature layers with the enhanced fusion in 2b) using the concat function to form a residual network structure.

[0018] 2d) The backbone network processed by the attention mechanism in step 2a), the ASPP-6 structure formed by fusing multi-scale small target features in step 2b), and the residual network structure formed in step 2c) are cascaded in sequence to form a small target segmentation model for the earth background.

[0019] (3) Training the small target segmentation model for the background:

[0020] (3a) Using the training set of the existing Pascal VOC2007 public dataset, the ground background small target segmentation model is pre-trained by adopting the training method of forward propagation and then backward propagation, so as to obtain the pre-trained ground background small target segmentation model.

[0021] 3b) Using the training set of the semantic segmentation model, a two-stage training method of first freezing the backbone network and then unfreezing the backbone network is used to train the pre-trained earth background small target segmentation model again, so as to obtain the trained earth background small target semantic segmentation model and generate the model weight file trained by the training set of the semantic segmentation model.

[0022] (4) Based on the validation set of the semantic segmentation model, the trained semantic segmentation model for small targets in the background is validated using the prediction script file of the background segmentation model to achieve coarse segmentation of the background and small targets in the region.

[0023] (5) Based on the training set of the yolov5-s small target detection network model in 1c), the yolov5-s small target detection network model is trained using the mixed precision training method to obtain a trained small target recognition model in the background of the earth, and a model weight file trained by the training set of the small target detection network model is generated.

[0024] (6) The trained small target recognition model under the background is used to detect the image dataset after coarse segmentation by the semantic segmentation model under the background. The location and category of small targets under the background in the image dataset are detected. That is, based on the coarse segmentation of the background and small target regions, the recognition of small targets under the background is carried out to realize the cascaded combination detection of background segmentation and target recognition.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] First, by performing image enhancement and software annotation preprocessing on a dataset of visible light images of small targets against a background, this invention can improve the edge sharpness of small targets and the category proportion of small targets in the image dataset.

[0027] Secondly, this invention constructs a small target segmentation model for the earth background based on the existing Deeplabv3plus semantic segmentation model. Specifically, it processes the three feature reference channels and spatial attention mechanisms extracted by the backbone network of the segmentation model at different scales and depths, making the network more likely to notice the features of small targets and increasing the depth feature weights of small targets. At the same time, by adding a new small target feature fusion branch to the feature fusion structure and forming an ASPP-6 structure, more scale feature information can be retained to prevent the loss of boundary detail information. This is not only beneficial for segmenting large targets with convolutional kernels with large dilatations, but also more beneficial for segmenting small targets with convolutional kernels with small dilatations. In addition, the design of a residual network structure can effectively improve the model's generalization ability.

[0028] Third, this invention uses the large public dataset Pascal VOC2007 for pre-training and employs mixed-precision training to effectively train the cascaded combination of small target segmentation models in the ground background. This significantly reduces memory consumption and accelerates model training. By utilizing the trained small target segmentation and recognition models, the accuracy of segmentation and recognition between complex and variable ground backgrounds and small targets in air-to-ground observation is improved. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0030] Figure 2 This is a diagram of the small target segmentation model against the ground background in this invention:

[0031] Figure 3 This is the ASPP-6 structure diagram of multi-scale small target feature fusion in this invention;

[0032] Figure 4 This is a diagram of the attention mechanism and residual structure in this invention;

[0033] Figure 5 This is a graph showing the average intersection-union ratio (IUR) of the existing Deeplabv3plus model and the earth background semantic segmentation model of this invention.

[0034] Figure 6 The average classification accuracy index of the existing Deeplabv3plus model and the earth background semantic segmentation model of this invention for small targets is shown in the figure. Detailed Implementation

[0035] The embodiments and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] Reference Figure 1 The implementation steps for this example are as follows:

[0037] Step 1: Construct a dataset of small target images against a background of the earth.

[0038] The specific implementation of this step is as follows:

[0039] 1.1) Obtain a dataset of visible light images of small targets against a background of the earth:

[0040] Based on the scenario of small targets being blurry and difficult to identify during simulated airborne and missile-borne air-to-ground observations, and occupying a small proportion in the whole image, or even being difficult to distinguish in the dark and complex background of the earth, a total of 15,000 images of small targets in the earth background were acquired using industrial cameras.

[0041] 1.2) Preprocessing for image enhancement on the acquired small target visible light image dataset:

[0042] To address the issues of small targets in images being dark, blurry, and having a small or insignificant proportion in the overall image, the Mosaic algorithm is used. Four images are randomly selected from a constructed dataset of small targets against a background of the earth. These four images are then flipped, scaled, and have their color gamut changed. They are then arranged in four directions and randomly stitched together to form a single image with a resolution of 1024 pixels x 768 pixels. This enriches the features of the small targets. Furthermore, the model can compute data from four images simultaneously, not only enriching the information about the small targets, background, and features, but also improving the model's computational speed and generalization ability.

[0043] To address the issues of small targets being disturbed, having poor outline clarity, and being difficult to distinguish from the ground background, the median filtering algorithm from the OpenCV library is used to process the dataset to improve the edge clarity of four types of small targets: house, plane, tank, and car.

[0044] To address the imbalance between large and small object categories in images, a method of tiling small objects is used to increase the number of small objects in the image and increase the model's weight for small objects, so that the network can treat objects of different scales more equally.

[0045] Step 2: The acquired small target visible light image dataset is labeled, transformed, and divided according to different models.

[0046] This example uses the existing Deeplabv3plus semantic segmentation model and the yolov5-s small object detection network model.

[0047] The semantic segmentation model consists of two cascaded parts: encoding and decoding.

[0048] The encoding part consists of a segmentation backbone network, a feature fusion ASPP structure, and a 1x1 convolutional layer cascaded together. The segmentation backbone network, used to extract small target features, is mainly composed of cascaded convolutional layers, batch normalization, activation functions, and pooling modules. The feature fusion ASPP structure, composed of a 1x1 convolutional layer, three 3x3 dilated convolutions with different dilation rates, and a pooling module cascaded together, is used to fuse high-level and low-level feature layers, enhancing the feature extraction capability.

[0049] The decoding part consists of several cascaded 1x1 convolutions, concat functions, 3x3 convolutions, and upsampling modules. The 1x1 convolutions adjust the number of channels in the feature layer to the number of small object segmentation categories. The concat function is used to fuse the feature layers. The upsampling uses the resize function to output a feature layer with the same width and height as the input image.

[0050] The YOLOv5-S small object detection network model consists of three main parts: a detection backbone network, enhanced feature extraction and fusion, and a detection head.

[0051] The detection backbone network structure mainly extracts features of small targets at different scales in the image through a series of convolutions, batch normalization, activation functions, and pooling.

[0052] This enhanced feature extraction and fusion structure consists of a pyramid and several upsampling and downsampling modules, used to achieve enhanced extraction and fusion of multi-scale features of small targets;

[0053] This detection head section mainly consists of multi-scale detection heads, used to predict the location and category of small targets in the ground background.

[0054] The specific implementation of this step is as follows:

[0055] 2.1) Preprocessing of the acquired small target visible light image dataset using software annotation:

[0056] Object segmentation requires precise pixel-level annotations, including detailed information such as outlining the contours of each small object with points and lines. This example uses Labelme software to annotate different datasets, where:

[0057] For the semantic segmentation model dataset, Labelme software was used to assign a color to each small target. There are 5 target categories: "car", "house", "plane", and "tank", with corresponding colors of red, green, yellow, and blue, respectively. The background is black by default. A JSON file is generated after annotation.

[0058] For the YOLOv5-S small object detection network model dataset, the LabelImg software was used to outline the four types of small objects (house, plane, tank, and car) with rectangles of different sizes. After the annotation was completed, an XML format file was generated.

[0059] Due to the high cost of creating datasets, this example currently only labels 4,000 images for the semantic segmentation model dataset and 12,000 images for the YOLOv5-S small object detection network model dataset. In future work, the background segmentation dataset will be labeled based on the performance of the network models.

[0060] 2.2) Convert the preprocessed annotation files:

[0061] The labeled files must be converted into label files before model training can be performed. This example converts different labeled files separately:

[0062] The conversion of the annotation files for the semantic segmentation model dataset involves converting the input JSON annotation file and the corresponding original JPG image into a PNG label file using a segmentation conversion script file. For example, if the input is a 1.json annotation file and a 1.jpg image, the output of the segmentation conversion script file is a 1.png label file, thus completing the label file conversion.

[0063] The conversion of the YOLOv5-S small object detection network model dataset annotation files involves converting the input XML annotation files into TXT format files using a detection conversion script file, thus completing the label file conversion work.

[0064] 2.3) Divide the dataset into training and validation sets.

[0065] The semantic segmentation model dataset and the YOLOv5-s small object detection network model dataset were divided into training and validation sets according to different ratios, where:

[0066] The semantic segmentation model dataset is divided based on the segmentation script file. The PNG label files generated in 2.1) are divided into training and validation sets in an 8:2 ratio, generating train.txt and val.txt images. The train.txt is used as the input training data for the model, and the val.txt is used to validate the model and visualize the segmentation results of small objects.

[0067] The dataset partitioning for the YOLOv5-S small object detection network model is based on the detection partitioning script file. The txt format file generated in 2.2) is divided into training, validation, and test sets in a ratio of 8:1:1. Train.txt, val.txt, and test.txt images are generated, where train.txt is used to train the model, val.txt is used to evaluate the model, and test.txt is used to visualize the small object detection results.

[0068] Step 3: Construct a small target segmentation model for the earth background.

[0069] Reference Figure 2 This step is based on an improvement to the existing Deeplabv3plus semantic segmentation model, and its implementation is as follows:

[0070] 3.1) A channel attention mechanism is used to process one shallow high-resolution feature layer in the backbone network of the semantic segmentation network model:

[0071] 3.1.1) For shallow high-resolution feature layers, first perform global average pooling, then perform two fully connected processing steps;

[0072] 3.1.2)) The sigmoid function is used to calculate the weight w of each feature point in the shallow high-resolution feature layer after processing in 2.1.1), and the weight is multiplied by the input shallow high-resolution feature layer to complete the attention mechanism processing of the shallow high-resolution feature layer, where the value of w is between 0 and 1.

[0073] 3.2) For the two deep low-resolution feature layers in the backbone network of the semantic segmentation network model, channel attention and spatial attention mechanisms are used for processing:

[0074] 3.2.1) Channel attention mechanism processing for two deep low-resolution feature layers:

[0075] The channel attention mechanism is divided into two branches. Each deep low-resolution feature layer first goes through the first branch and is processed sequentially through global average pooling, the first fully connected layer, the ReLU activation function, and the second fully connected layer; then it goes through the second branch and is processed sequentially through max pooling, the first fully connected layer, the ReLU activation function, and the second fully connected layer.

[0076] Next, the processing results of the two branches are added together, and the weight w of each channel in each input deep low-resolution feature layer is calculated using the sigmoid function. , , where w , The value ranges from 0 to 1;

[0077] 3.2.2) Spatial attention mechanism processing for two deep low-resolution feature layers:

[0078] For each feature point in each deep low-resolution feature layer, the maximum value and average value are taken first, and these maximum value and average value are stacked using the concat function. That is, the channel number of the two is added by the concat function, and the channel number is adjusted by a convolution with a channel number of 1.

[0079] For each deep low-resolution feature layer after adjusting the number of channels, the weight w of each feature point is calculated using the sigmoid function. ,, , where w ,, The value of is between 0 and 1;

[0080] 3.2.3) Each input deep low-resolution feature layer is sequentially compared with the weights w obtained by the channel attention mechanism. , The weights w obtained by the spatial attention mechanism ,, Multiplication is performed to complete the attention mechanism processing of the two deep low-resolution feature layers;

[0081] By processing the attention mechanism of one shallow high-resolution feature layer and two deep low-resolution feature layers in the backbone network of the semantic segmentation network model through steps 2.1) and 2.2), a deeper feature layer is obtained.

[0082] 3.3) Based on the small target feature fusion (ASPP) feature fusion structure in the semantic segmentation model, a new small target feature fusion branch is added, and the dilation rate parameter of its internal dilated convolution is adjusted to 4, 8, 12, and 16, forming a multi-scale small target feature fusion ASPP-6 structure, such as... Figure 3 As shown, it consists of two parts: the first part is composed of six cascaded branches: 1x1 convolution, 3x3 convolution with a dilation rate of 4, 3x3 convolution with a dilation rate of 8, 3x3 convolution with a dilation rate of 12, 3x3 convolution with a dilation rate of 16, and global average pooling. Each branch generates a feature layer, and each 3x3 convolution branch consists of convolution, batch normalization, and ReLU activation function. The second part uses the contact function to perform small target feature fusion operation on the feature layer processed by these six branches to strengthen the first deep low-resolution feature layer after fusion.

[0083] 3.4) A shallow, high-resolution feature layer suitable for small object segmentation is extracted using the backbone network of the semantic segmentation network model. The number of channels is adjusted using 1x1 convolutions. This shallow, high-resolution feature layer with adjusted channels is then stacked with the enhanced low-resolution feature layer obtained in 2.3) using the concat function. This forms a residual network structure consisting of a shallow, high-resolution feature layer and a low-resolution feature layer obtained from the ASPP-6 structure connected in parallel. Figure 4 As shown;

[0084] 3.5) The backbone network processed by the attention mechanism in steps 3.1) and 3.2), the ASPP-6 structure formed by fusing multi-scale small target features in step 3.3), the residual network structure formed in step 3.4), and the 1x1 convolution are sequentially cascaded to form the encoding part of the improved semantic segmentation model. This encoding part and the decoding part of the original model together form the small target segmentation model for the earth background.

[0085] Step 4: Train the small target segmentation model for the ground background.

[0086] In this example, the testing and evaluation of the small target segmentation model for the earth background and the YOLOv5-S small target detection network model were all built on the Python 3.9, PyTorch 1.11, and PyCharm platforms.

[0087] 3.1) Using the existing Pascal VOC2007 public dataset training set, a pre-training method of forward propagation followed by backward propagation is used to train the small target segmentation model in the ground background, resulting in a pre-trained small target segmentation model in the ground background, and generating a pre-trained weight file:

[0088] 3.1.1) Place the existing Pascal VOC2007 public dataset training set into the training file path of the model;

[0089] 3.1.2) Set the relevant training parameters for the model:

[0090] The training rounds are set to 50, the maximum learning rate is 0.001, the minimum learning rate is 0.0001, the Adam optimizer is used, the input image size is 640 pixels x 640 pixels, the total number of small object and background segmentation categories is 21, the downsampling factor is 8, the number of training threads is 8, and the unified computing device architecture acceleration module is referenced.

[0091] 3.1.3) Begin training:

[0092] The training rounds start from 1. The entire model training will proceed through forward propagation and backward propagation in sequence. First, forward propagation will proceed through the backbone network of the small target segmentation model, the residual network structure, and the ASPP-6 structure for multi-scale small target feature fusion in sequence. Then, backward propagation will proceed through the ASPP-6 structure for multi-scale small target feature fusion, the residual network structure, and the backbone network in sequence.

[0093] After each forward and backward propagation, the current training round number is incremented by 1 until the training round number reaches 50. At this point, the pre-training of the small target segmentation model for the earth background is completed, and a pre-trained weight file based on the Pascal VOC2007 public dataset is generated.

[0094] 3.2) Retrain the small target segmentation model against the background using the training set of the semantic segmentation dataset to obtain a retrained small target segmentation model against the background, and generate a retraining weight file:

[0095] To enable the small target segmentation model for background terrain to learn the features of small targets in the images of the training set of the semantic segmentation dataset, and to use the trained model to segment images in the validation set, the background terrain segmentation model needs to be retrained using the training set of the semantic segmentation dataset. The weight file used for retraining the model is the pre-trained weight file obtained in section 3.1.3). The dataset used is the training set of the semantic segmentation dataset, which is placed in the training file path of the background terrain segmentation model. The total number of training rounds for the background terrain segmentation model is set to 50. The specific implementation is divided into two stages: frozen training and unfrozen training.

[0096] 3.2.1) Freeze Training:

[0097] The first 25 training epochs are designated as the frozen training phase, with a maximum learning rate of 0.001 and a minimum learning rate of 0.0001. The number of data samples captured in one training session is 4, and the Adam optimizer is used. The input image size is 1024x768 pixels, the total number of categories for segmenting small objects and background is 5, the downsampling factor is 8, the number of training threads is 8, and the focal loss function is used. To prevent imbalance between positive and negative samples of small objects, a unified computing device architecture acceleration module is used.

[0098] Running the training script file indicates the start of training. At this point, the number of training rounds starts from 1. When the earth background small object segmentation model completes one traversal of the training set in the semantic segmentation dataset, it represents the end of one training round, and then the next round of training begins. This process is repeated until the number of training rounds reaches 25, at which point the training freeze phase is complete.

[0099] 3.2.2) Thawing training:

[0100] The unfreezing training phase begins at training epoch 26. Compared to the frozen training phase in 3.2.1), the difference lies in adjusting the model's training parameters, primarily by setting the learning rate to 0.0005, using 2 data samples per training session, and doubling the GPU memory usage. When training epochs reach 50, the unfreezing training phase is complete. This also signifies the completion of retraining the entire ground background small object segmentation model, generating a retraining weight file. This file is used for model validation and segmentation of the semantic segmentation model validation set.

[0101] Step 4: Based on the validation set of the semantic segmentation model, the trained semantic segmentation model for small objects against a background is validated using the validation script file of the small object segmentation model against a background to achieve coarse segmentation of the background and small objects.

[0102] 4.1) The average intersection-over-union ratio (IoU) and average coarse segmentation accuracy metrics were used to effectively validate the small target segmentation model for the Earth's background:

[0103] The semantic segmentation model's validation set was used to validate the small target segmentation model against the background. Specifically, an existing validation script file was referenced from the model. First, the number of segmentation categories within the file was set to 5. Then, the internal weights were adjusted to the retrained weights file generated in step 3.2. The validation script file was then run, ultimately yielding the average intersection-union ratio (OCR) and the average coarse segmentation accuracy of the target and background regions. These two metrics complete the validation of the small target segmentation model against the background based on the semantic segmentation validation set. The average OCR is the ratio of the intersection to the union of the labeled small target area and the area predicted by the model.

[0104] 4.2) Achieve coarse segmentation of the terrain background and small targets:

[0105] This is achieved by using an existing prediction script on any image in the validation set. Specifically, running the prediction script opens a window where input is given in the format "img / image name.image format". The output image shows a segmented image where each small target is covered by a color (e.g., black for the background, red for armored vehicles, green for houses, yellow for airplanes, and blue for tanks). This completes the coarse segmentation between the background and the small targets, demonstrating that the background-small-target segmentation model in this invention has achieved coarse segmentation of complex and variable backgrounds and small targets.

[0106] Step 5: Train the YOLOv5-S small object detection network model.

[0107] 5.1) Place the labeled XML format tag file in the Annotation file under the VOC2007 folder in the VOCdevkit folder; place the original image file in the JPEGImages folder under the VOC2007 folder in the VOCdevkit folder; then create a cls_classes.txt file, which contains the four types of targets that the yolov5-s small object detection network model needs to distinguish and identify, namely house, car, plane and tank.

[0108] 5.2) Set the relevant training parameters for this model:

[0109] The total number of training epochs is set to 50, the maximum learning rate is 0.001, the minimum learning rate is 0.0001, the Adam optimizer is selected, the model weights are from an existing model weight file, the input image size is 640x640 pixels, the number of small object categories to be identified is 4, the downsampling factor is 8, the number of GPU threads is 8, and the Unified Computing Device Architecture acceleration module is referenced.

[0110] 5.3) Start training the model:

[0111] The training of this part of the model is twofold: firstly, to enable the model to learn the features of small objects in the image, and secondly, to use the YOLOv5-S small object detection network model to detect the segmented image.

[0112] In the first half of the model training, the torch.cuda.amp module, which comes with the deep learning PyTorch framework, is used. The gradient descent function used by the backbone network during model training is used as the input of the torch.cuda.amp function. The function is then passed through the backbone network, the enhanced small target feature extraction part, and the detection head part of the yolov5-s small target detection network model in sequence to optimize it.

[0113] Each time an iteration of these three parts is completed, the training round number is incremented by 1. When the training round number reaches 50 or the torch.cuda.amp function converges, the model training stops, and a trained small target recognition model against a ground background is obtained. At the same time, a weight file for the small target recognition model against a ground background is generated.

[0114] Step 6: Use the trained small target recognition model against a background of the earth to detect the image dataset after coarse segmentation by the semantic segmentation model against a background of the earth.

[0115] In complex, dynamic, and random terrain environments, the segmentation and recognition of small targets of the same category but different forms, as well as the segmentation and recognition of different buildings, can be performed using only the existing Deeplabv3plus semantic segmentation model.

[0116] When dealing with small targets that are small in size, occupy a small area in the image, and have unclear edges and contours, there may be problems with missegmentation or even missed segmentation of small targets. The YOLOv5-S object detection model can be used for detection, but when dealing with small targets that are mixed together or occluded with each other, there may be problems with low detection accuracy, misdetection, or even missed detection.

[0117] It is evident that both single semantic segmentation models and object detection models have room for optimization and cannot achieve accurate segmentation and recognition of small targets and large backgrounds.

[0118] This example addresses the problems of existing technologies that use single models that only segment or only detect, and proposes a small target cascade detection technique for special terrain scenes. This technique first segments the target from the terrain background before performing target recognition. The specific implementation is as follows:

[0119] 6.1) For any image segmented in 4.2), after random flipping, cropping, merging, and color gamut changes, the processed segmented image is obtained;

[0120] 6.2) The trained small target recognition model against the background is used to further detect the segmented image processed in 6.1), and the location and category of small targets against the background are identified. That is, based on the coarse target segmentation, a good result of small target recognition against the background is achieved, and finally the task of coarse segmentation and small target recognition between the complex and variable background and small targets in air-to-ground observation is completed.

[0121] The effectiveness of this invention can be further illustrated by the following comparative simulation experiments:

[0122] I. Simulation Conditions

[0123] All simulation experiments conducted in this invention are based on an experimental platform consisting of an Intel i9-10980XE 3.0GHz*18 processor, 128GB RAM, and an NVIDIA RTX3090 24GB VRAM, using Jupyter Notebook, Python 3.9, PyTorch 1.11 deep learning framework, CUDA 10 module library, cuDNN 11 module library, and PyCharm 2020 integrated development environment software.

[0124] II. Simulation Content

[0125] Simulation 1: Under the above simulation conditions, using existing segmentation verification scripts and based on the validation set of the semantic segmentation dataset, the existing Deeplabv3plus model and the geodetic background semantic segmentation model of this invention are used to identify five types of small targets—background, car, house, plane, and tank—in a complex and variable special geodetic background environment. The average intersection-over-union (IoU) index parameters are obtained as follows: Figure 5 As shown. Among them. Figure 5 (a) is a plot of the average intersection-union ratio of the existing Deeplabv3plus model; Figure 5 (b) is the average intersection-union index of the geodetic background semantic segmentation model of the present invention.

[0126] from Figure 5 It can be seen that the existing Deeplabv3plus model has an average intersection-over-union (IoU) ratio of 0.85 for tank targets, 0.84 for plane targets, 0.77 for house targets, and 0.86 for car targets, with an overall average IoU of 86.42%. In contrast, the earth background semantic segmentation network model of this invention achieves an average IoU of 87.38%, representing an overall improvement of 0.96% compared to existing technologies, with improvements specifically for smaller targets in the tank, plane, house, and car categories.

[0127] Experimental results show that the earth background semantic segmentation network model of the present invention can make the area coverage of different colors on each small target more accurate, the coverage rate greater, and the intersection-union ratio higher.

[0128] Simulation 2: Under the above simulation conditions, using existing segmentation validation scripts and based on the validation set of the semantic segmentation dataset, the existing Deeplabv3plus model and the earth background semantic segmentation model were used to identify five types of small targets—background, car, house, plane, and tank—in complex and varied special earth background environments. The average coarse segmentation accuracy index parameters are as follows: Figure 6 As shown. Among them. Figure 6 (a) is a graph showing the average classification accuracy of the existing Deeplabv3plus model for small targets; Figure 6 (b) is a graph showing the average classification accuracy of the ground background semantic segmentation model of the present invention for small targets.

[0129] from Figure 6 It can be seen that the existing Deeplabv3plus model has an average classification accuracy of 0.91 for tank-type targets, an average intersection-over-union (IoU) ratio of 0.90 for plane-type targets, an average IoU ratio of 0.75 for house-type targets, and an average IoU ratio of 0.83 for car-type targets, with an average classification accuracy of 89.93%. In contrast, the earth background semantic segmentation network model of this invention achieves an average classification accuracy of 90.98%, representing an overall improvement of 1.05% compared to existing technologies.

[0130] Experimental results show that the geodetic background semantic segmentation model of the present invention can achieve high-accuracy inter-region coarse segmentation and classification of geodetic background and small targets.

[0131] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the protection scope of the claims of the present invention.

Claims

1. A method for identifying small targets in the background of the earth based on cascaded combined networks, characterized in that, Includes the following steps: (1) Construct a dataset of small ground background images using a semantic segmentation model and a YOLOv5-S small object detection network model: 1a) Preprocessing of a dataset of visible light images of small targets against a background of the earth, including image enhancement and software annotation; 1b) Divide the preprocessed dataset into a training set and a validation set for the semantic segmentation model in an 8:2 ratio; 1c) Divide the preprocessed dataset into training, validation and test sets for the YOLOv5-S small object detection network model in an 8:1:1 ratio; (2) Improve the existing Deeplabv3plus semantic segmentation model and construct a small target segmentation model for the ground background: 2a) In the backbone network of the semantic segmentation network model, one shallow high-resolution feature layer and two deep low-resolution feature layers are processed by an attention mechanism to obtain a deeper feature layer. 2b) Based on the small target feature fusion (ASPP) feature fusion structure in the semantic segmentation model, a new small target feature fusion branch is added, and the dilation rate parameter of its internal dilated convolution is adjusted to 4, 8, 12, and 16 to form a multi-scale small target feature fusion ASPP-6 structure, which is used to strengthen the first deep low-resolution feature layer after fusion; the ASPP-6 structure includes two parts: a front part and a back part. The first half consists of six cascaded branches: 1x1 convolution, 3x3 convolution with a dilation rate of 4, 3x3 convolution with a dilation rate of 8, 3x3 convolution with a dilation rate of 12, 3x3 convolution with a dilation rate of 16, and global average pooling. Each branch generates a feature layer, and each 3x3 convolution branch consists of convolution, batch normalization, and ReLU activation function. The latter part involves using the concat function to perform small target feature fusion on the feature layer after processing these 6 branches; 2c) Use the backbone network in the semantic segmentation network model to extract shallow high-resolution feature layers suitable for small object segmentation, use 1x1 convolution to adjust the number of channels, and stack the shallow high-resolution feature layers with the enhanced fusion in 2b) using the concat function to form a residual network structure. 2d) The backbone network processed by the attention mechanism in step 2a), the ASPP-6 structure formed by fusing multi-scale small target features in step 2b), and the residual network structure formed in step 2c) are cascaded in sequence to form a small target segmentation model for the earth background. (3) Training the small target segmentation model for the earth background: 3a) Using the training set of the existing Pascal VOC2007 public dataset, the small target segmentation model of the earth background is pre-trained by adopting the training method of forward propagation and then backward propagation, so as to obtain the pre-trained small target segmentation model of the earth background. 3b) Using the training set of the semantic segmentation model, a two-stage training method of first freezing the backbone network and then unfreezing the backbone network is used to train the pre-trained earth background small target segmentation model again, so as to obtain the trained earth background small target semantic segmentation model and generate the model weight file trained by the training set of the semantic segmentation model. (4) Based on the validation set of the semantic segmentation model, the trained semantic segmentation model for small targets in the background is validated using the prediction script file of the background segmentation model for small targets in the background, so as to achieve coarse segmentation of the background and small targets in the region. (5) Based on the training set of the yolov5-s small target detection network model in 1c), the yolov5-s small target detection network model is trained using the mixed precision training method to obtain a trained small target recognition model in the background of the earth, and a model weight file trained by the training set of the small target detection network model is generated. (6) Use the trained small target recognition model under the background to detect the image dataset after coarse segmentation by the semantic segmentation model under the background. Detect the location and category of small targets under the background in the image dataset. That is, on the basis of coarse segmentation of the background and small target regions, identify small targets under the background to realize the cascaded combination detection of background segmentation and target recognition.

2. The method according to claim 1, characterized in that, Step 1a) involves preprocessing the visible light image dataset of small targets against a background of the earth, including image enhancement and software annotation, as follows: 1a1) Using the median filtering function from the OpenCV library, the median filtering kernel is convolved with each pixel in the image to be processed. The number of various small objects in the image is increased, and some small objects in the image are randomly flipped and scaled to achieve image enhancement processing of the dataset. 1a2) Labelme software is used to draw the outline of each small target with lines, that is, to outline the small target to be detected with a rectangle, and to set a category for each type of small target, generating an image annotation dataset for the small target detection model in the ground background.

3. The method according to claim 1, characterized in that, Step 2a) uses an attention mechanism to process shallow high-resolution feature layers, specifically through a channel attention mechanism, as detailed below: 2a1) For shallow high-resolution feature layers, first perform global average pooling, then perform two fully connected processing steps; 2a2) The sigmoid function is used to calculate the weight w of each feature point in the feature layer after processing by 2a1), and the weight is multiplied by the input shallow high-resolution feature layer to complete the attention mechanism processing of the shallow high-resolution feature layer, where the value of w is between 0 and 1.

4. The method according to claim 1, characterized in that, Step 2a) employs an attention mechanism to process the two deep low-resolution feature layers, using both channel attention and spatial attention mechanisms. The specific implementation is as follows: 2a3) Channel attention mechanism processing for two deep low-resolution feature layers: The channel attention mechanism is divided into two branches. Each deep low-resolution feature layer first undergoes global average pooling, a first fully connected layer, a ReLU activation function, and a second fully connected layer sequentially through the first branch. Then, it undergoes max pooling, a first fully connected layer, a ReLU activation function, and a second fully connected layer sequentially through the second branch. Finally, the processing results from the two branches are summed, and the sigmoid function is used to calculate the channel weight w for each input deep low-resolution feature layer. , , where w , The value ranges from 0 to 1; 2a4) Spatial attention mechanism processing for two deep low-resolution feature layers: For each feature point in each deep low-resolution feature layer, first take the maximum value and average value in each channel, and then stack these maximum values ​​and average values ​​using the concat function; The number of channels is adjusted by performing a convolution with 1 channel again; then, the weight w of each feature point in each deep low-resolution feature layer is calculated using the sigmoid function. ,, , where w ,, The value of is between 0 and 1; 2a5) Each input deep low-resolution feature layer is sequentially compared with the weights w obtained by the channel attention mechanism. , The weights w obtained by the spatial attention mechanism ,, Multiplication is performed to complete the attention mechanism processing of two deep low-resolution feature layers.

5. The method according to claim 1, characterized in that, In step 2c), the shallow high-resolution feature layer with adjusted channel count and the enhanced fused low-resolution feature layer are stacked using the concat function, which adds the channel counts of the two layers together.

6. The method according to claim 1, characterized in that, In step 3a), the existing Pascal VOC2007 public dataset is used as the training set. A forward propagation and then backward propagation training method is used to pre-train the small target segmentation model of the ground background, as follows: 3a1) Place the training set of the existing Pascal VOC2007 public dataset into the training file path of the model; 3a2) Set the relevant training parameters for the model: The training rounds are set to 50, the maximum learning rate is 0.001, the minimum learning rate is 0.0001, the Adam optimizer is used, the input image size is 640 pixels x 640 pixels, the total number of small object and background segmentation categories is 21, the downsampling factor is 8, the number of training threads is 8, and the unified computing device architecture acceleration module is referenced. 3a3) Start training the model The training rounds start from 1. The entire model training will proceed through forward propagation and backward propagation in sequence. First, forward propagation will proceed through the backbone network of the small target segmentation model, the residual network structure, and the ASPP-6 structure for multi-scale small target feature fusion in sequence. Then, backward propagation will proceed through the ASPP-6 structure for multi-scale small target feature fusion, the residual network structure, and the backbone network in sequence. After each forward and backward propagation, the current training round number is incremented by 1 until the training round number reaches 50. At this point, the pre-training of the small target segmentation model for the earth background is completed, and a pre-trained weight file based on the Pascal VOC2007 public dataset is generated.

7. The method according to claim 1, characterized in that, In step 4), the trained semantic segmentation model for small objects against a background is validated using its prediction script file. This involves first adjusting the internal weight file of the existing validation script to the final model weight file; then, a random image is selected from the validation set, and its validation script file is run. A terminal window pops up, where input is given in the format "img / image name". The output is a segmented image, thus completing the validation of the small object segmentation model against a background.

8. The method according to claim 1, characterized in that, Step (5) Based on the training set of the yolov5-s small object detection network model in 1c), the yolov5-s small object detection network model is trained using the mixed precision training method, as follows: 5a) Place the training set of the yolov5-s small object detection network model dataset in the training file path of the model; 5b) Set the relevant training parameters for this model: The total number of training epochs is set to 50, the maximum learning rate is 0.001, the minimum learning rate is 0.0001, the Adam optimizer is selected, the model weights are from an existing model weight file, the input image size is 640x640 pixels, the number of small object categories to be identified is 4, the downsampling factor is 8, the number of GPU threads is 8, and the Unified Computing Device Architecture acceleration module is referenced. 5c) Start training the model In the first half of the model training, the torch.cuda.amp module, which comes with the deep learning PyTorch framework, is used. The gradient descent function used by the backbone network during model training is used as the input of the torch.cuda.amp function. The function is then passed through the backbone network, the enhanced small target feature extraction part, and the detection head part of the yolov5-s small target detection network model in sequence to optimize it. Each time an iteration of these three parts is completed, the training round number is incremented by 1. When the training round number reaches 50 or the torch.cuda.amp function converges, the model training stops, and a well-trained small target recognition model against a ground background is obtained.

Images