A multi-stage collaborative fine-grained ship class imbalance problem optimization method

Abstract

In the fine-grained ship target detection task, the number of instances of different ship target categories is very different, and the network cannot pay enough attention to the very small number of rare categories in the network training process, resulting in a decline in detection performance. To solve this problem, the invention provides a multi-stage collaborative fine-grained ship category imbalance optimization method. In the forward propagation stage, a sample replacement method based on ship shape mask is provided, and the rare ship is used to replace the common ship to alleviate the sample imbalance problem. In the back propagation stage, a loss weight method based on the number of meta-ship samples is provided, so that the network pays more attention to rare samples. In the inference decision stage, based on the number of meta-ship samples, a decision boundary optimization method is provided to reduce the influence of prior probability on model prediction. The multi-stage category imbalance optimization method provided by the invention can effectively improve the learning ability of the neural network to rare samples and improve the performance of fine-grained ship detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, and particularly relates to a multi-stage cooperative fine-grained ship class imbalance problem optimization method. BACKGROUND

[0002] Remote sensing technology uses satellite detection instruments to record, analyze and interpret the electromagnetic wave characteristics of target objects at a distance without contacting the target objects, and is applied to various fields such as resource investigation, ground environment monitoring and human activity monitoring. In recent years, with the continuous development of remote sensing technology, the number and resolution of visible light remote sensing images have also greatly improved. These images contain rich earth observation information, and how to automatically and accurately interpret this information has important value and significance. Benefiting from the powerful feature representation capability of deep convolutional neural networks, target detection algorithms based on deep learning have achieved excellent results. The deep learning method is data-driven, and in particular in the fine-grained ship detection task, due to the large number of ship target classes and the large difference in the number of instances of different classes, a long-tail distribution is presented, which leads to a sample imbalance problem. The number of rare classes is usually very small, and the network will not pay enough attention to these classes during the training process of the neural network, which will lead to a decrease in detection performance. SUMMARY

[0003] In view of the deficiencies in the prior art, the purpose of the present application is to provide a multi-stage cooperative fine-grained ship detection sample imbalance problem optimization method, which can balance the attention of deep neural networks to different classes of samples, better learn the features of rare samples, and further improve the performance of ship target detection.

[0004] The technical scheme adopted by the present application is:

[0005] A multi-stage cooperative fine-grained ship class imbalance problem optimization method, comprising the following processes:

[0006] Step 1: prepare a training set and a target detection model, the training set contains ship targets, the target detection model contains a feature extraction backbone network and a detection head, and the input of the target detection model is a three-channel optical remote sensing image;

[0007] Step 2: in the forward propagation stage, the ship classes in the training set are divided into common classes and rare classes, and the rare class ships are modeled using a two-dimensional Gaussian model, the rare class ships are used to replace the common ships, and then the backgrounds and directions of different ships are adjusted to be consistent;

[0008] Step 3: In the back propagation stage, the number of meta samples of different categories of ships is calculated, and the weight factor of the ship is inversely proportional to the number of meta samples, and the balanced loss function of different categories is obtained, and then the training set samples are used to train the target detection model to obtain the updated target detection model.

[0009] Step 4: In the inference decision stage, the updated target detection model is used to predict the input sample, the detection frame output by the model is sorted in descending order according to the score, and the non-maximum suppression algorithm is used to remove redundant frames to obtain the final accurate detection result.

[0010] Further, step 2 specifically includes the following process:

[0011] Step 2-1: In the forward propagation stage, the number of different categories of ships in the training set is counted, and the categories higher than the average number of samples are taken as common categories, and the categories lower than the average number of samples are taken as rare categories; then the ship samples of the rare categories are cropped to establish a sample library and index, and the size, aspect ratio and color information of the cropped rare ship samples are calculated;

[0012] Step 2-2: The height and width of the cropped rare ship sample are respectively subjected to one-dimensional Gaussian, and the Gaussian kernel in the height direction is The Gaussian kernel in the width direction is And the mask M of the ship shape is constructed through the Gaussian kernels in two directions h×w =G h ·G w T ; wherein w is the width of the rare ship sample, h is the height of the rare ship sample, w η and h η are the fusion weights of the Gaussian kernel;

[0013] Step 2-3: A set probability is selected to select a common ship instance S c in the training set, the size, aspect ratio and color information are calculated, and the calculated information is integrated into a feature vector and sent into a KNN matching tree to match with the rare ship, and a set proportion of rare ships are screened out according to the distance from near to far, and a sample S r is randomly selected from them; S c is taken as a reference to perform affine transformation on S r to unify the size and direction, and then S c and S r are fused to obtain a new sample S n =S r ×M h×w +S c ×(1-M h×w );

[0014] Step 2-4: fusing the new sample S n Transforming back to the original training set remote sensing image I k In the process, I k The class of the corresponding position ship in I v is modified to the replaced rare ship class, and the brightness I v of the original sample and the brightness O v of the replaced ship sample are calculated. Then the output pixel value is obtained Where I is the input pixel value, I = O i , and the brightness of the replaced ship sample is adjusted according to the output pixel value using gamma correction.

[0015] Further, step 3 specifically includes the following process:

[0016] Step 3-1: calculating the meta-sample number of different classes of ships, wherein the meta-sample number is the minimum number of ships that can completely represent the characteristics of different classes; the calculation method of the ship meta-sample number is as follows: Where n i is the number of ships of class i in the training set, δ y is a hyperparameter, and M j is the meta-sample number of ships of class i.

[0017] Step 3-2: assuming that the weight factor of the ship is inversely proportional to the meta-sample number, the balanced classification loss of a certain class y is obtained Where z η is the true label, and z j is the predicted value.

[0018] Step 3-3: training the target detection model by using the training set samples in a back propagation manner, and updating the parameters of the target detection model by using a stochastic gradient descent optimizer to obtain an updated target detection model.

[0019] Further, step 4 specifically includes the following process:

[0020] Step 4-1: predicting the input vector by using the updated target detection model to obtain a predicted probability distribution, and dividing the predicted probability distribution by a prior probability to obtain the scores of different detection boxes; wherein the prior probability is represented by the meta-sample number.

[0021] Step 4-2: sorting the detection boxes output by the model in step 4-1 according to the scores from large to small, and removing redundant boxes by using a non-maximum suppression algorithm to obtain a final accurate detection result, wherein the detection result includes the final rotation box coordinates and the target class.

[0022] The present application has the following advantages:

[0023] The application uses a sample replacement method based on a ship shape mask in the neural network forward propagation stage, and the rare ship sample can be randomly used to replace the common ship during the training process;

[0024] In the back propagation stage, the distribution of the meta-ship sample is calculated, and the loss of different types of ships is adjusted according to the distribution, so as to enhance the learning ability of the neural network to the rare type of ship sample;

[0025] In the decision reasoning stage, the prediction probability distribution of the model is corrected, and then the decision boundary is adjusted, so that the prediction score of the network to different types of ships is more balanced, and the three stages are optimized cooperatively, so as to improve the detection performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The sample replacement schematic diagram based on the ship shape mask provided by the application is shown.

[0027] Figure 2 The ship shape modeling process schematic diagram of the application is shown.

[0028] Figure 3 The gamma transformation schematic diagram of the application is shown.

[0029] Figure 4 The logistic regression schematic diagram of the application is shown.

[0030] Figure 5 The decision boundary correction schematic diagram of the application is shown.

[0031] Figure 6 The detection result schematic diagram of the application is shown. DETAILED DESCRIPTION

[0032] The application provides a multi-stage cooperative fine-grained ship class imbalance problem optimization method, in order to make the purpose, technical scheme and effect of the application more clear and explicit, the application will be further described in detail below with reference to the drawings.

[0033] The application provides a multi-stage cooperative fine-grained ship class imbalance problem optimization method. In the forward propagation stage, a sample replacement method based on ship shape mask is provided, the ship target is modeled by two-dimensional Gaussian, and the rare ship is used to replace the common ship and the consistency of the background and direction of different ships in the replacement process is ensured. In the back propagation stage, a loss reweighting method based on the number of meta-ship samples is provided, an adaptive weight function is designed for rare samples, so that the network pays more attention to rare samples during back propagation. In the reasoning and decision-making stage, based on the number of meta-ship samples, a decision boundary optimization method is provided to reduce the influence of prior probability on model prediction. The multi-stage class imbalance optimization method provided by the application can effectively improve the learning ability of the neural network to rare samples and improve the performance of fine-grained ship detection. Specifically, the following processes are included:

[0034] Step 1: Prepare the training set and the target detection model, the training set contains ship targets, the target detection model contains a feature extraction backbone network and a detection head, and the input of the target detection model is a three-channel optical remote sensing image;

[0035] In this embodiment, the optical images containing ships in the training set are cropped to 1024 pixels x 1024 pixels, the number of samples in each batch is 16, and it is ensured that no repeated images appear in each batch.

[0036] Step 2: In the forward propagation stage, the ship classes in the training set are divided into common classes and rare classes, and the rare class ships are modeled by two-dimensional Gaussian, the rare class ships are used to replace the common ships, and then the background and direction of different ships are adjusted to keep consistent;

[0037] As shown in Figure 1 , step 2 specifically includes the following processes:

[0038] Step 2-1: Count the number of different classes of ships in the training set in step 1, define the classes higher than the average number of samples as common classes, and the classes lower than the average number of samples as rare classes (the rare classes can also be defined by the user). Crop the ship samples of the rare classes to establish a sample library and an index, and calculate the size, aspect ratio and color information of the cropped ships. The short side of the ship is recorded as w, the long side is recorded as h, the size is defined as w x h, and the aspect ratio is defined as h / w. Convert the RGB format of the image to HSV format, where H represents the hue of the image, S represents the saturation of the image, and V represents the brightness of the image;

[0039] Step 2-2: Two-dimensional Gaussian modeling is performed on the cropped rare ship samples, that is, assuming that the rare ship sample has a size of w x h, one-dimensional Gaussian is performed on w and h respectively, the Gaussian kernel in the height direction is and the Gaussian kernel in the width direction is where w η and h η can be regarded as the fusion weight of the Gaussian kernel, w η and h η are larger, the weight of the center part is higher, and then the mask M h×w of the ship shape is constructed by the Gaussian kernel in two directions h · G w T . w η and h η are free to adjust, and in this method w η and h η are set to 4 and 3.5 respectively; as shown in Figure 2 , (a) is the visualization of the Gaussian kernel G h , (b) is the visualization of the Gaussian kernel G w , (c) is the visualization of the generated mask M h×w , (d) is a rare ship sample selected from the sample library, (e) is the original ship sample, and (f) is the result of fusion.

[0040] Step 2-3: Perform the ship shape mask-based sample replacement method with a certain probability, and set the probability to 0.2 in this method. Select a common ship instance S c in the training set with the set probability, calculate its size, aspect ratio and color information, integrate these information into a feature vector and send it into the KNN matching tree for distance matching with the rare ship, and randomly select a sample S r from the nearest 10% of the rare ships; perform affine transformation on S c based on S r to unify its size and direction, and then fuse the two samples with the mask calculated in step 2-2 to obtain a new sample S n =S r ×M h×w +S c ×(1-M h×w ). The KNN matching tree used in this method is a Ball Tree; Step 2-4: Transform S n back to the original training set remote sensing image I k , and modify the class of the corresponding ship in I k to the rare ship class after replacement. Adopt gamma correction to adjust the brightness, calculate the brightness I v of the original sample and the brightness O v of the ship sample used for replacement, calculate the compensation coefficient according to the two, and then obtain the output pixel value , where I is the input pixel value, I = O v . As shown in Figure 3As shown, the gamma transformation is nonlinear, and the change intensity is different for different original brightness values. When γ < 1, the sample brightness is increased; and when γ > 1, the sample brightness is decreased, so that the brightness of the replaced ship sample is corrected to increase the authenticity of data synthesis.

[0041] Step 3: In the back propagation phase, the meta sample number of different categories of ships is calculated, and the weight factor of the ship is inversely proportional to the meta sample number, a balanced loss function of different categories is obtained, and then the training set sample is used to train the target detection model to obtain an updated target detection model.

[0042] Step 3 specifically includes the following processes:

[0043] Step 3-1: The meta sample number of different categories of ships is calculated, wherein the meta sample number of ships of category i is defined as the minimum number of ships that can completely represent the characteristics of the category, and the meta sample number of ships of category i is M i . The sample replacement in steps 2-5 is an idea of image augmentation, and the meta ship sample number will not change due to the above steps. The calculation method of the meta sample number of ships of category i is as follows: Where n i is the number of ships of category i in the training set, and δ i is a hyperparameter, which is set to 0.95 in this method.

[0044] Step 3-2: The loss function of the target detection model includes a classification loss L cls and a positioning loss L reg , and the model can be fine-tuned and corrected for the detection frame by deriving the loss function.

[0045] According to the loss function of the ship meta sample number optimized target detection model, in order to balance the classification loss, a weight factor α c is introduced, which is inversely proportional to the meta sample number M i of the corresponding i-th ship. In summary, assuming that the number of samples of ships of category i S i in the data set is N i , the balanced classification loss corresponding to this category is The balanced classification loss of category y of this method is Where z y is the true label, and z j is the predicted value.

[0046] Step 3-3: The training set sample is used to train the target detection model by back propagation, and the random gradient descent optimizer is used to update the parameters of the target detection model to obtain an updated target detection model.

[0047] Step 4: In the reasoning and decision-making stage, the updated target detection model is used to predict the input samples. The detection boxes output by the model are sorted from largest to smallest score, and the non-maximum suppression algorithm is used to remove redundant boxes to obtain the final accurate detection results.

[0048] Step 4-1: Balance the neural network's prediction scores for different ship categories based on the number of sample ships. For datasets with class imbalance, the classifier scores high for head classes and low for tail classes, causing the classifier to over-predict head classes. Assume a neural network is being trained with logarithm f. y (x). For example... Figure 4 As shown, the label argmax can typically be predicted as the input vector x passes through all neural network layers. y∈[L] f y (x), where y is the output class label y = [L] = {1, 2, ..., L}. When training using softmax cross-entropy, p can be... y (x)∝exp(f y (x) is considered an approximation of the latent P(y|x), thus this is equivalent to predicting the label with the highest estimated class probability, where P(y|x) represents the probability that the network input x will output y. After obtaining the predicted probability distribution, this probability distribution is divided by the prior probability P(y) to adjust the logistic regression boundary before decision, i.e., to optimize argmax. y∈[L] exp(f y (x)) / P(y). Using the number of meta-samples π y As an estimate of the class prior P(y), it is only necessary to optimize Where τ is a hyperparameter used to control the scale of the classification boundary movement. For example... Figure 5 As shown, the unadjusted boundary is represented by the black dashed line. Due to the influence of prior probability, rare ships are easily misidentified as common ships at this decision boundary. However, after boundary adjustment at an appropriate scale, at the ideal boundary shown by the green dashed line, most decision errors caused by class imbalance can be avoided.

[0049] Step 4-2: Sort the detection boxes output by the model in Step 4-1 from largest to smallest score, and use the non-maximum suppression algorithm to remove redundant boxes to obtain the final accurate detection result. The detection result includes the final rotated box coordinates and the target category.

[0050] We validate the effectiveness of the proposed method on the ShipRSImageNet dataset, which is a large-scale fine-grained dataset for ship detection in high-resolution optical remote sensing images. The dataset contains 3,435 images from various sensors, satellite platforms, locations, and seasons, each of which is approximately 930x930 pixels and contains ships of different scales, orientations, and aspect ratios. The images are annotated by satellite image interpretation experts and divided into 50 target classes. The fully annotated ShipRSImageNet contains 17,573 ship instances. The ship instances in the dataset are labeled in OBB format, which is defined by a 5-dimensional vector (x, y, w, h, θ), where (x, y) is the center of the OBB, and (w, h) is the width and height. The mean average precision is used to measure the performance of the algorithm. The detection results of the model are sorted by confidence, and the confidence threshold is gradually adjusted from high to low so that each sample is exactly on the positive / negative boundary. The precision and recall at this time are calculated one by one and recorded in the coordinate plane with recall as the horizontal axis and precision as the vertical axis. The curve formed by these points is the precision-recall curve, and the area enclosed by the curve and the coordinate axes is the average precision (AP). The mean average precision (mAP) is the average of the AP of each class, where the subscript of AP refers to the IoU threshold.

[0051] Figure 6 For the visualization of the detection results, (a) and (c) are the visualization of the baseline method, and (b) and (d) are the results of the method proposed in this paper. Compared with the baseline method, the method proposed in this paper has a significant advantage in detection accuracy and positioning accuracy. The specific experimental results are shown in Table 1. From the qualitative and quantitative analysis of the detection accuracy, it can be seen that the accuracy of the method on ShipRSImageNet is at a high level, and compared with the benchmark algorithm and other detection algorithms, the average precision of the proposed algorithm has been significantly improved. It should be understood that the parts not elaborated in this specification are all prior art.

[0052] Table 1 Experimental results of different detection algorithms on ShipRSImageNet

[0053]

Claims

1. A multi-stage collaborative fine-grained optimization method for the ship category imbalance problem, characterized in that, Includes the following processes: Step 1: Prepare the training set and the target detection model. The training set contains ship targets, and the target detection model contains a feature extraction backbone network and a detection head. The input to the target detection model is a three-channel optical remote sensing image. Step 2: In the forward propagation phase, the ship categories in the training set are divided into common categories and rare categories. Two-dimensional Gaussian models are performed on the rare category ships. The common ships are replaced with rare category ships. Then, the background and orientation of different ships are adjusted to be consistent. Step 3: In the backpropagation stage, calculate the number of meta-samples for different categories of ships, and assume that the weight factor of the ship is inversely proportional to the number of meta-samples to obtain the balance loss function for different categories. Then, use the training set samples to train the target detection model to obtain the updated target detection model. Step 4: In the reasoning and decision-making stage, the updated target detection model is used to predict the input samples. The detection boxes output by the model are sorted from largest to smallest score, and the non-maximum suppression algorithm is used to remove redundant boxes to obtain the final accurate detection results. Step 2 specifically includes the following process: Step 2-1: In the forward propagation phase, count the number of ships of different categories in the training set, classify the categories with a sample size higher than the mean as common categories, and classify the categories with a sample size lower than the mean as rare categories; then cut out the ship samples of rare categories to build a sample library and index, and calculate the size, aspect ratio and color information of the cut out rare ship samples. Step 2-2: Apply a one-dimensional Gaussian kernel to the height and width of the cut-out rare ship samples, with the Gaussian kernel in the height direction being... The Gaussian kernel in the width direction is And construct a ship-shaped mask M using Gaussian kernels in two directions. h×w =G h ·G w T Where w is the width of the rare ship sample, h is the height of the rare ship sample, and w η and h η These are the fusion weights of the Gaussian kernel; Steps 2-3: Select common ship instances S from the training set with a set probability. c The system calculates size, aspect ratio, and color information, integrates this information into a feature vector, and feeds it into a KNN matching tree to perform distance matching with rare ships. It then filters out a set proportion of rare ships based on distance, from closest to furthest, and randomly selects a sample S from these. r ; with S c As a benchmark for S r Perform an affine transformation to unify the size and direction, then S c and S r The two samples are fused using the mask calculated in step 2-2 to obtain a new sample S. n =S r ×M h×w +S c ×(1-M h×w ); Steps 2-4: Fuse to obtain a new sample S n Transform back to the original training set remote sensing image I k In the middle, I k The corresponding ship category in the image is modified to the replaced rare ship category, and the brightness I of the original sample is calculated. v And the brightness O of the ship sample used for replacement v Calculate the compensation coefficient based on both. Then the output pixel value is obtained. Where I is the input pixel value, I = 0 v The brightness of the ship sample used for replacement is adjusted using gamma correction based on the output pixel values.

2. The multi-stage collaborative fine-grained optimization method for the ship category imbalance problem according to claim 1, characterized in that, Step 3 specifically includes the following processes: Step 3-1: Calculate the number of meta-samples for different categories of ships, where the number of meta-samples is the minimum number of ships that can completely represent the characteristics of different categories; the calculation method for the number of ship meta-samples is as follows: Where n i Let δ be the number of ships of category i in the training set. i M is a hyperparameter. i Let i be the number of ship element samples of category i; Step 3-2: Assume the weight factor of the ship is inversely proportional to the number of meta-samples, and obtain the balanced classification loss for a certain category y. Where z y For real labels, z j This is a predicted value; Step 3-3: Train the object detection model using the training set samples through backpropagation, and update the object detection model parameters using a stochastic gradient descent optimizer to obtain the updated object detection model.

3. The multi-stage collaborative fine-grained optimization method for the ship category imbalance problem according to claim 1, characterized in that, Step 4 specifically includes the following processes: Step 4-1: Use the updated object detection model to predict the input vector, obtain the predicted probability distribution, and divide the predicted probability distribution by the prior probability to obtain the score of different detection boxes; where the prior probability is represented by the number of meta-samples. Step 4-2: Sort the detection boxes output by the model in Step 4-1 from largest to smallest score, and use the non-maximum suppression algorithm to remove redundant boxes to obtain the final accurate detection results. The detection results include the final rotated box coordinates and the target category.

Images