Optical remote sensing target detection method based on active learning

Abstract

This invention relates to an active learning-based optical remote sensing target detection method, comprising: acquiring a sample set of optical remote sensing target images; constructing a test sample set and an unlabeled sample pool; randomly selecting a batch of samples from the unlabeled sample pool for expert annotation as an initial training sample set; building a single-stage optical remote sensing target detection model; designing an active learning sampling strategy; training the single-stage optical remote sensing target detection model using the training sample set to obtain an initial optical remote sensing target detection model; testing the optical remote sensing target detection model using the test sample set to determine whether the model has reached the training termination condition; if not, sampling from the unlabeled sample pool according to the active learning sampling strategy, adding the samples to the training sample set after expert annotation, and continuing iteratively training the optical remote sensing target detection model until the training termination condition is reached. This method can effectively reduce sample annotation costs while maintaining the model's detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and remote sensing image processing, and particularly relates to an optical remote sensing target detection method based on active learning. BACKGROUND

[0002] Ship is an important sea transport tool equipment, and is also a key target and attack object in sea military activities. Intelligent target detection and identification of ship through optical remote sensing image has the significant characteristics of wide detection range, long dynamic monitoring time and strong timeliness, and assists sea traffic control, search and rescue, enhances key sea area monitoring and target accurate attack, and has important application value in civil and military fields.

[0003] With the rapid development of high-resolution optical remote sensing satellites, the number of optical remote sensing ship images increases day by day, providing a large number of sample image datasets for optical remote sensing ship target detection. The optical remote sensing ship target detection method can be divided into a traditional target detection method and a deep learning-based method.

[0004] The traditional remote sensing ship target detection method uses artificial definition of target features or feature extraction for target detection. The feature diversity of this method is limited, which leads to weak robustness of the model, low accuracy in complex scenes, and certain degree of limitation of the application of this method.

[0005] In recent years, as a method of automatically learning feature representation directly from data, deep learning technology has made significant progress and breakthroughs in the field of remote sensing ship target detection and identification. Deep learning target detection algorithms are divided into single-stage target detection models and two-stage target detection models according to whether a region proposal network is needed. For remote sensing ship target detection, it is divided into rotating frame target detection and horizontal frame target detection. Compared with horizontal frame, rotating frame target detection has better recognition accuracy and recall rate for densely arranged ship targets in the port. SSD (Single Shot MultiBox Detector) is a horizontal frame single-stage target detection model, and R3Det (Refined Single-Stage Detector) is a rotating frame single-stage target detection model. MMDetection is an open source target detection framework based on Pytorch for target detection tasks. The processes of dataset construction, model building and training strategy are encapsulated into modules, and a new algorithm is obtained through module calling with a small amount of code, which greatly improves the code reuse rate.

[0006] Deep learning is adopted to detect remote sensing ship targets, wherein a deep convolutional neural model usually contains million-level parameters. Under a traditional supervised learning framework, a model must be trained on a given labeled training sample set, and the information value of all training samples is considered equal. Therefore, in order to make the model obtain good performance, large-scale manual labeling of sample image data is required to construct a training sample set as the input of the model to train the model.

[0007] However, in the training process of a deep learning optical remote sensing ship target detection model, the remote sensing image data has a large number of unlabeled samples and the ship target labeling is relatively professional, which requires the assistance of expert knowledge to accurately label. However, there are few professional interpreters, and the time and energy are limited. The daily intelligence production task of the massive remote sensing images downloaded from the satellite every day is heavy, which has far exceeded the work load of the ground interpreters, and it is difficult to perform large-scale labeling in a short time. Moreover, long-time interpretation leads to fatigue, which causes the decline of interpretation quality. Therefore, it is urgent to solve the problem of "reducing labeling cost" as much as possible under the premise that the model reaches the target performance. SUMMARY

[0008] To solve the technical problems existing in the prior art, the purpose of the present application is to provide an optical remote sensing target detection method based on active learning, which effectively reduces the sample labeling cost while maintaining the detection performance of the remote sensing target detection model.

[0009] To achieve the above-mentioned application purpose, the technical scheme of the present application is:

[0010] The present application provides an optical remote sensing target detection method based on active learning, comprising:

[0011] Obtaining a sample set of optical remote sensing target images, constructing a test sample set and an unlabeled sample pool, randomly selecting a batch of samples in the unlabeled sample pool for expert labeling, and taking them as an initial training sample set;

[0012] Based on the MMDetection target detection framework, a single-stage optical remote sensing target detection model is built;

[0013] An active learning sampling strategy of the single-stage optical remote sensing target detection model is designed;

[0014] The training sample set is used to train the single-stage optical remote sensing target detection model to obtain an initial optical remote sensing target detection model;

[0015] The test sample set is used to test the optical remote sensing target detection model to determine whether the optical remote sensing target detection model reaches the termination condition of training;

[0016] If not, the unlabeled sample pool is sampled according to the active learning sampling strategy, the expert-labeled samples are added to the training sample set, and the optical remote sensing target detection model is iteratively trained until the termination condition of training is reached.

[0017] According to an aspect of the present application, the initial training sample set contains all sample categories, and each sample category contains at least one sample.

[0018] According to an aspect of the present application, the single-stage optical remote sensing target detection model is a horizontal box single-stage target detection model or a rotated box single-stage target detection model.

[0019] According to an aspect of the present application, the active learning sampling strategy of the single-stage optical remote sensing target detection model includes: in the sampling process of unlabeled sample images, the calculation of the position uncertainty of the prediction box, the class uncertainty of the prediction box and the uncertainty of the unlabeled sample image, and the sorting criteria of the uncertainty of the unlabeled sample image.

[0020] According to an aspect of the present application, the process of designing the active learning sampling strategy of the single-stage optical remote sensing target detection model includes:

[0021] The single-stage optical remote sensing target detection model is randomly initialized N times, N initialized single-stage optical remote sensing target detection models are generated using a deep ensemble method, and all unlabeled sample images are input to obtain the target class and position information of the candidate box of the unlabeled sample, and the index value of the candidate box is recorded according to the anchor point;

[0022] According to the index value of the candidate box, the target class probability distribution and the position probability distribution of the candidate box are averaged, and the non-maximum suppression is performed on the averaged candidate box to generate a prediction box;

[0023] According to the index value of each prediction box, the corresponding candidate box is found to obtain the probability distribution of the prediction box, the position uncertainty of the prediction box is obtained by calculating the position variance of the candidate box, and the class uncertainty of the prediction box is obtained by calculating the information entropy of the target class probability output by the N single-stage optical remote sensing target detection models.

[0024] The class uncertainty and the position uncertainty of all prediction boxes of each unlabeled sample image are averaged to obtain the uncertainty of each unlabeled sample image.

[0025] According to an aspect of the present application, the position uncertainty of the prediction box includes: horizontal box target detection position uncertainty and rotated box target detection position uncertainty,

[0026] The calculation formula of the horizontal box target detection position uncertainty is:

[0027]

[0028] wherein P bj (O j ) represents the probability that the prediction box is a background category, and the coefficient assigns a higher weight to the positioning uncertainty of the foreground category target; represents the covariance matrix of the upper left corner of the bounding box, represents the covariance matrix of the right lower corner of the bounding box, size(O j ) is the mean of the Euclidean distance between the upper left corner and the right lower corner, for normalization;

[0029] The calculation formula of the position uncertainty of the rotating box target detection is as follows:

[0030]

[0031] wherein N represents the number of single-stage optical remote sensing target detection models, and a1, a2, a3, a4 and a5 represent weight parameters, and the five parameters (x, y, w, h, θ) represent the center point coordinates, length, width and rotation angle of the rotating bounding box, respectively.

[0032] According to an aspect of the present application, the algorithm of the category uncertainty of the prediction box is as follows:

[0033]

[0034] wherein, represents the classification uncertainty only on the foreground category, and is re-normalized after excluding the background probability; the coefficient 1-P bj (O j ) represents that a higher weight is assigned to the uncertainty of the prediction box being a foreground category; the greater the information entropy, the higher the category uncertainty of the prediction box.

[0035] According to an aspect of the present application, the algorithm of the uncertainty of each unlabeled sample image is as follows:

[0036] U(I)=μU loc +I cl

[0037] wherein μ is a weight coefficient, the greater μ, the greater the role of the position uncertainty in the active learning process, and vice versa, the smaller μ, the greater the role of the category uncertainty in the active learning process; the position uncertainty and the category uncertainty of each unlabeled sample image are both weighted and summed, and are respectively and

[0038] According to one aspect of the present application, the process of sampling the unlabeled sample pool according to the active learning sampling strategy and adding the labeled samples into the training sample set by experts includes:

[0039] According to the active learning sampling strategy, the uncertainty of the unlabeled sample image is evaluated and sorted, a part of samples with high uncertainty are selected and put into the labeled sample set, the corresponding samples are deleted from the unlabeled sample pool, and are added to the training sample set after being labeled by experts.

[0040] According to one aspect of the present application, the termination condition is that the unlabeled sample coverage rate is 50% and the map accuracy reaches 95% of the nominal value.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] According to the scheme of the present application, the uncertainty sampling strategy in active learning and the deep learning optical remote sensing target detection strategy are unified into the same framework for model training, which helps the target detection model to deeply mine large-scale unlabeled data, and through screening a small amount of typical samples with more information for labeling, the sample labeling cost is effectively reduced while the detection performance of the optical remote sensing target detection model is maintained.

[0043] According to one aspect of the present application, a prediction box class and position uncertainty sampling algorithm is proposed for horizontal box target detection and rotating box target detection. According to the characteristics of the single-stage deep learning target detection algorithm, a depth integration method is used to generate N initial optical remote sensing target detection models, all unlabeled sample images are input into the N initial optical remote sensing target detection models to obtain the target class and position information of the unlabeled sample candidate boxes. The index value of the candidate box is established through the anchor point box before NMS of the single-stage target detection model, and the mapping relationship between the prediction box after NMS and the candidate box is obtained by using the index value. Finally, the probability distribution of the prediction box is obtained through the target class and position information of the candidate box output by the N models, and then the position and class uncertainty of the prediction box are obtained. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0045] Figure 1 A flowchart schematically showing an optical remote sensing target detection method based on active learning provided by an embodiment of the present application;

[0046] Figure 2 A network architecture diagram schematically representing an optical remote sensing target detection framework based on active learning provided by an embodiment of the present application;

[0047] Figure 3 A schematic diagram schematically representing a rotated bounding box and its five parameters provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The description of the embodiments of the present application should be combined with the corresponding drawings, which should be considered as a part of the complete description of the present application. In the drawings, the shape or thickness of the embodiments can be exaggerated and simplified or convenient for illustration. Moreover, parts of the structures in the drawings will be described separately, and it should be noted that the elements not shown or not described by words in the drawings are in the form known by those skilled in the art.

[0049] The description of the embodiments herein, any reference to direction and position, is only for the convenience of description, and cannot be understood as any limitation on the scope of protection of the present application. The following description of the preferred embodiments will involve combinations of features, which can exist independently or in combination, and the present application is not particularly limited to the preferred embodiments. The scope of the present application is defined by the claims.

[0050] As shown in Figure 1 and Figure 2 The present embodiment discloses an optical remote sensing target detection method based on active learning, for example, detecting a ship target. The specific process of the method includes the following steps:

[0051] Step 100, obtaining a sample set of optical remote sensing ship target images, constructing a test sample set and an unlabeled sample pool, randomly selecting a batch of samples in the unlabeled sample pool for expert labeling, and taking them as an initial training sample set.

[0052] In one embodiment, the initial training sample set in step 100 contains all sample categories, and each sample category contains at least one sample. The balance of sample categories can be ensured.

[0053] Step 200, based on the MMDetection target detection framework, a single-stage optical remote sensing target detection model is built. Further, the single-stage optical remote sensing target detection model is a horizontal box single-stage target detection model, such as SSD, or a rotated box single-stage target detection model, such as RetinaNet. 3 Det.

[0054] At step 300, an active learning sampling strategy of the single-stage optical remote sensing target detection model is designed. Specifically, the active learning sampling strategy of the single-stage optical remote sensing target detection model includes: in the sampling process of the unlabeled sample image, calculation of position uncertainty of a prediction box, class uncertainty of the prediction box, and uncertainty of the unlabeled sample image, and an uncertainty sorting criterion of the unlabeled sample image.

[0055] In one embodiment, the specific implementation process of designing the active learning sampling strategy of the single-stage optical remote sensing target detection model at step 300 includes:

[0056] First, the single-stage optical remote sensing target detection model is randomly initialized N times, N initialized single-stage optical remote sensing target detection models are generated using a deep ensemble method, and all unlabeled sample images are input to obtain target class and position information D_i of the candidate box of the unlabeled sample. The index value i of the candidate box is recorded according to the anchor box. In this embodiment, N is 8.

[0057] Then, the target class probability distribution and the position probability distribution of the candidate box are averaged according to the index value of the candidate box, and the non-maximum suppression (NMS) is performed on the averaged candidate box to generate a prediction box. The candidate box here is generated by the N single-stage optical remote sensing target detection models.

[0058] Next, the probability distribution of the prediction box is obtained by finding the corresponding candidate box output by the N models according to the index value of each prediction box, the position uncertainty of the prediction box is obtained by calculating the position variance of the candidate box, and the class uncertainty of the prediction box is obtained by calculating the information entropy of the target class probability output by the N single-stage optical remote sensing target detection models.

[0059] Finally, the class uncertainty and the position uncertainty of all prediction boxes of each unlabeled sample image are averaged to obtain the uncertainty of each unlabeled sample image.

[0060] Specifically, the position uncertainty of the prediction box includes: horizontal box target detection position uncertainty and rotating box target detection position uncertainty. The specific calculation process of the prediction box position uncertainty is as follows:

[0061] In the horizontal box target detection, the horizontal boundary box is expressed in the form of (x_min, y_min, x_max, y_max) in pascal_voc, and (x_min, y_min) and (x_max, y_max) represent the position coordinates of the upper left corner and the lower right corner of the boundary box respectively. The covariance matrix is calculated according to the probability distribution of the boundary box of N sampling points, and the trace of the matrix represents the variance in the horizontal direction and the vertical direction. The variance represents the degree of uncertainty, that is, the greater the variance, the more uncertain N models are for the prediction in the mean range; the smaller the variance, the higher the confidence of N models for the position of the mean. The calculation formula of the position uncertainty of the horizontal box target detection is:

[0062]

[0063] wherein P bj (O j ) represents the probability that the prediction box is a background category, and the coefficient assigns a higher weight to the positioning uncertainty of the foreground category target; represents the covariance matrix of the upper left corner of the boundary box, represents the covariance matrix of the lower right corner of the boundary box, and size(O j ) is the mean of the Euclidean distance between the upper left corner and the lower right corner, which is used for normalization.

[0064] As Figure 3 shown, in the rotated box target detection, the rotated boundary box is expressed in five parameters (x, y, w, h, θ), and the calculation formula of the position uncertainty of the rotated box target detection is:

[0065]

[0066] wherein N represents the number of single-stage optical remote sensing target detection models, α1, α2, α3, α4 and α5 represent weight parameters, and the five parameters (x, y, w, h, θ) represent the center point coordinates, length, width and rotation angle of the rotated boundary box respectively. The core idea here is to calculate the variance of different prediction results, normalize them, and then multiply them by the weight sum.

[0067] Specifically, the specific calculation process of the class uncertainty of the prediction box is as follows:

[0068] The class uncertainty of the prediction box adopts the strategy of information entropy uncertainty, and the classification information entropy of each prediction box is calculated according to the probability distribution of all classes of the prediction box output by N models, which is:

[0069]

[0070] The greater the information entropy, the higher the class uncertainty of the prediction box.

[0071] For each unlabeled sample image, the maximum value of information entropy of all prediction boxes is taken as the evaluation score of the image. The image with the highest evaluation score is finally selected as the sample that needs to request labeling. The algorithm of the class uncertainty of the prediction box is:

[0072]

[0073] wherein, represents the classification uncertainty only on the foreground class, and is re-normalized after excluding the background probability; the coefficient 1-P bj (O j ) represents assigning a higher weight to the uncertainty that makes the prediction box a foreground class.

[0074] Specifically, the specific calculation process of the uncertainty of the unlabeled sample image is as follows:

[0075] The uncertainty of each unlabeled sample image is obtained by the comprehensive calculation of the class uncertainty and the position uncertainty of all prediction boxes thereof. In order to reduce the influence of the accumulation of multiple small uncertainties, the position uncertainty and the class uncertainty of each unlabeled sample image are both weighted summation. Wherein, the position uncertainty and the class uncertainty are respectively:

[0076]

[0077] At this point, the algorithm of the uncertainty of each unlabeled sample image is:

[0078] U(I)=μU loc +U cl

[0079] Wherein, μ is a weight coefficient, the greater μ is, the greater the role of the position uncertainty in the active learning process is; on the contrary, the smaller μ is, the greater the role of the class uncertainty in the active learning process is.

[0080] Step 400, training the single-stage optical remote sensing target detection model by using the training sample set to obtain an initial optical remote sensing target detection model.

[0081] Step 500, testing the optical remote sensing target detection model by using the test sample set to determine whether the optical remote sensing target detection model reaches the termination condition of training.

[0082] Specifically, the termination condition of model training in step 500 is that 50% of the unlabeled sample coverage rate and 95% of the nominal value of the map accuracy rate are reached. In order to balance the cost of labeled samples and the detection performance of the model, the training termination condition is set.

[0083] If not reached, the unlabeled sample pool is sampled according to the active learning sampling strategy, the expert-labeled training sample set is added, the optical remote sensing target detection model is iteratively trained until the termination condition of training is reached.

[0084] In one embodiment, the specific implementation process of sampling the unlabeled sample pool according to the active learning sampling strategy in step 600 and adding the expert-labeled training sample set includes:

[0085] According to the active learning sampling strategy, the uncertainty of the unlabeled sample image calculated according to the step 300 is evaluated and sorted according to the uncertainty of all unlabeled sample images, a part of samples with higher uncertainty is selected according to the active learning sampling strategy, for example, the top K samples are selected and put into the labeled sample set, the corresponding samples are deleted from the unlabeled sample pool and added to the training sample set after being labeled by an expert. The training sample set is used for the next round of training of the optical remote sensing target detection model, and the above process is iterated until the training termination condition of the model is met.

[0086] In summary, the above-mentioned optical remote sensing target detection method is proposed by incorporating the idea of active learning, which helps the target detection model to deeply mine large-scale unlabeled data, and by selecting a small number of typical samples with more information for labeling, the accuracy of the optical remote sensing target detection model is maintained while the cost of sample labeling is effectively reduced.

[0087] The serial numbers of the above-mentioned steps of the method of the present application do not mean the order of execution of the method, the execution order of each step should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0088] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An optical remote sensing target detection method based on active learning, comprising: obtaining a sample set of optical remote sensing target images, constructing a test sample set and an unlabeled sample pool, randomly selecting a batch of samples in the unlabeled sample pool for expert annotation, and taking them as an initial training sample set; based on the MMDetection target detection framework, a single-stage optical remote sensing target detection model is built; an active learning sampling strategy for the single-stage optical remote sensing target detection model is designed; the training sample set is used to train the single-stage optical remote sensing target detection model to obtain an initial optical remote sensing target detection model; the test sample set is used to test the optical remote sensing target detection model to determine whether the optical remote sensing target detection model meets the termination condition of training; if not, the unlabeled sample pool is sampled according to the active learning sampling strategy, the samples are added to the training sample set after expert annotation, and the optical remote sensing target detection model is iteratively trained until the termination condition of training is met; wherein the single-stage optical remote sensing target detection model is a horizontal box single-stage target detection model or a rotated box single-stage target detection model; the active learning sampling strategy for the single-stage optical remote sensing target detection model includes the calculation of the position uncertainty of the prediction box, the class uncertainty of the prediction box and the uncertainty of the unlabeled sample image in the sampling process of the unlabeled sample image, and the sorting criteria of the uncertainty of the unlabeled sample image; the process of designing the active learning sampling strategy for the single-stage optical remote sensing target detection model includes: N times of random initialization are performed on the single-stage optical remote sensing target detection model, N initialized single-stage optical remote sensing target detection models are generated using a deep ensemble method, and all unlabeled sample images are input to obtain the target class and position information of the candidate box of the unlabeled sample, and the index value of the candidate box is recorded according to the anchor point; the mean value of the target class probability distribution and the position probability distribution of the candidate box is calculated according to the index value of the candidate box, and the non-maximum suppression is performed on the mean value of the candidate box to generate a prediction box; the probability distribution of the prediction box is obtained by finding the corresponding candidate box according to the index value of each prediction box, the position uncertainty of the prediction box is obtained by calculating the position variance of the candidate box, and the class uncertainty of the prediction box is obtained by calculating the information entropy of the target class probability output by the N single-stage optical remote sensing target detection models; the uncertainty of each unlabeled sample image is obtained by calculating the mean value of the class uncertainty and the position uncertainty of all prediction boxes of each unlabeled sample image; the sampling of the unlabeled sample pool according to the active learning sampling strategy and the addition of the training sample set after expert annotation specifically include: the uncertainty of the unlabeled sample image is calculated according to the active learning sampling strategy, all unlabeled sample images are evaluated and sorted according to the uncertainty, Top K samples are selected according to the active learning sampling strategy and put into the labeled sample set, the corresponding samples are deleted from the unlabeled sample pool, and are added to the training sample set after expert annotation; The termination condition is that 50% of the unlabeled sample coverage and the map accuracy reaches 95% of the nominal value.

2. The method of claim 1, wherein, The initial training sample set contains all sample categories, and each sample category contains at least one sample.

3. The method of claim 1, wherein, The position uncertainty of the prediction box includes a horizontal box target detection position uncertainty and a rotating box target detection position uncertainty, The calculation formula of the horizontal box target detection position uncertainty is where P bj (O j ) denotes the probability that the prediction box is a background class, and this coefficient assigns a higher weight to the localization uncertainty of the foreground class target; denotes the covariance matrix of the left upper corner of the bounding box, denotes the covariance matrix of the right lower corner of the bounding box, and size(O j ) is the mean of the Euclidean distance between the left upper corner and the right lower corner, used for normalization. The calculation formula of the rotating box target detection position uncertainty is Wherein, N represents the number of single-stage optical remote sensing target detection models, α1, α2, α3, α4 and α5 represent weight parameters, and the five parameters (x, y, w, h, θ) represent the center point coordinates, length, width and rotation angle of the rotating bounding box.

4. The method of claim 3, wherein, The category uncertainty of the prediction box is calculated according to the following algorithm: where, represents classification uncertainty only on foreground classes, renormalized after excluding background probabilities; coefficient 1-P bj (O j ) represents assigning higher weights to the uncertainty degree that makes the prediction box a foreground class; the greater the information entropy, the higher the class uncertainty of the prediction box.

5. The method of claim 4, wherein, The uncertainty of each unlabeled sample image is calculated according to the following algorithm: U(I) = μU loc + U cl Wherein, μ is a weight coefficient, the greater μ is, the greater the role of the position uncertainty in the active learning process represents, and vice versa μ is smaller, the greater the role of the class uncertainty in the active learning process represents; the position uncertainty and the class uncertainty of each unlabeled sample image are both weighted summation, respectively and and The position uncertainty and the class uncertainty of each target prediction box in the unlabeled sample image are respectively 6. The method of claim 1, wherein, The process of sampling the unlabeled sample pool according to the active learning sampling strategy and adding the training sample set by expert labeling includes: According to the active learning sampling strategy, the uncertainty of the unlabeled sample image is evaluated and sorted, a part of the samples with high uncertainty are selected and put into the labeled sample set, the corresponding samples are deleted from the unlabeled sample pool, and are added to the training sample set after being labeled by experts.

Images