Object detection method for small sample low-quality images based on multi-resolution ensemble self-training

Abstract

The present invention relates to a small sample low-quality image target detection method based on multi-definition integrated self-training, which belongs to the field of image processing technology. First, the target detection model is preliminarily trained using labeled data, and then the training model is used to predict the unlabeled data, and then the predicted data is added to the original data, and the target detection model is retrained together, and so on, and the final detection model is obtained by continuous iteration and updating; at the same time, each time the new model is used to predict the five labeled data, a multi-definition integrated self-training method is adopted. The method of the present invention effectively combines labeled low-quality image data and unlabeled low-quality image data, thereby improving the accuracy of small sample low-quality image target detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a small sample low-quality image target detection method based on multi-definition integrated self-training. Background Art

[0002] With the advancement of science and technology and the rapid development of digital information technology, digital equipment products are not only widely used in all walks of life, but also become an indispensable part of people's daily life. From the middle of the last century to the present, computer image target detection technology has shown a vigorous vitality and has been widely used in aerospace, terrain exploration and traffic monitoring. In the era of high-speed development of information, especially the popularization of electronic products such as digital cameras and handheld cameras, target detection technology is used in all aspects of life.

[0003] In practical applications, computer image imaging conditions are diverse, and the quality of the images obtained is often poor. For example, images obtained in extreme weather such as rain, snow, and fog have reduced contrast, blurred details, and severely reduced quality, which greatly limits the subsequent application of computer vision, especially in outdoor navigation, traffic monitoring, and target recognition. Video images on the Internet are often poor in clarity and have serious information loss after frequent copying, transmission, and format conversion. Slight movement of the shooting equipment causes image jitter and blurred pictures. For example, in night environments, due to insufficient light, a single light source, and the influence of shooting equipment, the collected images generally have the characteristics of low contrast, high noise, and color distortion, which greatly reduces the utilization rate of the images. In summary, low-quality images refer to unclear images caused by poor imaging conditions of the shooting scene or unstable shooting equipment. Research on target detection problems under reduced image quality is of great significance in the field of computer vision and practical applications.

[0004] On the other hand, compared with target detection in traditional scenarios, it is often more difficult to obtain low-quality scene images in extreme weather conditions, and the cost of manually annotating data is often high. Therefore, it is very necessary to study a method for target detection with very small low-quality image annotation data. Summary of the invention

[0005] Technical issues to be solved

[0006] In order to avoid the shortcomings of the prior art, the present invention proposes a small sample low-quality image target detection method based on multi-definition integrated self-training. Using the detection framework proposed by the present invention, data can be fully utilized for low-quality image target detection tasks under limited small sample annotated data and computing resources, and it can be adapted to different target detection models, taking into account the balance between efficiency and detection accuracy.

[0007] Technical Solution

[0008] A method for object detection in small sample low-quality images based on multi-resolution integrated self-training, characterized by the following steps:

[0009] Step 1: Assume that the input image data is a labeled data pair (X1, Y1) and unlabeled data X2. First, use the labeled data to perform preliminary training on the target detection model Faster-R-CNN. The optimization goal is:

[0010] MIN Loss(Y1,F1(X1))

[0011] After obtaining the first training model F1, use it to predict the unlabeled data, namely:

[0012] Y2=F1(X2)

[0013] Step 2: Then add (X2, Y2) as labeled data to the original data to obtain the enhanced data set D1 = (X1, Y1, X2, Y2), use D1 to retrain the target detection model Faster-R-CNN to obtain F2, and then use F2 to predict better labeling information for X2 to obtain Y3;

[0014] Step 3: Similarly, continuously perform iterative updates to obtain the final detection model F n ;

[0015] Step 4: Use the final detection model F n The image data to be detected is detected to obtain the final image target.

[0016] A further technical solution of the present invention is as follows: after each training, the latest model F is obtained, and the unlabeled data is predicted using it: first, the original low-quality foggy picture is clarified using the dark channel defogging model, and defogging pictures I1I2…I with different clarity are generated by controlling its window parameters. k, where k represents a total of k kinds of clarity. Then, the pictures of the above k kinds of clarity are respectively input to F for prediction, which will generate k groups of (x, y, w, h, c) five-tuple prediction results, where the first four numbers predict the position, and the last number c predicts the confidence of the current category. When integrating the k groups of five-tuple prediction results, according to the size of the confidence c: when c is greater than the given threshold of 0.8, the current prediction result is retained in the final result set; when c is less than the given threshold of 0.3, the current prediction result is added to the set to be manually corrected; for the remaining prediction results, the intra-class fusion is performed according to the size of the intersection-over-union ratio, that is, for any two prediction boxes of the same type whose intersection-over-union ratio is greater than the given threshold of 0.7, the one with the larger c value is retained and added to the final result set; finally, the "wrong" prediction results in the set to be manually corrected generated in the above process are corrected and added to the final result set.

[0017] A further technical solution of the present invention: the number of iterations in step 3 is 6 times.

[0018] Beneficial Effects

[0019] The present invention proposes a small sample low-quality image target detection method based on multi-definition integrated self-training, which is improved based on the self-training method in semi-supervised technology, and adds active learning to specifically utilize the data that is mislabeled during self-training. At the same time, a multi-definition integrated self-training method is proposed for low-quality image scenes to effectively combine labeled low-quality image data and unlabeled low-quality image data, thereby improving the accuracy of small sample low-quality image target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0021] Figure 1 Schematic diagram of the overall structure of the model;

[0022] Figure 2 The main process of the semi-supervised low-quality image target detection algorithm;

[0023] Figure 3 Multi-resolution ensemble self-training method. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The technical solution module of the present invention is described from two aspects: the overall structure of the model and multi-definition integrated self-training. The overall model structure is built based on the existing two-stage target detection framework Faster-R-CNN. First, the model is trained using the provided labeled data, and then the trained model is used to predict the labels on the unlabeled samples. Finally, these "labeled" data are added to the original training data for retraining, and the final model is obtained for testing.

[0026] 1. Overall structure of the model

[0027] like Figure 1 As shown, the model of the present invention is constructed based on the Faster-R-CNN framework. Assuming that the input data is a labeled data pair (X1, Y1) and unlabeled data X2, the first step is to use the labeled data to perform preliminary training on the model, and the optimization goal is:

[0028] MIN Loss(Y1,F1(X1))

[0029] After obtaining the first training model F1, use it to predict the unlabeled data, namely:

[0030] Y2=F1(X2)

[0031] Then (X2, Y2) is added to the original data as labeled data to obtain the enhanced data set D1 = (X1, Y1, X2, Y2), where the labeling information of Y2 is noisy, and its credibility can be increased by increasing the final confidence threshold. At the same time, in order to improve the prediction quality of the labeled information, the above process can be repeated, and the model can be retrained using D1 to obtain F2, and then F2 can be used to predict better labeling information for X2 to obtain Y3, and so on, and iterative updates are continuously performed until there is a better data labeling. There is no doubt that the increase in the amount of data will bring a greater improvement in the results of the model.

[0032] After the above multiple rounds of iterative training, the final detection model F is obtained. n , the model is used to detect on the test data set without any annotation, and the final low-quality image detection result is obtained. The main steps of the whole algorithm are as follows Figure 2 shown.

[0033] 2. Multi-resolution integrated self-training

[0034] There are two reasons why deep learning technology can be widely used: large amounts of data and advances in hardware technology. Generally speaking, data in deep learning is divided into three different categories: training data, validation data, and test data. Training data needs to be labeled according to the task type, which is very time-consuming. At the same time, in low-quality scenes, it is difficult to obtain sufficient image data due to weather and other reasons. Therefore, for this scene, self-training can be used to increase the amount of labeled data. However, general self-training methods do not take into account the application scenarios of low-quality images and lack effective use of the factor of "image quality". This method is improved on the basis of general self-training combined with low-quality image scenes. Specifically, there are the following two points: First, for low-quality images, it is proposed to use multiple resolutions to jointly predict their labeling information to increase the model's fitting ability for images of different resolutions; second, an integrated method combining active learning and confidence voting is used to fuse the prediction results of low-quality images at different resolutions. Based on the above two points, the present invention proposes a multi-definition integrated self-training strategy as follows: use a dehazing algorithm to dehaze low-quality images, generate images of different clarity levels for predicting labels, and then use active learning to fuse these label information to obtain the final annotation information.

[0035] In the integrated self-training method, when the detector performance is low, samples close to the class boundary may be mislabeled by voting, and samples close to the class center are selected by confidence. Although this can reduce the mislabeling rate, even for samples with high confidence judged by each integrated detector, the labels predicted by the heterogeneous detector for the sample may be inconsistent. In addition, only samples with high confidence are selected in the iterative process and added to the training set of samples. Such samples have a high similarity with the training set samples. Adding these samples may not improve the performance of the algorithm. On the contrary, samples with low confidence have little similarity with the training set samples. Correctly labeling these samples and then adding them to the training set can greatly improve the performance of the classifier. Active learning is to let experts label a small number of unlabeled samples, so as to obtain the correct labeling information of the unlabeled samples. Combining the two, using an integrated self-training algorithm that combines active learning with confidence voting can effectively solve the above problems of integrated self-training.

[0036] Finally, after obtaining relatively accurate data labels, they are added to the original labeled data set and sent to the target detection model Faster-R-CNN for collaborative training. This is equivalent to using two different parts of data (one part is accurately labeled, and the other part is less accurate) to optimize the final model together. Therefore, the final optimization goal can be written as the following expression:

[0037]

[0038] where Yi is the true label, f is the model output, Loss1 and Loss2 are the respective loss functions in the target detection model, n is the number of labeled training data, and the latter part is the loss calculation of unlabeled data, where Y i ′ is a relatively accurate pseudo label obtained through the aforementioned optimized integrated self-training method.

[0039] In order to enable those skilled in the art to better understand the present invention, the present invention is described in detail below in conjunction with specific embodiments.

[0040] The whole method process is divided into three parts: the initial network training phase, the integrated self-training phase, and the testing phase. The overall framework is improved based on the two-stage target detection framework Faster-R-CNN.

[0041] 1) Initial network training phase:

[0042] The present invention uses the Faster-R-CNN model in the original training stage of the network. In order to increase the generalization performance of the model, the model is first pre-trained on the Pascal-VOC and COCO data sets. Since the former has 20 categories and the latter has 80 categories, they cannot be directly merged. Therefore, the data set labels are clarified, and similar labels are merged. Finally, all are classified into 80 categories of COCO. There are 135412 pictures in total, and a total of 12 Epochs are pre-trained. The SGD optimizer with a learning rate of 0.001 and a batch-size of 16 is used. The learning rate is decayed to half of the original in the ninth and eleventh Epochs, respectively, to achieve a better convergence condition; then the trained weights are used as the initialization parameters of the network, and the output categories of the model are adjusted to 5 categories in the RTTS data set, and re-training is performed. Since this training is fine-tuning, only 6 Epochs are trained in total, and the initialization learning rate is 0.0005, and the others are consistent with the previous ones.

[0043] The data used in the initial training stage include two general target detection datasets with larger data volume and the RTTS training dataset with smaller data volume, with a total of 500 images. The training data here has manually annotated information.

[0044] 2) Multi-resolution integrated self-training stage:

[0045] In the multi-resolution integrated self-training stage, low-quality images are first sharpened to obtain images of different resolutions, and then the model trained in the first step is used to annotate the unlabeled data of different resolutions. A total of 4,000 unlabeled images are used. This step uses three models with different confidence threshold scores of 0.3, 0.5, and 0.7. In the final integration, the NMS algorithm is first used to perform non-maximum suppression and de-duplication, and the four coordinates generated by the model under the three parameters are weighted and fused to obtain the final coordinates (x1, y1, x2, y2), and the label y with the highest probability is selected. c As the final category label, we can get y′=(x1,y1,x2,y2)+y c Added to the training set as the final pseudo label.

[0046] The above self-training process requires a total of 6 rounds of iterations. The main purpose is to obtain more realistic annotation information for unlabeled data, increase the effective information content of the data set, and finally use all the data for training to obtain the final test model parameters.

[0047] 3) Testing phase:

[0048] The testing phase is used to finally verify the effectiveness of the above-mentioned multi-resolution integrated self-training method. The parameter settings of this part remain consistent with those of frameworks such as Faster-R-CNN.

[0049] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A small sample low-quality image target detection method based on multi-resolution ensemble self-training, characterized by Here are the steps: Step 1: The input image data is a labeled data pair (X1, Y1) and an unlabeled data X2. First, the labeled data is used to perform preliminary training on the target detection model Faster-R-CNN. The optimization goal is: MIN Loss(Y1,F1(X1)) After obtaining the first training model F1, use it to predict the unlabeled data, namely: Y2=F1(X2) Step 2: Then add (X2, Y2) as labeled data to the original data to obtain the enhanced data set D1 = (X1, Y1, X2, Y2), use D1 to retrain the target detection model Faster-R-CNN to obtain F2, and then use F2 to predict better labeling information for X2 to obtain Y3; Step 3: Continuously perform iterative updates to obtain the final detection model F n ; Step 4: Use the final detection model F n Detect the image data to be detected and obtain the final image target: each training obtains the latest model F i After that, 1≤i≤n, use model F i Predict the unlabeled data, specifically: First, the dark channel defogging model is used to clarify the original low-quality foggy image, and the defogging images with different clarity are generated by controlling its window parameters. k , where k represents a total of k kinds of clarity, and then the above k kinds of clarity images are input to F respectively. i When making predictions, k groups of (x, y, w, h, c) five-tuple prediction results will be generated, where the first four numbers predict the position, and the last number c predicts the confidence of the current category; when the k groups of five-tuple prediction results are integrated, according to the size of the confidence c: when c is greater than the first given threshold, the current prediction result is retained in the final result set; When c is less than the second given threshold, the current prediction result is added to the set to be manually corrected; for the remaining prediction results, intra-class fusion is performed according to the size of the intersection-and-union ratio, that is, for any two prediction boxes in the same category whose intersection-and-union ratio is greater than the third given threshold, the one with the larger c value is retained and added to the final result set; finally, the "wrong" prediction results in the set to be manually corrected generated in the above process are corrected and added to the final result set.

2. According to claim 1, a small sample low-quality image target detection method based on multi-resolution integrated self-training is characterized in that The first given threshold is 0.

8.

3. The method for detecting a target in a small sample low-quality image based on multi-resolution integrated self-training according to claim 1 is characterized in that The second given threshold is 0.

3.

4. The method for detecting a target in a small sample low-quality image based on multi-resolution integrated self-training according to claim 1 is characterized in that The third given threshold is 0.

7.

5. The method for detecting a target in a small sample low-quality image based on multi-resolution integrated self-training according to claim 1 is characterized in that The number of iterations in step 3 is 6.

Images