A road pavement image labeling method based on semi-supervised unbalanced learning technology

Abstract

The application discloses a road pavement image labeling method based on semi-supervised unbalanced learning technology, aiming at the problems of small quantity and great difficulty of road pavement data labeling in the field of unmanned driving, designing a semi-supervised classification model for data with unbalanced class distribution, having the characteristics of easy implementation and high efficiency, and being capable of realizing rapid performance improvement of the model based on a small quantity of labeled samples. Main steps include: based on an image data set with only a small quantity of labels, a benchmark model is trained by using a semi-supervised learning technology, pseudo-label distribution is predicted for unlabeled samples; the benchmark model result is optimized, high-quality pseudo-labels are obtained by updating based on prototype learning, and the benchmark model prediction is balanced by logical adjustment based on class proportion; a reuse model is constructed, and learning based on label distribution is performed, so that the reuse model sufficiently learns the minority class information in the benchmark model prediction label distribution. The final reuse model is used as a machine learning model on a target image data set.

Description

TECHNICAL FIELD

[0001] The application relates to a road surface image labeling method and device based on semi-supervised unbalanced learning technology, and belongs to the technical field of machine learning and the technical field of image classification. BACKGROUND

[0002] Unmanned driving is an important application direction of artificial intelligence, and the recognition and classification of road conditions are important components of the practical application of unmanned driving technology. At present, one of the major challenges of road surface recognition is the small amount and great difficulty of data labeling, which hinders the application of artificial intelligence technology represented by deep learning methods such as VGG and ResNet and the performance improvement of unmanned driving technology.

[0003] In addition, another major challenge of road surface recognition is the serious imbalance of data categories. For example, the data proportion of common dry asphalt pavement is large, and the data proportion of mud ground covered with ice and snow is small, and the data proportion of different road conditions is greatly different. Compared with the category with rich data, the rare category has poor model generalization performance due to lack of training data, and the recognition difficulty is large. Correctly judging the rare road surface condition is very necessary for ensuring the safety of car owners and pedestrians and maintaining the stability of traffic conditions, and only relying on simple semi-supervised learning methods cannot achieve the goal of high-precision road surface image labeling.

[0004] In previous semi-supervised learning research, researchers from Google proposed Fixmatch, Remixmatch and other advanced technologies, which usually assume that the known categories have relatively balanced labeled data. However, in the real road surface scene, the category data is no longer balanced, and directly applying existing typical works to the open road surface scene will result in poor performance or even serious performance decline. SUMMARY

[0005] The application aims to solve the problem that semi-supervised learning in the prior art leads to performance decline in the case of unbalanced road surface labeling data categories, and proposes an image labeling method and device capable of robustly recognizing road surfaces, which alleviates the problems caused by a small amount of labeling and unbalanced data, effectively reduces the computational cost in the training process, and improves the utilization efficiency of labeled data.

[0006] Technical scheme: A road surface image labeling method based on semi-supervised unbalanced learning technology, specifically comprising:

[0007] 1) Construct a road surface image dataset of the target task, hereinafter referred to as the target image dataset, wherein only a small number of image samples (not more than 10% of the total amount of the image dataset) are labeled;

[0008] 2) using a semi-supervised learning technique to establish a benchmark model for the road pavement image dataset, and predicting a pseudo-labeled label distribution for unlabeled samples;

[0009] 3) optimizing the benchmark model result, updating a high-quality pseudo-label based on prototype learning, and adjusting the benchmark model prediction to be more balanced based on the class proportion logic;

[0010] 4) constructing a reuse model, learning based on the label distribution, and making the reuse model learn the minority class information in the benchmark model predicted label distribution. Based on the convolutional network parameters in the benchmark model, the reuse model is initialized, the similarity between the predicted label distribution of the benchmark model and the reuse model is measured, and the reuse model is trained;

[0011] 5) using the final reuse model as a machine learning model on the target image dataset for road pavement image labeling.

[0012] Optionally, in the target image dataset, part of the labeled image samples and unlabeled image samples have the same distribution.

[0013] Optionally, the semi-supervised learning technique includes but is not limited to Fixmatch and other deep learning techniques to obtain the predicted label distribution.

[0014] Optionally, the high-quality pseudo-label obtained based on prototype learning is updated by the following steps:

[0015] 31) using a neural network feature learner to obtain labeled data features;

[0016] 32) calculating the class center using the labeled data features of each class;

[0017] 33) using a neural network feature learner to obtain unlabeled sample features;

[0018] 34) calculating the cosine distance between the unlabeled sample and each class center, and updating the nearest class as the pseudo-label of the unlabeled sample.

[0019] Optionally, the label distribution learning refers to measuring the similarity between the pseudo-labeled label distribution predicted by the benchmark model and the predicted label distribution of the reuse model, calculating the distribution difference loss using KL divergence, and making the reuse model learn the minority class information in the benchmark model predicted label distribution.

[0020] A road pavement image labeling device based on a semi-supervised unbalanced learning technique, the device comprising:

[0021] An acquisition unit is configured to acquire an image dataset of a target task, i.e., a road surface image dataset in the field of unmanned driving, and a small part of sample data in the image dataset is labeled.

[0022] A first selection unit is configured to select an initial image sample set from the image dataset and select a semi-supervised learning technique.

[0023] A first determination unit is configured to train a baseline model based on the initial image sample set by using the semi-supervised learning technique, predict pseudo-labels of unlabeled samples, update and optimize the baseline model based on prototype learning, obtain high-quality pseudo-labels, and balance the prediction results of the baseline model based on the logic of the class ratio, i.e., obtain a prediction label distribution of the baseline model.

[0024] A second selection unit is configured to obtain the prediction label distribution of the baseline model and a feature learner of the baseline model. The prediction label distribution of the baseline model is the prediction result of the baseline model obtained by the first determination unit.

[0025] A second determination unit is configured to train a reuse model. The reuse model is initialized by using the feature learner parameters of the baseline model to train a full connection layer. The prediction label distribution of the baseline model and the reuse model is measured by using similarity, and the reuse model is updated by using the distribution difference.

[0026] A third determination unit takes the final reuse model as a machine learning model on the target image dataset, trains the final machine learning model on the target image dataset according to a machine learning algorithm, and uses the final machine learning model for road surface image labeling.

[0027] The implementation process of the device is the same as the method, and thus will not be described in detail.

[0028] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the computer device implements the road surface image labeling method based on the semi-supervised unbalanced learning technique as described above.

[0029] A computer-readable storage medium stores a computer program for executing the road surface image labeling method based on the semi-supervised unbalanced learning technique as described above.

[0030] Beneficial effects: Compared with the prior art, the road pavement image labeling method and device based on semi-supervised imbalance learning technology provided by the application utilize the internal information of labeling distribution, transfer the labeling distribution of the balanced benchmark model, alleviate the data class imbalance problem and fully learn the minority class information, optimize the training of the target reuse model, and based on the same feature learner, realize the faster improvement of the target task performance. The application effectively improves the performance of different road surface condition classification in the case of lack of sufficient labeled data, for common scenes such as dry or wet smooth pavement, and rare scenes such as water accumulation and extremely uneven road surface, thereby effectively helping the vehicle to more accurately judge the road surface condition in the unmanned driving. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the main algorithm flowchart of the road pavement image labeling method based on semi-supervised imbalance learning technology in the embodiment of the application;

[0032] Figure 2 is the flowchart of generating and screening pseudo-labeling in steps (2) and (3) in the embodiment of the application;

[0033] Figure 3 is the benchmark model training flowchart of steps (2) and (3) in the embodiment of the application;

[0034] Figure 4 is the reuse model training flowchart of step (4) in the embodiment of the application;

[0035] Figure 5 is the component block diagram of the road pavement image labeling device based on semi-supervised imbalance learning technology realized by the embodiment of the application. DETAILED DESCRIPTION

[0036] The application will be further illustrated below in conjunction with specific embodiments, which should be understood as only for illustrating the application and not for limiting the scope of the application. After reading the application, those skilled in the art can make various equivalent modifications of the application, which all fall within the scope defined by the claims attached to the application.

[0037] The embodiment of the application provides a road pavement image labeling method based on semi-supervised imbalance learning technology. The specific steps of the method are as shown in Figure 1 , mainly including:

[0038] (1) Obtain a road surface image dataset of a target task, also referred to as a target image classification dataset. Part of the sample data in the target image dataset is labeled, and no more than 10% of the sample data in the dataset is labeled. In an embodiment of the present application, the target image dataset is a road surface image dataset in the field of unmanned driving. Among the plurality of sample data included in the dataset, there are labeled samples and unlabeled samples. In an embodiment of the present application, the labeled samples are data with known classification results, and the unlabeled samples are data with unknown classification results. In addition, the dataset presents a long-tail class distribution.

[0039] In an embodiment of the present application, the process of obtaining the target image dataset can be performed according to the existing acquisition method, for example, setting an interface dedicated to the target image data to obtain the target dataset.

[0040] (2) Pseudo-label generation and screening. The labeled samples are trained using a residual neural network. In this example, a ResNet50 neural network is used to train the baseline model. First, the labeled samples are randomly cropped and randomly horizontally flipped for weak augmentation, and then input into the model. The labeled image samples after weak augmentation are input into the neural network model to obtain the prediction results. The prediction results and the true labels are calculated by the standard cross-entropy loss, the model is trained, and the input after weak augmentation makes the model more robust. Where N, f θ is the feature learner of the baseline model, F base is the classifier of the baseline model, and the loss of the labeled samples in the baseline model is as follows:

[0041]

[0042] Where x i is the image sample input, y i is the prediction result, l CE is the cross-entropy loss. Cross-entropy is mainly used to determine the closeness of the predicted output and the expected output. The predicted distribution is made close to the labeled distribution through gradient descent, and the parameters of the model are constantly updated. The standard cross-entropy loss is:

[0043]

[0044] The trained baseline model is used to predict the unlabeled samples, and the prediction results are set as the pseudo-labels of the unlabeled samples to fully utilize the information of the unlabeled samples and meet the needs of semi-supervised learning. Pseudo-labels with low confidence cannot be identified as valid. In order to avoid misleading the model training, it is necessary to ensure that the pseudo-labels have high accuracy, so it is necessary to screen the pseudo-labels. Here, the unlabeled dataset is M, the confidence t is usually set to 95%, and the pseudo-label screening formula is as follows:

[0045]

[0046] wherein, if the pseudo-label confidence is greater than t, is 1, otherwise is 0.

[0047] In addition, the pseudo-label is updated again by using the class center, so as to further ensure the accuracy of the pseudo-label. First, the feature of the labeled sample is extracted, and the class center c l is calculated, and the formula is as follows:

[0048]

[0049] wherein, N l is all samples of the l-th class, f θ (x) is the sample feature, and Normalize() is to normalize the feature to eliminate the unit and scale difference between the features. Then, the pseudo-label is updated by sorting the distance between the class center and the unlabeled sample, and the specific steps are as follows:

[0050]

[0051] wherein, the cosine distance (Cosine Distance) cos() is used to measure the distance between the class center and the unlabeled sample, if the nearest class center of the unlabeled sample is the pseudo-label of the unlabeled sample, the pseudo-label is unchanged, otherwise, the pseudo-label result is updated to the nearest class center. argmax() is the variable c l that makes cos() take the maximum value. The closer the unlabeled sample is to the class center, the closer the cosine value is to 1; when the directions of two vectors are completely opposite, the included angle cosine takes the minimum value -1.

[0052] (3) Training of the baseline model. The training of the baseline model is composed of the training of the labeled sample and the unlabeled sample. In the training of the unlabeled sample, the input x is obtained by using weak enhancement (i.e. random cropping and random horizontal flipping of the road surface image), and the pseudo-label is generated by training the baseline model; the input is obtained by using strong enhancement (i.e. random deletion and random selection of the transformed road surface image), and the predicted label is generated by training the baseline model. The prediction result is obtained by training the input after processing by using the weak enhancement and the strong enhancement, and the loss is calculated by using the standard cross-entropy loss, and the model is trained. The loss of the training of the unlabeled sample is as follows:

[0053]

[0054] Here, the consistency regularization is adopted, which plays a role in the robustness of semi-supervised learning. The formula of the training of the baseline model is as follows:

[0055]

[0056] To construct a balanced baseline model, the prediction results are biased towards the minority class by smoothing with class proportion based logistic adjustment. Among them, The smoothed results are converted into a probability distribution ranging from [0, 1] and summing to 1, i.e. the label distribution; π y is the estimate of the class prior probability, τ is the parameter controlling the degree of smoothing, and the prediction results of the baseline model are as follows:

[0057]

[0058] (4) Reuse model training. The reuse model is based on the convolutional network parameters of the baseline model to construct a feature learner f θ , and train a new classifier F reuse , i.e. train the updated neural network feature learner, and initialize and train a new fully connected layer. The prediction label distribution of the baseline model is compared with the prediction label distribution of the reuse model for similarity measurement, and the reuse model is trained and updated.

[0059] The similarity measurement of the strong augmented sample prediction label distribution of the baseline model and the reuse model is performed by using the KL divergence, the distribution difference loss is calculated, and the reuse model is trained. The KL divergence is a measure of the similarity between two probability distributions, and is as follows:

[0060]

[0061] Therefore, the reuse model fully learns the minority class information, thereby alleviating the unbalanced distribution problem. In the reuse model classifier, the training of the labeled data is the same as that of the baseline model, and the main difference is that the unlabeled sample training is based on the similarity measurement of the label distribution, which is as follows:

[0062]

[0063] is the training loss of the reuse model, N is the labeled data, and M is the unlabeled data. Therefore, the training of the entire model is as follows:

[0064]

[0065] is the training loss of the entire model. In the training of the present example, each iteration includes 64 batches, and in the process of parameter learning and updating in each round, there are 500 iterations, and a total of 500 rounds of training.

[0066] (5) The final reuse model is used as a machine learning model on the target image dataset for road surface image labeling. After the learning process ends, the final reuse model is the machine learning model obtained by the application, as follows:

[0067] y = F reuse (f θ (x))

[0068] As Figure 5 shown, the application is used for road surface image labeling device based on semi-supervised imbalance learning technology, specifically comprising:

[0069] An acquisition unit is configured to acquire a target image dataset, i.e., a road surface image dataset in the field of unmanned driving. A small part of sample data in the target image dataset has a label.

[0070] A first selection unit is configured to select an initial image sample set from the image dataset and select a semi-supervised learning technology.

[0071] A first determination unit is configured to train and determine a benchmark model based on the initial image sample set.

[0072] A second selection unit is configured to obtain a balanced prediction label distribution of the benchmark model and a feature learner of the benchmark model.

[0073] A second determination unit is configured to train a reuse model.

[0074] A third determination unit uses the final reuse model as a machine learning model on the target image dataset. According to a machine learning algorithm, a final machine learning model is trained on the target image dataset for road surface image labeling.

[0075] Further, as Figure 5 shown, the first determination unit comprises:

[0076] An acquisition module is configured to determine and select pseudo-labels of unlabeled samples.

[0077] A construction module is configured to construct a training and updating benchmark model.

[0078] The second determination unit comprises:

[0079] An acquisition module is configured to determine the prediction label distribution of the benchmark model and the reuse model.

[0080] A construction module is configured to construct a label distribution similarity measurement classifier and train and update the reuse model through distribution difference.

[0081] The present example is used to solve the problems of strong data annotation dependence, high manual annotation cost and unbalanced data distribution in the classification of road surface conditions in the field of unmanned driving based on deep learning, effectively reducing the resource consumption in the training process and improving the utilization efficiency of the annotated samples. The method and device of the present application are mainly based on the following observations: existing studies have shown that semi-supervised learning can improve performance when the number of annotated samples is small, but this performance improvement is limited because existing semi-supervised learning methods are usually based on class-balanced data, while real-world class-imbalance data can lead to model prediction bias towards the majority class. The present application migrates the annotation distribution of the benchmark model to alleviate the problem of class imbalance and fully learn the minority class information, optimizes the training of the model, and based on the pre-trained feature learner parameters, the target task can quickly obtain performance improvement.

[0082] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0083] Obviously, those skilled in the art should understand that each step of the road surface image annotation method based on semi-supervised imbalance learning technology of the above-mentioned embodiments of the present application or each unit of the road surface image annotation device based on semi-supervised imbalance learning technology can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0084] The present example verifies the effectiveness on the road surface condition classification dataset from Tsinghua University. In the experiment, 27 kinds of road surface conditions are classified. The most numerous class, wet smooth asphalt pavement, has 79404 images, while the least numerous class, severely uneven wet asphalt pavement, has only 2161 images, with an imbalance ratio of 36. Only 10% of the labeled samples are used, and the remaining 90% of the data are learned using only their images without annotation. The model is established. The present application and the benchmark fully supervised learning method (using all data annotation, based on cross-entropy loss for training), the benchmark semi-supervised learning method (also randomly sampling 10% of the data to retain its annotation, based on Fixmatch for training) are compared. The experimental results report the recall rate (mean ± standard deviation) of different models.

[0085] Baseline fully supervised learning method Baseline semi-supervised learning method The method of the invention Experimental results 75.83±0.76 70.12±1.76 76.61±1.21

Claims

1. A road pavement image labeling method based on semi-supervised unbalanced learning technology, characterized in that, Specifically comprising: 1) construct a road surface image dataset of a target task, i.e. a target image dataset, wherein only a small number of image samples are labeled; 2) use a semi-supervised learning technique to establish a benchmark model for the road surface image dataset, and predict a pseudo-labeled label distribution for unlabeled samples; 3) optimize the benchmark model result, update the high-quality pseudo-label based on prototype learning, and make the benchmark model prediction more balanced based on the logic adjustment of the class proportion; The high-quality pseudo-label obtained by updating based on prototype learning has the following steps: 31) obtain the labeled data features by using a neural network feature learner; 32) calculate the class center by using each class of labeled data features; 33) obtain the unlabeled sample features by using a neural network feature learner; 34) calculate the cosine distance between the unlabeled sample and each class center, and update the nearest class as the pseudo-label of the unlabeled sample; The prediction result is biased to the minority class by smoothing with class proportion based logistic adjustment; wherein, The smoothed result is converted into a probability distribution ranging from [0, 1] and summing to 1, i.e. a label distribution; is a feature learner of the benchmark model, is a classifier of the benchmark model, is obtained after strong enhancement of the input sample; is an estimate of the class prior probability, is a parameter controlling the smoothing degree, and the prediction result of the benchmark model is as follows: 4) Construct a reuse model, learn the minority class information in the benchmark model's predicted label distribution based on the annotation distribution learning, initialize the reuse model based on the convolutional network parameters in the benchmark model, measure the similarity between the predicted label distributions of the benchmark model and the reuse model, and train the reuse model; the reuse model constructs a feature learner based on the convolutional network parameters of the benchmark model , and trains a new classifier , that is, trains the feature learner of the updated neural network, and initializes and trains a new fully connected layer; the reuse model is used to predict the unlabeled samples, the prediction result is the pseudo-label of the unlabeled samples, and the pseudo-label is screened; the predicted label distribution is obtained by predicting the strongly augmented samples, and the predicted label distribution of the benchmark model is measured with the predicted label distribution of the reuse model, and the updated reuse model is trained; the similarity measurement calculates the distribution difference loss by using the KL divergence, and trains the model; the KL divergence is a measure of the similarity of two probability distributions, is the sample category, L is the total number of sample categories, and the specific process is as follows: In the reuse model classifier, the training of the labeled data is the same as that of the benchmark model, and the difference is that the training of the unlabeled sample is based on the similarity measurement of the label distribution, which is as follows: a training loss for the model, labeled data, unlabeled data; 5) use the final reuse model as a machine learning model on the target image dataset for road surface image labeling. 2.The road pavement image labeling method based on semi-supervised imbalanced learning technology according to claim 1, wherein, In the target image dataset, the labeled image samples and the unlabeled image samples have the same distribution. 3.The road pavement image labeling method based on semi-supervised imbalanced learning technology according to claim 1, characterized in that, In the updating based on the prototype learning, high-quality pseudo labels are obtained, and label data features are obtained, and class centers are calculated The formula is as follows: wherein, is the first class of all samples, is the sample feature, is the normalization of the feature, is the image sample input, is the prediction result; Then, the pseudo-label is updated by sorting the distance between the class center and the unlabeled sample, and the specific steps are as follows: Wherein, the cosine distance is utilized The distance between the class center and the unlabeled sample is measured, and if the nearest class center of the unlabeled sample is the pseudo-label of the unlabeled sample The pseudo-label is unchanged, otherwise, the pseudo-label result is updated to the nearest class center. is the variable corresponding to the maximum value is the variable corresponding to the maximum value . 4.The road pavement image labeling method based on semi-supervised imbalanced learning technology according to claim 1, wherein, The label distribution learning refers to measuring the similarity of the pseudo-label distribution predicted by the benchmark model and the predicted label distribution of the reuse model, using KL divergence to calculate the distribution difference loss, and making the reuse model fully learn the minority class information in the benchmark model prediction label distribution.

5. An apparatus for performing the road pavement image labeling method based on the semi-supervised imbalanced learning technique of claim 1, characterized in that, The device comprises: An acquisition unit is configured to acquire an image dataset of a target task, i.e. a road surface image dataset in the field of unmanned driving, wherein a small part of sample data in the image dataset is labeled; the small part of sample data being labeled means that no more than 10% of the sample data in the dataset is labeled; A first selection unit is configured to select an initial image sample set from the image dataset and select a semi-supervised learning technique; A first determination unit is configured to train a benchmark model based on the initial image sample set by using the semi-supervised learning technique, predict a pseudo-label of an unlabeled sample, optimize the benchmark model based on prototype learning to obtain a high-quality pseudo-label, and make the benchmark model prediction result more balanced based on the logic adjustment of the class proportion; A second selection unit is configured to obtain a predicted label distribution of the benchmark model and a feature learner of the benchmark model; A second determination unit is configured to train a reuse model; the reuse model is initialized by training a full connection layer based on the feature learner parameters of the benchmark model; the reuse model is updated by training the distribution difference based on the similarity measurement of the predicted label distribution of the benchmark model and the reuse model; A third determination unit uses the final reuse model as a machine learning model on the target image dataset, trains a final machine learning model on the target image dataset according to a machine learning algorithm, and uses the final machine learning model for road surface image labeling.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the road surface image labeling method based on the semi-supervised imbalanced learning technology according to any one of claims 1-4 when executing the computer program.

7. A computer readable storage medium, storing a computer program for executing the road surface image labeling method based on the semi-supervised imbalanced learning technology according to any one of claims 1-4.

Images