An abnormal weather-based road scene semantic segmentation method based on category grouping

Abstract

The application discloses a kind of abnormal weather under the method for grouping based on road scene semantic segmentation of category, steps are as follows: (1) preparation data;(2) the importance of road scene category is grouped according to the safety of abnormal weather under automatic driving, guarantee important category and non-important category are separated, according to the grouping result, construct road scene semantic segmentation model;(21) the importance of category is grouped according to the safety of abnormal weather under automatic driving;(22) according to the model of category grouping result;(3) input data into model, obtain segmentation result;(31) input data, extract data features;(32) obtain full category features;(33) use full category features to encode category relationship, obtain category relationship features;(34) complete category relationship feature decoding, obtain segmentation result;(4) model iterative training;(5) model test.The application can guarantee the segmentation effect of important grouping under abnormal weather, maximum degree guarantee the safety of automatic driving task is carried out.Meanwhile, under the condition that abnormal degree is smaller, the overall segmentation effect equivalent to normal model can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision image segmentation, and particularly relates to a road scene semantic segmentation method based on category grouping under abnormal weather. BACKGROUND

[0002] For the automatic driving task, most of the existing road scene semantic segmentation methods are based on the research under normal weather conditions, without considering the more difficult and more common abnormal weather conditions. Moreover, due to the difficulty of data collection and labeling under abnormal weather, various related data sets are lacking. Therefore, the general semantic segmentation model can only process the scene under normal weather conditions, and it is difficult to play a role under abnormal weather conditions, the segmentation effect is very poor, and it cannot be applied to the road scene semantic segmentation task under abnormal weather. Considering that under abnormal weather, the most important thing for the automatic driving task is to ensure driving safety, and some categories in the road scene semantic segmentation task are irrelevant to the safety of the automatic driving task, and relatively, some categories must ensure a certain segmentation effect. Therefore, under abnormal weather conditions, the precision of irrelevant categories can be discarded, and the segmentation effect of more important categories for safe driving is ensured.

[0003] In recent years, some methods have emerged to solve various problems under abnormal weather conditions. First, in order to solve the problem that the road scene dataset is mainly collected under normal weather conditions, not only various methods of generating abnormal weather data have emerged, but also multiple real scene datasets under abnormal weather conditions have been proposed. For example, synthetic datasets of rain and fog, BDD100k dataset and ACDC dataset. The emergence of these methods and datasets has to some extent solved the problem of data missing, but the problem of insufficient labeled data due to difficult data labeling still needs to be solved, and this problem becomes more serious as the abnormality of weather conditions increases. In the field of computer vision, in order to solve the problem of insufficient labeled data, some methods have emerged: semi-supervised learning, weakly supervised learning, and transfer learning. Based on mature semantic segmentation models, using the above various methods, some excellent methods for solving road scene semantic segmentation under abnormal weather conditions using limited data are proposed: a new method for road scene segmentation under abnormal weather conditions using self-supervised learning under abnormal weather conditions, a semi-supervised method for solving the problem of semantic ambiguity scene understanding based on synthetic data under abnormal weather conditions. Secondly, how to deal with the problem that the normal semantic segmentation model has poor segmentation effect under abnormal weather conditions. A new curriculum model adaptation method under abnormal weather conditions is proposed, which gradually adapts the semantic segmentation model from synthetic fog to dense real fog in multiple steps, and gradually improves the model by training in multiple stages, using a model with low abnormality to predict data with high abnormality. These methods have to some extent solved the problem of road scene under partial abnormal weather conditions, but there are still problems: all classes are considered equally, without considering the importance of classes to the task. The above various methods do not consider the importance of classes to the task when segmenting, and it is impossible to guarantee the segmentation effect of all classes under abnormal weather conditions, and it is also meaningless to guarantee the segmentation accuracy of some irrelevant classes. SUMMARY

[0004] The present application is directed to the problem of road scene semantic segmentation under abnormal weather conditions, and proposes a new method based on class grouping under abnormal weather conditions to guarantee the segmentation effect of important classes.

[0005] The technical solution for achieving the object of the present application is: a road scene semantic segmentation method based on class grouping under abnormal weather conditions, comprising the following steps:

[0006] 1) Prepare data, including simulated dataset under abnormal weather conditions and real scene dataset, for training and testing of the method;

[0007] 2) Group road scene classes according to their importance to the safety of autonomous driving under abnormal weather conditions, to ensure that important classes are separated from non-important classes, and to build a road scene semantic segmentation model based on the grouping results;

[0008] 3) input data into the model to obtain the grouping segmentation result;

[0009] 4) train the model, input the training set data of the data set into the constructed model, train the model through the loss function, and obtain the trained model;

[0010] 5) test the model, input the test set data of the data set into the trained model, and test the overall effect of the model.

[0011] The step 1) is specifically implemented as follows:

[0012] The rain and fog generation method is used to generate rainy and foggy data on the Cityscapes data set, and real scene data sets under two abnormal weather conditions, ACDC data set and BDD100K data set, are obtained. The four data sets are used for model training and testing.

[0013] The step 2) groups the categories according to the importance of the safety of autonomous driving under abnormal weather; the specific implementation is as follows:

[0014] 21) Group the categories of the road scene data set according to the importance of the safety of autonomous driving under abnormal weather, and use the categories of the Cityscapes data set as the standard. 19 categories are divided into four groups according to the importance: (1) road, person, car; (2) rider, truck, bus, train, motorcycle, bicycle; (3) sidewalk, pole, traffic light, traffic sign; (4) building, wall, fence, vegetation, terrain, sky;

[0015] 22) Construct the model according to the category grouping result; the specific implementation is as follows:

[0016] According to the grouping result, a road scene semantic segmentation model is constructed, which includes a feature extraction module constructed using dilated ResNet50, a category relationship encoding module constructed using covariance channel attention, and a multi-head decoder module constructed using convolutional layers and up-sampling layers according to the category grouping situation.

[0017] The step 3) inputs data into the model to obtain the segmentation result, including the following steps:

[0018] (31) input simulated data and real scene data under abnormal weather, and extract data features;

[0019] (32) obtain full-category features;

[0020] (33) Use full-category feature encoding to obtain category relationship features;

[0021] (34) Complete the category relationship feature decoding and obtain the segmentation result.

[0022] Step 31) is implemented as follows:

[0023] Features F∈R of the input data are extracted using an inflated ResNet50 network pre-trained on the ImageNet dataset. C×h×w Where C represents the number of channels of the feature, and h and w represent the length and width of the feature map.

[0024] Step 32) is implemented as follows:

[0025] For the extracted features F∈R C×h×w Using a 1×1 convolution operation, we obtain the full-class feature F with the same number of channels as the total number of classes. N During model training, the F-values ​​are evaluated using the true labels of the data. N Supervised training is performed to ensure that all class features possess valid information from all classes. The loss function formula is expressed as:

[0026]

[0027] Where T represents the total number of pixels, Y i P represents the true label. i This represents the predicted value for all categories.

[0028] Step 33) is implemented as follows:

[0029] In the full category feature F N Using covariance channel attention to encode class relationships, firstly in F... N Two original features F1∈R are generated by using two 1×1 convolutions. W×H×N ,F2∈R W×H×N Then, the covariance matrix Cov in both directions is obtained using the image covariance formula. W and Cov H After standardization, the covariance attention R in both directions is obtained. i,j W and R i,j H Multiply by the two original features F1 and F2, and add the full-class feature F N Obtain the category relationship feature F that encodes category relationship information. C .

[0030] Step 34) is implemented as follows:

[0031] The multi-head decoder is configured by using a convolutional layer and an up-sampling layer to decode the category relation feature F C to obtain segmentation results of each group; the segmentation results of different groups are represented as P i ∈R w×h×k , i ∈ [1, 4], and the real label information is represented as Y i ∈R w ×h×1 , i ∈ [1, 4], the model is learned and optimized by using a weighted cross-entropy loss, and a specific loss function formula is as follows:

[0032]

[0033] where T represents the total number of pixel points, h and w represent the length and width of the feature map, η i represents the weight of the i-th group.

[0034] The step 4) is specifically implemented as follows:

[0035] During training of the model, the data of the training set of the abnormal weather data set is used, the image is input into the model to obtain the segmentation results of different category groups, the model is learned and optimized according to the defined loss function, and the parameters of the model are updated; during training, the real label needs to be processed according to the category grouping, the categories belonging to the group in the real label are reserved, and the categories of other groups are set to-1, to obtain the real label data of each group, then the importance is weighted for training, and the total loss includes two parts: the feature loss of the whole category and the final segmentation loss, and the specific formula is as follows:

[0036] The step 5) is specifically implemented as follows:

[0037] The data set test set data is input into the trained model to obtain the segmentation results of each group and the overall segmentation result obtained by the combination method, the average intersection over union of each segmentation result is calculated by using the real label data, and the effects of each group segmentation and overall segmentation are obtained.

[0038] Compared with the prior art, the present application has the following advantages: 1) the present application proposes a new idea of class grouping, mainly considering that different classes are different in importance to the automatic driving task, so more attention should be paid to important classes. Under abnormal weather conditions, it is more meaningful to ensure the segmentation effect of important classes because data is more difficult to process. Good segmentation effect of important classes can maximize the safety of the automatic driving task. The specific implementation is to group classes according to importance, then decode and segment each group through an independent decoder to obtain the respective segmentation result. In order to emphasize that different groups have different importance, a weighted loss function is used to give more weight to important class groups during training, so as to ensure that the training is more biased towards important groups, thereby ensuring the effect of important classes; 2) the present application proposes a class relationship encoding module, which encodes the class relationship into the feature, so that the relationship is integrated into the feature in advance during the grouping and segmentation task, so that the class relationship information is encoded into the feature, which improves the result of the model to some extent. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is an abnormal weather-based road scene semantic segmentation method flowchart based on class grouping.

[0040] Figure 2 is a display of abnormal weather data.

[0041] Figure 3 is the class grouping used by the present application.

[0042] Figure 4 is the overall structure diagram of the model of the present application.

[0043] Figure 5 is the feature extraction module of the present application.

[0044] Figure 6 is the class relationship encoding module of the present application.

[0045] Figure 7 is the decoder module of the present application. DETAILED EMBODIMENT

[0046] The overall operation flow of the present application is shown in Figure 1 The present application is described in detail below in conjunction with the drawings in the specification 4:

[0047] Step 1: Prepare data; prepare data under abnormal weather conditions as training set and test set. Use the method of rain and fog simulation generation to generate rain and fog weather data on the Cityscapes dataset, and obtain two real scene datasets under abnormal weather conditions: BDD100K dataset and ACDC dataset. Figure 2Some data is shown, and the generated data set includes 4 intensity of rainy day data and 4 different visibility of foggy day data.

[0048] Step 2: Group the road scene categories according to the importance of autonomous driving safety under abnormal weather, ensure that important categories are separated from non-important categories, and build a road scene semantic segmentation model according to the grouping results;

[0049] Step 21: Group the road scene categories according to the importance of autonomous driving safety under abnormal weather; take the categories of the Cityscapes data set as the standard, 19 categories are divided into 4 groups according to the importance, and the corresponding importance coefficients are set, and the specific grouping situation is as shown in Figure 3

[0050] Step 22: According to the category grouping situation, build a road scene semantic segmentation model, and the overall model architecture is as shown in Figure 4

[0051] Step 3: Input data into the model to obtain the segmentation result;

[0052] Step 31: Use the dilated ResNet50 network pre-trained on the ImageNet data set to extract the features F of the input data, F ∈ R C×h×w , where C represents the number of channels of the features, generally 2048, h and w represent the length and width of the feature map, and the specific feature extraction network structure is as shown in Figure 5

[0053] Step 32: Use a 1x1 convolution operation on the extracted feature F ∈ R C×h×w to obtain the full-class feature F N , which has the same number of channels as the total number of categories. Through the real label of the data, the F N is supervised training to ensure that the full-class feature has effective information of all categories. The commonly used cross-entropy loss is used to complete the supervised learning and optimization of the full-class feature, and the loss function formula is represented as:

[0054]

[0055] , where T represents the total number of pixels, Y i represents the real label, and P i represents the predicted value of the full-class.

[0056] Step 33: Encode the category relationship on the full-class feature F N , and the category relationship module is as shown in​​​Figure 6 As shown, the specific operation is as follows: first, in the full-class feature F N Two 1x1 convolutions are used above to generate two original features F1∈R W×H×N ,F2∈R W ×H×N , for solving the image covariance matrix in two directions, where N is the number of full classes, W, H are the length and width of the feature respectively. Then fix the channel dimension of F1, F2, decompose F1 by row and F2 by column, calculate the covariance matrix, and finally average to get the covariance attention in two directions. After decomposition, F1 can be expressed as F1=(P 1 ,P 2 ,P 3 ,...,P W ), and F2 is expressed as F2=(Q 1 ,Q 2 ,Q 3 ,...,Q H ), where P i ∈R H×N ,Q i ∈R W×N . According to the definition of the covariance matrix, the two image covariance matrices can be expressed as:

[0057]

[0058]

[0059] Where Cov W ∈R N×N ,Cov H ∈R N×N . After obtaining the covariance attention in two directions, standardization is performed, and the value is mapped to (0, 1) through the softmax function:

[0060]

[0061]

[0062] R i,j W , R i,j H represents the correlation coefficient of the i-th class and the j-th class, that is, the attention matrix representing the class relationship. Multiplying the original features F1, F2 with the class relationship attention matrix can obtain the feature fused with the class relationship. Finally, the features fused with the class relationship in two directions are simply added to obtain the class relationship feature F C :

[0063]

[0064] wherein represents F C The jth category feature in the middle is mapped through the channel covariance matrix, and γ is a trainable scalar. The obtained feature F C , called the category relationship feature, models the relationship of the whole category.

[0065] Step 34: complete the decoding of the category relationship feature F C using the decoder to obtain the segmentation result. According to the grouping result, a multi-head decoder module is constructed using a convolutional layer and an upsampling layer to decode the category relationship feature F C to obtain a high-quality segmentation result. The specific decoder structure is shown in Figure 7 . The segmentation results of different groups are represented as P i ∈R w×h×k , i ∈ [1, 4], and the true label information is represented as Y i ∈R w×h×1 , i ∈ [1, 4], and a weighted cross-entropy loss is used to learn and optimize the model. According to the importance of different groups, different degrees of weighting are performed on different groups during loss calculation to ensure that the training is biased towards groups with high importance, so that the segmentation effect of groups with high importance is better. The specific formula of the loss function is as follows:

[0066]

[0067] where T represents the total number of pixels, h and w represent the length and width of the feature map, and η i represents the weight of the ith group.

[0068] Step 4: train the model; when training the model, use the data in the training set of the abnormal weather dataset to input the image into the model to obtain the segmentation result of different category groups, learn and optimize according to the defined loss function, and update the parameters of the model. When training, the true label needs to be processed according to the category grouping, by retaining the class of the true label belonging to the group, and setting the other group class to -, obtaining the true label data of each group, and then weighting according to the importance. The total loss includes two parts: the whole category feature loss and the final segmentation loss, and the specific formula is:

[0069] Step 5: test the model; input the prepared dataset test set into the trained model to obtain the segmentation result of each group, and then use the combination method to obtain the overall segmentation result. Calculate the average intersection over union of each segmentation result through the true label data to obtain the segmentation effect.

Claims

1.A method for semantic segmentation of road scenes based on category grouping under abnormal weather, characterized in that The method comprises the following steps: 1) preparing data, including a simulated data set under abnormal weather and a real scene data set, for training and testing of the method; 2) grouping road scene categories according to their importance to the safety of autonomous driving under abnormal weather, ensuring that important categories are separated from non-important categories, and constructing a road scene semantic segmentation model based on the grouping results; 3) inputting data into the model to obtain grouping segmentation results; The method comprises the following steps: (31) inputting simulated data under abnormal weather and real scene data, and extracting data features; step 31) is specifically implemented as follows: The features F e R of the input data are extracted using a pre-trained dilated ResNet50 network on the ImageNet dataset C ×h×w where C denotes the number of channels of the features, h, w denote the height and width of the feature map. (32) obtaining full-category features; Step 32) is specifically implemented as follows: For the extracted features F∈R C×h×w Using a 1×1 convolution operation, we obtain the full-class feature F with the same number of channels as the total number of classes. N During model training, the F-values ​​are evaluated using the true labels of the data. N Supervised training is performed to ensure that all class features possess valid information from all classes. The loss function formula is expressed as: where T represents the total number of pixel points, Y i represents the true label, P i represents the predicted value of the whole class; (33) using full-category feature encoding category relationship to obtain category relationship features; Step 33) is specifically implemented as follows: In the full-class feature F N The category relationship is encoded using covariance channel attention. First, two 1x1 convolutions are used to generate two original features F1, F2 N w×h×N w×h×N Then, the covariance matrices Cov w and Cov h in two directions are obtained through the image covariance formula, and the normalized covariance attentions R i,j and R w are obtained. The category relationship feature F i,j h is obtained by multiplying the two original features F1, F2 with the two normalized covariance attentions R N and adding the full-class feature F C .​ (34) completing category relationship feature decoding to obtain segmentation results; Step 34) is specifically implemented as follows: A multi-head decoder composed of convolutional layers and up-sampling layers decodes the category relationship features F C , to obtain segmentation results of each group; the segmentation results of different groups are represented as P i ∈R w×h×k , i∈[1,4], k represents the number of categories contained in each group; the real label information is represented as Y i ∈R w×h×1 , i∈[1,4], the model is learned and optimized by using a weighted cross-entropy loss, and a specific loss function formula is as follows: where T denotes the total number of pixels, h, w denote the length and width of the feature map, η i denotes the weight of the i-th group. 4) training the model, inputting the training set data of the data set into the constructed model, training the model through a loss function, and obtaining a trained model; The step 4) is specifically implemented as follows: When training the model, the data of the abnormal weather data training set is used, the image is input into the model, the segmentation result of different category grouping is obtained, learning and optimization are performed according to the defined loss function, and the parameters of the model are updated; during training, the real label needs to be processed according to the category grouping, the class belonging to the grouping in the real label is reserved, and the other grouping classes are set to-1, the real label data of each grouping is obtained, and then weighted training is performed according to the importance, and the total loss includes two parts: the feature loss of the whole category and the final segmentation loss, and the specific formula is: 5) testing the model, inputting the test set data of the data set into the trained model, and testing the overall effect of the model. 2.The category-group-based road scene semantic segmentation method according to claim 1, characterized in that, The step 1) is specifically implemented as follows: Rain and fog generation methods are used to generate rain and fog data on the Cityscapes data set, and real scene data sets under two abnormal weather conditions are obtained: ACDC data set and BDD100K data set. These four data sets are used for training and testing of the model. 3.The category-group-based road scene semantic segmentation method according to claim 1, characterized in that, The step 2) groups categories according to their importance to the safety of autonomous driving under abnormal weather; the specific implementation is as follows: 21) grouping the categories of the road scene data set according to their importance to the safety of autonomous driving under abnormal weather, using the categories of the Cityscapes data set as the standard, and grouping the 19 categories into four groups according to their importance: (1) road, person, car; (2) rider, truck, bus, train, motorcycle, bicycle; (3) sidewalk, pole, traffic light, traffic sign; (4) building, wall, fence, vegetation, terrain, sky; 22) constructing a model according to the category grouping results; the specific implementation is as follows: A road scene semantic segmentation model is constructed according to the grouping results. The model includes a feature extraction module constructed using an expanded ResNet50, a category relationship encoding module constructed using a covariance channel attention, and a multi-head decoder module constructed using convolutional layers and up-sampling layers according to the category grouping. 4.The method of claim 1, wherein, The step 5) is specifically implemented as follows: Input the data test set data into the trained model to obtain the segmentation results of each group and the overall segmentation result obtained by the combination method. Calculate the average intersection over union of each segmentation result through the real label data to obtain the effect of each group segmentation and overall segmentation.

Images