Road weather identification method based on lightweight convolutional neural network

Abstract

The invention discloses a road weather recognition method based on a lightweight convolutional neural network, and the method comprises the following steps: S1, building a road weather data set of a training network model, and dividing the road weather data set into a training set and a verification set; s2, establishing a three-channel composite convolutional neural network and training the three-channel composite convolutional neural network; s3, establishing a lightweight convolutional neural network; s4, in combination with the trained three-channel composite convolutional neural network, training the lightweight convolutional neural network; s5, performing weather identification by using the trained lightweight convolutional neural network; according to the method, knowledge learned by the three-channel composite convolutional neural network with stronger learning ability is migrated to the lightweight convolutional neural network with relatively weak learning ability, so that the generalization ability of the lightweight convolutional neural network is improved, the recognition precision of the lightweight convolutional neural network is improved, and the lightweight convolutional neural network is easily deployed at a mobile terminal with limited resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of weather recognition, and in particular relates to a road weather recognition method based on a lightweight convolutional neural network. Background Art

[0002] Bad weather is one of the important factors that affect road driving safety. For example, rainy days will make the road slippery and the friction coefficient of the road surface will decrease, and foggy days will cause road visibility to decrease, all of which will cause potential traffic accidents. Therefore, it is crucial to study the technology of real-time identification of weather conditions around the vehicle. This technology can be applied to assisted driving systems. When bad weather conditions are detected, it can assist the driver to limit the speed, turn on the wipers or defogger lights, and reduce the possibility of traffic accidents.

[0003] In recent years, deep learning technology has developed rapidly, and technicians have begun to apply it widely in the field of weather recognition. Common weather recognition network models include AlexNet, VGG16, ResNet101, ConvNeXt, etc., which can achieve good results, but the model parameters are generally large. For example, the parameter size of VGG16 is as high as 527M, which is difficult to apply in mobile and embedded devices. In order to solve the problem of large model parameters and difficulty in deploying to mobile terminals, lightweight CNN models have been proposed at home and abroad in recent years, such as the MobileNet and MnasNet series proposed by Google, the SqueezeNet series proposed by DeepScale, and the ShuffleNet series proposed by Megvii Technology. These models are lightweight in terms of volume and speed to adapt to devices with limited resources, thus achieving good results. However, lightweight CNN improves the efficiency of the model by reducing the model size and computational complexity, which is bound to lead to the model accuracy not being as good as large models such as VGG and ResNet. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a road weather recognition method based on a lightweight convolutional neural network, which transfers the knowledge learned by the three-channel composite convolutional neural network with stronger learning ability to the lightweight convolutional neural network with relatively weaker learning ability. It not only retains the high-precision advantages of the large model, but also retains the lightweight characteristics of the small model, solving the problems that the large model is difficult to apply on the mobile terminal and the small model is insufficient in accuracy.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] A road weather recognition method based on a lightweight convolutional neural network comprises the following steps:

[0007] S1. Establish a road weather dataset for training the network model and divide it into a training set and a validation set;

[0008] S2. Establish and train a three-channel composite convolutional neural network, wherein the three-channel composite convolutional neural network includes a trunk network and two branch networks, wherein the trunk network input is a global road weather picture, wherein one of the branch networks inputs a road weather picture of the upper half of the sky area, and the other branch network inputs a road weather picture of the lower half of the ground area, and splices the output feature maps of the trunk network and the two branch networks according to the channel dimension to achieve three-channel composite, and then passes the spliced ​​feature maps through the classification layer to output the weather image classification recognition result;

[0009] S3. Establish a lightweight convolutional neural network. The lightweight convolutional neural network uses MobileNetV3 as the initial network. On the basis of the initial network, a Hardswish activation function layer, a Dropout regularization layer, and a Linear full connection layer are added in sequence after the last layer.

[0010] S4, combining the trained three-channel composite convolutional neural network to train the lightweight convolutional neural network;

[0011] S5. Use the trained lightweight convolutional neural network for weather recognition;

[0012] Preferably, step S2 includes the following sub-steps:

[0013] S2-1. The backbone network of the three-channel composite convolutional neural network selects the ResNext101_64x4d network model and modifies the network: after the last convolution layer of all residual structures in the third residual layer, add the SE attention mechanism module, and delete the last two global average pooling layers and fully connected layers of the original network; the input of the backbone network is the global road weather picture;

[0014] S2-2. The upper and lower branches of the three-channel composite convolutional neural network both use the ResNet50 network model, and modify the network: after the last convolution layer of all residual structures in the third residual layer, add the SE attention mechanism module, and delete the last two global average pooling layers and fully connected layers of the original network; the input of the upper branch network is the road weather picture of the upper half of the sky area, and the input of the lower branch network is the road weather picture of the lower half of the ground area;

[0015] S2-3, splice the output feature maps of the main network and the upper and lower branch networks according to the channel dimension to achieve three-channel compounding, and then pass the spliced ​​feature maps through the classification layer: global average pooling layer, Linear fully connected layer, Relu activation function layer, Dropout regularization layer, Linear fully connected layer, where the number of neurons in the first fully connected layer is 1000, the number of neurons in the second fully connected layer is 4, and the regularization layer parameter p is set to 0.2; thereby achieving weather image classification and recognition;

[0016] S2-4. According to steps S2-1, S2-2 and S2-3, a three-channel composite convolutional neural network is established, and then the road weather data set established in step S1 is used for training.

[0017] Preferably, step S3 includes the following sub-steps:

[0018] S3-1. Use the PyTorch framework to build a lightweight convolutional neural network;

[0019] S3-2. The lightweight convolutional neural network uses MobileNetV3 as the initial network. On the basis of the initial network, a Hardswish activation function layer, a Dropout regularization layer, and a Linear fully connected layer are added in sequence after the last layer, wherein the parameter p of the regularization layer is set to 0.2, and the number of neurons in the fully connected layer is set to 4.

[0020] Preferably, step S4 includes the following sub-steps:

[0021] S4-1. Calculate the soft loss Lsoft, the calculation formula is: Lsoft = f1(g(student_scores / temp), g(teacher_scores / temp)), where f1 function is the KL divergence loss function, g function is the softmax classifier, student_scores is the prediction output of the lightweight convolutional neural network, teacher_scores is the prediction output of the three-channel composite convolutional neural network, and temp is the softening temperature;

[0022] S4-2. Calculate the hard loss Lhard. The calculation formula is: Lhard = f2(student_scores, labels), where f2 is the cross entropy loss function and labels is the training set label.

[0023] S4-3. Combine the soft loss Lsoft and the hard loss Lhard to calculate the total loss Ld. The calculation formula is: Ld = temp 2*alpha*Lsoft+(1-alpha)*Lhard, where alpha is the weight parameter of Lsoft and Lhard losses;

[0024] S4-4. The gradient of the parameters of each layer of the lightweight convolutional neural network is calculated by the total loss Ld, and then the optimizer updates the parameters to achieve the training purpose.

[0025] The present invention has the following characteristics and beneficial effects:

[0026] By adopting the above technical scheme, the road weather recognition method based on the lightweight convolutional neural network of the present invention transfers the knowledge learned by the three-channel composite convolutional neural network with stronger learning ability to the lightweight convolutional neural network with relatively weaker learning ability, thereby improving the generalization ability of the lightweight convolutional neural network and improving its recognition accuracy. In addition, the lightweight convolutional neural network can be easily deployed on mobile terminals with limited resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 This is a flowchart of a road weather recognition method based on a lightweight convolutional neural network according to an embodiment of the present invention.

[0029] Figure 2 A schematic diagram of the structure of a three-channel composite convolutional neural network for a road weather recognition method based on a lightweight convolutional neural network according to an embodiment of the present invention.

[0030] Figure 3 A schematic diagram of a lightweight convolutional neural network structure of a road weather recognition method based on a lightweight convolutional neural network according to an embodiment of the present invention.

[0031] Figure 4 This is a flow chart of training a lightweight convolutional neural network for a road weather recognition method based on a lightweight convolutional neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0035] The embodiment of the present invention provides a road weather recognition method based on a lightweight convolutional neural network. Figure 1 As shown, the following steps are included:

[0036] S1. Establish a road weather dataset for training the network model and divide it into a training set and a validation set;

[0037] Specifically, in this embodiment, step S1 includes the following sub-steps:

[0038] S1-1, integrate weather images from Baidu Gallery, Sogou Gallery and the public weather dataset MWI;

[0039] S1-2, filtering the weather images integrated in step S1-1, retaining only the weather images with the sky in the upper half and the road in the lower half, and finally obtaining a road weather data set of four categories: sunny, rainy, foggy, and snowy, with 9,000 images for each weather category;

[0040] S1-3. In this embodiment, the road weather data set obtained in step S1-2 is divided into a training data set and a verification data set at a ratio of 7:3;

[0041] S1-4, perform image preprocessing on the training data set, randomly cropping to 224*224 resolution, randomly horizontally inverting, and normalizing operations in sequence, wherein the normalization operation sets the mean and variance of the image R channel to [0.485, 0.229], the mean and variance of the G channel to [0.456, 0.224], and the mean and variance of the B channel to [0.406, 0.225];

[0042] S1-5. Perform image preprocessing on the validation data set, scaling to 256*256 resolution, center cropping to 224*224 resolution, and normalization. The normalization operation sets the mean and variance of the image R channel to [0.485, 0.229], the mean and variance of the G channel to [0.456, 0.224], and the mean and variance of the B channel to [0.406, 0.225].

[0043] S2. Establish and train a three-channel composite convolutional neural network, wherein the three-channel composite convolutional neural network includes a trunk network and two branch networks, wherein the trunk network input is a global road weather picture, wherein one of the branch networks inputs a road weather picture of the upper half of the sky area, and the other branch network inputs a road weather picture of the lower half of the ground area, and splices the output feature maps of the trunk network and the two branch networks according to the channel dimension to achieve three-channel composite, and then passes the spliced ​​feature maps through the classification layer to output the weather image classification recognition result;

[0044] The step S2 comprises the following sub-steps:

[0045] S2-1. The backbone network of the three-channel composite convolutional neural network selects the ResNext101_64x4d network model, which is built using the PyTorch framework and modified: after the last convolution layer of all residual structures in the third residual layer, the SE attention mechanism module is added, and the last two global average pooling layers and fully connected layers of the original network are deleted; wherein, the SE attention mechanism first passes through the global average pooling layer to convert the input feature map of N*C*H*W into a feature vector of N*C*1*1, and then passes through two fully connected layers (the number of neurons in the first fully connected layer is 0.25*C, and the number of neurons in the second fully connected layer is C) to obtain a weight vector of N*C*1*1, and finally multiplies the input feature map by the corresponding value of the channel and the weight vector to obtain an output feature map of N*C*H*W, where N is the batch size batch_size, C is the number of channels of the feature map, H is the height of the feature map, and W is the width of the feature map; the input of the backbone network is a global road weather picture;

[0046] S2-2. The upper and lower branches of the three-channel fusion convolutional neural network both use the ResNet50 network model, which is built using the PyTorch framework, and the network is modified: after the last convolution layer of all residual structures in the third residual layer, the SE attention mechanism module is added, and the last two global average pooling layers and fully connected layers of the original network are deleted; the input of the upper branch network is the road weather picture of the upper half of the sky area, and the input of the lower branch network is the road weather picture of the lower half of the ground area;

[0047] S2-3. According to step S1, the size of the weather image in the training set is 224*224, so the dimensions of the output feature maps of the main network and the upper and lower branch networks are N*2048*7*7. The three output feature maps are spliced ​​according to the channel dimension to obtain N*6144*7*7 to achieve three-channel compounding, and then the spliced ​​feature maps are passed through the classification layer: global average pooling layer, Linear fully connected layer, Relu activation function layer, Dropout regularization layer, Linear fully connected layer, where the number of neurons in the first fully connected layer is 1000, the number of neurons in the second fully connected layer is 4, and the regularization layer parameter p is set to 0.2; thereby achieving weather image classification and recognition;

[0048] S2-4. According to steps S2-1, S2-2 and S2-3, a three-channel composite convolutional neural network is established, and its structure is as follows: Figure 2 As shown; then the data set established in step S1 is used for training. The training method is to delete the last layer of fully connected parameters from the weight parameters obtained by training the ResNext101_64x4d and ResNet50 network models on the ImageNet data set, and then use them as the initialization parameters of the backbone network and the upper and lower branch networks in the teacher network respectively. The remaining parameters of the teacher network use the Kaiming initialization method. During the training process, the total number of training rounds epochs is set to 50, the batch size batch_size is set to 64, the optimizer selects Adma, the loss function loss_function selects the cross entropy loss function, and the learning rate lr is set to 0.0001.

[0049] S3. Establish a lightweight convolutional neural network;

[0050] Use the PyTorch framework to build it. Specifically, its structure is as follows Figure 3 As shown, in this embodiment, the lightweight convolutional neural network uses MobileNetV3 as the initial network. On the basis of the initial network, a Hardswish activation function layer, a Dropout regularization layer, and a Linear fully connected layer are added in sequence after the last layer, where the parameter p of the regularization layer is set to 0.2, and the number of neurons in the fully connected layer is set to 4.

[0051] S4, combining the trained three-channel composite convolutional neural network to train the lightweight convolutional neural network;

[0052] The step S4 comprises the following sub-steps:

[0053] S4-1. Calculate the soft loss Lsoft, the calculation formula is: Lsoft = f1(g(student_scores / temp), g(teacher_scores / temp)), where f1 function is the KL divergence loss function, g function is the softmax classifier, student_scores is the prediction output of the lightweight convolutional neural network, teacher_scores is the prediction output of the three-channel composite convolutional neural network, and temp is the softening temperature;

[0054] S4-2. Calculate the hard loss Lhard. The calculation formula is: Lhard = f2(student_scores, labels), where f2 is the cross entropy loss function and labels is the training set label.

[0055] S4-3. Combine the soft loss Lsoft and the hard loss Lhard to calculate the total loss Ld. The calculation formula is: Ld = temp 2 *alpha*Lsoft+(1-alpha)*Lhard, where alpha is the weight parameter of Lsoft and Lhard losses;

[0056] S4-4 and the weather pictures of the training set are respectively input into the three-channel composite convolutional neural network and the lightweight convolutional neural network, and then according to steps S4-1, S4-2 and S4-3, Lsoft, Lhard and Ld are calculated in turn, and then the gradient of the parameters of each layer of the lightweight convolutional neural network is calculated by Ld, and finally the optimizer updates the parameters of the lightweight convolutional neural network, thereby realizing the training of the lightweight convolutional neural network.

[0057] In this embodiment, the training process is as follows: Figure 4 As shown; during the training process, the distillation temperature temp is set to 5, the weight parameter alpha is set to 0.3, the total number of training epochs is set to 50, the batch size batch_size is set to 64, the optimizer optimizer selects Adma, and the learning rate lr is set to 0.0001.

[0058] S5. Use the trained lightweight convolutional neural network for weather recognition;

[0059] The step S5 comprises the following sub-steps:

[0060] S5-1. Deploy the trained lightweight convolutional neural network to the mobile terminal;

[0061] S5-2. The road weather images collected by the sensor are scaled to a resolution of 256*256, center cropped to a resolution of 224*224, and normalized. The normalization operation sets the mean and variance of the image R channel to [0.485, 0.229], the mean and variance of the G channel to [0.456, 0.224], and the mean and variance of the B channel to [0.406, 0.225]. They are then sent to a lightweight convolutional neural network for weather recognition and judgment.

[0062] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A road weather recognition method based on lightweight convolutional neural network, characterized in that: The following steps are involved: S1. Establish a road weather dataset for training the network model and divide it into a training set and a validation set; S2. Establish and train a three-channel composite convolutional neural network, wherein the three-channel composite convolutional neural network includes a trunk network and two branch networks, wherein the trunk network input is a global road weather picture, wherein one of the branch networks inputs a road weather picture of the upper half of the sky area, and the other branch network inputs a road weather picture of the lower half of the ground area, and splices the output feature maps of the trunk network and the two branch networks according to the channel dimension to achieve three-channel composite, and then passes the spliced ​​feature maps through the classification layer to output the weather image classification recognition result; S3. Establish a lightweight convolutional neural network. The lightweight convolutional neural network uses MobileNetV3 as the initial network. On the basis of the initial network, a Hardswish activation function layer, a Dropout regularization layer, and a Linear full connection layer are added in sequence after the last layer. S4, combining the trained three-channel composite convolutional neural network to train the lightweight convolutional neural network; S5. Use the trained lightweight convolutional neural network for weather recognition.

2. The road weather recognition method based on lightweight convolutional neural network according to claim 1 is characterized in that: The backbone network uses ResNext101_64x4d as the initial network model, adds an SE attention mechanism module after the last convolution layer of all residual structures in the third residual layer, and deletes the last two global average pooling layers and the fully connected layer of the original network; the two branch networks use ResNet50 as the initial network model, add an SE attention mechanism module after the last convolution layer of all residual structures in the third residual layer, and delete the last two global average pooling layers and the fully connected layer of the original network.

3. The road weather recognition method based on lightweight convolutional neural network according to claim 1 is characterized in that: The Dropout regularization layer parameter p is set to 0.2, and the parameter p refers to the probability that each neuron is set to 0; the number of Linear neurons is set to 4.

4. The road weather recognition method based on lightweight convolutional neural network according to claim 3 is characterized in that: The step S4 includes the following sub-steps: S4-1. Calculate the soft loss Lsoft, the calculation formula is: Lsoft = f1(g(student_scores / temp), g(teacher_scores / temp)), where f1 function is the KL divergence loss function, g function is the softmax classifier, student_scores is the prediction output of the lightweight convolutional neural network, teacher_scores is the prediction output of the three-channel composite convolutional neural network, and temp is the softening temperature; S4-2. Calculate the hard loss Lhard. The calculation formula is: Lhard = f2(student_scores, labels), where f2 is the cross entropy loss function and labels is the training set label. S4-3. Combine the soft loss Lsoft and the hard loss Lhard to calculate the total loss Ld. The calculation formula is: Ld = temp 2 *alpha*Lsoft+(1-alpha)*Lhard, where alpha is the weight parameter of Lsoft and Lhard losses; S4-4. The gradient of the parameters of each layer of the lightweight convolutional neural network is calculated by the total loss Ld, and then the optimizer updates the parameters to achieve the training purpose.

Images