Method and device for judging unevenness grade of front road surface based on image recognition and storage medium

Abstract

The invention discloses a method and device for judging the unevenness grade of a front road surface based on image recognition and a storage medium. The method comprises the steps that S11, a road surface image data set of various unevenness grades is acquired; s12, dividing the image data set into a training set, a verification set and a test set; s13, constructing a convolutional neural network (CNN) for road surface image recognition; s14, before the CNN is trained, performing normalization processing on the image data; s15, training a convolutional neural network (CNN) by using a training set in the image data; s20, in the vehicle driving process, front road surface images are collected in real time; and S21, recognizing the pavement image acquired in real time by using a convolutional neural network (CNN), and outputting a pavement unevenness grade. The method collects the image information of the front road surface as the input of the convolutional neural network, judges the unevenness grade of the front road surface in advance, supports the automobile to take corresponding measures in advance, and improves the stability and safety of the automobile.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for determining the roughness level of a road ahead based on image recognition. Background Art

[0002] When a car is driving, the ups and downs of the road surface will cause the car to vibrate vertically. The unevenness of the road surface has a direct impact on the smoothness and stability of the car's driving, as well as the driving comfort.

[0003] Existing methods for determining road roughness levels mainly include: measurement method and reverse analysis method.

[0004] The measurement method requires the use of specific measuring instruments to complete the measurement of road roughness. The measurement cost is high, the measurement efficiency is low, and some measurement methods are more complicated to implement.

[0005] The reverse analysis method is to install acceleration sensors, displacement sensors, etc. on the moving vehicle to obtain the vibration response of the vehicle as the input of the neural network, so as to identify the unevenness level of the road. The reverse analysis method has three main disadvantages: first, the reverse analysis method mostly uses traditional neural networks, which have poor robustness and generalization ability, and the accuracy of judging the unevenness level of the road is insufficient; second, when training the neural network, it is necessary to collect a large amount of sensor data on various unevenness levels of the road on the actual vehicle, or obtain the vehicle vibration response through a large number of simulation experiments as the input of the training set, so it is time-consuming to obtain the neural network training set; the last point is that the reverse analysis method determines the unevenness level based on the vibration response of the vehicle, and cannot play the role of pre-judgment to remind the driver or help the car implement longitudinal or vertical control in advance.

[0006] If the roughness level of the road ahead can be determined in advance during driving, it will help the car to implement acceleration and deceleration operations in advance, and more accurately control the vertical vibration of the vehicle, thereby improving the vehicle's stability, comfort, safety and adaptability to the road. Therefore, determining the roughness level of the road in advance is of great significance to vehicle dynamics control. Summary of the invention

[0007] In order to solve the above problems, the present invention provides a method for determining the roughness level of the road ahead based on image recognition, designs a lightweight feature fusion convolutional neural network, and uses the image information of the road ahead collected by the automobile vision sensor as the convolutional neural network input to determine the roughness level of the road ahead in real time, thereby supporting the automobile to take corresponding measures to cope with different road conditions and improve the stability, comfort and safety of the automobile.

[0008] In order to achieve the above purpose, the present invention provides a method for determining the roughness level of the road ahead based on image recognition, and the specific steps are as follows:

[0009] S11, obtaining a road surface image dataset with various roughness levels;

[0010] S12, dividing the image data set into a training set, a validation set, and a test set according to a preset ratio;

[0011] S13. Build a convolutional neural network (CNN) for road image recognition:

[0012] The network structure of the convolutional neural network CNN includes: the first module contains a convolution layer, a batch normalization (BN) layer, and a LeakyRelu activation function layer;

[0013] Next, there are multiple consecutive feature fusion grouped convolutional downsampling modules, each of which contains two convolutional branches:

[0014] The first branch includes three submodules from top to bottom. The first submodule includes a grouped convolution layer, a BN layer, and a LeakyRelu activation function layer. The second submodule is a zero padding layer. The third submodule includes a grouped convolution layer, a BN layer, and a Leaky Relu activation function layer.

[0015] The second branch includes grouped convolutional layer, BN layer, and Leaky Relu activation function layer;

[0016] After the two branches implement the feature map downsampling operation respectively, the output feature maps of the two branches are spliced ​​along the channel direction through the Concat operation to achieve the fusion of the features of different scales of the two branches;

[0017] Use the ChannelShuffle operation to shuffle the concatenated feature map channels to achieve information interaction between channels.

[0018] The feature fusion group convolution downsampling module not only realizes the fusion of features of different scales, but also reduces a large number of parameters through group convolution, ensuring the real-time determination of road roughness level.

[0019] The number of branches in the feature fusion group convolution downsampling module can be adjusted according to the difficulty of image feature extraction and the demand for multi-scale feature extraction, and more branches with different receptive field sizes can be introduced.

[0020] Next, the second module includes multiple sub-modules, each of which is a combination of a convolutional layer, a batch normalization (BN) layer, and a LeakyRelu activation function layer;

[0021] Next, there is the global average pooling layer and the Softmax classification layer;

[0022] S14. Before training the convolutional neural network CNN, the image data in the training set, validation set, and test set are normalized. The normalization method is as follows:

[0023] y(m,n)=x(m,n) / 255

[0024] Wherein, m=1,2,…,360;n=1,2,…,240;x(m,n) is the pixel value before normalization, and y(m,n) is the pixel value after normalization;

[0025] S15, using the training set to train the convolutional neural network CNN for road image recognition, using the validation set to validate the convolutional neural network CNN after each round of training, and using the test set to test the trained convolutional neural network CNN;

[0026] During the training process, select the appropriate optimization algorithm and loss function, and set the appropriate batch size and initial learning rate.

[0027] S20, during the driving process of the vehicle, real-time collection of images of the road ahead through the on-board visual sensor;

[0028] S21. Use the convolutional neural network (CNN) for road image recognition to recognize the road image collected by the on-board visual sensor in real time and output the road roughness grade.

[0029] Furthermore, the image dataset is divided into training set, validation set, and test set according to the preset ratio of 8:1:1.

[0030] Furthermore, the images of the image dataset are derived from an open source dataset and / or are obtained by photographing actual roads using a camera.

[0031] Furthermore, multiple road data sets or road images collected by real vehicle vision sensors can be used to supplement the training set, so that the network has stronger generalization ability.

[0032] Furthermore, in order to make the network have stronger generalization ability, multiple road data sets or road images collected by real vehicle vision sensors can be used to supplement the training set.

[0033] Furthermore, the image dataset is an image dataset augmented by data augmentation operations such as random horizontal or vertical translation, random rotation, random horizontal or vertical flipping, and random scaling.

[0034] Furthermore, a road surface image dataset with multiple roughness levels is obtained, including road surface image datasets with three roughness levels of A, B, and C, where level A is flat, level B is slightly rough, and level C is severely rough.

[0035] Furthermore, the number of grouped convolution kernels in multiple continuous feature fusion grouped convolution downsampling modules increases successively. If the number of groups is completely or partially different, the network parameter content and calculation complexity can be greatly reduced, thereby ensuring the real-time judgment of the road roughness level.

[0036] The present invention also provides a device for determining the roughness level of a road ahead based on image recognition, comprising:

[0037] An acquisition module is used to acquire a road surface image data set of various roughness levels, and divide the image data set into a training set, a validation set, and a test set according to a preset ratio;

[0038] Convolutional neural network (CNN) construction and training module, used to build a convolutional neural network (CNN) for road image recognition, normalize the image data set, and use the training set to train the convolutional neural network (CNN) for road image recognition;

[0039] A road surface image acquisition module is used to acquire the road surface image ahead in real time while the vehicle is driving;

[0040] The classification and recognition module is used to input the real-time collected front road surface image into the convolutional neural network CNN and output the road surface roughness grade.

[0041] The present invention also provides a computer-readable storage medium, which stores computer instructions. The computer instructions are used to enable a processor to implement the above-mentioned method for determining the roughness level of the road ahead based on image recognition when executed.

[0042] The present invention provides a method for determining the roughness level of the road ahead based on image recognition. The feature fusion group convolution downsampling module randomly selects samples from the majority class for deletion to reduce the number of samples in the majority class and make the data set more balanced. The Concat operation can be used to merge multiple objects together. The ChannelShuffle operation (channel random mixing operation ChannelShuffleOperation can be regarded as "reshape-transpose-reshape") is used to disrupt the spliced ​​feature map channels to achieve information interaction between channels.

[0043] In the feature fusion group convolution downsampling module, branches with different receptive field sizes are introduced, so that different branches have different understandings of the input image, affecting the network perception ability and computational complexity. A smaller receptive field can capture more detailed features, but may ignore information in a larger range; a larger receptive field can capture more global information, but may ignore more detailed features.

[0044] During training, setting the batch size refers to the number of samples selected during each training process in deep learning and machine learning. It directly affects the optimization speed of the model, memory usage, and stability during training.

[0045] Compared with the solutions in the known field, the present invention has the following substantial differences in structure or process:

[0046] (1) The present invention determines the road surface roughness level by identifying images collected by visual sensors. Compared with other methods, the method for obtaining the data set for network training is simpler and more efficient. It can collect road surface image data using non-vehicle cameras, without the need for real vehicle collection or data acquisition through simulation. More training data can be obtained in a shorter time.

[0047] (2) The present invention adopts a front road surface image recognition method based on deep learning to determine the level of road surface roughness. Compared with the method based on traditional neural network to identify vehicle vibration response, the present invention has stronger robustness and generalization ability. At the same time, the vehicle has the ability to predict the road surface roughness level, so as to respond to different road surface environments in advance and have a strong adaptability to changes in road surface roughness.

[0048] (3) The convolutional neural network designed in the present invention has the characteristics of feature fusion and lightweight. The network's ability to extract features of different scales is improved through feature fusion, and the real-time determination of road roughness level is ensured through model lightweighting. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to better understand the above and other purposes, features, advantages and functions of the present invention, reference may be made to the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings refer to the same components. It should be understood by those skilled in the art that the accompanying drawings are intended to schematically illustrate the preferred embodiments of the present invention and have no limiting effect on the scope of the present invention, and the components in the drawings are not drawn to scale.

[0050] Figure 1 A flow chart of a method for determining the roughness level of a road ahead based on image recognition according to the present invention;

[0051] Figure 2 Partial road surface images of road roughness levels A, B, and C;

[0052] Figure 3 Schematic diagram of the convolutional neural network structure of the present invention;

[0053] Figure 4 Schematic diagram of the structure of the feature fusion group convolution downsampling module. DETAILED DESCRIPTION

[0054] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0055] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0056] In order to at least partially solve one or more of the above problems and other potential problems, an embodiment of the present disclosure proposes a method for determining the roughness level of a road ahead based on image recognition.

[0057] like Figure 1 As shown, the specific steps are as follows:

[0058] S11, obtaining a road surface image dataset with various roughness levels;

[0059] This implementation uses the RSCD road image dataset released and maintained by the research team of the School of Vehicle and Transportation Engineering of Tsinghua University. This is the first large-scale road classification dataset RSCD, and the road images contain precise annotations.

[0060] like Figure 2 As shown, the image dataset contains three types of roughness levels: A, B, and C, where level A is flat, level B is slightly uneven, and level C is severely uneven.

[0061] The image dataset was also expanded through data augmentation operations such as random horizontal or vertical translation, random rotation, random horizontal or vertical flipping, random scaling, etc. Finally, 10,000 images of each road roughness level were selected, totaling 30,000 images, and the size of each image was 360*240.

[0062] S12. Divide the image data set into a training set, a validation set, and a test set in a ratio of 8:1:1; the image training set, validation set, and test set of each roughness level contain 8,000, 1,000, and 1,000 images, respectively.

[0063] S13. Build a convolutional neural network (CNN) for road image recognition:

[0064] like Figure 3 As shown, the convolutional neural network CNN includes multiple modules.

[0065] The first module contains a convolution layer with a kernel size of 33 and 16 kernels, a batch normalization (BN) layer, and a LeakyRelu activation function layer;

[0066] Next, there are four consecutive feature fusion grouped convolutional downsampling modules, each of which contains two convolutional branches:

[0067] like Figure 4 As shown, the first branch includes three sub-modules from top to bottom.

[0068] The first submodule includes a grouped convolution layer with a convolution kernel size of 11 and a step size of 1, a BN layer, and a LeakyRelu activation function layer;

[0069] The second submodule is the zero-filling layer;

[0070] The third submodule contains a grouped convolution layer with a kernel size of 33 and a stride of 2, a BN layer, and a LeakyRelu activation function layer.

[0071] The second branch includes, from top to bottom: a grouped convolution layer with a kernel size of 11 and a stride of 2, a BN layer, and a LeakyRelu activation function layer.

[0072] The two branches implement feature map downsampling operations respectively, and the output feature maps of the two branches are spliced ​​along the channel direction through the Concat operation to achieve the fusion of different scale features of the two branches;

[0073] Then, the ChannelShuffle operation is used to shuffle the concatenated feature map channels to achieve information interaction between channels.

[0074] In the four consecutive feature fusion group convolution downsampling modules, the number of group convolution kernels is 16, 32, 64, and 128, and the number of groups is 4, 8, 8, and 8, respectively, which greatly reduces the network parameter content and calculation complexity, thereby ensuring the real-time judgment of the road roughness level.

[0075] After 4 consecutive feature fusion group convolution downsampling modules, there is the second module, which includes 3 submodules, each of which is a combination of convolution layer, batch normalization (BN) layer, and LeakyRelu activation function layer;

[0076] The convolution kernel sizes of the three sub-modules of the second module are 11, 33, and 3*3 respectively, the number of convolution kernels are 64, 3, and 3 respectively, and the step sizes used are 1, 1, and 2 respectively.

[0077] Finally, there is a global average pooling layer and a Softmax output layer.

[0078] S14. Before training the convolutional neural network CNN, the image data in the training set, validation set, and test set are normalized. The normalization method is as follows:

[0079] y(m,n)=x(m,n) / 255

[0080] Among them, m=1,2,…,360;n=1,2,…,240;x(m,n) is the pixel value before normalization, and y(m,n) is the pixel value after normalization.

[0081] S15, using the training set to train the convolutional neural network CNN for road image recognition, using the validation set to validate the convolutional neural network CNN after each round of training, and using the test set to test the trained convolutional neural network CNN;

[0082] The network training uses the Adam optimization algorithm, which is an adaptive learning rate optimization algorithm that combines the characteristics of momentum and adaptive learning rate. The main idea is to dynamically adjust the learning rate of each parameter according to the gradient of the parameter;

[0083] The loss function is the cross-entropy loss function, which measures the difference between the probability distribution predicted by the model and the true probability distribution, and is used to evaluate the accuracy of the model prediction;

[0084] The batch size is set to 100 and the initial learning rate is 0.001.

[0085] The experimental environment uses the Keras deep learning framework, with Tensorflow as the computing backend and GPU for acceleration.

[0086] S20, during the driving process of the vehicle, real-time collection of images of the road ahead through the on-board visual sensor;

[0087] S21. Use the convolutional neural network (CNN) for road image recognition to recognize the road image collected by the on-board visual sensor in real time and output the road roughness grade.

[0088] Furthermore, the images of the image dataset are derived from an open source dataset and / or are obtained by photographing actual roads using a camera.

[0089] It is also possible to use multiple road data sets or road images collected by real vehicle vision sensors to supplement the training set, so that the network has stronger generalization capabilities.

[0090] For example, when using multiple road datasets, the above RSCD and RSRD image data can be combined. RSRD is a high-precision road reconstruction dataset designed specifically for autonomous driving. It contains binocular images, lidar data, and motion information. It is used to improve vehicle driving performance and decision support. It covers a variety of road conditions and provides rich samples and detailed annotations. It can be used for testing and research on multiple computer vision tasks. The combination of the above two datasets can provide more accurate and comprehensive road status information, enabling high-level autonomous driving.

[0091] Advantages of the invention:

[0092] The present invention determines the roughness level of the road ahead in advance by identifying the road ahead image acquired by the vehicle visual sensor. Compared with the determination method based on the vehicle vibration response, the present invention has the prediction capability, provides information for the longitudinal control and vertical control of the vehicle, helps the vehicle to implement corresponding operations in advance in response to different road environments, and enhances the stability, smoothness and comfort of the vehicle.

[0093] The present invention uses a deep learning algorithm to identify the road image ahead, which has stronger robustness and generalization ability than traditional neural networks;

[0094] In order to solve the problem that the key features of road images are not obvious, the convolutional neural network designed in the present invention has the characteristics of feature fusion. By fusing features of different scales, the global features are combined with fine-grained features, the feature extraction capability of the network is enhanced, and the level judgment accuracy is higher.

[0095] In order to ensure the real-time determination of the road surface roughness level, the present invention reduces the computational complexity of the convolutional neural network through a lightweight structure, thereby achieving rapid recognition of road surface images.

[0096] The road surface roughness grade determination method based on image recognition can also be used by highway departments to evaluate road construction quality and performance.

Claims

1. A method for determining the roughness level of a road ahead based on image recognition, characterized in that: The steps include: S11, obtaining a road surface image dataset with various roughness levels; S12, dividing the image data set into a training set, a validation set, and a test set according to a preset ratio; S13. Build a convolutional neural network (CNN) for road image recognition: The network structure of the convolutional neural network CNN includes: the first module contains a convolution layer, a batch normalization (BN) layer, and a LeakyRelu activation function layer; Next, there are multiple consecutive feature fusion grouped convolutional downsampling modules, each of which contains two convolutional branches: The first branch includes three submodules from top to bottom. The first submodule includes a grouped convolution layer, a BN layer, and a LeakyRelu activation function layer. The second submodule is a zero padding layer. The third submodule includes a grouped convolution layer, a BN layer, and a Leaky Relu activation function layer. The second branch includes grouped convolutional layer, BN layer, and Leaky Relu activation function layer; After the two branches implement the feature map downsampling operation respectively, the output feature maps of the two branches are spliced ​​along the channel direction through the Concat operation to achieve the fusion of the features of different scales of the two branches; Use the ChannelShuffle operation to disrupt the spliced ​​feature map channels to achieve information interaction between channels; Next, the second module includes multiple sub-modules, each of which is a combination of a convolutional layer, a batch normalization (BN) layer, and a LeakyRelu activation function layer; Next, there is the global average pooling layer and the Softmax classification layer; S14. Before training the convolutional neural network CNN, the image data in the training set, validation set, and test set are normalized. The normalization method is as follows: y(m,n)=x(m,n) / 255 Wherein, m=1,2,…,360;n=1,2,…,240;x(m,n) is the pixel value before normalization, and y(m,n) is the pixel value after normalization; S15, using the training set to train the convolutional neural network CNN for road image recognition, using the validation set to validate the convolutional neural network CNN after each round of training, and using the test set to test the trained convolutional neural network CNN; During the training process, select the appropriate optimization algorithm and loss function, set the appropriate batch size and initial learning rate; S20, during the driving process of the vehicle, real-time collection of images of the road ahead through the on-board visual sensor; S21. Use the convolutional neural network (CNN) for road image recognition to recognize the road image collected by the on-board visual sensor in real time and output the road roughness grade.

2. A method for determining the roughness level of a road ahead based on image recognition as claimed in claim 1, characterized in that: The image dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

3. The method for determining the roughness level of the road ahead based on image recognition as claimed in claim 1, characterized in that: The images in the image data set are derived from an open source data set and / or are obtained by photographing an actual road surface using a camera.

4. The method for determining the roughness level of the road ahead based on image recognition according to claim 1, characterized in that: Use multiple road data sets or road images collected by real vehicle vision sensors to supplement the training set.

5. The method for determining the roughness level of the road ahead based on image recognition according to claim 1, characterized in that: Use multiple road data sets or road images collected by real vehicle vision sensors to supplement the training set.

6. The method for determining the roughness level of the road ahead based on image recognition as claimed in claim 1, characterized in that: The image data set is an image data set expanded by data enhancement operations such as random horizontal or vertical translation, random rotation, random horizontal or vertical flipping, and random scaling.

7. The method for determining the roughness level of the road ahead based on image recognition as claimed in claim 1, characterized in that: The image data set includes a road surface image data set with three roughness levels of A, B, and C, wherein level A is flat, level B is slightly uneven, and level C is severely uneven.

8. The method for determining the roughness level of the road ahead based on image recognition as claimed in claim 1, characterized in that: The number of grouped convolution kernels in the multiple consecutive feature fusion grouped convolution downsampling modules increases successively.

9. A device for determining the roughness level of the road ahead based on image recognition, characterized in that: include: An acquisition module is used to acquire a road surface image data set of various roughness levels, and divide the image data set into a training set, a validation set, and a test set according to a preset ratio; Convolutional neural network (CNN) construction and training module, used to build a convolutional neural network (CNN) for road image recognition, normalize the image data set, and use the training set to train the convolutional neural network (CNN) for road image recognition; A road surface image acquisition module is used to acquire the road surface image ahead in real time while the vehicle is driving; The classification and recognition module is used to input the real-time collected front road surface image into the convolutional neural network CNN and output the road surface roughness grade.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for determining the roughness level of a road surface ahead based on image recognition according to any one of claims 1 to 8 when executed.

Images