Road surface state detection method based on video image and weather data

Abstract

The invention discloses a road surface state detection method based on video images and weather data. The method comprises the following steps: S1, inputting road surface state video data into edge computing equipment to obtain road surface feature data; s2, fusing the weather data with the output data in the step S1, constructing a fusion model, and inputting the fusion model into an edge computing device to obtain a road surface real-time state detection result; s3, the steps S1 and S2 are implemented through detection equipment, and the detection equipment comprises edge computing equipment, road video acquisition equipment and weather data acquisition equipment; and the edge computing device is in communication connection with the road video acquisition device and the weather data acquisition device, and is used for respectively acquiring a road surface image and weather data at an edge end in real time. According to the invention, the data does not pass through the cloud, the road surface does not need to be damaged during detection, the road surface state information can be accurately obtained in real time, and the method is of great significance to traffic safety guarantee.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic detection technology, and more specifically to a road surface state detection method based on video images and weather data. Background Art

[0002] Road pavement detection is mainly based on experience and manual judgment, specifically inspectors or professional road information collection vehicles, but they are either costly or highly subjective, and are not comprehensive, with inevitable time delays. They have been gradually abandoned in today's rapid development of various methods. With the gradual improvement of transportation infrastructure, there are also methods of manually viewing video images or installing various sensors on the road, such as lidar, to identify road conditions. These methods save labor costs compared to before, but the equipment requirements for this method are still high. Considering the cost and benefit, it is only suitable for deployment in some extremely important sections of the road. Today, the rapid development of deep neural networks has provided many possibilities for computer vision processing, making it a reality to judge road conditions through video images.

[0003] The detection method based on video images is to install cameras in certain key sections of the road and input digital images into the computer. Its advantages are powerful functions, intuitive images, and easy upgrades. It can be used independently or built into a detection network. Installation and maintenance have no impact on the road. Many deep learning networks proposed so far, such as InceptionNet, ResNet, and the newer Vision Transformer network, can accurately identify the road surface status. However, the implementation of this method is easily affected by weather and requires relatively powerful equipment to support it. Many methods consider using the cloud to solve related calculations, but this will also increase bandwidth costs and there are data security issues.

[0004] Therefore, a series of lightweight CNN networks were proposed, such as MobileNet, ShuffleNet, EfficentNet, etc. These neural networks are designed to run on the edge. They have the characteristics of low computational complexity, small model architecture, and fast computing speed. However, they are generally less accurate and are more susceptible to weather effects, resulting in poor detection results.

[0005] Therefore, how to provide an edge real-time online road pavement condition detection method that integrates video images and weather data, improve the accuracy and anti-interference ability of the video image detection method, and realize edge real-time online road pavement condition detection is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides a road surface condition detection method based on video images and weather data, which can realize online, edge, and real-time road surface condition detection, overcomes the problems of long lag, non-intuitiveness, low precision, complex and expensive equipment, etc. of other methods, and can quickly and accurately detect a variety of road surface conditions, reducing traffic risks caused by poor road conditions.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A road surface condition detection method based on video images and weather data comprises the following steps:

[0009] S1. Real-time edge road surface status detection based on road video image data;

[0010] Inputting the road surface status video data into the edge computing device to obtain the road surface feature data;

[0011] S2, real-time edge road surface status detection based on fusion data of road video images and weather information;

[0012] The weather data is integrated with the output data in step S1 to construct a fusion model of the road video image and the weather information data, and the fusion model is input into the edge computing device to obtain the real-time road surface status detection result;

[0013] S3, steps S1 and S2 are all implemented through a detection device, which includes an edge computing device, a road video acquisition device, and a weather data acquisition device; the edge computing device and the road video acquisition device are communicatively connected via a CSI / USB interface, and are communicatively connected to the weather data acquisition device via VCC, DATA, and GND pins, so as to respectively obtain road pavement images and weather data in real time at the edge.

[0014] The beneficial effect of the above technical solution is that through the combination of road video and weather information, road surface status information can be obtained in real time and with greater accuracy and anti-interference. For high-risk sections such as tunnels, bridges, long and steep slopes, and sharp bends near water and cliffs, the efficient acquisition of such information can significantly improve the response speed of relevant traffic management measures and the level of traffic safety.

[0015] Preferably, step S1 comprises:

[0016] S11, real-time road video image data acquisition: road video information is collected in real time by a road video acquisition device and stored, and transmitted to an edge computing device;

[0017] S12, video image data preprocessing: extracting images and inputting them into the edge computing device for image preprocessing; divided into three steps:

[0018] Step 1: Use the vector median filtering method to denoise the current image;

[0019] Step 2: Determine the lighting conditions of the image, use the pixel value statistics method, and set the threshold according to the historical image conditions to determine whether the current image is a low-brightness or overexposed image. If it is an image with unsatisfactory lighting, use the S-curve adjustment method to adjust the brightness and contrast;

[0020] Step 3: To facilitate model training and prediction, the image is normalized and standardized. The standardization makes the image mean 0 and variance 1, which conforms to the Gaussian distribution.

[0021] The data from the road video image database or the data input into the edge computing device in real time is preprocessed as above.

[0022] S13, extracting the pre-processed video image data to perform road surface condition detection.

[0023] Preferably, in step S13, the following steps are also included:

[0024] S131. Build a lightweight CNN network RepShuffleNet model. Build a new lightweight CNN model RepShuffleNet based on the lightweight CNN model ShuffleNetV2 and Repconv.

[0025] S132, constructing a historical road video image database and performing model training optimization; using historical data to construct a road pavement video image database, and training and optimizing the RepShuffleNet model;

[0026] S133, loading the trained RepShuffleNet model into the edge computing device to perform real-time feature extraction of road pavement video image data and road pavement status detection.

[0027] The beneficial effect of the above technical solution is that by improving ShuffleNetV2, one of the best lightweight models, and integrating the Repconv structure, the model accuracy becomes higher and the inference time becomes shorter because it uses a multi-branch structure during training and an equivalent single branch during prediction, making the real-time and accuracy of the entire method stronger.

[0028] Preferably, in step S132, video image data of various road types are collected and classified and marked according to different road pavement conditions, each image file and its category are respectively matched and recorded, and a training set and a test set are divided to make a historical road video image database; the RepShuffleNet model is trained using the historical road video image database. By fusing multi-source data, namely weather data and road pavement condition prediction data, and matching and optimizing multiple hyperparameters of machine learning, the accuracy of the model is greatly improved compared to that based only on road video image data.

[0029] Preferably, step S2 comprises:

[0030] S21, weather data collection: obtaining weather information of the current time and location through a weather data collection device, and inputting the collected data into an edge computing device for sorting and storage;

[0031] S22, fusion data construction: fusing the collected real-time weather information and real-time road video image feature data to obtain real-time fusion data;

[0032] S23. Load the fused data obtained in step S22 into the edge computing device to obtain the current road surface status.

[0033] Preferably, step S23 also includes the following steps:

[0034] S231, fusion detection model construction: based on the machine learning model with prediction function, a real-time road surface state detection model based on fusion weather information and road video images is constructed;

[0035] S232, historical fusion data set construction and model training;

[0036] The historical fusion dataset is a historical road video image database that is combined with image feature extraction, detection and weather data through the RepShuffleNet model to form classification labels;

[0037] The hyperparameter optimization method is used to optimize and train the fusion detection model;

[0038] S233, road surface condition detection based on real-time fusion data; loading the trained fusion detection model into the edge computing device to detect the real-time road video image and weather information fusion data to obtain the real-time condition of the road surface.

[0039] Preferably, the fused data in step S22 is obtained by concatenating and merging the output data of step S133 and the weather data of step S21 as two separate lists.

[0040] Preferably, in step S231, the input of the fusion detection model is the fusion data in step S22; the data is input in vector form.

[0041] Preferably, in step S3, the edge computing device includes an edge computing processing module, a communication module, a monitoring and visual module, a data recording and storage module, and a power management module;

[0042] The edge computing processing module is used for data processing and calculation; the communication module is used to provide wireless network connection services and transmit the detection results to the back-end server; the monitoring visual module is used for video signal output and visual display of video analysis and processing; the data recording storage module is used to transmit the detection results to the storage device and load data files; the power management module is used to provide a power source for the detection equipment.

[0043] Preferably, the road video acquisition device acquires the video images of the traffic scene in real time through the camera, and transmits the video images to the edge computing processing module after preliminary processing of the video; the weather data acquisition device is electrically connected to the temperature and humidity sensor, acquires the current temperature and humidity information in real time, organizes and stores it, and transmits it to the edge computing processing module in real time.

[0044] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a road surface condition detection method based on video images and weather data. Data collection is obtained in real time using a module connected to an edge computing device, and an original lightweight CNN network RepShuffleNet obtained by combining ShuffleNet with the improved Repconv structure is used to extract real-time road surface video image features. At the same time, the addition of weather data makes the detection method based on video images more accurate and more resistant to interference.

[0045] Compared with existing road pavement status detection methods, the data of this method does not pass through the cloud, the relevant model is lightweight, and there is no need to damage the road surface. At the same time, the addition of weather data makes the accuracy of this method higher than the general detection method based only on video data. It can accurately and real-time obtain road pavement status information, which is of great significance to the protection of traffic safety. It overcomes the problems of long lag, non-intuitiveness, low accuracy, complex and expensive equipment, etc. of other methods, and therefore has strong application prospects.

[0046] The detection equipment used with this method is suitable for deployment in sections prone to severe weather, sections prone to waterlogging, and sections at high altitudes with changeable weather. It can quickly and accurately detect a variety of road surface conditions such as snow-covered, semi-snow-covered, iced, waterlogged, wet, dry, etc. By quickly and accurately notifying and warning the road maintenance department of poor road surface conditions, rapid road maintenance can be carried out on these special sections, which has important application value for maintaining traffic safety and reducing traffic risks caused by poor road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A road surface condition detection flow chart provided by the present invention;

[0049] Figure 2 This is a schematic diagram of the structure of the detection equipment provided by the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See attached Figures 1-2 The embodiment of the present invention discloses a road surface condition detection method based on video images and weather data, comprising the following steps:

[0052] S1. Real-time edge-end road surface state detection based on road video image data; inputting road surface state video data into an edge computing device to obtain road surface feature data;

[0053] S11, real-time road video image data acquisition: road video information is collected in real time by a road video acquisition device, stored as needed, and transmitted to an edge computing device connected to the camera;

[0054] S12, video image data preprocessing: preprocessing the collected real-time road video, extracting images as needed for training and prediction of subsequent road state recognition modules, reading the images into the edge computing device, and performing image preprocessing, including but not limited to image screening, noise reduction and other operations;

[0055] In order to further optimize the above technical solution, in step S12, the edge computing device obtains the video collected in real time by the road video acquisition device, extracts the frames therein according to actual needs, and performs a series of processing on the image. The preprocessing of the image may include denoising, brightness and contrast adjustment, normalization and standardization, and other image preprocessing methods that can be beneficial to the training and optimization of the deep neural network model.

[0056] In order to further optimize the above technical solution, in step S12, the adapted image data is preprocessed, the edge computing device obtains the video collected in real time by the road video acquisition module, extracts the frames therein, and performs a series of processing on the image. The processing method is as follows:

[0057] First, the image is denoised by using the vector median filter method, with the filter window set to 3×3 and the vector distance within the window using the L1 distance. Specifically,

[0058] 1. A pixel in an image can be represented as a three-dimensional vector:

[0059] I=(R,G,B)

[0060] For a 3×3 filter window, there are 9 such vectors:

[0061] I1, I2, ..., I9

[0062] 2. The distance between two vectors is calculated using L1 distance:

[0063] D(Ii,·Ij)·=·|Ri-Rj|·+·|Gi-Gj|·+·|Bi-Bj|

[0064] In this way, we can calculate the distances between 9 vectors, a total of 36 distance values. Then we find the two vectors with the smallest distance, set as Im and In.

[0065] 3. The final filtering output is the median vector of Im and In

[0066] 4. Perform this operation on each pixel to complete the vector median filtering as described above.

[0067] Secondly, the lighting conditions of the image are determined by using pixel value statistics and setting thresholds based on historical image conditions to determine whether the current image is a low-brightness or overexposed image. If the image is not ideally lit, the S-curve adjustment method is used to adjust the brightness and contrast.

[0068] In this embodiment, selecting video image data of road surface conditions that are close to the prediction target as a database can effectively improve the recognition accuracy of the model for road surface conditions. In addition, denoising, brightness adjustment and normalization operations on graphics can make the model converge more quickly and reduce the impact of bad image data.

[0069] S13, extracting the pre-processed video image data to perform road surface condition detection;

[0070] S131. Build a lightweight CNN network RepShuffleNet model. Build a new lightweight CNN model RepShuffleNet based on the lightweight CNN model ShuffleNetV2 and Repconv.

[0071] In order to further optimize the above technical solution, the RepShuffleNet model in step S131 performs feature extraction of the image, and the model can be run online in real time on the edge computing device, and can basically judge the road surface condition as one of snow-covered, semi-snow-covered, icy, water-logged, wet, and dry through the extracted data features.

[0072] S132, constructing a historical road video image database and performing model training optimization; using historical data to construct a road pavement video image database, and training and optimizing the RepShuffleNet model;

[0073] Collect video image data of various road types, classify and mark them according to different road surface conditions, correspond and record each image file and its category, divide it into training set and test set, and make a historical road video image database; use the historical road video image database to train the RepShuffleNet model;

[0074] Video image data of various road types such as highways, river bridges, overpasses, tunnel entrances, intersections, mountain roads, etc. are collected and photographed on the Internet. They are classified and marked according to categories such as snow-covered, semi-snow-covered, iced, water-logged, wet, and dry. Each image file is matched with its category and recorded, divided into training sets and test sets, and a road video image database is created;

[0075] The real-time road status recognition module based on video images is trained using the historical road video image database. The deep neural network is mainly composed of convolutional layers and fully connected layers, and these layers are composed of convolution kernel parameters. The optimization and training targets are these parameters. The deep neural network uses the Loss function as an indicator for training optimization. The parameters of the Loss function and the model are directly related to the detection accuracy. The detection accuracy is the degree of consistency between the data predicted by the model and the actual result. The more accurate the prediction, the higher the accuracy.

[0076]

[0077] By inputting the data of the model, the parameters are continuously updated according to the direction of the Loss gradient in the model through forward propagation operations and back propagation operations, so as to reduce the Loss value and improve the accuracy.

[0078] In order to further optimize the above technical solution, the RepShuffleNet model training process is as follows:

[0079] 1) The road video image database is read into the lightweight road condition recognition module of the edge computing device in the form of RGB three channels using the Opencv library.

[0080] 2) Preprocess the read image data.

[0081] 3) The parameters of the model have initialization values. During the training process, each neuron in each layer undergoes the following calculation:

[0082] With Z as input, after the activation function operation, we get:

[0083]

[0084] y=δ(Z)

[0085] δ(Z) is the activation function, and the ReLU activation function is used in training.

[0086] ReLU = max(0, x) #(3)

[0087] Between each layer, by treating the first layer input as a 0 , the corresponding weights and biases are w 1 and b 1 , the next neuron to enter is z 1 , after being activated by the activation function, it becomes a 1 . This is another cycle until the last layer outputs

[0088] The specific formula is:

[0089]

[0090] After each cycle, the cross entropy function is used to calculate the loss value:

[0091]

[0092] Through the calculation of the entire neural network, we will get n H functions depending on how many neurons there are in the output layer, where n is the number of neurons in the output layer. Then the total loss function is:

[0093]

[0094] It is concluded that this function starts from the output layer and calculates the gradient of the parameters backward, that is, the partial derivative of cov with respect to the layer before the output layer w, b is calculated by the chain derivation rule. and Then update the parameter weights of this layer

[0095]

[0096] The weights and biases are constantly changing, so that the loss function is continuously reduced, so as to achieve the effect of model training.

[0097] The specific settings for model training are shown in the following table:

[0098] Table model training hyperparameter settings

[0099] Parameter name explain Settings batch_size The number of images to read into the model each time 32 Epochs The number of training rounds (the number of the above cycles) 50 Lr Learning rate, the weight of updating parameters 0.001 Moment Momentum for the SGD optimizer 0.9

[0100] The pre-processed real-time road pavement video image data that needs to be detected can be directly input into the trained RepShuffleNet model to obtain the corresponding feature data.

[0101] S133, loading the trained RepShuffleNet model into the edge computing device to perform real-time feature extraction of road pavement video image data and road pavement status detection;

[0102] In this embodiment, in step S13, a lightweight CNN network is obtained by improving the ShuffleNetV2_0.5 model by combining the Repcov structure, and the model is named RepShuffleNet. The model is installed on an edge computing device and can be run online in real time. It can determine the road surface conditions such as snow-covered, semi-snow-covered, icy, water-logged, wet, dry, etc.

[0103] The construction method of the RepShuffleNet model is:

[0104] The general CNN model continuously abstracts and extracts images through operations such as convolution, pooling, and full connection, and uses activation functions to map the model from linear to nonlinear, so that the model can fit quite complex data. The convolution operation can extract local features and combine these features to form a more advanced feature representation. The pooling layer is used to reduce the spatial dimension of the feature map, reduce computational complexity, and extract the most significant features. The fully connected layer is used to accept features from the convolution and pooling layers and use them for classification, regression, or other tasks. The model converts the input image into a tensor, and finally obtains various possible results through the softmax function. The general CNN model has a large amount of calculation and can only be run on a computer equipped with a discrete graphics card or a CPU of a certain quality.

[0105] ShuffleNetV2 is known as the champion of lightweight CNN networks. It performs particularly well among a number of lightweight CNN networks and is particularly suitable for deployment on lightweight edge devices with only CPUs.

[0106] ShuffleNetV2 improves the arrangement of convolution operations and general CNN operations, and adds some channel operations. Its convolution operation uses depthwise separable convolution, which changes the number of channels of ordinary convolution from the number of input channels n to 1 / n, so its computational complexity is only about 1 / n of ordinary convolution. The same operation process can improve the performance of the CNN network, so a fixed set of convolution, activation and channel operations are designed in the model, and this set of operations is called the bottleneck structure. The channel operation includes channel split and channel shuffle operations to further mix features, making the model detection more accurate.

[0107] However, ShuffleNetV2 still has low accuracy in road pavement status detection based on video images, so the patent holder proposed RepShuffleNet, which introduced the Repconv structure based on ShuffleNetV2, also known as model reparameterization. This operation uses a multi-branch structure during model training, and reparameterizes it to a single-branch structure during model prediction. In this way, the multi-branch structure is used to extract image features at multiple scales during training, and the single-branch structure is used during training to reduce the amount of calculation and reasoning time. This improves the model's accuracy and reasoning speed to a certain extent, and can better meet the needs of fast and accurate recognition of road pavement status.

[0108] ShuffleNetV2 continuously deepens the network structure by stacking bottleneck structures to achieve feature extraction of the input image. The bottleneck structure in ShuffleNetV2 is improved by combining it with the Repconb structure and called the RepShuffleNet basic unit. As the new bottleneck structure in RepShuffleNet, it is divided into RepShuffleNet basic unit and RepShuffleNetX2 basic unit according to the number of channels output by the structure.

[0109] It is worth noting that there are two units corresponding to ShuffleNetV2, and the other is called ×2 basic unit, through which the number of channels of the feature map is doubled. Its structure is to change the left short connection of the ShuffleNetV2 basic unit to a superposition of a 3×3 depth-separable convolution and a 1×1 ordinary convolution. Similarly, RepShuffleNet also has the same RepShuffleNet×2 basic unit.

[0110] By improving ShuffleNetV2, one of the best lightweight models, and integrating the Repconv structure, the model accuracy becomes higher and the inference time becomes shorter because it uses a multi-branch structure during training and an equivalent single branch during prediction, making the entire method more real-time and accurate.

[0111] S2. Real-time edge-end road surface condition detection based on fusion data of road video images and weather information; fusing the weather data with the output data in step S1, constructing a fusion model of road video images and weather information data, and inputting the fusion model into the edge computing device to obtain real-time road surface condition detection results;

[0112] S21, weather data collection: obtaining weather information of the current time and location through a weather data collection device, and inputting the collected data into an edge computing device for sorting and storage;

[0113] Weather data is the local weather information obtained in real time by real-time weather data collection equipment at the edge. The weather information may include but is not limited to information related to weather changes such as temperature, humidity, air pressure, wind speed, etc. The collected information can be input into the edge computing device in real time for sorting and storage.

[0114] S22, fusion data construction: fusing the collected real-time weather information and real-time road video image feature data to obtain real-time fusion data;

[0115] In order to further optimize the above technical solution, the fused data in step S22 is obtained by concatenating and merging the output data of step S133 and the weather data of step S21 as two separate lists.

[0116] S23, loading the fused data obtained in step S22 into the edge computing device to obtain the current road surface status;

[0117] S231. Construction of fusion detection model; based on a machine learning model with prediction function, a real-time road surface condition detection model based on fusion weather information and road video images is constructed; the basic model can use any machine learning method with prediction function, including but not limited to random forest RF, support vector machine SVM, naive Bayes, clustering algorithm and other methods that can use vectors as input and output results.

[0118] In order to further optimize the above technical solution, in step S231, the input of the fusion detection model is the fusion data in step S22; the data is input in vector form.

[0119] S232, historical fusion data set construction and model training;

[0120] The historical fusion dataset is a combination of the historical road video image database, which is obtained by extracting and detecting image features through RepShuffleNet, and the corresponding weather data, and also has the correct real classification labels;

[0121] The hyperparameter optimization method is used to optimize and train the fusion detection model. The machine learning algorithm (RF, SVM, Naive Bayes) selected by hyperparameter optimization is used to retain the model with the best training effect.

[0122] Fusion detection model training. Generally, machine learning models with prediction functions have multiple hyperparameters that can be manually selected. Different hyperparameters will significantly affect the detection effect of the model. In order to target the road surface condition detection task, the hyperparameter optimization method is used to optimize and train the model. Specifically, the range and change value of each hyperparameter are selected for each hyperparameter, and then the hyperparameter corresponding to the high accuracy is selected based on the model detection accuracy as the selection standard, and it is used as a trained detection model for subsequent real-time detection.

[0123] The fused data is the output of the lightweight road surface state prediction module and weather data. The model used is the machine learning model SVM support vector machine method, which has the advantages of less training data required, the classification result is only related to the support vector, good anti-data interference ability, strong applicability, and can be used for linear and nonlinear data. A small amount of data can train a vector machine model with good results. The specific function is:

[0124]

[0125] st * yi (wx i +b) * ≥γ **

[0126] The specific training parameters of the model are set as the kernel function using the 'RBF' radial basis function kernel, the regularization strength is used to balance the weights of the classification interval edge and the misclassified points, and is searched and selected from three numbers [1, 10, 100] through experiments. Gamma- is a parameter in the RBF kernel that controls the degree of influence of a single training instance on the model, and is set to be searched and selected from a smaller [0.1, 0.01, 0.001] through experiments. Call the fit method for training to find the best parameter combination. Output the best parameter combination best_params. Output the cross-validation score mean and standard deviation std of each parameter combination. Finally, use the best parameters to make predictions on the test set.

[0127] S233, road surface condition detection based on real-time fusion data; loading the trained fusion detection model into the edge computing device to detect the real-time road video image and weather information fusion data, and obtaining the final road surface detection result of snow-covered, semi-snow-covered, iced, water-logged, wet, dry, etc.;

[0128] In order to further optimize the above technical solution, the hyperparameters of the fusion detection model in step S2 are determined after multiple experiments. Taking the support vector machine method as an example of the fusion model, the kernel function uses the 'RBF' radial basis function kernel, and the regularization strength is used to balance the weights of the classification interval edge and the misclassified points. It is searched and selected from three numbers [1, 10, 100] through experiments. Gamma is a parameter in the RBF kernel, which controls the influence of a single training instance on the model. It is set to search and select from a smaller [0.1, 0.01, 0.001] through experiments. Call the fit method for training to find the best parameter combination. Output the best parameter combination best_params. Output the cross-validation score mean and standard deviation std of each parameter combination. Finally, use the best parameters to predict on the test set.

[0129] By integrating multi-source data, namely weather data and road pavement condition prediction data, and matching and optimizing multiple hyperparameters through machine learning, the accuracy of the model is greatly improved compared to that based only on road video image data.

[0130] S3, steps S1 and S2 are all implemented through a detection device, which includes an edge computing device, a road video acquisition device, and a weather data acquisition device; the edge computing device and the road video acquisition device are communicatively connected via a CSI / USB interface, and are communicatively connected to the weather data acquisition device via VCC, DATA, and GND pins, so as to respectively obtain road pavement images and weather data in real time at the edge.

[0131] In step S3, the edge computing device includes an edge computing processing module, a communication module, a monitoring and visual module, a data recording and storage module, and a power management module;

[0132] The edge computing processing module is used for data processing and calculation; the communication module is used to provide wireless network connection services, transmit the detection results to the back-end server, and realize unattended road surface condition detection; the monitoring visual module is connected to the display screen and provides an HDMI physical interface to facilitate video signal output, which is used for visual display of video analysis and processing; the data recording storage module is used to transmit the detection results to a large-capacity storage device, provides a USB host and SATAIII interface for connecting a large-capacity external storage device, and also provides a MicroSD memory card interface for loading the file system; the power management module is used to provide a power source for the detection equipment.

[0133] The road video acquisition device obtains traffic scene video images in real time through the camera, and transmits the video images to the edge computing processing module after preliminary processing of the video; the weather data acquisition device is electrically connected to the DH22 temperature and humidity sensor to obtain the current temperature and humidity information in real time, organize and store it, and transmit it to the edge computing processing module in real time.

[0134] In order to further optimize the above technical solution, the road video acquisition device includes a CSI / USB camera, which is used to collect traffic video images in real time and transmit the video images to the real-time road video acquisition device; it is connected to the edge computing processing module through a CSI connection line; the expansion camera module, as redundancy and supplement, assists the main camera to collect video images in real time, and transmits the video images to the real-time road video acquisition device; the camera needs to have a resolution of at least 307px wide × 209px high.

[0135] The weather data collection device includes a DH22 temperature and humidity sensor, which is used to collect weather information in real time and transmit it to the real-time weather data collection device. It is connected to the edge computing processing module through the three pins of VCC, DATA, and GND. It uses a low-power demand sensor device. The information obtained includes real-time temperature and humidity, and the weather information collection module can be called through the code, and the detection interval can be manually selected.

[0136] The edge computing device is at least one processor with a storage function; and the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can control and obtain the camera and weather collection equipment connected to it, can run a pre-built road state recognition model, and the processing speed of the running model is close to the video frame rate, and can store video and weather data information for at least a period of time until the results of the previous round of model are output.

[0137] In this embodiment, the weather data collection device uses a low-power demand sensor device, and the information obtained includes real-time temperature and humidity. The weather information collection module can be called through code, and the detection interval can be manually selected.

[0138] The terminal processor is an edge computing device such as Raspberry Pi; the edge computing device needs to have a certain computing power, can be equipped with only a CPU, and can run a lightweight deep neural network. It also has an interface with the camera device and a pin that can be connected to the temperature detection module and networking function, and has at least 4GB of memory and 16GB of storage space. The camera of the road video acquisition module needs to have a resolution of at least 307px wide × 209px high.

[0139] The present invention can obtain road pavement status information in real time and with greater accuracy and anti-interference through the combination of road video and weather information. For high-risk sections such as tunnels, bridges, long and steep slopes, and sharp bends near water or cliffs, the efficient acquisition of such information can significantly improve the response speed of relevant traffic management measures and the level of traffic safety, and contribute to protecting people's lives and property.

[0140] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0141] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A road condition detection method based on video images and weather data, characterized in that: The following steps are involved: S1. Real-time edge road surface status detection based on road video image data; Inputting the road surface status video data into the edge computing device to obtain the road surface feature data; S2, real-time edge road surface status detection based on fusion data of road video images and weather information; The weather data is integrated with the output data in step S1 to construct a fusion model of the road video image and the weather information data, and the fusion model is input into the edge computing device to obtain the real-time road surface status detection result; S3, steps S1 and S2 are all implemented through a detection device, which includes an edge computing device, a road video acquisition device, and a weather data acquisition device; the edge computing device and the road video acquisition device are communicatively connected via a CSI / USB interface, and are communicatively connected to the weather data acquisition device via VCC, DATA, and GND pins, so as to respectively obtain road pavement images and weather data in real time at the edge.

2. A road condition detection method based on video images and weather data according to claim 1, characterized in that: Step S1 includes: S11, real-time road video image data acquisition: road video information is collected in real time by a road video acquisition device and stored, and transmitted to an edge computing device; S12, video image data preprocessing: extracting images and inputting them into edge computing devices for image preprocessing; S13, extracting the pre-processed video image data to perform road surface condition detection.

3. The road surface condition detection method based on video images and weather data according to claim 2 is characterized in that: In step S13, the following steps are also included: S131. Build a lightweight CNN network RepShuffleNet model. Build a new lightweight CNN model RepShuffleNet based on the lightweight CNN model ShuffleNetV2 and Repconv. S132, constructing a historical road video image database and performing model training optimization; using historical data to construct a road pavement video image database, and training and optimizing the RepShuffleNet model; S133, loading the trained RepShuffleNet model into the edge computing device to perform real-time feature extraction of road pavement video image data and road pavement status detection.

4. The road surface condition detection method based on video images and weather data according to claim 3 is characterized in that: In step S132, video image data of various road types are collected and classified and labeled according to different road surface conditions. Each image file and its category are respectively matched and recorded, and the training set and test set are divided to produce a historical road video image database; the historical road video image database is used to train the RepShuffleNet model.

5. The road surface condition detection method based on video images and weather data according to claim 4 is characterized in that: Step S2 includes: S21, weather data collection: obtaining weather information of the current time and location through a weather data collection device, and inputting the collected data into an edge computing device for sorting and storage; S22, fusion data construction: fusing the collected real-time weather information and real-time road video image feature data to obtain real-time fusion data; S23. Load the fused data obtained in step S22 into the edge computing device to obtain the current road surface status.

6. A road surface condition detection method based on video images and weather data according to claim 5, characterized in that: Step S23 also includes the following steps: S231, fusion detection model construction: based on the machine learning model with prediction function, a real-time road surface state detection model based on fusion weather information and road video images is constructed; S232, historical fusion data set construction and model training; The historical fusion dataset is a historical road video image database that is combined with image feature extraction, detection and weather data through the RepShuffleNet model to form classification labels; The hyperparameter optimization method is used to optimize and train the fusion detection model; S233, road surface condition detection based on real-time fusion data; loading the trained fusion detection model into the edge computing device to detect the real-time road video image and weather information fusion data to obtain the real-time condition of the road surface.

7. A road surface condition detection method based on video images and weather data according to claim 6, characterized in that: The fused data in step S22 is obtained by concatenating the output data of step S133 and the weather data of step S21 as two separate lists.

8. The method for detecting road conditions based on video images and weather data according to claim 7, characterized in that: In step S231, the input of the fusion detection model is the fusion data in step S22; the data is input in vector form.

9. The road surface condition detection method based on video images and weather data according to claim 1, characterized in that: In step S3, the edge computing device includes an edge computing processing module, a communication module, a monitoring and visual module, a data recording and storage module, and a power management module; The edge computing processing module is used for data processing and calculation; the communication module is used to provide wireless network connection services and transmit the detection results to the back-end server; the monitoring visual module is used for video signal output and visual display of video analysis and processing; the data recording storage module is used to transmit the detection results to the storage device and load data files; the power management module is used to provide a power source for the detection equipment.

10. The road surface condition detection method based on video images and weather data according to claim 9, characterized in that: The road video acquisition device obtains traffic scene video images in real time through the camera, and transmits the video images to the edge computing processing module after preliminary processing of the video; the weather data acquisition device is electrically connected to the temperature and humidity sensor to obtain the current temperature and humidity information in real time, organize and store it, and transmit it to the edge computing processing module in real time.

Images