Real-time estimation method of adhesion coefficient on mixed ice-snow road based on video data

By building a pavement state detection system, combining surveillance video and on-board video, using technologies such as hybrid Gaussian background modeling and attention mechanism 3D CNN, the sensors in estimation of ice and snow pavement attachment coefficients are solved, the sensors are expensive, dynamic modeling is complex and machine vision recognition are complex in estimation of ice and snow pavement adhesion coefficients. The problem of low accuracy achieves accurate and timely detection of the adhesion coefficient of ice and snow roads, improving driving safety and decision-making accuracy of intelligent systems.

CN114332722BActive Publication Date: 2024-11-05JILIN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111668767.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-31
Publication Date
2024-11-05
Estimated Expiration
2041-12-31

AI Technical Summary

Technical Problem

In the estimation of the adhesion coefficient of ice and snow pavement, there are problems such as expensive sensors and susceptible to environmental influences, complex dynamic modeling and not real-time, low machine vision recognition accuracy and susceptible to shadow interference, making it difficult to achieve accurate and timely adhesion of ice and snow pavement. Coefficient detection.

Method used

Using a real-time estimation method of mixed ice and snow pavement adhesion coefficient based on video data, the road state detection system is constructed, and the combination of surveillance video and on-board video is used to perform backgrounds using hybrid Gaussian background modeling, attention mechanism 3D CNN, generative adversarial network and other technologies. Extraction, pavement area segmentation, shadow detection and elimination and status recognition to realize real-time estimation of adhesion coefficients of ice and snow pavement.

Benefits of technology

Improve the accuracy and real-time detection of adhesion coefficient of ice and snow pavement, and can accurately identify typical ice and snow pavement, especially "black ice" pavement, enhance the ability to predict future road conditions, improve driving safety and accurate decision-making of intelligent systems. sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114332722B_ABST
    Figure CN114332722B_ABST
Patent Text Reader

Abstract

The present invention relates to a real-time estimation method for the adhesion coefficient of a mixed ice and snow road surface based on video data, and the steps are: collecting historical monitoring video data containing mixed ice and snow roads and their corresponding environmental parameters, as well as different road surface shadow images, constructing a mixed ice and snow road historical video data and its corresponding environmental parameter data set, and a road surface shadow image data set; offline training, respectively obtaining a background extraction model, a road surface area segmentation model, a road surface shadow detection and elimination model, and a road surface state recognition model, and constructing a road surface state recognition system. The present invention not only realizes the estimation of the adhesion coefficient of a typical ice and snow road surface based on machine vision, but also accurately segments ice and snow roads in mixed states, estimates their adhesion coefficients, ensures the accuracy and real-time nature of the estimation, and is of great significance to improving the safety of early warning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to a real-time estimation method for the adhesion coefficient of a mixed ice and snow road surface based on video data. Background Art

[0002] Highway traffic in northern my country is generally affected by ice and snow weather. The harsh driving environment in ice and snow weather poses a huge safety hazard to winter driving safety. Studies have shown that about 15% to 30% of traffic accidents each year are caused by snow and ice on the road in winter. For example, "black ice" roads are difficult to identify, which poses a certain safety hazard to driving vehicles. Therefore, accurate and real-time estimation of the adhesion coefficient of ice and snow roads is of great significance to improving driving safety, and can also improve the accuracy of intelligent system planning and decision-making.

[0003] At present, there are three main methods for estimating the road adhesion coefficient:

[0004] The first method is to obtain the required road surface data through sensors (acoustic, optical, temperature sensors, etc.), and establish the relationship between the sensor data and the road adhesion coefficient for estimation. This method has a wide detection range, but the sensors are expensive and easily affected by the environment.

[0005] The second is an estimation method based on dynamic response, which estimates based on system response and mathematical models. Although the estimation of road adhesion coefficient using dynamic modeling is accurate and reliable, it is difficult to accurately obtain a large number of parameters due to the complexity of the model, and it is difficult to ensure real-time performance. In addition, the method based on dynamic estimation requires the vehicle tire to be in contact with the road surface to estimate. In this way, the adhesion coefficient of the road surface that is currently in contact with the tire can only be estimated, but the road adhesion coefficient of the road surface that will be passed in the future cannot be predicted, and the vehicle cannot be intervened and controlled in time.

[0006] The third method is to directly estimate the road adhesion coefficient based on machine vision sensors with the help of deep learning. The machine vision-based method does not rely on the vehicle dynamics model, has fast recognition, does not need to meet certain incentive conditions, and has a certain degree of advance predictiveness in the recognition results. The recognition method only needs a classification recognition model to recognize the road state, but the road image contains multiple nonlinear data, and the existing convolutional neural network has insufficient generalization ability, which will lead to low accuracy in road state recognition. In addition, shadows are one of the common interference factors of road images. Road shadows will change the characteristics of road images, which can easily lead to recognition errors based on machine vision methods. Summary of the invention

[0007] The purpose of the present invention is to provide a real-time estimation method for the adhesion coefficient of mixed ice and snow roads based on video data, construct a road condition detection system, input monitoring video and real-time vehicle-mounted video into the system, obtain a real-time road condition image, and thus estimate the adhesion coefficient of the ice and snow road.

[0008] The technical solution of the present invention is: a real-time estimation method of the adhesion coefficient of a mixed ice-snow road surface based on video data, comprising the following steps:

[0009] S1: Construct historical monitoring video data of mixed ice and snow roads and their corresponding environmental parameter datasets, real-time vehicle video data and their corresponding environmental parameter datasets, and road shadow image datasets;

[0010] S2: Offline training, respectively obtaining the background extraction model, the road area segmentation model, the road shadow detection and elimination model, and the road state recognition model, and building a road state detection system;

[0011] S3: Online identification, transmitting the real-time monitoring video recorded by the monitoring equipment at each site to the road surface condition detection system 1, outputting the road surface condition detection result 1 and saving it;

[0012] S4: The vehicle-mounted camera takes a real-time video of the road ahead, transmits it to the road surface state detection system 2, outputs the road surface state detection result 2, uses the road surface state detection result 1 to correct the road surface state detection result 2, and obtains the final road surface state detection result 3, thereby achieving the purpose of real-time detection of the adhesion coefficient of the ice and snow road surface;

[0013] S5: According to the mapping relationship between the road surface type and the road surface adhesion coefficient, a real-time estimated value of the ice and snow road surface adhesion coefficient in front of the vehicle is obtained.

[0014] The ice and snow road in step S1 includes six types of road: loose snow road, snow slab road, slurry road, ice slab road, semi-melted ice slab road, and "black ice" road; in addition, environmental parameters include: solar radiation intensity, time, road surface temperature, air temperature, precipitation, and snowfall. The collection method is: using road monitoring equipment and vehicle-mounted cameras to collect mixed ice and snow road video image data sets, while using meteorological stations to collect environmental parameters corresponding to different video images, and using handheld cameras and web crawlers to collect shadow data under different road conditions.

[0015] Step S2 specifically includes the following steps:

[0016] S2-1. Use the mixed Gaussian background modeling algorithm to obtain a background extraction model. Use the background extraction model to obtain the static background from the surveillance video to prepare for road area segmentation. The specific steps of the mixed Gaussian background modeling algorithm are as follows:

[0017] Step 1: Use multiple single Gaussian models to describe each pixel in the video image. The corresponding mathematical expression defines the probability of the pixel as shown in formula (1-1):

[0018] p(x)={[ω i (x,y,t),μ i (x,y,t),σ i (x,y,z) 2 ]}(i=1,2,…,N) (1-1)

[0019] Where N is the number of single Gaussian models in the mixed Gaussian model, and its value is between 3 and 5. i (x, y, t) is the weight coefficient of each single Gaussian model and satisfies equation (1-2); μ i (x, y, t) is the mean of this Gaussian distribution, σ i (x,y,z) 2 is the covariance of this Gaussian distribution.

[0020]

[0021] Step 2: In order to match each new pixel value I(x, y, t) to a suitable high-speed distribution model, compare it with the current N models according to formula (1-3);

[0022]

[0023] Among them, For the pixels in the new video frame that satisfy equation (1-3), they will be judged as background, and those that do not satisfy equation (1-3) will be considered as noise or moving vehicles.

[0024] Step 3: If the pixel is determined to be background, the weight coefficients of each mode need to be updated, and the new weight is formula (1-4);

[0025] ω i (x,y,t)=(1-α)ω i (x,y,t-1)+α (1-4)

[0026] Among them, α∈(0,1) is the learning factor in the coefficient update process, that is, the update speed;

[0027] Step 4: Update the mean and variance of each mode according to formula (1-5) and formula (1-6) respectively;

[0028] μ i (x,y,t)=(1-p)μ i-1 (x,y,t)+pI(x,y,t) (1-5)

[0029] σ i (x,y,t) 2 =(1-p)σ i-1 (x,y,t) 2 +p(I(x,y,t)-μ i (x,y,t) 2 (1-6)

[0030] Among them, ρ is the learning rate of the mean and variance, which reflects the convergence speed of the Gaussian distribution parameters and can be calculated from α to obtain formula (1-7);

[0031]

[0032] Step 5: If the new pixel does not meet the matching conditions of any Gaussian model, replace it with the Gaussian model with the smallest weight, that is, the current pixel value is the mean of the model, the weight is a smaller value, and the standard deviation is the initial larger value;

[0033] Step 6: According to ω i (x,y,t) / σ i (x, y, t) Arrange the order of each model in descending order, and put the model with large weight and small standard deviation at the top;

[0034] Step 7: Finally, take the first N models as the background after video modeling and combine them, where the parameter T is the proportion of the background, ω k is the weight of the kth Gaussian distribution, and the definition of parameter N satisfies formula (1-8);

[0035]

[0036] S2-2, annotate the historical monitoring video dataset of mixed ice and snow roads, annotate the road area and non-road area, and establish a road surface area segmentation dataset for training the 3D CNN (convolutional neural network) based on the attention mechanism, that is, integrate the 3D CNN with the dual-channel CBAM attention mechanism module, which includes the channel attention mechanism and the spatial attention mechanism, so as to obtain the road surface area segmentation model. Since the pixels need to be divided into two categories: road surface area and non-road surface area, two neurons are selected for convolution, and the Softmax function is used to calculate the probability, which is input into the cross entropy for back propagation training, and finally the binary road surface area segmentation result is output.

[0037] S2-3. Use the road shadow dataset to train a shadow detection algorithm based on shadow DSC features to detect the shadow part of the road surface. On the basis of the detected binary marking mask, two generative adversarial networks are constructed. One generative adversarial network is used to realize the conversion of the shadowed road surface to the shadow-free road surface under the guidance of the shadow mask, and the other generative adversarial network is used to reconstruct the shadow-free road surface to the shadowed road surface image under the guidance of the same shadow mask, generate a road shadow detection and elimination model, and finally realize the shadow elimination of the shadowed road surface image.

[0038] S2-4. Perform pixel-level annotation on historical monitoring video data of mixed ice and snow roads and their corresponding environmental parameter data sets to construct a road surface state data set. At the same time, use data enhancement technology to amplify the data set and train a multi-module convolutional neural network. That is, use the road surface state data set to train a 3D CNN (convolutional neural network) based on a residual structure, and the environmental parameters are used to train a fully connected neural network. Finally, the 3D CNN based on the residual structure and the fully connected neural network are connected by the fully connected layer to obtain a road surface state recognition model.

[0039] Step S3 specifically includes the following steps:

[0040] S3-1, transmitting the monitoring video recorded in real time by the monitoring equipment at each site to the road surface condition detection system 1;

[0041] S3-2, extracting the static background of the video through the background extraction model;

[0042] S3-3, loading the road segmentation model from the extracted background to complete the segmentation of the road area, and saving the segmented road area video frame locally;

[0043] S3-4, the road surface shadow detection and elimination model detects and eliminates the shadows appearing in the road surface area data, and saves the road surface area video frame after the shadows are eliminated locally;

[0044] S3-5, the road surface state recognition model loads the video frame of the road surface area with the shadows removed, recognizes its state, and uses colors of different depths to represent different road surface states, and outputs a state diagram of the monitored road surface segment;

[0045] S3-6. Save the output road surface condition detection result 1.

[0046] Step S4 specifically includes the following steps:

[0047] S4-1, transmitting the front road video captured by the vehicle-mounted camera in real time to the road surface state detection system 2;

[0048] S4-2, extracting the static background of the video through the background extraction model;

[0049] S4-3, loading the road segmentation model from the extracted background to complete the segmentation of the road area, and saving the segmented road area video frame locally;

[0050] S4-4, the road surface shadow detection and elimination model detects and eliminates the shadows appearing in the road surface area video frame, and saves the road surface area video frame after the shadows are eliminated locally;

[0051] S4-5, the road surface state recognition model loads the road surface area video frame with the shadows removed, recognizes its state, and uses colors of different depths to represent different road surface states, and outputs the road surface state detection result 2;

[0052] S4-6. Use the road surface condition detection result 1 to correct the road surface condition detection result 2 to obtain the final road surface condition detection result 3.

[0053] The specific mapping relationship between the road surface type and the road surface adhesion coefficient in step S5 is shown in Table 1-1.

[0054] Table 1-1 Adhesion coefficients of different road conditions

[0055]

[0056] The present invention provides a real-time estimation method for the adhesion coefficient of a mixed icy and snowy road surface based on video data, which estimates the adhesion coefficient of an icy and snowy road surface based on a background extraction model, a road area segmentation model, a road shadow detection and elimination model, and a road state recognition model. At the same time, the video recordings collected by a monitoring device and a vehicle-mounted camera are loaded into a road state detection system, the road state detection results are corrected, and the accuracy of the estimation of the adhesion coefficient of an icy and snowy road surface is improved.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1) The present invention makes full use of the video acquired in real time by the monitoring equipment and the vehicle-mounted camera, and innovatively proposes a road surface condition detection system based on the establishment. The road surface condition detection results of the monitored section taken by the monitoring equipment are used to correct the road surface condition detection results ahead taken by the vehicle-mounted camera. The "black ice" road surface that is difficult to detect in typical icy and snowy road surfaces can also be accurately detected, thereby improving the detection accuracy and real-time performance of the mixed icy and snowy road surface conditions.

[0059] 2) The road surface condition recognition model in the present invention takes into account the influence of environmental parameters on the road surface condition recognition results, combines the 3D CNN (convolutional neural network) based on the residual structure and the fully connected neural network to form a multi-module neural network to classify the road surface condition, further improving the detection accuracy of the mixed ice and snow road surface condition. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of a method for real-time estimation of adhesion coefficient of mixed ice and snow road surface based on video data of the present invention;

[0061] Figure 2 This is a road surface condition detection system diagram;

[0062] Figure 3 It is the architecture of 3D CNN (Convolutional Neural Network);

[0063] Figure 4 It is a flow chart of road shadow detection and elimination;

[0064] Figure 5 is a schematic diagram of the residual structure;

[0065] Figure 6 Schematic diagram of a multi-module convolutional neural network. DETAILED DESCRIPTION

[0066] The present invention is described in detail below in conjunction with the embodiments and drawings. The protection scope of the present invention is not limited to the embodiments, and any changes made by those skilled in the art within the scope defined by the claims also fall within the protection scope of the present invention.

[0067] The present invention provides a real-time estimation method for the adhesion coefficient of a mixed ice and snow road surface based on video data, such as Figure 1 As shown, the following steps are included:

[0068] S1: Construct historical monitoring video data of mixed ice and snow roads and their corresponding environmental parameter datasets, real-time vehicle video data and their corresponding environmental parameter datasets, and road shadow image datasets;

[0069] S2: Offline training, respectively obtain the background extraction model, road area segmentation model, road shadow detection and elimination model, road state recognition model, and build a road state detection system, such as Figure 2 As shown;

[0070] S3: Online identification, transmitting the real-time monitoring video recorded by the monitoring equipment at each site to the road surface condition detection system 1, outputting the road surface condition detection result 1 and saving it;

[0071] S4: The vehicle-mounted camera takes a real-time video of the road ahead, transmits it to the road surface state detection system 2, outputs the road surface state detection result 2, uses the road surface state detection result 1 to correct the road surface state detection result 2, and obtains the final road surface state detection result 3, thereby achieving the purpose of real-time detection of the adhesion coefficient of the ice and snow road surface;

[0072] S5: According to the mapping relationship between the road surface type and the road surface adhesion coefficient, a real-time estimated value of the ice and snow road surface adhesion coefficient in front of the vehicle is obtained.

[0073] The specific embodiment is developed based on the PyQt architecture, and the core algorithms in the background extraction model, the road surface area segmentation model, the road surface shadow detection and elimination model, and the road surface state recognition model are integrated into the PyQt-based GUI system to obtain a road surface state detection system. Specific embodiment:

[0075] Step 1: Collect historical monitoring video data of mixed ice and snow roads and their corresponding environmental parameters, as well as different road shadow images, to construct historical video data of mixed ice and snow roads and their corresponding environmental parameter datasets and road shadow image datasets. Mixed ice and snow roads collected using road monitoring equipment and vehicle-mounted cameras include: loose snow roads, snow slab roads, slurry roads, ice slab roads, semi-melted ice slab roads, and "black ice" roads; environmental parameters collected using meteorological stations include: solar radiation intensity, time, road surface temperature, air temperature, precipitation, and snowfall; shadow data under different road conditions are collected using handheld cameras and web crawlers.

[0076] Step 2: Offline training, obtain the background extraction model, road area segmentation model, road shadow detection and elimination model, and road state recognition model, and build a road state recognition system.

[0077] The background extraction model is obtained by the mixed Gaussian background modeling algorithm. The background extraction model is used to obtain the static background from the surveillance video to prepare for the road area segmentation. The specific steps of the mixed Gaussian background modeling algorithm are as follows:

[0078] Step 1: Use multiple single Gaussian models to describe each pixel in the video image. The corresponding mathematical expression defines the probability of the pixel as shown in formula (1-1):

[0079] p(x)={[ω i (x,y,t),μ i (x,y,t),σ i (x,y,z) 2 ]}(i=1,2,…,N) (1-1)

[0080] Where N is the number of single Gaussian models in the mixed Gaussian model, and its value is between 3 and 5. i (x, y, t) is the weight coefficient of each single Gaussian model and satisfies equation (1-2); μ i (x, y, t) is the mean of this Gaussian distribution, σ i (x,y,z) 2 is the covariance of this Gaussian distribution.

[0081]

[0082] Step 2: In order to match each new pixel value I(x, y, t) to a suitable high-speed distribution model, compare it with the current N models according to formula (1-3);

[0083]

[0084] Among them, For the pixels in the new video frame that satisfy equation (1-3), they will be judged as background, and those that do not satisfy equation (1-3) will be considered as noise or moving vehicles.

[0085] Step 3: If the pixel is determined to be background, the weight coefficients of each mode need to be updated, and the new weight is formula (1-4);

[0086] ω i (x,y,t)=(1-α)ω i (x,y,t-1)+α (1-4)

[0087] Among them, α∈(0,1) is the learning factor in the coefficient update process, that is, the update speed;

[0088] Step 4: Update the mean and variance of each mode according to formula (1-5) and formula (1-6) respectively;

[0089] μ i (x,y,t)=(1-p)μ i-1 (x,y,t)+pI(x,y,t) (1-5)

[0090] σ i (x,y,t) 2 =(1-p)σ i-1 (x,y,t) 2 +p(I(x,y,t)-μ i (x,y,t) 2 (1-6)

[0091] Among them, ρ is the learning rate of the mean and variance, which reflects the convergence speed of the Gaussian distribution parameters and can be calculated from α to obtain formula (1-7);

[0092]

[0093] Step 5: If the new pixel does not meet the matching conditions of any Gaussian model, replace it with the Gaussian model with the smallest weight, that is, the current pixel value is the mean of the model, the weight is a smaller value, and the standard deviation is the initial larger value;

[0094] Step 6: According to ω i (x,y,t) / σ i(x, y, t) Arrange the order of each model in descending order, and put the model with large weight and small standard deviation at the top;

[0095] Step 7: Finally, the first N models are taken as the background after video modeling for combination, where parameter T is the proportion of the background, and the definition of parameter N satisfies formula (1-8);

[0096]

[0097] Then, the historical monitoring video dataset of mixed ice and snow roads is annotated to mark the road area and non-road area, and a road surface area segmentation dataset is established for training the 3D CNN (convolutional neural network) based on the attention mechanism. The 3D CNN architecture built based on Python is as follows: Figure 3 As shown, it contains a hardwired layer, 3 convolutional layers, 2 pooling layers and 1 fully connected layer. Each 3D convolution kernel convolution cube takes 7 consecutive frames, each frame size is 60×40, and extracts information of 5 channels through the first layer, namely: grayscale, gradients in the x and y directions, and optical flows in the x and y directions; each layer of convolution kernel u×v×3 (u×v represents the spatial dimension, 3 represents the time dimension) performs convolution operations on the 5 channel information respectively; the pooling layer is used to abstract the features, and after multiple layers of convolution and pooling, each consecutive 7 frames of input images are converted into a 128-dimensional feature vector; the 3D CNN based on the attention mechanism is: the 3D CNN is integrated with the dual-channel CBAM attention mechanism module. The dual-channel attention mechanism module includes the channel attention mechanism and the spatial attention mechanism. Since the pixels need to be divided into two categories: road area and non-road area, two neurons are selected for convolution, and the Softmax function is selected to calculate the probability, as shown in formula (1-9), and it is input into the cross entropy for back propagation training, and finally the binary road area segmentation result is output.

[0098] The calculation formula of Softmax function is:

[0099]

[0100] Where P(y=i|x) is the probability that the input road surface x to be detected is judged to belong to the i-th road surface state in the classifier, and K=6 is the six road surface states to be detected in this paper.

[0101] The road shadow dataset is then used to train a shadow detection algorithm based on shadow DSC features to detect the shadow part of the road surface. On the basis of the binary marking mask detected, two generative adversarial networks are constructed. One generative adversarial network is used to realize the conversion of the road surface with shadows to the road surface without shadows under the guidance of the shadow mask. The other generative adversarial network is used to reconstruct the road surface without shadows to the road surface with shadows under the guidance of the same shadow mask, generate a road shadow detection and elimination model, and finally realize the shadow elimination of the road surface with shadows. The road shadow detection and elimination process is as follows: Figure 4 shown.

[0102] Finally, the historical monitoring video data of mixed ice and snow roads and their corresponding environmental parameter data sets are annotated at the pixel level to construct a road state data set. At the same time, the data set is amplified by using data enhancement technology to train a multi-module convolutional neural network, that is, the road state data set is used to train a 3D CNN (convolutional neural network) based on a residual structure, and the environmental parameters are used to train a fully connected neural network, in which the solar radiation intensity is divided into 1-5, the time is divided into 0-23.5 according to 0.5h, and the road surface temperature, air temperature, precipitation, and snowfall are real-time measurement values. Finally, the 3D CNN based on the residual structure and the fully connected neural network are connected by the fully connected layer to obtain a road state recognition model. The residual structure is as follows Figure 5 As shown, the multi-module convolutional neural network architecture is as follows Figure 6 shown.

[0103] Step 3: a) The monitoring video recorded in real time by the monitoring equipment at each site is transmitted to the road surface condition detection system 1; b) The static background of the video is extracted through the background extraction model; c) The road surface segmentation model is loaded from the extracted background to complete the segmentation of the road surface area, and the segmented road surface area video frame is saved locally; d) The road surface shadow detection and elimination model detects and eliminates the shadows appearing in the road surface area data, and saves the road surface area video frame after the shadow is eliminated locally; e) The road surface condition recognition model loads the road surface area video frame with the shadow eliminated, identifies its state, and uses colors of different depths to represent different road surface states, and outputs the state diagram of the monitored road surface section; f) Save the output road surface condition detection result 1.

[0104] Step 4: a) The front road video captured by the on-board camera in real time is transmitted to the road surface condition detection system 2; b) The static background of the video is extracted through the background extraction model; c) The road surface segmentation model is loaded from the extracted background to complete the segmentation of the road surface area, and the segmented road surface area video frame is saved locally; d) The road surface shadow detection and elimination model detects and eliminates the shadows appearing in the road surface area video frame, and saves the road surface area video frame after the shadow is eliminated locally; e) The road surface condition recognition model loads the road surface area video frame with the shadow eliminated, identifies its state, and uses colors of different depths to represent different road surface states, and outputs the road surface condition detection result 2; f) The road surface condition detection result 1 is used to correct the road surface condition detection result 2 to obtain the final road surface condition detection result 3.

[0105] Step 5: Match the final road condition detection result with the road adhesion coefficient. The mapping relationship between the road type and the road adhesion coefficient is shown in Table 1-1.

[0106] Table 1-1 Adhesion coefficients of different road conditions

[0107]

Claims

1. A real-time estimation method for the adhesion coefficient of mixed ice and snow roads based on video data, It is characterized in that The steps include: S1: Construct historical monitoring video data of mixed ice and snow roads and their corresponding environmental parameter datasets, real-time vehicle video data and their corresponding environmental parameter datasets, and road shadow image datasets; S2: Offline training, respectively obtaining the background extraction model, the road area segmentation model, the road shadow detection and elimination model, and the road state recognition model, and building a road state detection system; S3: Online identification, transmitting the real-time monitoring video recorded by the monitoring equipment at each site to the road surface condition detection system 1, outputting the road surface condition detection result 1 and saving it; S4: The vehicle-mounted camera takes a real-time video of the road ahead, transmits it to the road surface state detection system 2, outputs the road surface state detection result 2, uses the road surface state detection result 1 to correct the road surface state detection result 2, and obtains the final road surface state detection result 3, thereby achieving the purpose of real-time detection of the adhesion coefficient of the ice and snow road surface; S5: obtaining a real-time estimated value of the adhesion coefficient of the icy and snowy road ahead of the vehicle according to the mapping relationship between the road type and the road adhesion coefficient; The road surface condition detection system described in step 2 has the following training steps: S2-1. Use the mixed Gaussian background modeling algorithm to obtain a background extraction model, and use the background extraction model to obtain a static background from the surveillance video to prepare for road area segmentation; S2-2, annotate the historical monitoring video dataset of mixed ice and snow roads, mark the road area and non-road area, establish a road surface area segmentation dataset, which is used to train the 3D CNN convolutional neural network based on the attention mechanism to obtain a road surface area segmentation model; S2-3, using the road shadow dataset, training a shadow detection algorithm based on the shadow DSC feature of global information analysis to detect the shadow part of the road, and then training a cyclic generative adversarial network to eliminate the road shadow, thereby obtaining a road shadow detection and elimination model; S2-4. Perform pixel-level annotation on historical monitoring video data of mixed ice and snow roads and their corresponding environmental parameter data sets to construct a road condition data set. At the same time, use data enhancement technology to amplify the data set and train a multi-module convolutional neural network to obtain a road condition recognition model.

2. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, Features: In step S1, the icy and snowy road surface includes: loose snow road surface, snow slab road surface, slurry road surface, ice slab road surface, semi-melted ice slab road surface, and black ice road surface; the environmental parameters include: solar radiation intensity, time, road surface temperature, air temperature, precipitation, and snowfall. Road surface monitoring equipment and vehicle-mounted cameras are used to collect video data of different icy and snowy road surfaces. At the same time, a meteorological station is used to collect environmental parameters corresponding to different video data. Handheld cameras and web crawlers are used to collect shadow data under different road conditions.

3. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, It is characterized in that The background extraction model described in step S2-1 adopts a mixed Gaussian background modeling algorithm, and the specific steps are as follows: Step 1: Use multiple single Gaussian models to describe each pixel in the video image. The corresponding mathematical expression defines the probability of the pixel as shown in formula (1-1): p(x)=*[ω i (x,y,t),μ i (x,y,t),σ i (x,y,z) 2 ]}(i=1,2,…,N) (1-1) Where N is the number of single Gaussian models in the mixed Gaussian model, and its value is between 3 and 5. i (x, y, t) is the weight coefficient of each single Gaussian model and satisfies equation (1-2); μ i (x, y, t) is the mean of this Gaussian distribution, σ i (x,y,z) 2 is the covariance of this Gaussian distribution; Step 2: In order to match each new pixel value I(x, y, t) to a suitable high-speed distribution model, compare it with the current N models according to formula (1-3); Among them, For the pixels in the new video frame that satisfy equation (1-3), they will be judged as background, and those that do not satisfy equation (1-3) will be considered as noise or moving vehicles. Step 3: If the pixel is determined to be background, the weight coefficients of each mode need to be updated, and the new weight is formula (1-4); oh i (x,y,t)=(1-α)ω i (x,y,t-1)+α(1-4) Among them, α∈(0,1) is the learning factor in the coefficient update process, that is, the update speed; Step 4: Update the mean and variance of each mode according to formula (1-5) and formula (1-6) respectively; μ i (x,y,t)=(1-p)μ i-1 (x,y,t)+pI(x,y,t)(1-5) σ i (x,y,t) 2 =(1-p)σ i-1 (x,y,t) 2 +p(I(x,y,t)-μ i (x,y,t)) 2 (1-6) Among them, ρ is the learning rate of the mean and variance, which reflects the convergence speed of the Gaussian distribution parameters and can be calculated from α to obtain formula (1-7): Step 5: If the new pixel does not meet the matching conditions of any Gaussian model, replace it with the Gaussian model with the smallest weight, that is, the current pixel value is the mean of the model, the weight is a smaller value, and the standard deviation is the initial larger value; Step 6: According to ω i (x,y,t) / σ i (x, y, t) Arrange the order of each model in descending order, and put the model with large weight and small standard deviation at the top; Step 7: Finally, take the first N models as the background after video modeling and combine them, where the parameter T is the proportion of the background, ω k is the weight of the kth Gaussian distribution, and the definition of parameter N satisfies formula (1-8); 4. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, It is characterized in that The 3D CNN convolutional neural network based on the attention mechanism described in step S2-2; specifically: integrating the 3D CNN with the dual-channel CBAM attention mechanism module, the dual-channel attention mechanism module includes a channel attention mechanism and a spatial attention mechanism.

5. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, It is characterized in that The cyclic generative adversarial network described in step S2-3 is specifically as follows: two generative adversarial networks are constructed, one generative adversarial network is used to realize the conversion of the shadowed road surface to the shadow-free road surface under the guidance of the shadow mask, and the other generative adversarial network is used to reconstruct the shadow-free road surface to the shadowed road surface image under the guidance of the same shadow mask, generate a road surface shadow detection and elimination model, and finally realize the shadow elimination of the shadowed road surface image.

6. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, It is characterized in that The multi-module convolutional neural network described in step S2-4 is specifically as follows: the road state data set is used to train the 3D CNN convolutional neural network based on the residual structure, and the environmental parameters are used to train the fully connected neural network. Finally, the 3D CNN based on the residual structure and the fully connected neural network are connected by the fully connected layer to obtain the final road state recognition result, wherein the 3D CNN based on the residual structure is specifically as follows: based on the feedforward neural network, a jump connection is added to bypass a part of the original structure.

7. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, It is characterized in that The data enhancement techniques described in step S2-4 include: translation conversion, flip change, data cropping, multi-scale scaling, distortion, and use of mean files and other methods.

8. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, It is characterized in that Step S3 specifically includes the following steps: S3-1, transmitting the monitoring video recorded in real time by the monitoring equipment at each site to the road surface condition detection system 1; S3-2, extracting the static background of the video through the background extraction model; S3-3, loading the road segmentation model from the extracted background to complete the segmentation of the road area, and saving the segmented road area video frame locally; S3-4, the road surface shadow detection and elimination model detects and eliminates the shadows appearing in the road surface area data, and saves the road surface area video frame after the shadows are eliminated locally; S3-5, the road surface state recognition model loads the video frame of the road surface area with the shadows removed, recognizes its state, and uses colors of different depths to represent different road surface states, and outputs a state diagram of the monitored road surface segment; S3-6. Save the output road surface condition detection result 1.

9. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, It is characterized in that Step S4 specifically includes the following steps: S4-1, transmitting the front road video captured by the vehicle-mounted camera in real time to the road surface state detection system 2; S4-2, extracting the static background of the video through the background extraction model; S4-3, loading the road segmentation model from the extracted background to complete the segmentation of the road area, and saving the segmented road area video frame locally; S4-4, the road surface shadow detection and elimination model detects and eliminates the shadows appearing in the road surface area video frame, and saves the road surface area video frame after the shadows are eliminated locally; S4-5, the road surface state recognition model loads the road surface area video frame with the shadows removed, recognizes its state, and uses colors of different depths to represent different road surface states, and outputs the road surface state detection result 2; S4-6. Use the road surface condition detection result 1 to correct the road surface condition detection result 2 to obtain the final road surface condition detection result 3.

10. The real-time estimation method of the adhesion coefficient of mixed ice and snow road surface based on video data according to claim 1, It is characterized in that The mapping relationship between the road surface type and the road surface adhesion coefficient described in step S5 is specifically shown in Table 1-1 Table 1-1 Adhesion coefficients of different road conditions