Road adhesion coefficient estimation method and system based on multiple modes

Abstract

The invention discloses a pavement adhesion coefficient estimation method and system based on multiple modes, and relates to the technical field of pavement recognition. The invention aims to solve the problem of low accuracy of the current road adhesion coefficient acquisition method. The method comprises the following steps: acquiring visual information and non-visual information of a pavement to be detected; inputting the visual information and the non-visual information of the pavement to be detected into the pavement state recognition model to obtain a pavement state; and obtaining a mean value and a variance of the road adhesion coefficient estimation value according to the road state and the to-be-measured road temperature. The pavement adhesion coefficient estimation system based on multiple modes comprises a power supply, a processor, a temperature sensor, a GPS positioner, an RTK measurer and a camera, and the processor is used for operating the road adhesion coefficient estimation method based on multiple modes. The method is used for estimating the road adhesion coefficient.

Description

Technical Field

[0001] The present invention relates to the technical field of road surface recognition, and in particular to a road surface adhesion coefficient estimation method and system based on multimodality. Background Art

[0002] During vehicle operation, the road adhesion coefficient significantly impacts driving safety and is related to multiple factors. Road surfaces are categorized by material, with vehicles typically traveling on asphalt, concrete, gravel, and other road types. Weather changes can cause road surfaces to become dry, wet, flooded, icy, and snowy. As roads age, they experience varying degrees of wear. Temperature fluctuations alter the properties of road materials, so accurately estimating the adhesion coefficient for different road types and conditions can improve autonomous driving safety.

[0003] Road surface condition recognition typically relies on data-driven methods, but these methods lack the ability to understand the dynamics of adhesion coefficient changes. They can only identify the road surface's state and type, but are unable to derive the adhesion coefficient based on the road surface's state. With technological advancements, dynamics-based methods have emerged, primarily using control parameters during vehicle travel to predict road surface conditions and thereby derive the adhesion coefficient. However, these methods are not easily transferable between different vehicle types, resulting in variations in the adhesion coefficients obtained from different vehicles and low accuracy. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of low accuracy in the current road adhesion coefficient acquisition method, and propose a road adhesion coefficient estimation method and system based on multimodality.

[0005] The multimodal-based road adhesion coefficient estimation methods include:

[0006] Step 1: Obtain visual information and non-visual information of the road surface to be tested;

[0007] The visual information is an image of the road surface in front of the test vehicle;

[0008] The non-visual information includes: temperature, test vehicle speed and geographic location;

[0009] The temperature includes: air temperature and ground surface temperature;

[0010] The geographical locations include: the northern permafrost zone, the eastern warm seasonal frozen zone, the Loess Plateau dry-wet transition zone, the southeastern hot and humid zone, the southwestern warm and humid zone, the northwestern arid zone, and the Qinghai-Tibet Plateau cold zone;

[0011] Step 2: Input the visual information and non-visual information of the road surface to be tested into the road surface state recognition model to obtain the road surface state;

[0012] Step 3: Obtain the mean and variance of the estimated value of the road adhesion coefficient according to the road surface state and the road surface temperature to be measured.

[0013] Furthermore, the road surface state recognition model is obtained by:

[0014] A1. Build a dataset and divide it into a training set and a test set.

[0015] The dataset includes: visual information, non-visual information and labels;

[0016] The label is the road surface status;

[0017] The road surface conditions include: dry, wet, waterlogged, icy and snowy;

[0018] A2. Use the training set and test set to train and test the recognition model to obtain a road condition recognition model.

[0019] Furthermore, the recognition model includes: a first convolutional layer, a temperature data processing unit, a first splicing layer, a second convolutional layer, a first CN_V2 Block unit, a first downsampling layer, a second CN_V2 Block unit, a second downsampling layer, a third CN_V2 Block unit, a third downsampling layer, a fourth CN_V2 Block unit, a fourth downsampling layer, other non-visual information processing units, a second splicing layer, and a third convolutional layer;

[0020] The first convolutional layer is a convolutional layer with a convolution kernel of 4×4 and a stride of 4, and is used to extract a visual information feature map; the input of the first convolutional layer is visual information;

[0021] The temperature data processing unit is used to process the temperature data, construct a temperature characteristic graph, and perform channel expansion on the temperature characteristic graph to obtain a first temperature characteristic graph;

[0022] The first splicing layer is used to splice the first temperature feature map and the visual information feature map to obtain a first splicing result; the input of the first splicing layer is the output of the first convolutional layer and the output of the temperature data processing unit;

[0023] The second convolutional layer is a 1×1 convolution, and its input is the first concatenation result;

[0024] The first CN_V2 Block unit includes three CN_V2 Block subunits; the input of the first CN_V2 Block unit is the output of the second convolutional layer;

[0025] The output of the first CN_V2 Block subunit serves as the input of the second CN_V2 Block subunit, and the output of the second CN_V2 Block subunit serves as the input of the third CN_V2 Block subunit. The three CN_V2 Block subunits have the same structure.

[0026] The first downsampling layer is a convolution layer with a convolution kernel of 2×2 and a stride of 2; the input of the first downsampling layer is the output of the first CN_V2 Block unit;

[0027] The second CN_V2 Block unit includes three CN_V2 Block subunits;

[0028] The third CN_V2 Block unit includes 9 CN_V2 Block subunits;

[0029] The fourth CN_V2 Block unit includes three CN_V2 Block subunits;

[0030] The second sampling layer, the third sampling layer, and the fourth sampling layer are the same as the first downsampling layer;

[0031] The input of the second CN_V2 Block unit is the output of the first downsampling layer, the output of the second CN_V2 Block unit is the input of the second downsampling layer, the output of the second downsampling layer is the input of the third CN_V2 Block unit, the output of the third CN_V2 Block unit is the input of the third downsampling layer, the output of the third downsampling layer is the input of the fourth CN_V2Block unit, and the output of the fourth CN_V2 Block unit is the input of the fourth downsampling layer;

[0032] The other non-visual information processing unit is used to process the non-visual information to obtain a geographic location feature map, a second temperature feature map, and a vehicle speed feature map;

[0033] The second splicing layer is used to splice the geographic location feature map, the second temperature feature map, the vehicle speed feature map and the output of the fourth downsampling layer to obtain a second splicing result;

[0034] The convolution kernel of the third convolution layer is a 1×1 convolution, the input is the second splicing result, and the output is the road surface state.

[0035] Furthermore, the temperature data processing unit includes: a first normalization layer and a first channel expansion layer;

[0036] The first normalization layer normalizes the surface temperature and the air temperature to between 0 and 1, respectively, to obtain a temperature characteristic map composed of the normalized surface temperature and air temperature;

[0037] The first channel expansion layer is used to expand the number of channels of the temperature feature map so that the number of channels of the temperature feature map is consistent with the number of channels of the road image, thereby obtaining a first temperature feature map; the input of the first channel expansion layer is the output of the first normalization layer.

[0038] Furthermore, the CN_V2 Block subunit includes: a fourth convolutional layer, a second normalization layer, a fifth convolutional layer, an activation function layer, a global response normalization layer, and a sixth convolutional layer;

[0039] The fourth convolutional layer is a convolutional layer with a convolution kernel of 7×7;

[0040] The second normalization layer is used to normalize the output of the fourth convolutional layer;

[0041] The fifth convolutional layer is a convolution with a convolution kernel of 1×1; the input of the fifth convolutional layer is the output of the second normalization layer;

[0042] The activation function layer is a GELU activation function, and the input of the activation function layer is the output of the fifth convolutional layer;

[0043] The global response normalization layer is used to perform global response normalization processing on the output of the activation function layer;

[0044] The sixth convolutional layer is a convolution with a convolution kernel of 1×1, and the input of the sixth convolutional layer is the output of the global response normalization layer.

[0045] Furthermore, the other non-visual information processing units include: a one-hot encoding layer, a third normalization layer, a second channel expansion layer, and a third channel expansion layer;

[0046] The one-hot encoding layer is used to perform one-hot encoding on the geographic location information to obtain a geographic location feature map;

[0047] The third normalization layer is used to normalize the vehicle speed information, surface temperature, and air temperature to between 0 and 1, respectively, to obtain a temperature characteristic map and a vehicle speed characteristic map;

[0048] The temperature characteristic map includes: surface temperature and air temperature;

[0049] The second channel expansion layer is used to expand the dimension of the geographic location feature map so that the dimension of the geographic location feature map is the same as the dimension of the feature map output by the fourth downsampling layer;

[0050] The third channel expansion layer is used to expand the vehicle speed feature map and the temperature feature map so that the dimensions of the vehicle speed feature map and the temperature feature map are the same as the output feature map of the fourth downsampling layer.

[0051] Furthermore, in step 3, the mean and variance of the estimated value of the road adhesion coefficient are obtained according to the road surface state and the road surface temperature to be measured, specifically:

[0052] Step 31: Determine the road surface condition. If the road surface condition is dry, wet, or wet, execute step 32. If the road surface condition is snowy or icy, directly obtain the mean and variance of the estimated road adhesion coefficient.

[0053] Step 32: correcting the mean of the estimated road adhesion coefficient based on the mean of the original estimated road adhesion coefficient, and then obtaining the variance of the estimated adhesion coefficient;

[0054] The method of correcting the mean of the estimated road adhesion coefficient based on the mean of the original estimated road adhesion coefficient is as follows:

[0055] u j =u mj +Δu

[0056] Among them, u j is the corrected mean of the estimated road adhesion coefficient, Δu is the corrected value of the mean of the estimated road adhesion coefficient;

[0057] The mean correction value of the estimated road adhesion coefficient is obtained according to the surface temperature, specifically:

[0058] If T≤0℃, then Δu=0.05;

[0059] If 0℃<T≤5℃, then Δu=0.04;

[0060] If 5℃<T≤10℃, then Δu=0.03;

[0061] If 10℃<T≤15℃, then Δu=0.02;

[0062] If 15℃<T≤20℃, then Δu=0.01;

[0063] If 20℃<T≤25℃, then Δu=0;

[0064] If 25℃<T≤30℃, then Δu=-0.01;

[0065] If 30℃<T≤35℃, then Δu=-0.02;

[0066] If T>35℃, then Δu=-0.03;

[0067] Where T is the surface temperature of the road to be measured.

[0068] Furthermore, if the road surface condition in step 31 is snow or ice, the mean and variance of the estimated road adhesion coefficient are directly obtained, specifically:

[0069] If the road surface state is snow, the mean value of the road surface adhesion coefficient estimate is 0.400, and the variance of the adhesion coefficient estimate is 0.0500;

[0070] If the road surface state is ice, the mean value of the road surface adhesion coefficient estimate is 0.175, and the variance of the adhesion coefficient estimate is 0.0375.

[0071] Further, the mean value of the original road surface adhesion coefficient estimate in step three is specifically:

[0072] If the road surface state is dry, the mean value of the original road surface adhesion coefficient estimate is 0.650, and the variance of the adhesion coefficient estimate is 0.0750;

[0073] If the road surface state is wet, the mean value of the original road surface adhesion coefficient estimate is 0.575, and the variance of the adhesion coefficient estimate is 0.0625;

[0074] If the road surface state is water, the mean value of the original road surface adhesion coefficient estimate is 0.450, and the variance of the adhesion coefficient estimate is 0.0750.

[0075] The multi-modal based road surface adhesion coefficient estimation system comprises a power supply, a processor, a temperature sensor, a GPS locator, an RTK measurer and a camera.

[0076] The temperature sensor is used to collect road surface temperature and air temperature, and is connected to the processor.

[0077] The GPS locator is used to obtain a geographic position, and is connected to the processor.

[0078] The RTK measurer is used to obtain a test vehicle speed, and is connected to the processor.

[0079] The camera is used to obtain an image in front of the test vehicle, and is connected to the processor.

[0080] The processor is used to estimate a road surface adhesion coefficient.

[0081] The power supply is used to provide electric energy, and is connected to the processor.

[0082] The present application has the following beneficial effects:

[0083] The present invention proposes a multimodal road adhesion coefficient estimation method and system. The present invention designs a road state recognition model. By fusing the visual information and non-visual information of the road surface to be tested, the state type of the road surface is obtained. The present invention finally obtains the mean and variance of the road adhesion coefficient of the road surface to be tested based on the road surface state. The present invention can quickly obtain the road adhesion coefficient and improve the efficiency of road adhesion coefficient acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 Flowchart of the present invention;

[0085] Figure 2 A structural diagram of the road surface state identification model;

[0086] Figure 3 Schematic diagram of the system hardware structure;

[0087] Figure 4 This is a rendering of the present invention;

[0088] Figure 5 Schematic diagram of the distribution of road adhesion coefficient under different road types. DETAILED DESCRIPTION

[0089] Specific implementation method 1: Figure 1 As shown, the specific process of the multi-modal road adhesion coefficient estimation method of this embodiment is as follows:

[0090] Step 1: Obtain visual information and non-visual information of the road surface to be tested through the on-board visual device and the on-board non-visual device;

[0091] The visual information is an image of the road surface in front of the vehicle;

[0092] The non-visual information includes: temperature, vehicle speed and geographic location;

[0093] The temperature includes: air temperature and ground surface temperature;

[0094] The geographical locations include: the northern permafrost zone, the eastern warm seasonal frozen zone, the Loess Plateau dry-wet transition zone, the southeastern humid and hot zone, the southwestern humid and warm zone, the northwestern arid zone and the Qinghai-Tibet Plateau cold zone.

[0095] Those skilled in the art correlated the positioning information with the natural zoning of highways. The first-level zoning divides my country into seven regions based on frozen soil, water and heat balance, and geographical location;

[0096] Step 2: Input the visual information and non-visual information of the road surface to be tested into the road surface state recognition model to obtain the road surface state;

[0097] The road surface condition recognition model is obtained by:

[0098] A1. Build a dataset and divide it into training and test sets in a ratio of 7:3:

[0099] The dataset includes: visual information, non-visual information and labels;

[0100] The label is the road surface status;

[0101] The road surface conditions include: dry, wet, waterlogged, icy and snowy;

[0102] If there is no water film in the road surface image, the road surface state is dry; if there is a water film in the road surface image but the water film is discontinuous, the road surface state is wet; if there is a continuous water film in the road surface image, the road surface state is waterlogged;

[0103] Different road surface conditions may exist on the road at the same time. When constructing the dataset, images with controversial road surface conditions are deleted and only a single road surface condition is retained for annotation.

[0104] The thickness of the water film on the road surface is related to meteorological conditions and road drainage capacity. When the water film thickness reaches 1mm to 4mm, the adhesion coefficient between the tire and the road surface is mainly determined by fluid viscosity. A continuous water film may form between the tread and the road surface, and even hydroplaning may occur. Since the thickness of the water film cannot be measured in the collected images, the visual appearance of the image is used to classify it: a wet road surface has moisture but no continuous water film, that is, no visible liquid water layer; a flooded road surface has a continuous water film, and the waterlogged area appears white and reflective. When both wet and flooded roads are present in the image, the waterlogged road surface is used as the road surface status.

[0105] A2. Use the training set and test set to train and test the recognition model to obtain a road condition recognition model;

[0106] like Figure 2 As shown, the recognition model includes: a first convolutional layer, a temperature data processing unit, a first splicing layer, a second convolutional layer, a first CN_V2 Block unit, a first downsampling layer, a second CN_V2 Block unit, a second downsampling layer, a third CN_V2 Block unit, a third downsampling layer, a fourth CN_V2 Block unit, a fourth downsampling layer, other non-visual information processing units, a second splicing layer, and a third convolutional layer;

[0107] The first convolutional layer is a convolutional layer with a convolution kernel of 4×4 and a stride of 4, and is used to extract a visual information feature map; the input of the first convolutional layer is visual information; the size of the visual information feature map is one-fourth the size of the visual information image;

[0108] The temperature data processing unit is used to process the temperature data, construct a temperature characteristic graph, and perform channel expansion on the temperature characteristic graph to obtain a first temperature characteristic graph;

[0109] The temperature data processing unit includes: a first normalization layer and a first channel expansion layer;

[0110] The first normalization layer normalizes the surface temperature and the air temperature to between 0 and 1, respectively, to obtain a temperature characteristic map composed of the normalized surface temperature and air temperature;

[0111] The first channel expansion layer is used to expand the number of channels of the temperature feature map to be consistent with the number of channels of the visual information feature map, thereby obtaining a first temperature feature map; the input of the first channel expansion layer is the output of the first normalization layer;

[0112] The first splicing layer is used to splice the first temperature feature map and the visual information feature map to obtain a first splicing result; the input of the first splicing layer is the output of the first convolutional layer and the output of the temperature data processing unit;

[0113] The second convolutional layer is a 1×1 convolution, and its input is the first concatenation result, which is used to adjust the number of channels;

[0114] The first CN_V2 Block unit includes three CN_V2 Block subunits; the input of the first CN_V2 Block unit is the output of the second convolutional layer;

[0115] The output of the first CN_V2 Block subunit is used as the input of the second CN_V2 Block subunit, and the output of the second CN_V2 Block subunit is used as the input of the third CN_V2 Block subunit. The three CN_V2 Block subunits have the same structure. The input of the first CN_V2 Block unit is the output of the second convolutional layer.

[0116] The CN_V2 Block subunit includes: the fourth convolutional layer, the second normalization layer, the fifth convolutional layer, the activation function layer, the global response normalization layer, and the sixth convolutional layer;

[0117] The fourth convolutional layer is a depthwise convolution with a convolution kernel of 7×7;

[0118] The second normalization layer is used to perform LN normalization processing on the output of the fourth convolutional layer;

[0119] The fifth convolutional layer is a convolution with a convolution kernel of 1×1; the input of the fifth convolutional layer is the output of the second normalization layer;

[0120] The activation function layer is a GELU activation function, and the input of the activation function layer is the output of the fifth convolutional layer;

[0121] The global response normalization layer is used to perform GRN processing on the output of the activation function layer to enhance feature competition between channels;

[0122] The sixth convolutional layer is a convolution with a convolution kernel of 1x1, and an input of the sixth convolutional layer is an output of the global response normalization layer.

[0123] The first down-sampling layer is a convolutional layer with a convolution kernel of 2x2 and a stride of 2, used to change the feature map size to half of the original size and the depth to twice the original depth; an input of the first down-sampling layer is an output of the first CN_V2 Block unit.

[0124] The second CN_V2 Block unit includes three CN_V2 Block sub-units.

[0125] The third CN_V2 Block unit includes nine CN_V2 Block sub-units.

[0126] The fourth CN_V2 Block unit includes three CN_V2 Block sub-units.

[0127] The second sampling layer, the third sampling layer, and the fourth sampling layer are the same as the first down-sampling layer.

[0128] An input of the second CN_V2 Block unit is an output of the first down-sampling layer, an output of the second CN_V2 Block unit is an input of the second down-sampling layer, an output of the second down-sampling layer is an input of the third CN_V2 Block unit, an output of the third CN_V2 Block unit is an input of the third down-sampling layer, an output of the third down-sampling layer is an input of the fourth CN_V2 Block unit, and an output of the fourth CN_V2 Block unit is an input of the fourth down-sampling layer.

[0129] The other non-visual information processing unit is used to process non-visual information to obtain a geographical position feature map, a second temperature feature map, and a vehicle speed feature map.

[0130] The other non-visual information processing unit includes a one-hot encoding layer, a third normalization layer, a second channel expansion layer, and a third channel expansion layer.

[0131] The one-hot encoding layer is used to one-hot encode geographical position information to obtain a geographical position feature map with a channel of 7.

[0132] The third normalization layer is used to normalize vehicle speed information, temperature information (ground temperature, air temperature) to 0-1 respectively, to obtain a vehicle speed feature map with a channel number of 1, a temperature feature map with a dimension of 2, and a vehicle speed feature map with a dimension of 1.

[0133] The second channel expansion layer is used to expand the dimension of the geographical position feature map, so that the dimension of the geographical position feature map is the same as the dimension of the output feature map of the fourth down-sampling layer.

[0134] The third channel expansion layer is used to expand the vehicle speed feature map and the temperature feature map, so that the vehicle speed feature map and the temperature feature map have the same dimension as the output feature map of the fourth down-sampling layer.

[0135] The second splicing layer is used to splice the geographical position feature map, the second temperature feature map, the vehicle speed feature map and the output of the fourth down-sampling layer to obtain a second splicing result.

[0136] The third convolution layer is a convolution with a convolution kernel of 1x1, which is used to adjust the channel number of the second splicing result to obtain the road surface state.

[0137] The splicing layer in this step refers to channel splicing, which is implemented by using a contact function.

[0138] Step three, according to the road surface state and the to-be-measured road surface temperature, the mean and variance of the road surface adhesion coefficient estimate value are obtained, specifically:

[0139] Step three one, the road surface state is judged, if the road surface state is dry, wet or water accumulation, step three two is executed, if the road surface state is snow accumulation or icing, the mean and variance of the road surface adhesion coefficient estimate value are directly obtained, specifically:

[0140] If the road surface state is snow accumulation, the mean of the road surface adhesion coefficient estimate value is 0.400, and the variance of the adhesion coefficient estimate value is 0.0500;

[0141] If the road surface state is icing, the mean of the road surface adhesion coefficient estimate value is 0.175, and the variance of the adhesion coefficient estimate value is 0.0375.

[0142] Step three two, if the road surface state is dry, wet or water accumulation, the mean of the road surface adhesion coefficient estimate value is corrected on the basis of the mean of the original road surface adhesion coefficient estimate value, and then the variance of the adhesion coefficient estimate value is obtained.

[0143] The variance of the adhesion coefficient estimate value and the mean of the original road surface adhesion coefficient estimate value are obtained by the following method:

[0144] If the road surface state is dry, the mean of the original road surface adhesion coefficient estimate value is 0.650, and the variance of the adhesion coefficient estimate value is 0.0750;

[0145] If the road surface state is wet, the mean of the original road surface adhesion coefficient estimate value is 0.575, and the variance of the adhesion coefficient estimate value is 0.0625;

[0146] If the road surface state is water accumulation, the mean of the original road surface adhesion coefficient estimate value is 0.450, and the variance of the adhesion coefficient estimate value is 0.0750;

[0147] The mean of the estimated road adhesion coefficient is corrected based on the mean of the original estimated road adhesion coefficient, specifically:

[0148] u j =u mj +Δu

[0149] Among them, u j is the corrected mean of the estimated road adhesion coefficient, Δu is the corrected value of the mean of the estimated road adhesion coefficient;

[0150] The road adhesion coefficient is related to the surface temperature. Within a certain range, the road adhesion coefficient decreases as the surface temperature increases. The Δu is specifically obtained by the following method:

[0151] If T≤0℃, then Δu=0.05;

[0152] If 0℃<T≤5℃, then Δu=0.04;

[0153] If 5℃<T≤10℃, then Δu=0.03;

[0154] If 10℃<T≤15℃, then Δu=0.02;

[0155] If 15℃<T≤20℃, then Δu=0.01;

[0156] If 20℃<T≤25℃, then Δu=0;

[0157] If 25℃<T≤30℃, then Δu=-0.01;

[0158] If 30℃<T≤35℃, then Δu=-0.02;

[0159] If T>35℃, then Δu=-0.03;

[0160] Where T is the surface temperature of the road to be measured;

[0161] The identification results of the present invention are as follows Figure 4 As shown in the figure, the probability density distribution of the road adhesion coefficient under different road conditions is different, such as Figure 5 shown.

[0162] Specific implementation method 2: Figure 3 As shown, the multi-modal road adhesion coefficient estimation system includes: a power supply 300, a processor 310, a temperature sensor 320, a GPS locator 330, an RTK measurer 340 and a camera 350;

[0163] The temperature sensor 320 is used to collect road surface temperature and air temperature; the output interface of the temperature sensor is connected to the first input interface of the processor 310;

[0164] The GPS locator is used to obtain the current geographical position of the vehicle; the output interface of the GPS locator is connected to the second input interface of the processor 310;

[0165] The RTK measurer 340 is used to obtain the current speed of the vehicle; the output interface of the RTK measurer 340 is connected to the third input interface of the processor 310;

[0166] The camera 350 is used to obtain the image in front of the vehicle; the output interface of the camera 350 is connected to the fourth input interface of the processor;

[0167] The processor 310 is used to run a multi-modal based road surface adhesion coefficient estimation method;

[0168] The power supply 350 is used to provide electric energy; the output interface of the power supply is connected to the input interface of the processor 310.

[0169] At present, AlexNet, ResNet, MobileNet V3, EfficientNet V2, ConvNeXtV2, ShuffleNetV2, Swin-Transformer, Swin-Transformer V2, DaViT or MobileViT V3 are used as road surface recognition models in the prior art, cross-entropy loss and label smoothing technology are adopted, and images and labels are learned. The model is converted into TensorRT format, and inference is accelerated using CUDA, but this method has limitations of the camera itself, is difficult to be associated with the change mechanism of the road surface adhesion coefficient, and is easily affected by the shooting environment. In order to improve the accuracy of the model and enhance the robustness of the model in extreme environments, the road surface recognition model of the present application is proposed.

Claims

1. A multi-modal road adhesion coefficient estimation method, characterized by The specific process of the method is: Step 1: Obtain visual information and non-visual information of the road surface to be tested; The visual information is an image of the road surface in front of the test vehicle; The non-visual information includes: temperature, test vehicle speed and geographic location; The temperature includes: air temperature and ground surface temperature; The geographical locations include: the northern permafrost zone, the eastern warm seasonal frozen zone, the Loess Plateau dry-wet transition zone, the southeastern hot and humid zone, the southwestern warm and humid zone, the northwestern arid zone, and the Qinghai-Tibet Plateau cold zone; Step 2: Input the visual information and non-visual information of the road surface to be tested into the road surface state recognition model to obtain the road surface state; Step 3: Obtain the mean and variance of the estimated value of the road adhesion coefficient according to the road surface state and the road surface temperature to be measured.

2. The multimodal road adhesion coefficient estimation method according to claim 1, characterized in that: The road surface condition recognition model is obtained by: A1. Build a dataset and divide it into a training set and a test set. The dataset includes: visual information, non-visual information and labels; The label is the road surface status; The road surface conditions include: dry, wet, waterlogged, icy and snowy; A2. Use the training set and test set to train and test the recognition model to obtain a road condition recognition model.

3. The multimodal road adhesion coefficient estimation method according to claim 2, characterized in that: The recognition model includes: a first convolutional layer, a temperature data processing unit, a first splicing layer, a second convolutional layer, a first CN_V2Block unit, a first downsampling layer, a second CN_V2Block unit, a second downsampling layer, a third CN_V2Block unit, a third downsampling layer, a fourth CN_V2Block unit, a fourth downsampling layer, other non-visual information processing units, a second splicing layer, and a third convolutional layer; The first convolutional layer is a convolutional layer with a convolution kernel of 4×4 and a stride of 4, which is used to extract a visual information feature map; The input of the first convolutional layer is visual information; The temperature data processing unit is used to process the temperature data, construct a temperature characteristic graph, and perform channel expansion on the temperature characteristic graph to obtain a first temperature characteristic graph; The first splicing layer is used to splice the first temperature feature map and the visual information feature map to obtain a first splicing result; the input of the first splicing layer is the output of the first convolutional layer and the output of the temperature data processing unit; The second convolutional layer is a 1×1 convolution, and its input is the first concatenation result; The first CN_V2Block unit includes three CN_V2Block subunits; the input of the first CN_V2Block unit is the output of the second convolutional layer; The output of the first CN_V2Block subunit serves as the input of the second CN_V2Block subunit, and the output of the second CN_V2Block subunit serves as the input of the third CN_V2Block subunit. The three CN_V2Block subunits have the same structure. The first downsampling layer is a convolution layer with a convolution kernel of 2×2 and a stride of 2; the input of the first downsampling layer is the output of the first CN_V2Block unit; The second CN_V2Block unit includes three CN_V2Block subunits; The third CN_V2Block unit includes 9 CN_V2Block subunits; The fourth CN_V2Block unit includes three CN_V2Block subunits; The second sampling layer, the third sampling layer, and the fourth sampling layer are the same as the first downsampling layer; The input of the second CN_V2Block unit is the output of the first downsampling layer, the output of the second CN_V2Block unit is the input of the second downsampling layer, the output of the second downsampling layer is the input of the third CN_V2Block unit, the output of the third CN_V2Block unit is the input of the third downsampling layer, the output of the third downsampling layer is the input of the fourth CN_V2Block unit, and the output of the fourth CN_V2Block unit is the input of the fourth downsampling layer; The other non-visual information processing unit is used to process the non-visual information to obtain a geographic location feature map, a second temperature feature map, and a vehicle speed feature map; The second splicing layer is used to splice the geographic location feature map, the second temperature feature map, the vehicle speed feature map and the output of the fourth downsampling layer to obtain a second splicing result; The convolution kernel of the third convolution layer is a 1×1 convolution, the input is the second splicing result, and the output is the road surface state.

4. The multimodal road adhesion coefficient estimation method according to claim 3, characterized in that: The temperature data processing unit includes: a first normalization layer and a first channel expansion layer; The first normalization layer normalizes the surface temperature and the air temperature to between 0 and 1, respectively, to obtain a temperature characteristic map composed of the normalized surface temperature and air temperature; The first channel expansion layer is used to expand the number of channels of the temperature feature map so that the number of channels of the temperature feature map is consistent with the number of channels of the road image, thereby obtaining a first temperature feature map; the input of the first channel expansion layer is the output of the first normalization layer.

5. The multimodal road adhesion coefficient estimation method according to claim 4, characterized in that: The CN_V2Block subunit includes: a fourth convolutional layer, a second normalization layer, a fifth convolutional layer, an activation function layer, a global response normalization layer, and a sixth convolutional layer; The fourth convolutional layer is a convolutional layer with a convolution kernel of 7×7; The second normalization layer is used to normalize the output of the fourth convolutional layer; The fifth convolutional layer is a convolution with a convolution kernel of 1×1; the input of the fifth convolutional layer is the output of the second normalization layer; The activation function layer is a GELU activation function, and the input of the activation function layer is the output of the fifth convolutional layer; The global response normalization layer is used to perform global response normalization processing on the output of the activation function layer; The sixth convolutional layer is a convolution with a convolution kernel of 1×1, and the input of the sixth convolutional layer is the output of the global response normalization layer.

6. The multi-modal road adhesion coefficient estimation method according to claim 5, characterized in that: The other non-visual information processing units include: a one-hot encoding layer, a third normalization layer, a second channel expansion layer, and a third channel expansion layer; The one-hot encoding layer is used to perform one-hot encoding on the geographic location information to obtain a geographic location feature map; The third normalization layer is used to normalize the vehicle speed information, surface temperature, and air temperature to between 0 and 1, respectively, to obtain a temperature characteristic map and a vehicle speed characteristic map; The temperature characteristic map includes: surface temperature and air temperature; The second channel expansion layer is used to expand the dimension of the geographic location feature map so that the dimension of the geographic location feature map is the same as the dimension of the feature map output by the fourth downsampling layer; The third channel expansion layer is used to expand the vehicle speed feature map and the temperature feature map so that the dimensions of the vehicle speed feature map and the temperature feature map are the same as the output feature map of the fourth downsampling layer.

7. The multimodal road adhesion coefficient estimation method according to claim 6, characterized in that: In step 3, the mean and variance of the estimated road adhesion coefficient are obtained based on the road surface state and the road surface temperature to be measured, specifically: Step 31: Determine the road surface condition. If the road surface condition is dry, wet, or wet, execute step 32. If the road surface condition is snowy or icy, directly obtain the mean and variance of the estimated road adhesion coefficient. Step 32: correcting the mean of the estimated road adhesion coefficient based on the mean of the original estimated road adhesion coefficient, and then obtaining the variance of the estimated adhesion coefficient; The method of correcting the mean of the estimated road adhesion coefficient based on the mean of the original estimated road adhesion coefficient is as follows: u j =u mj +Δu Among them, u j is the corrected mean of the estimated road adhesion coefficient, Δu is the corrected value of the mean of the estimated road adhesion coefficient; The mean correction value of the estimated road adhesion coefficient is obtained according to the surface temperature, specifically: If T≤0℃, then Δu=0.05; If 0℃<T≤5℃, then Δu=0.04; If 5℃<T≤10℃, then Δu=0.03; If 10℃<T≤15℃, then Δu=0.02; If 15℃<T≤20℃, then Δu=0.01; If 20℃<T≤25℃, then Δu=0; If 25℃<T≤30℃, then Δu=-0.01; If 30℃<T≤35℃, then Δu=-0.02; If T>35℃, then Δu=-0.03; Where T is the surface temperature of the road to be measured.

8. The multi-modal road adhesion coefficient estimation method according to claim 7, characterized in that: If the road surface condition in step 31 is snow or ice, the mean and variance of the estimated road adhesion coefficient are directly obtained, specifically: If the road condition is snow, the mean of the estimated road adhesion coefficient is 0.400, and the variance of the estimated adhesion coefficient is 0.0500; If the road condition is icy, the mean of the estimated road adhesion coefficient is 0.175, and the variance of the estimated adhesion coefficient is 0.0375.

9. The multimodal road adhesion coefficient estimation method according to claim 8, characterized in that: The mean of the estimated values ​​of the original road adhesion coefficient in step 32 is specifically: If the road surface condition is dry, the mean of the estimated original adhesion coefficient of the road surface is 0.650, and the variance of the estimated adhesion coefficient is 0.0750; If the road surface condition is wet, the mean of the estimated value of the original adhesion coefficient of the road surface is 0.575, and the variance of the estimated value of the adhesion coefficient is 0.0625; If the road surface condition is waterlogged, the mean of the estimated original adhesion coefficient of the road surface is 0.450, and the variance of the estimated adhesion coefficient is 0.0750.

10. A multimodal road adhesion coefficient estimation system, characterized by: The system includes: a power supply, a processor, a temperature sensor, a GPS locator, an RTK measurer and a camera; The temperature sensor is used to collect road surface temperature and air temperature; the temperature sensor is connected to the processor; The GPS locator is used to obtain the geographic location; the GPS locator is connected to the processor; The RTK measurer is used to obtain the speed of the test vehicle; the RTK measurer is connected to the processor; The camera is used to obtain an image in front of the test vehicle; the camera is connected to the processor; The processor is used to estimate the road adhesion coefficient; The power supply is used to provide electrical energy, and the power supply is connected to the processor.

Images