A method for estimating a road adhesion coefficient in a road passable area based on machine vision and an electronic device

Abstract

The application discloses a kind of estimation method and electronic equipment of road passable area inside road surface adhesion coefficient based on machine vision, first according to semantic segmentation, road surface is divided into drivable area, and road drivable area dataset is obtained;Then the depth estimation of road drivable area is obtained, and grid division is carried out;Secondly, according to the lightweight classification network, each grid is classified, the category corresponding to the maximum number is selected as the final prediction result, and the adhesion coefficient corresponding thereto is estimated.The application does not depend on accurate dynamics model and the excitation between road surface and tire, and the road surface type is identified according to camera and the adhesion coefficient corresponding thereto is estimated with higher precision, which is very important for improving the performance of active safety control system and improving the driving safety of vehicle.

Description

Technical Field

[0001] This invention belongs to the field of road surface adhesion coefficient estimation, and particularly relates to a method and electronic device for estimating the road surface adhesion coefficient within the drivable area of ​​an intelligent vehicle. Background Technology

[0002] The coefficient of friction (COP) describes the maximum interaction force between the tire and the road surface, directly affecting vehicle braking performance, dynamics, and handling stability. Accurate and rapid identification of the COP not only optimizes the control effectiveness of intelligent vehicle active safety control systems but also improves the decision-making and planning performance of intelligent driving technologies.

[0003] The essence of road surface adhesion coefficient (LPC) identification is to directly or indirectly obtain the LPC between the tire and the road surface by observing changes in parameters such as tire force, tire slip ratio, and sideslip angle. Existing LPC identification methods are divided into experiment-based and model-based methods. Experiment-based LPC estimation relies on high-precision vision and position sensors, and can effectively identify LPC when tire force is very low, but it is costly and difficult to test. Compared with experimental methods, estimation methods based on tire models and dynamic models only utilize existing low-cost sensors on the vehicle to observe and infer the dynamic response of the whole vehicle and wheels caused by changes in road surface adhesion conditions, thus achieving real-time observation of the LPC. While model-based LPC estimation methods can accurately estimate the LPC at the current vehicle position, under normal driving conditions with low longitudinal or lateral tire excitation, the estimation results may have large errors and a certain lag, and they cannot predict the LPC at a distance. Vision-based LPC estimation has a certain predictive ability and does not rely on precise dynamic models or road-tire excitation. Moreover, identifying road surface types and estimating the corresponding LPC based on cameras has high accuracy. Summary of the Invention

[0004] To address the shortcomings of existing road surface adhesion coefficient estimation methods, this invention proposes a machine vision-based method for estimating road surface adhesion coefficients within drivable areas, which can accurately identify complex road conditions. First, the drivable area of ​​the road surface is segmented using semantic segmentation to obtain a road drivable area dataset. Then, depth estimation is performed on the drivable area to obtain depth information, and grid division is carried out. Next, each grid is classified using a lightweight classification network, and the category corresponding to the largest number of occurrences is selected as the final prediction result, and the corresponding adhesion coefficient is estimated. This method has significant theoretical research value for improving the performance of active safety control systems and enhancing vehicle driving safety.

[0005] The technical steps of the estimation method of the present invention are as follows:

[0006] Step 1: Delineation of the drivable area of ​​the road

[0007] The drivable area refers to the road surface area where all obstacles have been removed to ensure safe driving of vehicles. Accurate and real-time drivable area segmentation results can greatly enhance the environmental perception capabilities of intelligent driving vehicles within the drivable area, thereby providing necessary information for the planning and decision-making layers and the motion execution layer. Since there is currently no unified standard for classifying drivable areas, this invention determines a drivable area classification standard based on road material according to the needs of actual operational scenarios.

[0008] 1.1 Dataset Construction and Preprocessing

[0009] The Cityscapes dataset is a collection of street scenes from 50 cities in Germany, France, and Switzerland. The dataset contains 5000 frames with pixel-level annotations, divided into a training set of 2975 frames, a validation set of 500 frames, and a test set of 1525 frames. The pixel annotations are categorized into 19 classes, including cars, trains, buses, pedestrians, sky, and ground.

[0010] Because the aforementioned datasets suffer from insufficient quantity and suboptimal quality, failing to meet the data diversity requirements for model training, augmentation operations are necessary to increase the training data size. The data augmentation strategies used in this invention include: random scaling and cropping of the training data, horizontal flipping, and color changes; randomly altering the grayscale values ​​of the training data; and randomly adding small Gaussian noise to each pixel.

[0011] 1.2 Driving Region Segmentation Algorithm Based on Deeplabv3+

[0012] Deeplabv3+ is one of the most outstanding network frameworks for object segmentation tasks. Its biggest innovation is the introduction of dilated convolution and the design of a multi-aperture spatial pyramid pooling module (ASPP) based on dilated convolution. The ASPP module is used to generate multi-scale feature maps in parallel to achieve multi-scale object segmentation in the segmentation scene.

[0013] The specific structure of Deeplabv3+ based on the ResNet50 feature extraction module is as follows: Figure 2As shown. Assuming the initial input image size for DeepLabv3+ is 416×416×3, during the encoding stage, a ResNet50 network based on deformable convolution is used for feature extraction. The output of the last layer is then processed in parallel by the ASPP module to obtain multiple feature maps of different scales. The five feature map channels of the ASPP network output are merged along the channel direction, resulting in a feature map size of 15×15×512. Instead of layer-by-layer upsampling, the merged feature map undergoes an eight-fold upsampling and merging operation with lower-level feature channels to recover the boundary information of the target area. Finally, after convolution with a 3×3 kernel, it is upsampled four times to obtain the final segmentation result. Linear interpolation methods are used throughout the upsampling process.

[0014] Semantic segmentation is a pixel-by-pixel classification task in an image, so the loss function consists only of the classification loss function, L. seg Choose to calculate the cross-entropy loss, and the calculation formula is as follows:

[0015]

[0016] Where s(c,x,y) represents whether the pixel (x,y) belongs to category c; if it does, its value is 1, otherwise it is 0. n represents the probability that pixel (x, y) is predicted to be of class c; c is the number of segmentation categories; h and w are the length and width of the output prediction map, respectively.

[0017] The overall framework of the drivable region segmentation algorithm is designed based on DeepLabv3+. The backbone feature extraction network is based on ResNet50 with deformable convolution, and the encoder-decoder structure of DeepLabv3+ is optimized. After training on the Cityscapes training set, the model's road drivable region segmentation results are evaluated.

[0018] 1.4 Extraction of drivable road surface areas

[0019] The location of different semantic categories in the drivable area segmentation result image is clearly visible. The RGB value corresponding to the road surface area can be obtained by looking up the semantic category label in Table 1. This RGB value is regarded as the Region of Interest (ROI). The road surface area can be extracted by using the mask in digital image processing methods.

[0020] Table 1 Semantic Category Labels

[0021]

[0022] The RGB color standard is a commonly used image color system, consisting of three color channels: red (R), green (G), and blue (B). By varying and superimposing these three channels, a wide variety of colors, encompassing almost all colors perceptible to humans, can be produced. Each color channel has a value range of 0-255, which can be represented by 8 bits in computer language, making the entire image 24-bit in depth.

[0023] To retain only the road surface area in the original image, a mask needs to be created based on the semantic segmentation prediction result image. The ROI is then obtained by performing matrix operations between the mask and the original image. First, the semantic segmentation prediction result image is sliced ​​to obtain a two-dimensional matrix with three color channels. In OpenCV, these three channels correspond to BGR (Browser Gradients). The code implementation is as follows:

[0024] blue=prediction_image[:,:,0]

[0025] green=prediction_image[:,:,1]

[0026] red = prediction_image[:,:,2]

[0027] Then, look up the table to find the BGR value (128, 64, 128) corresponding to "road". Create a mask for the blue channel, where the mask value is 1 for pixels with a value of 128 in the prediction result image, and 0 for all other pixels. The code implementation is as follows:

[0028] mask_blue=np.empty(blue.shape).astype(np.uint8)

[0029] mask_blue[np.where(blue==128)]=1

[0030] mask_blue[np.where(blue!=128)]=0

[0031] Next, masks for the green and red channels are created sequentially, retaining only pixels where all three channels have a value of 1. When a pixel in one channel has a value of 0, the corresponding pixels in other channels are set to zero, ensuring that only the road surface area is extracted. Finally, the masks for the three color channels are stitched together, and matrix multiplication is performed with the corresponding channels of the original image to extract an image containing only the road surface area, while non-road surface areas are turned black.

[0032] The drivable region extraction effect after semantic segmentation and masking of images in the self-built database is as follows: Figure 3As shown, the semantic segmentation network was able to extract most of the effective road surface areas, removing background information. The overall road surface area extraction results are satisfactory, but there are still shortcomings. Because the semantic segmentation network was trained using the Cityscapes dataset instead of a self-built dataset, the trained model is not entirely applicable. Sometimes, fewer road surface areas are retained, or some non-road surface areas remain, requiring further filtering.

[0033] Step 2: Classify the types of drivable road areas.

[0034] 2.1 Self-supervised monocular depth estimation algorithm

[0035] This invention employs a self-supervised monocular depth estimation network. Based on the classic DispNet network structure, it uses an Encoder-Decoder architecture as the main framework, improving and upgrading the Encoder module to adapt to real-time monocular depth estimation tasks in complex scenes. Depth-separable convolutions and pointwise convolutions replace the basic modules in ResNet. The input image is scaled three times proportionally to form image pyramids of four sizes. The original input image outputs disparity maps of different sizes at different Decoder stages. Multi-scale optimization is achieved by constructing a multi-level loss function between the two to adapt to complex scenes with objects of varying sizes. Channel and spatial feature extraction mechanisms are designed between the Encoder and Decoder to expand the receptive field and enhance feature extraction. The decoder uses a combination of sub-pixel convolutional layers and bilinear interpolation for high-resolution upsampling, and uses low-lever features from the Encoder to skip connections to the Decoder to maintain higher resolution details. The self-supervised monocular depth estimation network structure diagram is shown below. Figure 4 As shown.

[0036] The segmented road surface image is divided into grids based on depth information. The initial grid size is m1, m2, a < m1 < b, a < m2 < b, where a and b are the dimensions of the road surface within the feasible area, and both a and b are smaller than the minimum size of the road surface image.

[0037] 2.2 Road Surface Image Classification Network

[0038] After processing with a semantic segmentation network and a mask, the original image sample set yields an image set containing only drivable areas. This will serve as the final dataset for training and evaluating the road surface type classification and recognition network. Considering that the preprocessed road surface images no longer contain other background information and that the features of different road surface types are relatively clear and distinguishable, the classification and recognition task is not complex. Therefore, the ShuffleNet V2 network in Table 2 is further simplified. The basic framework of the network is retained, and the model complexity is reduced by appropriately deleting some repeated (Repeat, see Table 2) basic units (stride=1) in Stage 2, Stage 3, and Stage 4. Finally, multiple experiments show that the network model accuracy after reducing Repeat to 1 still meets the classification accuracy requirements.

[0039] In Anaconda's Spyder programming software, a simplified ShuffleNet V2 network model was built. The network input image size was 224×224 pixels, the model channel scaling ratio was set to 1×, the batch size was set to batch_size=64, the softmax_cross_entropy loss function was selected (see formulas (2-1) and (2-2), the Adam adaptive gradient descent optimization algorithm was used, the base learning rate was 0.0001, the network training was started, and the model and training results were saved according to the number of iterations per epoch.

[0040]

[0041]

[0042] Where, x i Let y be the eigenvalue of the i-th class. i Let y be the probability value of the i-th class, T be the total number of classes, and y be the probability value of the i _ represents the actual value

[0043] Table 2 Overall Structure of ShuffleNet V2 Mesh

[0044]

[0045] Each grid is classified to obtain the road surface condition type for each grid. The road surface condition types include dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement, and ice film pavement.

[0046] The confidence scores of each grid's pavement condition type are fused. Specifically, the weights of each pavement condition type—dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement, and ice film pavement—are set to ω. dry-a ,ωwet-a ,ω dry-c ,ω wet-c ,ω brick ,ω snow-c ,ω snow-p ,ω ice The values ​​range from 0 to 1, and are updated according to the weight update strategy. Based on the updated weights of each pavement state type, a basic probability allocation function for the pavement state type of each candidate region grid is constructed. The basic probability allocation function for the pavement state type of each candidate region grid is fused to obtain the fused confidence scores for pavement state types such as dry asphalt pavement, wet and slippery asphalt pavement, and icy and snowy asphalt pavement. The specific weight update strategy is as follows:

[0047] α=k×n (2-1)

[0048]

[0049]

[0050]

[0051]

[0052] Where α is the weight update factor, k is the weight update rate constant, and n is the total number of grids in the segmented candidate region. r The number of grid cells with the highest confidence level for the r-th road surface type obtained from the classification is... This is the first intermediate variable for the r-th road surface state type in the current frame. This is the first intermediate variable for the r-th road surface state type in the previous frame. This is the second intermediate variable for the r-th road surface state type in the current frame. The weight of the r-th road surface state type in the previous frame is given by e, where e is a natural constant. Let r be the weight of the updated r-th state type. The probability assignment function is as follows:

[0053]

[0054]

[0055] Where, ω dry-a ,ω wet-a ,ω dry-c ,ω wet-c ,ω brick ,ω snow-c ,ω snow-p ,ω ice The weights for road surface condition types are as follows: dry asphalt road surface, wet asphalt road surface, dry cement road surface, wet cement road surface, paved road surface, solid snow road surface, loose snow road surface, and icy road surface. Represent The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement state type in the grid of dry asphalt pavement. The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement condition type in the grid of wet asphalt pavement. To determine the maximum confidence level obtained from the classification, P is the confidence level corresponding to the r-th pavement state type in the dry cement pavement grid. l r The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement state type in the wet cement pavement grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th pavement state type in the pavement grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th road surface state type in the actual snow road surface grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th road surface state type in the snowy road surface grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th pavement state type in the ice film pavement grid, n dry-a ,n wet-a ,n dry-c ,n wet-c ,n brick ,n snow-c ,n snow-p ,n ice The maximum confidence level obtained from the classification is determined by the number of grid cells corresponding to dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement, and icy pavement. This is a temporary intermediate variable for the basic assignment function of the r-th road surface state type. The basic probability assignment function for the r-th road surface state type is constructed; the fusion strategy is as follows:

[0056]

[0057]

[0058] Where K is the sum of the basic probability assignment functions for each pavement state type in the nth grid, and m n (r n Let ) be the basic probability assignment function for the r-th road surface condition type in the n-th grid, and m(r) be the basic probability assignment function for the r-th road surface condition type after fusion.

[0059] The confidence scores of the road surface condition type for each grid are fused, and the category corresponding to the maximum number is selected as the final prediction result, denoted as index.

[0060] Step 3: Mapping between road surface type and road surface adhesion coefficient

[0061] The result of image recognition networks is road surface type, while the research objective of this invention is to obtain the road surface adhesion coefficient. Therefore, it is necessary to provide a correspondence between road surface type and adhesion coefficient. Considering that the road surface adhesion coefficient is related to factors such as road surface wear, tire wear, and air temperature and humidity, and that the range of road surface adhesion coefficient values ​​varies at different vehicle speeds, the design of the road surface type and adhesion coefficient comparison table should take into account the influence of various uncertainties on the road surface adhesion coefficient. For conservatism, the range of adhesion coefficient values ​​corresponding to different road surface types is appropriately expanded, and two cases are distinguished: high-speed driving and low-speed driving. Finally, the comparison table of different road surface types and adhesion coefficient range values ​​is shown in Table 3.

[0062] Table 3 Comparison of Adhesion Coefficient Range Values ​​for Different Road Surface Types and Speeds

[0063]

[0064] Based on the road surface condition type obtained through fusion, the range of road adhesion coefficient values ​​corresponding to that road surface type is obtained by referring to Table 3.1. The lower limit is denoted as 'a', and the upper limit is denoted as 'b'. The average value is taken as the estimated value of the adhesion coefficient corresponding to different road surface types. Using the matrix [index, a, b, μ] e The road surface type classification and identification results data are stored in the form of ]

[0065] The present invention also proposes an electronic device capable of performing the above-described method.

[0066] The beneficial effects of this invention are:

[0067] 1. This invention is based on machine vision for estimating the road surface adhesion coefficient within the drivable area. It has a certain predictive ability and does not rely on an accurate dynamic model or the excitation between the road surface and the tire. Moreover, it has high accuracy in identifying the road surface type and estimating the corresponding adhesion coefficient based on the camera.

[0068] 2. The present invention first segments the drivable area, eliminating the influence of non-drivable areas on the classification and recognition task, and obtains a road surface type recognition network with high accuracy and high computational efficiency.

[0069] 3. The drivable area is divided into grids based on depth information. Each grid is classified, and the category corresponding to the maximum number is selected as the final prediction result. This eliminates the interference of environmental factors such as shadows and improves the accuracy of road surface type identification. Attached Figure Description

[0070] Figure 1This is a schematic diagram of the road surface adhesion coefficient estimation method of the present invention.

[0071] Figure 2 The Deeplabv3+ architecture based on the ResNet50 feature extraction module is shown in the diagram.

[0072] Figure 3 Extracting renderings of drivable areas

[0073] Figure 4 Diagram of the self-supervised monocular depth estimation network. Detailed Implementation

[0074] The invention will now be further described with reference to the accompanying drawings.

[0075] Figure 1 This is a schematic diagram of the road surface adhesion coefficient estimation method of the present invention, which includes the following steps:

[0076] Step 1: Delineation of the drivable area of ​​the road

[0077] The drivable area refers to the road surface area where all obstacles have been removed to ensure safe driving of vehicles. Accurate and real-time drivable area segmentation results can greatly enhance the environmental perception capabilities of intelligent driving vehicles within the drivable area, thereby providing necessary information for the planning and decision-making layers and the motion execution layer. Since there is currently no unified standard for classifying drivable areas, this invention determines a drivable area classification standard based on road material according to the needs of actual operational scenarios.

[0078] 1.1 Dataset Construction and Preprocessing

[0079] The Cityscapes dataset is a collection of street scenes from 50 cities in Germany, France, and Switzerland. The dataset contains 5000 frames with pixel-level annotations, divided into a training set of 2975 frames, a validation set of 500 frames, and a test set of 1525 frames. The pixel annotations are categorized into 19 classes, including cars, trains, buses, pedestrians, sky, and ground.

[0080] Because the aforementioned datasets suffer from insufficient quantity and poor quality, failing to meet the data diversity requirements for model training, augmentation operations are needed to increase the size of the training data. The data augmentation strategies of this invention include: randomly scaling and cropping the training data, horizontally flipping it, and changing its color; randomly altering the grayscale values ​​of the training data; and randomly adding small Gaussian noise to each pixel.

[0081] 1.2 Driving Region Segmentation Algorithm Based on Deeplabv3+

[0082] Based on Deeplabv3+, dilated convolution is proposed, and a multi-aperture spatial pyramid convolutional pooling module (ASPP) is designed using dilated convolution. Multi-scale feature maps generated in parallel by the ASPP module are used to segment multi-scale objects in the segmentation scene.

[0083] The specific structure of Deeplabv3+ based on the ResNet50 feature extraction module is as follows: Figure 2 As shown. Assuming the initial input image size for DeepLabv3+ is 416×416×3, during the encoding stage, a ResNet50 network based on deformable convolution is used for feature extraction. The output of the last layer is then processed in parallel by the ASPP module to obtain multiple feature maps of different scales. The five feature map channels of the ASPP network output are merged along the channel direction, resulting in a feature map size of 15×15×512. Instead of layer-by-layer upsampling, the merged feature map undergoes an eight-fold upsampling and merging operation with lower-level feature channels to recover the boundary information of the target area. Finally, after convolution with a 3×3 kernel, it is upsampled four times to obtain the final segmentation result. Linear interpolation methods are used throughout the upsampling process.

[0084] Semantic segmentation is a pixel-by-pixel classification task in an image, so the loss function consists only of the classification loss function, L. seg Choose to calculate the cross-entropy loss, and the calculation formula is as follows:

[0085]

[0086] Where s(c,x,y) represents whether the pixel (x,y) belongs to category c; if it does, its value is 1, otherwise it is 0. n represents the probability that pixel (x, y) is predicted to be of class c; c is the number of segmentation categories; h and w are the length and width of the output prediction map, respectively.

[0087] The overall framework of the drivable region segmentation algorithm is designed based on DeepLabv3+. The backbone feature extraction network is based on ResNet50 with deformable convolution, and the encoder-decoder structure of DeepLabv3+ is optimized. After training on the Cityscapes training set, the model's road drivable region segmentation results are evaluated.

[0088] 1.3 Extraction of drivable road surface areas

[0089] The location of different semantic categories in the drivable area segmentation result image is clearly visible. The RGB value corresponding to the road surface area can be obtained by looking up the semantic category label in Table 1. This RGB value is regarded as the Region of Interest (ROI). The road surface area can be extracted by using the mask in digital image processing methods.

[0090] Table 1 Semantic Category Labels

[0091]

[0092]

[0093] The RGB color standard is a commonly used image color system, consisting of three color channels: red (R), green (G), and blue (B). By varying and superimposing these three channels, a wide variety of colors, encompassing almost all colors perceptible to humans, can be produced. Each color channel has a value range of 0-255, which can be represented by 8 bits in computer language, making the entire image 24-bit in depth.

[0094] To retain only the road surface area in the original image, a mask needs to be created based on the semantic segmentation prediction result image. The ROI is then obtained by performing matrix operations between the mask and the original image. First, the semantic segmentation prediction result image is sliced ​​to obtain a two-dimensional matrix with three color channels. In OpenCV, these three channels correspond to BGR (Browser Gradients). The code implementation is as follows:

[0095] blue=prediction_image[:,:,0]

[0096] green=prediction_image[:,:,1]

[0097] red = prediction_image[:,:,2]

[0098] Then, look up the table to find the BGR value (128, 64, 128) corresponding to "road". Create a mask for the blue channel, where the mask value is 1 for pixels with a value of 128 in the prediction result image, and 0 for all other pixels. The code implementation is as follows:

[0099] mask_blue=np.empty(blue.shape).astype(np.uint8)

[0100] mask_blue[np.where(blue==128)]=1

[0101] mask_blue[np.where(blue!=128)]=0

[0102] Next, masks for the green and red channels are created sequentially, retaining only pixels where all three channels have a value of 1. When a pixel in one channel has a value of 0, the corresponding pixels in other channels are set to zero, ensuring that only the road surface area is extracted. Finally, the masks for the three color channels are stitched together, and matrix multiplication is performed with the corresponding channels of the original image to extract an image containing only the road surface area, while non-road surface areas are turned black.

[0103] The drivable region extraction effect after semantic segmentation and masking of images in the self-built database is as follows: Figure 3 As shown, the semantic segmentation network was able to extract most of the effective road surface areas, eliminating background information, and the overall road surface area extraction results were satisfactory.

[0104] Step 2: Classify the types of drivable road areas.

[0105] 2.1 Self-supervised monocular depth estimation algorithm

[0106] This invention employs a self-supervised monocular depth estimation network. Based on the classic DispNet network structure, it uses an Encoder-Decoder architecture as the main framework, improving and upgrading the Encoder module to adapt to real-time monocular depth estimation tasks in complex scenes. Depth-separable convolutions and pointwise convolutions replace the basic modules in ResNet. The input image is scaled three times proportionally to form image pyramids of four sizes. The original input image outputs disparity maps of different sizes at different Decoder stages. Multi-scale optimization is achieved by constructing a multi-level loss function between the two to adapt to complex scenes with objects of varying sizes. Channel and spatial feature extraction mechanisms are designed between the Encoder and Decoder to expand the receptive field and enhance feature extraction. The decoder uses a combination of sub-pixel convolutional layers and bilinear interpolation for high-resolution upsampling, and uses low-lever features from the Encoder to skip connections to the Decoder to maintain higher resolution details. The self-supervised monocular depth estimation network structure diagram is shown below. Figure 4 As shown.

[0107] The segmented road surface image is divided into grids based on depth information. The initial grid size is m1, m2, a < m1 < b, a < m2 < b, where a and b are the dimensions of the road surface within the feasible area, and both a and b are smaller than the minimum size of the road surface image.

[0108] 2.2 Road Surface Image Classification Network

[0109] After processing with a semantic segmentation network and a mask, the original image sample set yields an image set containing only drivable areas. This will serve as the final dataset for training and evaluating the road surface type classification and recognition network. Considering that the preprocessed road surface images no longer contain other background information and that the features of different road surface types are relatively clear and distinguishable, the classification and recognition task is not complex. Therefore, the ShuffleNet V2 network in Table 2 is further simplified. The basic framework of the network is retained, and the model complexity is reduced by appropriately deleting some repeated (Repeat, see Table 2) basic units (stride=1) in Stage 2, Stage 3, and Stage 4. Finally, multiple experiments show that the network model accuracy after reducing Repeat to 1 still meets the classification accuracy requirements.

[0110] In Anaconda's Spyder programming software, a simplified ShuffleNet V2 network model was built. The network input image size was 224×224 pixels, the model channel scaling ratio was set to 1×, the batch size was set to batch_size=64, the softmax_cross_entropy loss function was selected (see formulas (2-1) and (2-2), the Adam adaptive gradient descent optimization algorithm was used, the base learning rate was 0.0001, the network training was started, and the model and training results were saved according to the number of iterations per epoch.

[0111]

[0112]

[0113] Where, x i Let y be the eigenvalue of the i-th class. i Let y be the probability value of the i-th class, T be the total number of classes, and y be the probability value of the i The underscore represents the actual value.

[0114] Table 2 Overall Structure of ShuffleNet V2 Mesh

[0115]

[0116] Each grid is classified to obtain the road surface condition type for each grid. The road surface condition types include dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement, and ice film pavement.

[0117] The confidence scores of each grid's pavement condition type are fused. Specifically, the weights of each pavement condition type—dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement, and ice film pavement—are set to ω. dry-a ,ωwet-a ,ω dry-c ,ω wet-c ,ω brick ,ω snow-c ,ω snow-p ,ω ice The values ​​range from 0 to 1 and are updated according to the weight update strategy. Based on the updated weights of each pavement state type, a basic probability assignment function for the pavement state type of each candidate region grid is constructed. The basic probability assignment function for the pavement state type of each candidate region grid is fused to obtain the fusion confidence of pavement state types such as dry asphalt pavement, wet and slippery asphalt pavement, and icy and snowy asphalt pavement.

[0118] The weight update strategy is as follows: α = k × n (2-1)

[0119]

[0120]

[0121]

[0122]

[0123] Where α is the weight update factor, k is the weight update rate constant, and n is the total number of grids in the segmented candidate region. r The number of grid cells with the highest confidence level for the r-th road surface type obtained from the classification is... This is the first intermediate variable for the r-th road surface state type in the current frame. This is the first intermediate variable for the r-th road surface state type in the previous frame. This is the second intermediate variable for the r-th road surface state type in the current frame. The weight of the r-th road surface state type in the previous frame is given by e, where e is a natural constant. This represents the weight of the updated r-th state type.

[0124] The probability allocation function is specifically:

[0125]

[0126]

[0127] Where ω dry-a ω wet-a ω dry-c ω wet-c ω brick ω snow-c ω snow-p ω ieeThe weights for road surface condition types are as follows: dry asphalt road surface, wet asphalt road surface, dry cement road surface, wet cement road surface, paved road surface, solid snow road surface, loose snow road surface, and icy road surface. Represent The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement state type in the dry asphalt pavement grid. The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement condition type in the wet asphalt pavement grid. To determine the maximum confidence level obtained from the classification, P is the confidence level corresponding to the r-th pavement state type in the dry cement pavement grid. l r The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement state type in the wet cement pavement grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th pavement state type in the pavement grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th road surface state type in the actual snow road surface grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th road surface state type in the snowy road surface grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th pavement state type in the ice film pavement grid, n dry-a n wet-a n dry-c n wet-c n briek n snow-c n snow-p n ice The maximum confidence level obtained from the classification is determined by the number of grid cells corresponding to dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement, and icy pavement. This is a temporary intermediate variable for the basic assignment function of the r-th road surface state type. Let be the basic probability assignment function for the r-th road surface state type;

[0128] The specific integration strategy is as follows:

[0129]

[0130]

[0131] Where K is the sum of the basic probability assignment functions for each pavement state type in the nth grid, and m n (r n Let ) be the basic probability assignment function for the r-th road surface condition type in the n-th grid, and m(r) be the basic probability assignment function for the r-th road surface condition type after fusion.

[0132] The confidence scores of the road surface condition type for each grid are fused, and the category corresponding to the maximum number is selected as the final prediction result, denoted as index.

[0133] Step 3: Mapping between road surface type and road surface adhesion coefficient

[0134] The result of image recognition networks is road surface type, while the research objective of this invention is to obtain the road surface adhesion coefficient. Therefore, it is necessary to provide a correspondence between road surface type and adhesion coefficient. Considering that the road surface adhesion coefficient is related to factors such as road surface wear, tire wear, and air temperature and humidity, and that the range of road surface adhesion coefficient values ​​varies at different vehicle speeds, the design of the road surface type and adhesion coefficient comparison table should take into account the influence of various uncertainties on the road surface adhesion coefficient. For conservatism, the range of adhesion coefficient values ​​corresponding to different road surface types is appropriately expanded, and two cases are distinguished: high-speed driving and low-speed driving. Finally, the comparison table of different road surface types and adhesion coefficient range values ​​is shown in Table 3.

[0135] Table 3 Comparison of Adhesion Coefficient Range Values ​​for Different Road Surface Types and Speeds

[0136]

[0137] Based on the road surface condition type obtained through fusion, the range of road adhesion coefficient values ​​corresponding to that road surface type is obtained by referring to Table 3.1. The lower limit is denoted as 'a', and the upper limit is denoted as 'b'. The average value is taken as the estimated value of the adhesion coefficient corresponding to different road surface types. Using the matrix [index, a, b, μ] e The road surface type classification and identification results data are stored in the form of ]

[0138] In summary, this invention presents a vision-based method for estimating the road surface adhesion coefficient within the drivable area of ​​an intelligent vehicle. After semantic segmentation and masking of the image, the drivable area is obtained. Depth estimation is performed on the drivable area to obtain depth information, followed by grid division. Each grid is classified, and the road surface state type confidence scores of each grid are fused. The category corresponding to the highest number of road surface states is selected as the final prediction result. Finally, the road surface type is mapped to the road surface adhesion coefficient to obtain the road surface adhesion coefficient within the drivable area. The proposed method, while ensuring accuracy and speed, is more applicable to a variety of real-world driving environments, providing a new solution for vehicle dynamics state observation and dynamics control.

[0139] This invention also proposes an electronic device, which can be an in-vehicle controller, capable of executing the above-described methods.

[0140] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for estimating the road surface adhesion coefficient within a passable road area based on machine vision, characterized in that, Includes the following steps: S1 delineates the drivable area of ​​the road surface; S2 classifies the types of drivable road surfaces; S2 includes the following steps: S2.1 Design a self-supervised monocular depth estimation algorithm Based on the Encoder-Decoder architecture, the Encoder module is redesigned, replacing the basic module in ResNet with depthwise separable convolution and pointwise convolution. The input image is scaled three times proportionally to form image pyramids of four sizes. The original image is input, and disparity maps of different sizes are output at different Decoder stages. Multi-scale optimization is achieved by constructing a multi-level loss function between the two. Channel and spatial feature extraction mechanisms are designed between the Encoder and Decoder to expand the receptive field and enhance feature extraction. The decoder uses a combination of subpixel convolutional layers and bilinear interpolation for high-resolution upsampling, and uses low-lever features from the encoder to jump to the decoder to maintain higher resolution details. The segmented road surface image is divided into grids based on depth information, where the initial size of the grid is [size missing]. , a and b are the dimensions of the road surface within the feasible area of ​​the road surface, and both a and b are smaller than the minimum size of the road surface image; S2.2 Design a road surface image classification network; Based on the ShuffleNet V2 network, duplicate basic units in Stages 2, 3, and 4 were removed. The input image size was 224×224 pixels, the model channel scaling was 1×, and the batch size was 64. The softmax cross-entropy loss function was selected, and formulas (2-1) and (2-2) were used. The Adam adaptive gradient descent optimization algorithm was employed, with a base learning rate of 0.0001. Network training began, and the model and training results were saved according to the number of iterations (epochs). (2-1) (2-2) in, For the first Class feature values, For the first Class probability value, The total number of categories, The actual value; Each grid is classified to obtain the road surface condition type of each grid. The road surface condition types include dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement and ice film pavement. The confidence scores of each grid's pavement condition type are fused. Specifically, the weights of each pavement condition type—dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement, and ice film pavement—are set as follows: The values ​​range from 0 to 1 and are updated according to the weight update strategy. Based on the updated weights of each road surface condition type, a basic probability assignment function for the road surface condition type of each candidate region grid is constructed. The basic probability assignment functions for the road surface condition type of each candidate region grid are fused to obtain the fusion confidence scores of the road surface condition types of dry asphalt pavement, wet and slippery asphalt pavement, and icy and snowy asphalt pavement. The road surface condition type confidence scores of each grid are fused, and the category corresponding to the maximum number is selected as the final prediction result, denoted as index. S3, obtain the corresponding road adhesion coefficient based on the mapping between road surface type and road adhesion coefficient.

2. The method for estimating the road surface adhesion coefficient within a passable road area based on machine vision, as described in claim 1, is characterized in that... S1 includes: Construction and preprocessing of the S1.1 dataset Augmentation processing of the Cityscapes dataset includes: random scaling and cropping of the training data, horizontal flipping, and color changes; random alteration of the grayscale values ​​of the training data; and random addition of small Gaussian noise to each pixel.

3. The method for estimating the road surface adhesion coefficient within a passable road area based on machine vision, as described in claim 1 or 2, is characterized in that... S1 includes: S1.2 Design of a Driving Region Segmentation Algorithm Based on Deeplabv3+ Let DeepLabv3+ first input the size of the image. In the encoding stage, after feature extraction using a ResNet50 network based on deformable convolution, the output of the last layer is processed in parallel by the ASPP module to obtain multiple feature maps of different scales. The five feature maps output by the ASPP network are then merged along the channel direction. At this point, the feature map size is... The merged feature map undergoes an eight-fold upsampling and low-level feature channel merging operation to recover the target boundary information. Finally, it is processed... The final segmentation result is obtained by upsampling four times after convolution calculation of the convolution kernel. The upsampling process uses linear interpolation method. The loss function consists only of the classification loss function. Choose to calculate the cross-entropy loss, and the calculation formula is as follows: (1-1) in, Represents pixels Does it belong to a category? If its value is 1, then it is 0 otherwise; Represents pixels Predicted as category The probability of; It is the number of segmentation categories; These are the length and width of the output prediction image, respectively.

4. The method for estimating the road surface adhesion coefficient within a passable area based on machine vision according to claim 1, characterized in that, The weight update strategy is as follows: in, Here, k is the weight update factor, k is the weight update rate constant, and n is the total number of grid cells in the segmented candidate regions. The number of grid cells with the highest confidence level for the r-th road surface type obtained from the classification is... This is the first intermediate variable for the r-th road surface state type in the current frame. This is the first intermediate variable for the r-th road surface state type in the previous frame. This is the second intermediate variable for the r-th road surface state type in the current frame. The weight of the r-th road surface state type in the previous frame is given by e, where e is a natural constant. The updated weight for the r-th state type; The probability allocation function is: in, The weights for road surface condition types are as follows: dry asphalt road surface, wet asphalt road surface, dry cement road surface, wet cement road surface, paved road surface, solid snow road surface, loose snow road surface, and icy road surface. , The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement state type in the grid of dry asphalt pavement. The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement condition type in the grid of wet asphalt pavement. The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement state type in the dry cement pavement grid. The maximum confidence level obtained from the classification is the confidence level corresponding to the r-th pavement state type in the wet cement pavement grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th pavement state type in the pavement grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th road surface state type in the actual snow road surface grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th road surface state type in the snowy road surface grid. The maximum confidence score obtained from the classification is the confidence score corresponding to the r-th pavement state type in the ice film pavement grid. The maximum confidence level obtained from the classification is determined by the number of grid cells corresponding to dry asphalt pavement, wet asphalt pavement, dry cement pavement, wet cement pavement, paved pavement, solid snow pavement, loose snow pavement, and icy pavement. This is a temporary intermediate variable for the basic assignment function of the r-th road surface state type. Let be the basic probability assignment function for the r-th road surface state type; The specific integration strategy is as follows: Where K is the sum of the basic probability assignment functions for each road surface condition type in the nth grid. Let r be the basic probability assignment function for the r-th road surface condition type in the n-th grid. Let be the basic probability assignment function for the r-th road surface state type after fusion.

5. The method for estimating the road surface adhesion coefficient within a passable area based on machine vision according to claim 1, characterized in that, The mapping between road surface type and road surface adhesion coefficient in S3 is as follows: 。 6. An electronic device, characterized in that, The device is capable of performing the method described in any one of claims 1-5.

Images