A road surface adhesion coefficient prediction method based on a semantic segmentation network

Abstract

The application discloses a road surface adhesion coefficient prediction method based on a semantic segmentation network, first, a semantic segmentation network based on a multi-scale spatial attention mechanism is built; second, the built segmentation network is pre-trained on a public data set; then, the semantic segmentation data set is enriched and the segmentation network is specifically trained; next, the segmentation network is used to extract a road surface area to make a road surface classification network data set; then, a road surface type classification network is built and trained; finally, a mapping rule is established to obtain road surface adhesion coefficient information; the method enhances the generalization ability of the algorithm to the rain and snow driving scene, further improves the precision, real-time performance and robustness of the driving road surface extraction; meanwhile, in combination with a light-weight road surface recognition network, a semantic segmentation network, a road surface extraction and a road surface recognition network serial algorithm structure is designed, and the road surface adhesion information can be quickly and accurately predicted in a rich driving scene.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle technology and relates to a method for predicting road surface adhesion coefficient based on computer vision. More specifically, it relates to a method for predicting road surface adhesion coefficient based on semantic segmentation network. Background Technology

[0002] With the increase in vehicle ownership, while cars bring convenience and speed, traffic safety issues have also arisen. Traffic accidents have become one of the leading causes of injury and death both domestically and internationally. Especially under low friction coefficient conditions, vehicle instability conditions such as sideslip, drifting, and collisions are more likely to occur, leading to more severe loss of life and property. Therefore, obtaining accurate road friction coefficient information in advance can provide drivers with a reference and help improve driving safety. At the same time, advanced automotive active safety systems require accurate road friction coefficients as support. Accurately obtaining the road friction coefficient can expand the operating condition adaptation range of automotive active safety systems, and early perception of changes in the road surface condition ahead can help active safety systems adjust their control strategies in a timely manner. Therefore, accurately identifying and obtaining the road surface type and friction coefficient is key to improving driving safety and comfort.

[0003] Currently, the main methods for obtaining the road adhesion coefficient (LAC) are estimation methods based on dynamic response and image-based visual prediction methods based on neural networks. While dynamic model-based estimators can obtain accurate LAC estimates, their accuracy is limited by the precision of the vehicle and tire models, and they struggle to predict LAC in advance, exhibiting a certain degree of lag. Thanks to the rapid development of neural networks and the upgrading and improvement of intelligent vehicle hardware such as onboard cameras, vision-based LAC recognition methods are more reliable. Simultaneously, vision-based prediction methods effectively improve the lag of dynamic estimators, enabling early perception of road conditions and enhancing the ability to respond to hazardous situations. Furthermore, embedding semantic segmentation networks into visual prediction algorithms allows the algorithm to focus more on road information, eliminating redundant information and improving road surface recognition accuracy, thereby obtaining more accurate LAC prediction information. Summary of the Invention

[0004] To address the problems existing in the prior art and improve the accuracy, real-time performance, and robustness of road surface adhesion coefficient prediction algorithms, this invention proposes a road surface adhesion coefficient prediction method based on a semantic segmentation network. This method constructs a semantic segmentation network based on an attention mechanism to extract the road surface region of the road ahead, and improves the accuracy and robustness of the network as much as possible through pre-training and specific training. Then, a road surface classification network based on channel attention is constructed and trained, and the obtained road surface classification results are combined with mapping rules to obtain the predicted results of the road surface adhesion coefficient.

[0005] This invention is achieved using the following technical solution:

[0006] A road surface adhesion coefficient prediction method based on semantic segmentation network is proposed to predict the road surface adhesion coefficient based on the road type classification of the vehicle driving area under urban conditions. The specific steps are as follows:

[0007] Step 1: Construct a semantic segmentation network based on a multi-scale spatial attention mechanism:

[0008] The vehicle's onboard camera, equipped with an intelligent driving vehicle's perception system, collects video data of the road ahead during the vehicle's journey. A semantic segmentation network is then used to extract the drivable road surface area.

[0009] First, the network environment was configured and set up: the Linux operating system was chosen as the runtime environment for image processing, network setup, and training; Python was used to write the program code; and PyTorch, developed by the Meta AI team, was selected as the deep learning network framework. Anaconda was used to create a new virtual environment, and Python 3.10.8, PyTorch 1.11.0, and OpenCV 4.6.0.66 were installed in the new environment.

[0010] Secondly, a semantic segmentation network is built, using an advanced lightweight encoder-decoder framework to ensure the accuracy of the semantic segmentation network while meeting the real-time requirements of the algorithm.

[0011] Then, within the encoder-decoder framework, a hierarchical semantic segmentation network encoder structure based on the attention mechanism is specifically developed and constructed to generate rich multi-scale semantic features: first, the input image is scaled to a tensor input semantic segmentation network with a size of 1024×1024×3; then, after passing through 3×3 convolutional layers, BN layers, and Gelu activation processing, the initial feature extraction and downsampling functions of the input image are realized.

[0012] The obtained feature map is then used as input to the feature extraction module. The specific implementation process of the feature extraction module is as follows: First, a 1×1 convolution is used to adjust the number of input channels. After passing through the Gelu activation function, it is input into the spatial attention mechanism module, which consists of a 5×5 main branch convolution and subsequent 7×7 and 13×13 branch convolutions. Then, a 1×1 convolution is used to output the weight parameters representing spatial attention. The formula for the multi-scale spatial attention mechanism is shown below:

[0013]

[0014] In formula (1), Atten is the weight parameter representing spatial attention in the output, Atten0 is the weight output of the main branch of the spatial attention module, and Atten... i This is the weight output of the i-th branch convolution in the module, where i is the number of branch convolutions;

[0015] Next, the number of channels in the feature map is expanded by 1×1 convolution, and then 3×3 convolution is used to extract more detailed features from the feature map after spatial attention weighting. The extraction results are activated by the GeLu function, and finally the processing results of the feature extraction module are output by 1×1 convolution.

[0016]

[0017] In the formula, Output is the output of the bottleneck module, F is the input of the bottleneck module, Conv2D(·) represents using convolution to process the input feature map, and Gelu(·) represents using activation functions to further process the feature map;

[0018] Using the encoder bottleneck module constructed above, the network input tensor undergoes four sequential feature extractions to obtain feature maps representing four stages at different semantic levels. Stage 1 feature extraction is performed by three sequential feature extraction modules, outputting a 256×256×32 feature map. Then, through the aforementioned downsampling layer architecture, the number of convolutional layers, the number of convolutions within each layer, and the number of channels are adjusted to downsample the feature map. This is then processed by three sequential feature extraction modules to obtain a 128×128×64 feature map for Stage 2. Next, through a downsampling layer and five sequential feature processing modules, a 64×64×160 feature map for Stage 3 is obtained. Finally, through one more downsampling step and two more feature extraction modules, a 32×32×256 feature map for Stage 4 is obtained.

[0019] Finally, a lightweight decoder architecture is built based on the encoder-decoder framework: the feature maps extracted by the encoder at different stages are processed sequentially through upsampling, channel concatenation, a lightweight ham_head decoder module, and a fully connected classifier to fuse semantic information at different levels; in order to collect multi-scale semantic information and expand the receptive field, the first stage with more low-level semantic information is discarded, and the feature maps of the second, third, and fourth stages are processed into feature maps of the same size by bilinear interpolation upsampling method, and then channel concatenation is performed before input into the ham_head decoder module to process the integrated and concatenated 128×128×480 feature map; the specific implementation process of the ham_head decoder module is as follows: the 480-channel feature map obtained by the concat operation is compressed to 256 channels by a 1×1 convolutional layer, a 32-group GroupNorm layer, and a ReLU activation layer, and then the non-negative matrix factorization algorithm of NMF shown in formula (3) is used:

[0020] V m×n =P m×r ×Q r×n (3)

[0021] The characteristic matrix V m×n Decomposed into low-level matrices P of m×r and r×n m×r and Q r×n , where P m×r The characteristic basis matrix represents the main characteristics of the data; while Q... r×n The feature coefficient matrix represents the distribution of data features. The algorithm uses the feature basis matrix instead of the feature matrix to avoid interference from redundant matrices, accelerate the image processing process, and improve the real-time performance of the algorithm. Finally, the output is processed by a linearization layer consisting of a 1×1 convolutional architecture composed of a 1×1 convolutional layer, 32 groups of GroupNorm layers, and ReLU activation layers, which is the output of the ham_head decoder module.

[0022] The processed feature map is then fully connected through a 1×1 convolutional layer, followed by a softmax function to convert the class scores into a probability distribution, resulting in a 256×256×nls feature map. Here, the value of each pixel represents the probability of belonging to that semantic class, and nls represents the number of semantic classes to be classified. The resulting feature map is then upsampled using bilinear interpolation, and the ArgMax function is used to predict the semantic class of each pixel according to the maximum probability. The resulting 1024×1024 predicted segmentation image, which is the same size as the input image, is the encoder output.

[0023] Step 2: Pre-train the constructed segmentation network on a public dataset:

[0024] The Cityscapes dataset was selected for pre-training of the semantic segmentation network. This dataset, released by Mercedes-Benz, can be used for urban street scene image segmentation tasks. It contains street scene driving scene images of 50 cities and has precise annotations covering 19 categories of semantic information, including road surface, buildings, pedestrians, vehicles and buildings. At the same time, its image samples are similar to the daily urban road conditions, so the pre-trained semantic segmentation network has a certain generalization ability. A total of 5,000 precisely annotated image samples were used for network model training and validation, including 2,975 training set images, 500 validation set images, and 1,525 test set images. The specific pre-training process is as follows: First, the images are loaded from the dataset directory, then their corresponding annotation information is loaded, and then the training images and annotations are augmented. The augmentation methods include random scaling, random cropping of image tensors and random left and right flipping. Then, each pixel of the input image is normalized by the normalization function. Finally, the cross-entropy loss function as shown in formula (4) is designed:

[0025]

[0026] In the formula, nls represents the semantic category to be classified, which is 19 when trained on the Cityscapes dataset, and y c Let P be a vector with values ​​of 0 or 1. The value of each element is used to determine whether the class is the same as the sample class. c The probability that the predicted sample belongs to class c is c∈(1,nls);

[0027] The segmentation network is trained using a poly learning rate rule, where the poly learning rate decay expression is as follows:

[0028]

[0029] LR in formula (5) initial The initial learning rate for network training is set to 0.002 during pre-training. `iter` represents the number of training iterations, `max_iter` is the maximum number of training steps (set to 40K steps), and `power` is the decay coefficient used to control the shape of the learning rate curve (set to 0.9). `LR(iter)` is the learning rate calculated for each specific number of training steps, and the Adam optimization algorithm is used to dynamically adjust the learning rate of each parameter using the first and second moment estimates of the gradient. Based on computer hardware performance, the batch size is set to 32, and model parameters are saved every 4K steps. Simultaneously, a validation set is used to evaluate the network's performance.

[0030] For semantic segmentation tasks, the average pixel accuracy (Acc) and average intersection-union ratio (MIoU) based on the confusion matrix are used to evaluate the performance of the network pre-training results. The specific implementation process is as follows: For the binary classification confusion matrix shown in Table 1, each row represents the true class of the data, each column represents the predicted class, and the specific element value in the matrix represents the number of samples predicted as a certain class.

[0031]

[0032] Table 1: Schematic diagram of a binary confusion matrix

[0033] Accuracy Acc represents the percentage of pixels that correctly predict the class out of the total number of pixels, as shown in formula (6):

[0034]

[0035] MIoU represents the summation and averaging of the ratio of the intersection to the union of the predicted results and the actual values ​​for each category, as shown in formula (7) below:

[0036]

[0037] The training data was saved to a log file using the logging tool in PyTorch, and the training parameter curves were analyzed. The constructed semantic segmentation network showed excellent performance in urban driving scenarios with good weather.

[0038] Step 3: Enrich the semantic segmentation dataset and perform specific training on the segmentation network.

[0039] To address the issue of segmentation networks being susceptible to distortion under rainy and snowy weather conditions, this method creates a semantic segmentation dataset containing six road surface types: dry and wet asphalt, dry and wet cement, snow-covered roads, and icy roads. The pre-trained semantic segmentation network is then trained and evaluated on this dataset to further improve the network's generalization ability and better suit its application in everyday urban driving scenarios. The specific implementation process is as follows:

[0040] First, driving scene data under different weather conditions is collected using an in-vehicle camera. The collected video files are then broken down at a fixed frame rate to obtain an image sample database covering six road surface categories. Next, image samples with rich differences and distinct features are selected from the image database for detailed annotation. The specific annotation process is as follows: First, the EISeg interactive annotation tool is installed and launched in Anaconda. Then, the driving scene images to be annotated are imported and uploaded to the EISeg platform. Next, an annotation task is created, selecting the "Semantic Segmentation" task type and the Cityscapes dataset annotation format. Then, the "Annotate Data" function is selected to segment and annotate the imported images. The interactive annotation interface is then accessed and used... The image is labeled using a tool; after labeling, the "Export Data" function is used, selecting the Cityscapes dataset format and specifying the output path; the created dataset is structured according to the Cityscapes dataset format, with 100 precisely labeled images for each class of road surface information, totaling 600 dataset samples. 20% are randomly selected as the validation set, and the remaining 80% are used as the training set; the dataset is augmented using mirroring, translation, and brightness adjustment techniques, continuing the training strategies and optimization methods from the pre-training process, while the trained network is validated on the validation set; finally, the trained semantic segmentation network is used for model inference, processing the input driving scene images and outputting visualized pixel-level semantic segmentation results;

[0041] Step 4: Use a segmentation network to extract road surface regions and create a road surface classification network dataset:

[0042] The image sample library containing six road surface types obtained in step three is input into a semantic segmentation network for prediction processing according to road surface category. The process is similar to the online recognition process in daily driving environment. Then, the prediction results and the original image are simultaneously sent to the processor for road surface region extraction. The specific process of extracting the road surface region is as follows: First, the semantic segmentation prediction result image is split according to color channels, and the two-dimensional matrix corresponding to the three color channels is obtained. The BGR value (128, 64, 128) corresponding to "road" is found by looking up the table. Then, masks for the three color channels of BGR are made using OpenCV, and only pixels with the same BGR value as "road" are retained to ensure that only the road surface region is extracted. Finally, the masks of the three color channels are stitched together, and matrix multiplication is performed with the corresponding channels of the original image to extract the image containing only the road surface region. The non-road surface region is set as the background and turned into black. The extracted image sample library containing only the road surface region is stored in specific folders according to the six road surface information, and the corresponding 5-bit binary information of the specific road surface type and label information is shown in Table 2.

[0043]

[0044]

[0045] Table 2: Correspondence Table of Road Surface Type and Label Information

[0046] The prepared dataset was filtered according to image quality, and the final dataset contained 6,000 image samples, with 1,000 images for each road surface category. Finally, the road surface image dataset was shuffled and 20% of the images from each category were randomly selected as the validation set, with the remainder used as the training set.

[0047] Step 5: Build and train a road surface image type classification network:

[0048] The specific implementation process of building a road surface image type classification network is as follows: First, a road surface image type recognition network is built in the Anaconda environment. Considering the complexity of the road surface classification task and the real-time requirements of the classification process, a lightweight convolutional neural network is built as the backbone network of the classifier. At the same time, a channel attention mechanism is introduced to reduce the network parameters and floating-point operations as much as possible while ensuring recognition accuracy, thereby improving the real-time performance of the classification network. The specific road surface image classification network structure design is shown in Table 3.

[0049]

[0050]

[0051] Table 3: Road Surface Classification Network Structure Table

[0052] First, the network's input layer adjusts the road image to be recognized into a 288×288×3 tensor. Then, it passes through a convolutional layer with a stride of 2 and a 3×3 kernel, combined with batch normalization (BN) and Swish activation for initial feature extraction. The resulting feature map is then used as input to the bottleneck module. The bottleneck module extracts features from the input feature map. The specific process of building the network bottleneck module based on the channel attention mechanism is as follows: First, a 1×1 convolution is used to change the feature channel dimension according to the expansion ratio. After batch normalization (BN) and Swish activation, the input is fed into a 3×3 depthwise separable convolutional layer (DWConv) for feature extraction. The feature map is input into the channel attention mechanism SE module. The module performs global average pooling on the input feature map, then passes it through Swish activation to aggregate the global features of the input feature map. The resulting global features are then processed through a fully connected 1×1 convolutional layer with a certain activation rate. The sigmoid activation function is then used to obtain the attention weights for different channels. The obtained weights are then multiplied and weighted with the input feature map to retain the main features of the input, remove noise interference, reduce parameter calculation, improve the real-time performance of the network, and achieve the goal of focusing on the more important information channels. Finally, the output of the SE module is passed through a 1×1 convolution to obtain the output of the entire bottleneck module.

[0053] As shown in Table 3, the feature map extracted by a convolutional layer with a stride of 2 and a kernel size of 3×3 is input into two consecutive bottleneck modules with a dilation rate of 1 for feature extraction, resulting in a 144×144×24 feature map. The network structure is designed to use bottleneck modules for feature extraction in the first three layers, so that after each bottleneck layer, the size of the output feature map is reduced to half that of the input feature map, while the number of output channels is double that of the input channels. The first three layers yield a 36×36×64 feature map. The last three feature extraction layers use activation ratios of... Channel attention mechanisms of 0.35, 0.30, and 0.25 focus more on feature channels with rich content, resulting in a 9×9×256 feature map set containing high-level semantic information. This is then upscaled through a 1×1 convolutional layer, and the upscaled 9×9×1280 feature map set is pooled using a 9×9 global average pooling layer to obtain a 1×1×1280 feature sequence. Finally, a fully connected layer with a Softmax function is used to output the probability value of belonging to each road category. Finally, the Argmax function is used to determine the network classification result according to the maximum probability value.

[0054] Then, the road surface image type network was built and trained on a self-built dataset. The training process adopted the cross-entropy loss function and the Adam adaptive gradient descent optimization algorithm. The basic learning rate was 0.0001, and the model and training results were saved according to the number of iterations (epochs).

[0055] Step 6: Establish mapping rules to obtain road surface adhesion coefficient information:

[0056] Finally, the road surface adhesion coefficient is obtained based on the road surface image type classification results obtained from the above steps. In order to obtain the road surface adhesion coefficient of the identified road surface, the reference value table of vehicle longitudinal slip adhesion coefficient in "GA / T643-2006 Technical Appraisal of Vehicle Driving Speed ​​in Typical Traffic Accident Forms" and the reference value table of vehicle longitudinal slip adhesion coefficient for icy and snowy roads are referenced. Considering the influence of driving speed on the road surface adhesion coefficient, the mapping rule between road surface type and road surface adhesion coefficient as shown in Table 4 is defined. Considering the low driving speed of vehicles in urban conditions, which is often 48 km / h, the adhesion coefficient range of the current road surface is obtained by looking up the corresponding table based on the vehicle speed and road surface type identification results. The midpoint between the upper and lower limits of the adhesion coefficient is taken as the adhesion coefficient of the current road surface, which is the output result of the entire road surface adhesion coefficient prediction algorithm.

[0057]

[0058]

[0059] Table 4: Mapping table of road surface type and adhesion coefficient.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] This invention discloses a road surface adhesion coefficient prediction method based on semantic segmentation. By processing road images of the road surface being driven ahead, the method obtains the predicted road surface adhesion coefficient in advance, providing crucial road surface adhesion coefficient information for vehicle active safety control systems and driver assistance systems. This invention constructs a lightweight semantic segmentation network based on a multi-scale spatial attention mechanism and conducts specific training on a self-built dataset, enhancing the algorithm's generalization ability to driving scenarios in rain and snow, and further improving the accuracy, real-time performance, and robustness of road surface extraction. Simultaneously, combined with a lightweight road surface recognition network, a sequential algorithm structure of semantic segmentation network, road surface extraction, and road surface recognition network is designed, enabling rapid and accurate prediction of road surface adhesion information in rich driving scenarios. Attached Figure Description

[0062] Figure 1 The flowchart of a road surface adhesion coefficient prediction method based on semantic segmentation network provided by the present invention is shown.

[0063] Figure 2 This is a schematic diagram of the semantic segmentation network structure built in step one.

[0064] Figure 3This is a schematic diagram of the bottleneck module structure for feature extraction in the semantic segmentation network built in step one of this method.

[0065] Figure 4 This is a schematic diagram of the bottleneck module structure for feature extraction of the road surface classification network built in step five of this method. Detailed Implementation

[0066] This invention provides a method for predicting road surface adhesion coefficient based on semantic segmentation networks. It predicts the road surface adhesion coefficient based on road type classification in urban driving areas, and the specific steps are as follows:

[0067] Step 1: Construct a semantic segmentation network based on a multi-scale spatial attention mechanism:

[0068] The vehicle's perception system utilizes onboard cameras to collect video data of the road ahead during vehicle operation. A semantic segmentation network is then used to extract the drivable road surface area. The performance of the semantic segmentation network largely determines the overall performance of the road surface adhesion coefficient prediction algorithm. Therefore, building a high-precision, real-time, and robust semantic segmentation network is crucial. Its specific design is as follows:

[0069] First, the network environment was configured and set up: the Linux operating system was chosen as the runtime environment for image processing, network setup, and training. The specific hardware configuration included: an Intel(R) Core(TM) i9-9900K CPU, 16GB of RAM, an NVIDIA RTX A6000 graphics card with CUDA version 11.4, and a 256GB solid-state drive plus a 2TB hard disk drive. Python was used to write the program code, and PyTorch, developed by the Meta AI team, was chosen as the deep learning network framework. Next, the software was configured within this hardware environment: Anaconda was used to create a new virtual environment, and Python 3.10.8, PyTorch 1.11.0, and OpenCV 4.6.0.66 were installed in the new environment.

[0070] Next, the semantic segmentation network is constructed using an advanced lightweight encoder-decoder framework. This ensures both the accuracy of the semantic segmentation network and the real-time requirements of the algorithm. The specific network structure is as follows: Figure 1 As shown;

[0071] Then, within the encoder-decoder framework, a hierarchical semantic segmentation network encoder structure based on the attention mechanism is specifically developed and constructed to generate rich multi-scale semantic features: first, the input image is scaled to a tensor input semantic segmentation network with a size of 1024×1024×3; then, after passing through 3×3 convolutional layers, BN layers, and Gelu activation processing, the initial feature extraction and downsampling functions of the input image are realized.

[0072] The obtained feature map is then used as input to the feature extraction module. The specific implementation process of the feature extraction module is as follows: First, a 1×1 convolution is used to adjust the number of input channels. After passing through the Gelu activation function, it is input into the spatial attention mechanism module, which consists of a 5×5 main branch convolution and subsequent 7×7 and 13×13 branch convolutions. Then, a 1×1 convolution is used to output the weight parameters representing spatial attention. The formula for the multi-scale spatial attention mechanism is shown below:

[0073]

[0074] In formula (1), Atten is the weight parameter representing spatial attention in the output, Atten0 is the weight output of the main branch of the spatial attention module, and Atten... i This is the weight output of the i-th branch convolution in the module, where i is the number of branch convolutions;

[0075] Next, the number of channels in the feature map is expanded by 1×1 convolution, and then 3×3 convolution is used to extract more detailed features from the feature map after spatial attention weighting. The extraction results are activated by the GeLu function, and finally the processing results of the feature extraction module are output by 1×1 convolution.

[0076]

[0077] In the formula, Output is the output of the bottleneck module, F is the input of the bottleneck module, Conv2D(·) represents using convolution to process the input feature map, and Gelu(·) represents using activation functions to further process the feature map;

[0078] Using the encoder bottleneck module constructed above, the network input tensor undergoes four sequential feature extractions to obtain feature maps representing four stages at different semantic levels. Stage 1 feature extraction is performed by three sequential feature extraction modules, outputting a 256×256×32 feature map. Then, through the aforementioned downsampling layer architecture, the number of convolutional layers, the number of convolutions within each layer, and the number of channels are adjusted to downsample the feature map. This is then processed by three sequential feature extraction modules to obtain a 128×128×64 feature map for Stage 2. Next, through a downsampling layer and five sequential feature processing modules, a 64×64×160 feature map for Stage 3 is obtained. Finally, through one more downsampling step and two more feature extraction modules, a 32×32×256 feature map for Stage 4 is obtained.

[0079] Finally, a lightweight decoder architecture is built based on the encoder-decoder framework: the feature maps extracted by the encoder at different stages are processed sequentially through upsampling, channel concatenation, a lightweight ham_head decoder module, and a fully connected classifier to fuse semantic information at different levels; in order to collect multi-scale semantic information and expand the receptive field, the first stage with more low-level semantic information is discarded, and the feature maps of the second, third, and fourth stages are processed into feature maps of the same size by bilinear interpolation upsampling method, and then channel concatenation is performed before input into the ham_head decoder module to process the integrated and concatenated 128×128×480 feature map; the specific implementation process of the ham_head decoder module is as follows: the 480-channel feature map obtained by the concat operation is compressed to 256 channels by a 1×1 convolutional layer, a 32-group GroupNorm layer, and a ReLU activation layer, and then the non-negative matrix factorization algorithm of NMF shown in formula (3) is used:

[0080] V m×n =P m×r ×Q r×n (3)

[0081] The characteristic matrix V m×n Decomposed into low-level matrices P of m×r and r×n m×r and Q r×n , where P m×r The characteristic basis matrix represents the main characteristics of the data; while Q... r×nThe feature coefficient matrix represents the distribution of data features. The algorithm replaces the feature matrix with a feature basis matrix, which can map high-dimensional data to low-dimensional data while retaining important features of the data. This discards the interference of redundant matrices, enhances the generalization ability of the model, and accelerates the image processing process to improve the real-time performance of the algorithm. Finally, the output is processed by a linearization layer consisting of a 1×1 convolutional architecture, 32 groups of GroupNorm layers, and ReLU activation layers, which is the output of the ham_head decoder module.

[0082] The processed feature map is then fully connected through a 1×1 convolutional layer, followed by a softmax function to convert the class scores into a probability distribution, resulting in a 256×256×nls feature map. Here, the value of each pixel represents the probability of belonging to that semantic class, and nls represents the number of semantic classes to be classified. The resulting feature map is then upsampled using bilinear interpolation, and the ArgMax function is used to predict the semantic class of each pixel according to the maximum probability. The resulting 1024×1024 predicted segmentation image, which is the same size as the input image, is the encoder output.

[0083] Step 2: Pre-train the constructed segmentation network on a public dataset:

[0084] The Cityscapes dataset was selected for pre-training of the semantic segmentation network. This dataset, released by Mercedes-Benz, can be used for urban street scene image segmentation tasks. It contains street scene driving scene images of 50 cities and has precise annotations covering 19 categories of semantic information, including road surface, buildings, pedestrians, vehicles and buildings. At the same time, its image samples are similar to the daily urban road conditions, so the pre-trained semantic segmentation network has a certain generalization ability. A total of 5,000 precisely annotated image samples were used for network model training and validation, including 2,975 training set images, 500 validation set images, and 1,525 test set images. The specific pre-training process is as follows: First, the images are loaded from the dataset directory, then their corresponding annotation information is loaded, and then the training images and annotations are augmented. The augmentation methods include random scaling, random cropping of image tensors and random left and right flipping. Then, each pixel of the input image is normalized by the normalization function. Finally, the cross-entropy loss function as shown in formula (4) is designed:

[0085]

[0086] In the formula, nls represents the semantic category to be classified, which is 19 when trained on the Cityscapes dataset, and y c Let P be a vector with values ​​of 0 or 1. The value of each element is used to determine whether the class is the same as the sample class.c The probability that the predicted sample belongs to class c is c∈(1,nls);

[0087] The segmentation network is trained using a poly learning rate rule, where the poly learning rate decay expression is as follows:

[0088]

[0089] LR in formula (5) initial The initial learning rate for network training is set to 0.002 during pre-training. `iter` represents the number of training iterations, `max_iter` is the maximum number of training steps (set to 40K steps), and `power` is the decay coefficient used to control the shape of the learning rate curve (set to 0.9). `LR(iter)` is the learning rate calculated for each specific number of training steps, and the Adam optimization algorithm is used to dynamically adjust the learning rate of each parameter using the first and second moment estimates of the gradient. Based on computer hardware performance, the batch size is set to 16, and model parameters are saved every 4K steps. Simultaneously, a validation set is used to evaluate the network's performance.

[0090] For semantic segmentation tasks, the average pixel accuracy (Acc) and average intersection-union ratio (MIoU) based on the confusion matrix are often used to evaluate the performance of the network pre-training results. The specific implementation process is as follows: For the binary classification confusion matrix shown in Table 1, each row represents the true class of the data, each column represents the predicted class, and the specific element value in the matrix represents the number of samples predicted as a certain class.

[0091]

[0092] Table 1: Schematic diagram of a binary confusion matrix

[0093] Accuracy Acc represents the percentage of pixels that correctly predict the class out of the total number of pixels, as shown in formula (6):

[0094]

[0095] MIoU represents the summation and averaging of the ratio of the intersection to the union of the predicted results and the actual values ​​for each category, as shown in formula (7) below:

[0096]

[0097] The training data was saved to a log file using the logging tool in PyTorch, and the training parameter curves were analyzed. The constructed semantic segmentation network showed excellent performance in urban driving scenarios with good weather.

[0098] In this embodiment, the Acc value for the road surface category is 0.9909, the MIoU value is 0.9835, and the MIoU value for the entire semantic category is 0.7959. These values ​​all indicate that the constructed semantic segmentation network has high semantic segmentation accuracy. Meanwhile, the prediction time for a single image is 0.02s, which meets the real-time requirements of the algorithm during vehicle movement.

[0099] Step 3: Enrich the semantic segmentation dataset and perform specific training on the segmentation network.

[0100] During the experiment, it was found that the semantic segmentation network built and trained only through the above steps performed poorly in the task of segmenting waterlogged and snow-covered roads in rainy and snowy weather, and even showed serious distortion. The reason is that the training data in the Cityscapes dataset were all collected in urban driving scenarios under good weather conditions, and did not include driving scenarios under rainy and snowy weather conditions. As a result, the semantic segmentation network that has only been pre-trained is easily affected by adverse weather and special road conditions, resulting in misclassification and thus affecting the overall performance of the semantic segmentation network.

[0101] To address the aforementioned issues and better apply the method to everyday driving scenarios, and considering the susceptibility of segmentation networks to distortion under rainy or snowy conditions, this method creates a semantic segmentation dataset encompassing six road surface types: wet and dry asphalt, wet and dry cement, snow-covered roads, and icy roads. The pre-trained semantic segmentation network is then trained and evaluated on this dataset to further enhance its generalization ability and improve its applicability in everyday urban driving scenarios. The specific implementation process is as follows:

[0102] First, appropriate vehicle-mounted camera equipment is selected to collect driving scene data under different weather conditions. The collected video files are then broken down according to a fixed frame rate to obtain an image sample information library covering six road surface categories.

[0103] In this embodiment, the selected image acquisition device is the F1005-E Sensor Unit AXIS high-definition image sensor, with an operating temperature of -30℃ to 55℃, an adjustable focal length range of 28mm to 120mm, a video resolution of 1920×1200, a frame rate of 60fps, and a lens field of view (FOV) of 113°.

[0104] In this embodiment, the aforementioned image acquisition device was installed on an image acquisition experimental vehicle, and after a long period of road surface image data acquisition experiments, video files (daytime, evening, and nighttime) were finally collected, including different weather conditions such as sunny, cloudy, rainy, and snowy days, as well as different acquisition time periods such as daytime, evening, and nighttime. The video files were decomposed into images at fixed frame intervals, organized according to their categories, named uniformly, and placed in the same folder for similar road surface images. Finally, an image sample library of 6 road surface types was obtained.

[0105] Then, image samples with rich differences and obvious features are selected from the image database for fine annotation. The specific annotation process is as follows: First, install the EISeg interactive annotation tool in Anaconda and start the platform. Then, import the driving scene images to be annotated and upload the images to the EISeg platform. Next, create an annotation task, select the "Semantic Segmentation" task type, and select the Cityscapes dataset annotation format. Then, select the "Annotate Data" function to segment and annotate the imported images. Enter the interactive annotation interface and use the annotation tool to annotate the images. After the annotation is completed, use the "Export Data" function, select the Cityscapes dataset format, and specify the output path.

[0106] The dataset was structured according to the Cityscapes dataset format. 100 precisely labeled images were prepared for each class of road surface information, for a total of 600 dataset samples. 20% were randomly selected as the validation set, and the remaining 80% were used as the training set. The dataset was augmented by mirroring, translation, and adjusting the brightness. The training strategy and optimization methods used in the pre-training were continued. The trained network was then validated on the validation set. Finally, the trained semantic segmentation network was used for model inference, which can process the input driving scene images and output visualized segmentation results.

[0107] The experimental results in this embodiment show that the semantic segmentation network trained specifically adapts to the interference of rainy and snowy road surfaces. In the validation set of the dataset, the MIoU of the road surface part of the rainy and snowy road surface improved from 0.5532 to 0.7309, which further improved the accuracy of the constructed semantic segmentation network and made the algorithm more reliable.

[0108] Step 4: Use a segmentation network to extract road surface regions and create a road surface classification network dataset:

[0109] Steps one through three pertain to the construction and discrete training of the semantic segmentation network. However, in the specific application of road surface adhesion coefficient prediction technology, online recognition using the segmentation network is required. Therefore, we use an industrial control computer to simulate the online recognition process of the semantic segmentation network.

[0110] The image sample library containing six road surface types obtained in step three is input into a semantic segmentation network for prediction processing according to road surface category. The process is similar to the online recognition process in daily driving environment. Then, the prediction results and the original image are simultaneously sent to the processor for road surface region extraction. The specific process of extracting road surface regions is as follows: First, the semantic segmentation prediction result image is split according to color channels, and the two-dimensional matrix corresponding to the three color channels is obtained. The BGR value (128, 64, 128) corresponding to "road" is found by looking up the table. Then, masks for the three color channels of BGR are made using OpenCV, and only pixels with the same BGR value as "road" are retained to ensure that only road surface regions are extracted. Finally, the masks of the three color channels are stitched together, and matrix multiplication is performed with the corresponding channels of the original image to extract images containing only road surface regions. Non-road surface regions are set as background and turned black. The extracted image sample library containing only road surface regions is stored in specific folders according to the six types of road surface information, and the specific road surface type and label information with corresponding 5-bit binary information are shown in Table 2.

[0111]

[0112]

[0113] Table 2: Correspondence Table of Road Surface Type and Label Information

[0114] The prepared dataset was filtered according to image quality, and the final dataset contained 6,000 image samples, with 1,000 images for each road surface category. Finally, the road surface image dataset was shuffled and 20% of the images from each category were randomly selected as the validation set, with the remainder used as the training set.

[0115] Step 5: Build and train a road surface image type classification network:

[0116] Steps one through four complete the road surface region extraction process from the image information during the driving process. However, to implement a vision-based road surface adhesion coefficient prediction method, it is also necessary to identify the road surface type based on the extracted road surface region results. Therefore, building and training a road surface image type classification network is crucial. The specific implementation process is as follows:

[0117] The specific implementation process of building a road image type classification network is as follows: First, a road image type classification network is built in the Anaconda environment. Considering the complexity of the road classification task and the real-time requirements of the classification process, a lightweight convolutional neural network is built as the backbone network of the classifier. At the same time, a channel attention mechanism is introduced to reduce the network parameters and floating-point operations as much as possible while ensuring recognition accuracy, thereby improving the real-time performance of the classification network. The specific road image type classification network structure design is shown in Table 3.

[0118]

[0119]

[0120] Table 3: Road Surface Classification Network Structure Table

[0121] First, the input layer of the network adjusts the road image to be identified into a 288×288×3 tensor. Then, it passes through a convolutional layer with a stride of 2 and a 3×3 kernel, combined with batch normalization (BN) and the Swish activation function for initial feature extraction. The resulting feature map is then used as input to the bottleneck module, which extracts features from the input feature map. The structure and specific process of building the network bottleneck module based on the channel attention mechanism are as follows: Figure 3 As shown: First, a 1×1 convolution is used to change the dimension of the feature channels according to the expansion ratio. After batch normalization (BN) and Swish activation, the input is fed into a 3×3 depthwise separable convolutional layer (DWConv) for feature extraction. The obtained feature map is then fed into the channel attention mechanism (SE) module. The module performs global average pooling on the input feature map, followed by Swish activation to aggregate the global features of the input feature map. Then, the obtained global features are fully connected in a 1×1 convolutional layer with a certain activation rate. The sigmoid activation function is then used to obtain the attention weights for different channels. The obtained weights are then multiplied and weighted with the input feature map to retain the main features of the input, remove noise interference, reduce parameter calculation, improve the real-time performance of the network, and achieve the goal of focusing on the more important information channels. Finally, the output of the SE module is passed through a 1×1 convolution to obtain the output of the entire bottleneck module.

[0122] As shown in Table 3, the feature map extracted by a convolutional layer with a stride of 2 and a kernel size of 3×3 is input into two consecutive bottleneck modules with a dilation rate of 1 for feature extraction, resulting in a 144×144×24 feature map. The network structure is designed to use bottleneck modules for feature extraction in the first three layers, so that after each bottleneck layer, the size of the output feature map is reduced to half that of the input feature map, while the number of output channels is double that of the input channels. The first three layers yield a 36×36×64 feature map. The last three feature extraction layers successively adopt different activation ratios. Channel attention values ​​of 0.35, 0.30, and 0.25 were applied, focusing more on feature channels with rich content. This resulted in a 9×9×256 feature map set containing high-level semantic information. This map was then upscaled using a 1×1 convolutional layer. The upscaled 9×9×1280 feature map set was then pooled using a 9×9 global average pooling layer to obtain a 1×1×1280 feature sequence. Finally, the map was passed through a fully connected layer with a Softmax function to output the probability values ​​belonging to each road surface category. The Argmax function was then used to determine the network classification result based on the maximum probability value.

[0123] Then, the road surface image type network was built and trained on a self-built dataset. The training process adopted the cross-entropy loss function and the Adam adaptive gradient descent optimization algorithm. The basic learning rate was 0.0001, and the model and training results were saved according to the number of iterations (epochs).

[0124] In this embodiment, the trained model and parameters are saved, and the online recognition process is simulated. 1200 image samples are randomly selected from the sample library collected from real vehicles for verification. The specific performance of the constructed classification network is shown in Table 4. The numbers on the diagonal of the confusion matrix represent the number of correctly predicted images. That is, a total of 1179 images in the verification samples were correctly predicted. The average accuracy of the model verification is 0.9825, and the image processing time is 0.009s, which meets the accuracy and real-time requirements of road classification technology.

[0125]

[0126]

[0127] Table 4: Confusion Matrix of Road Surface Classification Network Classification Results

[0128] Step 6: Establish mapping rules to obtain road surface adhesion coefficient information:

[0129] Finally, the road surface adhesion coefficient is obtained based on the road surface image type classification results obtained from the above steps. In order to obtain the road surface adhesion coefficient of the identified road surface, the reference value table of vehicle longitudinal slip adhesion coefficient in "GA / T643-2006 Technical Appraisal of Vehicle Driving Speed ​​in Typical Traffic Accident Forms" and the reference value table of vehicle longitudinal slip adhesion coefficient for icy and snowy roads are referenced. Considering the influence of driving speed on the road surface adhesion coefficient, the mapping rule between road surface type and road surface adhesion coefficient as shown in Table 5 is defined. Considering that the vehicle driving speed is usually around 48 km / h in urban conditions, the range of adhesion coefficient values ​​of the current road surface is obtained by referring to Table 5 based on the vehicle speed and road surface type identification results. The midpoint between the upper and lower limits of the adhesion coefficient is taken as the adhesion coefficient of the current road surface, which is the output result of the entire road surface adhesion coefficient prediction algorithm.

[0130] Road surface type Adhesion coefficient Determined value of adhesion coefficient Dry asphalt 0.55-0.8 0.675 wet asphalt 0.45-0.7 0.575 dry cement 0.55-0.8 0.675 wet cement 0.45-0.75 0.600 Snow-covered roads 0.1-0.25 0.225 Icy roads 0.1-0.2 0.150

[0131] Table 5: Mapping Table of Road Surface Type and Adhesion Coefficient

[0132] In this embodiment, the average processing time of the semantic segmentation network, mask extraction and recognition network in the designed road surface adhesion coefficient prediction algorithm is 0.0314 seconds and the accuracy is high, which meets the requirements of the prediction algorithm for accuracy and real-time performance.

Claims

1. A method for predicting road surface adhesion coefficient based on semantic segmentation networks, which predicts the road surface adhesion coefficient based on road type classification in urban driving areas, characterized in that... The specific steps are as follows: Step 1: Construct a semantic segmentation network based on a multi-scale spatial attention mechanism: The vehicle's onboard camera, equipped with an intelligent driving vehicle's perception system, collects video data of the road ahead during the vehicle's journey. A semantic segmentation network is then used to extract the drivable road surface area. First, configure and set up the network environment; Secondly, a semantic segmentation network is built, using an advanced lightweight encoder-decoder framework to ensure the accuracy of the semantic segmentation network while meeting the real-time requirements of the algorithm. Then, within the encoder-decoder framework, a hierarchical semantic segmentation network encoder structure based on the attention mechanism is specifically developed and constructed to generate rich multi-scale semantic features: first, the input image is scaled to a tensor input semantic segmentation network with a size of 1024×1024×3; then, after passing through 3×3 convolutional layers, BN layers, and Gelu activation processing, the initial feature extraction and downsampling functions of the input image are realized. The obtained feature map is then used as input to the feature extraction module. The specific implementation process of the feature extraction module is as follows: First, a 1×1 convolution is used to adjust the number of input channels. After passing through the Gelu activation function, it is input into the spatial attention mechanism module, which consists of a 5×5 main branch convolution and subsequent 7×7 and 13×13 branch convolutions. Then, a 1×1 convolution is used to output the weight parameters representing spatial attention. The formula for the multi-scale spatial attention mechanism is shown below: In formula (1), Atten is the weight parameter representing spatial attention in the output, Atten0 is the weight output of the main branch of the spatial attention module, and Atten... i This is the weight output of the i-th branch convolution in the module, where i is the number of branch convolutions; Next, the number of channels in the feature map is expanded by 1×1 convolution, and then 3×3 convolution is used to extract more detailed features from the feature map after spatial attention weighting. The extraction results are activated by the GeLu function, and finally the processing results of the feature extraction module are output by 1×1 convolution. In the formula, Output is the output of the bottleneck module, F is the input of the bottleneck module, Conv2D(·) represents using convolution to process the input feature map, and Gelu(·) represents using activation functions to further process the feature map; Finally, a lightweight decoder architecture is built based on the encoder-decoder framework: the feature maps extracted by the encoder at different stages are processed sequentially through upsampling, channel concatenation, a lightweight ham_head decoder module, and a fully connected classifier to fuse semantic information at different levels; in order to collect multi-scale semantic information and expand the receptive field, the first stage with more low-level semantic information is discarded, and the feature maps of the second, third, and fourth stages are processed into feature maps of the same size by bilinear interpolation upsampling method, and then channel concatenation is performed before input into the ham_head decoder module to process the integrated and concatenated 128×128×480 feature map; the specific implementation process of the ham_head decoder module is as follows: the 480-channel feature map obtained by the concat operation is compressed to 256 channels by a 1×1 convolutional layer, a 32-group GroupNorm layer, and a ReLU activation layer, and then the non-negative matrix factorization algorithm of NMF shown in formula (3) is used: In m×n =P m×r ×Q r×n (3) The characteristic matrix V m×n Decomposed into low-level matrices P of m×r and r×n m×r and Q r×n , where P m×r The characteristic basis matrix represents the main characteristics of the data; while Q... r×n The feature coefficient matrix represents the distribution of data features. The algorithm uses the feature basis matrix instead of the feature matrix to avoid interference from redundant matrices, accelerate the image processing process, and improve the real-time performance of the algorithm. Finally, the output is processed by a linearization layer consisting of a 1×1 convolutional architecture composed of a 1×1 convolutional layer, 32 groups of GroupNorm layers, and ReLU activation layers, which is the output of the ham_head decoder module. The processed feature map is then fully connected through a 1×1 convolutional layer, followed by a softmax function to convert the class scores into a probability distribution, resulting in a 256×256×nls feature map. Here, the value of each pixel represents the probability of belonging to that semantic class, and nls represents the number of semantic classes to be classified. The resulting feature map is then upsampled using bilinear interpolation, and the ArgMax function is used to predict the semantic class of each pixel according to the maximum probability. The resulting 1024×1024 predicted segmentation image, which is the same size as the input image, is the encoder output. Step 2: Pre-train the constructed segmentation network on a public dataset: The Cityscapes dataset was selected for pre-training of the semantic segmentation network, and the cross-entropy loss function was designed as shown in Equation (4): In the formula, nls represents the semantic category to be classified, which is 19 when trained on the Cityscapes dataset, and y c Let P be a vector with values ​​of 0 or 1. The value of each element is used to determine whether the class is the same as the sample class. c The probability that the predicted sample belongs to class c is c∈(1,nls); The segmentation network is trained using a poly learning rate rule, where the poly learning rate decay expression is as follows: LR in formula (5) initial The initial learning rate for network training is set to 0.002 during pre-training. `iter` is the number of training iterations, `max_iter` is the maximum number of training steps (set to 40K steps), and `power` is the decay coefficient used to control the shape of the learning rate curve (set to 0.9). `LR(iter)` is the learning rate corresponding to the specific number of steps in the training process, calculated and updated. The Adam optimization algorithm is used to dynamically adjust the learning rate of each parameter using the first and second moment estimates of the gradient. For semantic segmentation tasks, the average pixel accuracy (Acc) and average intersection-union ratio (MIoU) based on the confusion matrix are used to evaluate the performance of the network pre-training results. The specific implementation process is as follows: For the binary classification confusion matrix shown in Table 1, each row represents the true class of the data, each column represents the predicted class, and the specific element value in the matrix represents the number of samples predicted as a certain class. Table 1: Schematic diagram of a binary confusion matrix Accuracy Acc represents the percentage of pixels that correctly predict the class out of the total number of pixels, as shown in formula (6): MIoU represents the summation and averaging of the ratio of the intersection to the union of the predicted results and the actual values ​​for each category, as shown in formula (7) below: The training data was saved to a log file using the logging tool in PyTorch, and the training parameter curves were analyzed. The constructed semantic segmentation network showed excellent performance in urban driving scenarios with good weather. Step 3: Enrich the semantic segmentation dataset and perform specific training on the segmentation network. To address the issue of segmentation networks being susceptible to distortion under rainy and snowy weather conditions, this method creates a semantic segmentation dataset containing six road surface types: dry and wet asphalt, dry and wet cement, snow-covered roads, and icy roads. The pre-trained semantic segmentation network is then trained and evaluated on this dataset to further improve the network's generalization ability and better suit its application in everyday urban driving scenarios. Step 4: Use a segmentation network to extract road surface regions and create a road surface classification network dataset: The image sample library containing 6 road surface types obtained in step 3 is input into the semantic segmentation network for prediction processing according to the road surface type. The process is similar to the online recognition process in the daily driving environment. Then, the prediction results and the original images are sent to the processor at the same time to extract the road surface area. Step 5: Build and train a road surface image type classification network: The specific implementation process of building a road surface image type classification network is as follows: First, a road surface image type recognition network is built in the Anaconda environment. Considering the complexity of the road surface classification task and the real-time requirements of the classification process, a lightweight convolutional neural network is built as the backbone network of the classifier. At the same time, a channel attention mechanism is introduced to reduce the network parameters and floating-point operations as much as possible while ensuring recognition accuracy, thereby improving the real-time performance of the classification network. The specific road surface image classification network structure design is shown in Table 3. Table 3: Road Surface Classification Network Structure Table Step 6: Establish mapping rules to obtain road surface adhesion coefficient information: Finally, the road surface adhesion coefficient is obtained based on the road surface image type classification results obtained from the above steps. In order to obtain the road surface adhesion coefficient of the identified road surface, the reference value table of vehicle longitudinal slip adhesion coefficient in "GA / T643-2006 Technical Appraisal of Vehicle Driving Speed ​​in Typical Traffic Accident Forms" and the reference value table of vehicle longitudinal slip adhesion coefficient for icy and snowy roads are referenced. Considering the influence of driving speed on the road surface adhesion coefficient, the mapping rule between road surface type and road surface adhesion coefficient as shown in Table 4 is defined. Considering the low driving speed of vehicles in urban conditions, which is often 48 km / h, the adhesion coefficient range of the current road surface is obtained by referring to Table 4 based on the vehicle speed and road surface type identification results. The midpoint between the upper and lower limits of the adhesion coefficient is taken as the adhesion coefficient of the current road surface, which is the output result of the entire road surface adhesion coefficient prediction algorithm. Table 4: Mapping table of road surface type and adhesion coefficient.

Images