A safe driving guidance method and system based on road surface wetness state detection

Abstract

The present application relates to the field of automobile environment perception, and particularly relates to a safe driving guidance method and system based on road surface wet and slippery state detection; real-time collection of road surface image signals, road surface temperature signals, vehicle speed signals and roadside information in front of the vehicle; input of the road surface image signals and the road surface temperature signals into a pre-trained multi-modal detection model to output a perception result; calculation of a comprehensive road surface-vehicle safety margin index based on the vehicle speed signals, a forward adhesion coefficient prediction curve and a forward risk interval; comparison of the safety margin index with a preset threshold to output a risk level; generation of corresponding driving guidance instructions according to the risk level. The present application overcomes the limitations of a single perception mode, can perform high robustness identification and spatial positioning on various wet and slippery working conditions, and can convert the perception result into a safety margin index to realize prediction and active guidance of driving risks, thereby significantly improving the active safety of the vehicle.

Description

Technical Field

[0001] This invention relates to the field of automotive environmental perception, specifically to a safe driving guidance method and system based on road surface slipperiness detection. Background Technology

[0002] Slippery road conditions are a significant factor affecting driving safety and pose a considerable threat. The grip between a car and the road surface depends on the coefficient of friction between the tires and the road surface, and slippery conditions (such as water accumulation, snow accumulation, and ice) reduce this coefficient of friction. A decrease in the coefficient of friction leads to longer braking distances, sluggish steering response, and a high risk of loss of vehicle control. Therefore, accurately and quickly assessing road slipperiness can significantly reduce the occurrence of traffic accidents.

[0003] To address the aforementioned issues, existing technologies have proposed several detection schemes: one is an analysis method based on cell grid division, which analyzes the state of the road surface by dividing it into grids; the second is a detection method based on acoustic imaging, which uses sound source arrays deployed in tunnels and other scenarios to acquire acoustic images of the road surface for judgment; the third is an analysis method based on monitoring audio, which assesses the degree of slipperiness by analyzing the audio signals generated by vehicles passing over the road surface; and the fourth is a recognition method based on machine learning, which uses trained models to analyze and classify the collected road surface images or sensor data.

[0004] However, methods based on cell grid partitioning rely excessively on grid scale division, resulting in high algorithm complexity, limited real-time performance, and unsuitability for fast-moving vehicles; methods based on acoustic image detection have limited acoustic imaging resolution, insufficient sensitivity to water thickness or thin ice, and require the deployment of sound source arrays, which are complex to maintain; analysis methods based on monitoring audio are easily affected by noise interference, have low signal-to-noise ratios, and are susceptible to the influence of different vehicle models and tire types, exhibiting poor noise robustness; and machine learning-based methods rely on large amounts of data annotation, resulting in high acquisition and annotation costs and poor generalization ability.

[0005] In addition, most existing solutions are limited to single perception or estimation of slippery conditions, failing to deeply integrate with vehicle dynamics and roadside information to form a quantitative safety assessment and driving guidance. At the same time, relying on single perception or estimation is easily affected by environmental factors such as lighting and weather. If multimodal perception fusion is adopted, the large amount of data and complex processing often lead to high system latency, making it difficult to meet the real-time perception needs of vehicles traveling at high speeds. Summary of the Invention

[0006] To address the problems mentioned in the prior art, this invention proposes a safe driving guidance method and system based on road surface slipperiness detection. By acquiring multiple signals in real time and processing the signal parameters using a multimodal detection model, when a change in road surface condition is detected that prevents the vehicle from safely passing under the current condition, the system provides prompts to the driver through a human-machine interface and gives corresponding driving guidance instructions, thereby improving driving safety.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: This invention proposes a safe driving guidance method based on road surface slipperiness detection, comprising the following steps: Real-time acquisition of road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information in front of the vehicle; The road surface image signal and road surface temperature signal are input into a pre-trained multimodal detection model, and the output includes the perception results containing road surface category, road surface region segmentation information and road surface thickness estimation. Based on the perception results, a road surface wetness level distribution map of the forward road surface is determined; a forward adhesion coefficient prediction curve is calculated based on the road surface wetness level distribution map; based on the forward adhesion coefficient prediction curve and the road surface wetness level distribution map, combined with roadside information, a forward risk zone is determined; based on the vehicle speed signal, the forward adhesion coefficient prediction curve, and the forward risk zone, a comprehensive road surface-vehicle safety margin index is calculated. The safety margin index is compared with a preset threshold to output a risk level; based on the risk level, corresponding driving guidance instructions are generated.

[0008] Preferably, the road surface image signal and road surface temperature signal are input into a pre-trained multimodal detection model, and the output is a perception result containing road surface category, road surface region segmentation information, and road surface thickness estimation, including: The multimodal detection model is an improvement on the YOLOv11 network and includes a backbone network, a feature aggregation module, a feature fusion module, a temperature information fusion branch, a thickness estimation regression head, and a detection and segmentation head. The backbone network sequentially performs convolution and attention processing on the road surface image signal to obtain multi-scale visual features. The feature aggregation module performs multi-scale pooling on the multi-scale visual features to obtain deep visual features. The feature fusion module sequentially performs cross-scale fusion and semantic enhancement processing on deep visual features and other levels of visual features in multi-scale visual features to obtain multi-level visual fusion features. The temperature information fusion branch encodes and expands the dimensions of the road surface temperature signal to generate high-dimensional temperature features; and then weights and fuses the high-dimensional temperature features with the multi-level visual fusion features to obtain a fusion feature vector. The thickness estimation regression head sequentially performs upsampling, feature stitching and convolutional regression processing on the multi-level visual fusion features to output a road surface thickness estimate. The detection and segmentation heads respectively perform road condition classification processing on the fused feature vector and segmentation processing on the multi-level visual fusion features to obtain road category and road area segmentation information; The system integrates road surface type, road surface area segmentation information, and road surface thickness estimation to output perception results.

[0009] Preferably, the road surface category in the perception result includes dry road surface, wet road surface, waterlogged road surface, snow-covered road surface, and icy road surface; The road surface thickness is estimated to be either the depth of water accumulation or the thickness of snow accumulation, with a value ranging from 0 mm to 50 mm.

[0010] Preferably, the forward adhesion coefficient prediction curve is calculated based on the road surface slipperiness level distribution map, including: Based on the mapping relationship between the road surface slipperiness level and the road surface adhesion coefficient in each region of the road surface slipperiness level distribution map, the road surface adhesion coefficient at each distance in front of the vehicle is estimated and arranged in spatial order to obtain the forward adhesion coefficient prediction curve.

[0011] Preferably, the formula for calculating the integrated road-vehicle safety margin index is as follows:

[0012] In the formula: For safety margin index; The predicted road surface adhesion coefficient; min is the minimum value; The minimum adhesion coefficient required for the forward risk zone corresponding to the vehicle's current speed; Forward risk zone; For safe braking distance; This represents the average rate of change of the adhesion coefficient over three consecutive adjacent frames. This is the threshold for the rate of change of the adhesion coefficient.

[0013] Preferably, the safety margin index is compared with a preset threshold to output a risk level; based on the risk level, corresponding driving guidance instructions are generated, including: When the safety margin index is ≥1.0, the risk level is safe, and no additional driving instructions are generated; When 0.7 ≤ safety margin index < 1.0, the risk level is mild, and a warning instruction to pay attention to changes in the road surface is generated; When 0.4 ≤ safety margin index < 0.7, the risk level is moderate, and a prompt instruction to reduce speed is generated. When the safety margin index is less than 0.4, the risk level is high risk, generating a warning instruction that the vehicle must slow down and triggering the vehicle's AEB system to intervene with braking.

[0014] Preferably, the system further includes a security monitoring step, which includes: Effectiveness of real-time monitoring of road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information; The cross-modal consistency of the perception results is verified based on road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information to obtain the verification results. When at least one of the road surface image signal, road surface temperature signal, vehicle speed signal, and roadside information is detected to be faulty, or when the verification results indicate that the perception result is inconsistent with any one of the road surface image signal, road surface temperature signal, vehicle speed signal, and roadside information, a downgrade strategy is executed.

[0015] This invention proposes a safe driving guidance system based on road surface slipperiness detection, comprising: The sensing module is used to collect real-time images of the road surface, road surface temperature, vehicle speed, and roadside information in front of the vehicle. The state reasoning module is used to input road surface image signals and road surface temperature signals into a pre-trained multimodal detection model, and output perception results including road surface category, road surface region segmentation information and road surface thickness estimation. The decision output module is used to determine the road surface slippage level distribution map of the forward road surface based on the perception results; calculate the forward adhesion coefficient prediction curve based on the road surface slippage level distribution map; determine the forward risk zone based on the forward adhesion coefficient prediction curve and the road surface slippage level distribution map, combined with roadside information; and calculate the comprehensive road surface-vehicle safety margin index based on the vehicle speed signal, the forward adhesion coefficient prediction curve and the forward risk zone. The driving guidance instruction output module is used to compare the safety margin index with a preset threshold and output the risk level; based on the risk level, it generates corresponding driving guidance instructions.

[0016] This invention proposes a safe driving guidance device based on road surface slipperiness detection, comprising a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the safe driving guidance method based on road surface slipperiness detection as described above.

[0017] This invention proposes a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the safe driving guidance method based on road surface slipperiness detection as described above.

[0018] Compared with the prior art, the present invention achieves the following technical effects: This invention first processes the collected road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information. The road surface image signals and road surface temperature signals are then input into a multimodal detection model to output the perception results of road surface condition detection. Finally, based on a quantified safety margin index, driving guidance instructions with different risk levels are generated, realizing a complete process from environmental perception and risk quantification to safety intervention. This invention overcomes the limitations of single perception, enabling highly robust identification and spatial positioning of various slippery conditions such as water accumulation, snow accumulation, and icing. The perception results are converted into a safety margin index, thereby achieving prediction and proactive guidance of driving risks, significantly improving the active safety of vehicles under complex weather and road conditions.

[0019] This invention constructs a multimodal detection model based on an improved YOLOv11 network and integrates a temperature information fusion branch and a thickness estimation regression head into the multimodal detection model. This enables the fusion of road surface image signals and road surface temperature signals, which can not only accurately distinguish between roads with similar appearances but different physical properties (such as reflective water-filled roads and low-temperature icy roads), but also obtain the perceived results of road surface thickness estimation. This provides a basis for subsequent assessment of the severity of slipperiness and thus improves the reliability of state perception.

[0020] This invention generates a forward adhesion coefficient prediction curve based on perception results, which can dynamically reflect the changing trend of the road surface grip ahead. In addition, by combining roadside information to determine the forward risk zone, it achieves the identification of high-risk road sections. Finally, by fusing vehicle speed signals, the forward adhesion coefficient prediction curve, and the safety margin index calculated from the forward risk zone, it achieves the quantification of the matching degree between the current driving state and the risk of the road surface ahead. Furthermore, the different risk level responses of this invention are triggered based on the safety margin index, forming corresponding driving guidance instructions. This not only provides auxiliary prompts to the driver but also allows for necessary intervention through the linkage of the vehicle's AEB system, achieving optimization of human-machine collaboration and ensuring safety redundancy. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a schematic diagram of the multimodal detection model architecture of the present invention; Figure 3 This is a schematic diagram of the C3k2_ECA module structure of the present invention; Figure 4 This is a schematic diagram of the feature aggregation module structure of the present invention; Figure 5 This is a schematic diagram of the feature fusion module structure of the present invention; Figure 6 This is a schematic diagram of the temperature information fusion branch structure of the present invention; Figure 7 This is a schematic diagram of the thickness estimation regression head structure of the present invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0023] See Figure 1 This embodiment proposes a safe driving guidance method based on road surface slipperiness detection, including the following steps: Real-time acquisition of road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information in front of the vehicle; The road surface image signal and road surface temperature signal are input into a pre-trained multimodal detection model, and the output includes the perception results containing road surface category, road surface region segmentation information and road surface thickness estimation. Based on the perception results, a road surface wetness level distribution map of the forward road surface is determined; a forward adhesion coefficient prediction curve is calculated based on the road surface wetness level distribution map; based on the forward adhesion coefficient prediction curve and the road surface wetness level distribution map, combined with roadside information, a forward risk zone is determined; based on the vehicle speed signal, the forward adhesion coefficient prediction curve, and the forward risk zone, a comprehensive road surface-vehicle safety margin index is calculated. The safety margin index is compared with a preset threshold to output a risk level; based on the risk level, corresponding driving guidance instructions are generated.

[0024] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments: Step 1: This step collects road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information in front of the vehicle. Specifically, the road surface image signals can be acquired using a camera with variable frame rate and exposure time. To improve perception efficiency and image clarity at different vehicle speeds, this embodiment dynamically adjusts the camera's frame rate and exposure time according to the vehicle speed, specifically as follows:

[0025] In the formula: For camera frame rate; This is the frame rate gain coefficient, set to 0.5; Vehicle speed; The minimum frame rate is set to 30. The maximum frame rate is set to 60; t is the exposure time. The base exposure time is set to 5ms. This is the exposure time adjustment factor, set to 0.01; This is a truncation function used to restrict the result to a certain upper and lower bound range.

[0026] This embodiment uses a non-contact road temperature sensor to collect road surface temperature signals. The non-contact road temperature sensor collects road surface temperature signals at a sampling frequency of 1Hz. The embodiment obtains vehicle speed signals through a CAN (Controller Area Network) bus and obtains roadside information of the current road segment through V2X (Vehicle-to-Everything) technology. The roadside information includes speed limit information and emergency event information of passing vehicles.

[0027] It should be noted that in this embodiment, the collected road surface image signal and road surface temperature signal are packaged into a unified data frame through a time synchronization mechanism and transmitted to the multimodal detection model via Ethernet.

[0028] Step 2: The multimodal detection model in this embodiment is an improvement on the YOLOv11 (You Only Look Once Version 11, end-to-end single-stage target detection version 11) network. Specifically, with the YOLOv11 network as the core, a feature fusion module, a temperature information fusion branch, and a thickness estimation regression head are introduced into the YOLOv11 network structure to achieve deep fusion and joint judgment of road surface image signals and road surface temperature signals, thereby improving the detection accuracy and environmental adaptability of the multimodal detection model under complex climate and lighting conditions.

[0029] See Figure 2 The multimodal detection model in this embodiment includes a backbone network, a feature aggregation module, a feature fusion module, a temperature information fusion branch, a thickness estimation regression head, and a detection and segmentation head.

[0030] In this embodiment, the backbone network adopts a five-layer structure, each layer containing a C3k2_ECA (C3k2 Module with Efficient Channel Attention) module and a convolutional layer; the C3k2_ECA module consists of a convolutional feature extraction unit, a channel attention modeling unit, and a residual fusion unit.

[0031] like Figure 3 As shown, specifically, the road surface image signal The input is fed into the convolutional feature extraction unit, where Let H be the real number field, W be the height, and C be the width. The convolutional feature extraction unit consists of two standard 3×3 convolutional layers with a stride of 1 and padding of 1, along with batch normalization and the SiLU (Sigmoid Linear Unit) activation function. After two convolutional processes, an enhanced feature map with unchanged spatial dimensions is obtained. This process can extract local road surface structure, providing more expressive base signals for the channel attention modeling unit.

[0032] Enhance feature maps The input is fed into the channel attention modeling unit; specifically, global average pooling is first performed in the spatial dimension to compress the enhanced feature map into channel description vectors. Then, a one-dimensional convolution with a kernel size of 3 is used to modify the channel description vector. Local features are extracted, and then the attention weight vector is obtained by activating the sigmoid function (S-shaped function). ; Integrate the attention weight vector with the enhanced feature map Multiply each channel individually to form a weighted feature. This process achieves a dynamic adjustment of "enhancing important channels and suppressing unimportant channels".

[0033] Finally, the residual fusion unit will integrate the road surface image signal. Features after weighting Element-wise addition is performed to obtain multi-scale visual features. The residual fusion unit further ensures the stability of the backbone network training, allowing the channel attention modeling unit to generate gains without destroying the original expression.

[0034] like Figure 2 As shown, after processing through the five-layer structure of the backbone network, multi-scale visual features are obtained; the output of the third layer of the multi-scale visual features is the shallow visual feature, and the output of the fourth layer is the medium visual feature. The shallow and medium visual features are used for subsequent feature fusion module processing.

[0035] Between the backbone network and the neck network, the SPPF (Spatial Pyramid Pooling-Fast) module in the original YOLOv11 network is replaced with the DySPPF (Dynamic Spatial Pyramid Pooling-Fast) module, which is the feature aggregation module in this embodiment.

[0036] like Figure 4As shown, the feature aggregation module in this embodiment includes four parts: an input preprocessing unit, a multi-scale pooling unit, a dynamic convolution weight generation unit, and a feature fusion output unit.

[0037] Specifically, preprocessing units reduce computational complexity and unify the channel dimensions of subsequent pooling branches. 1×1 convolutional layers are used to process the multi-scale visual features of the backbone network. Perform channel compression processing:

[0038] In the formula: It is a 1×1 convolution.

[0039] The compressed feature dimensions are:

[0040] In the formula: , This is the compressed channel; These are the compressed features.

[0041] The preprocessing unit in the embodiment can avoid parameter redundancy in multi-scale pooling features and improve the efficiency of feature aggregation.

[0042] The compressed features are input into a multi-scale pooling unit, which consists of three parallel pooling branches with kernel sizes of [5, 9, 13]. Small-scale pooling is used to capture fine textures and shallow water accumulation patterns; medium-scale pooling is suitable for medium-scale thin snow blocks and local structural changes; and large-scale pooling is used to expand the receptive field and capture large areas of reflection and low-texture surfaces. The three pooling operations are as follows:

[0043] In the formula: For max pooling operations, k For pooling core size, s Step size, p To fill; For small-scale pooling; For mesoscale pooling; This is for large-scale pooling.

[0044] The dimensions of multi-scale pooling features are:

[0045] Global average pooling is performed on the multi-scale pooling features, and the weight coefficients corresponding to the three pooling branches are obtained through nonlinear mapping. ,in These are the small-scale pooling weight coefficients. These are the weights for mesoscale pooling. These are the large-scale pooling weight coefficients; used to adjust the contribution of multi-scale pooling features (small-scale, medium-scale, and large-scale pooling) to subsequent processes.

[0046] The dynamic convolutional weight generation unit adaptively fuses multi-scale pooling features by controlling signals:

[0047] In the formula: This is a dynamic fusion feature.

[0048] It should be noted that in highly reflective scenes, the weight of large-scale pooling can be increased; in weak-texture scenes, the weight of medium-scale pooling can be increased; and in high-texture detail scenes, the weight of small-scale pooling can be increased, thus providing better scene adaptability.

[0049] The feature fusion output unit reassembles the dynamically fused features through a 3×3 convolutional layer to obtain deep visual features. :

[0050] In the formula: It is a 3×3 convolution.

[0051] The dimensions of deep visual features are as follows:

[0052] In this embodiment, the MSA (Multi-Scale Attention) module is used to replace the PAN-FPN (Path Aggregation Network with Feature Pyramid Network) module in the original neck network, which is the feature fusion module in this embodiment.

[0053] The feature fusion module consists of a channel compression layer, a self-attention computation layer, a spatial semantic enhancement layer, and a feature fusion output layer, such as... Figure 5 As shown below, the feature fusion module processing procedure is explained in detail: The input features include deep visual features F2 from the feature aggregation module, mid-level visual features F4 from the multi-scale visual features, and shallow visual features F3.

[0054] First, the deep visual feature F2 is upsampled to improve the spatial resolution, resulting in the first upsampled visual feature U2.

[0055] The first upsampled visual feature U2 and the middle-layer visual feature F4 are input into the channel compression layer and compressed to a uniform dimension of 128 channels through 1×1 convolution to obtain the compressed first upsampled visual feature. and compressed mid-level visual features .

[0056] Compress the first upsampled visual features and compressed mid-level visual features The input to the self-attention computation layer will compress the first upsampled visual features. By mapping the convolutional layers to query vectors, the compressed mid-layer visual features are... The convolutional layers map the vectors to key and value vectors respectively, as shown in the following formulas: , ,

[0057] In the formula: To compress the first upsampled visual features; For compressing mid-level visual features; K is the key vector; V is the value vector; Q is the query vector.

[0058] Calculate the attention weight matrix:

[0059] In the formula: A represents the attention weight matrix; T represents the transpose; This represents the normalized exponential function.

[0060] Attention-enhanced features are generated by weighted summation of value vectors based on the attention weight matrix. :

[0061] Attention-enhancing features The input is fed into the spatial semantic enhancement layer. In this embodiment, the spatial semantic enhancement layer consists of two 3×3 convolutional layers and a SiLU activation function, which is used to further extract local spatial context, enabling the attention features to have both global attention and local convolutional expressive power, and outputting the enhanced features. .

[0062] Enhanced features The enhanced features are first input to the feature fusion output layer. Compressed first upsampled visual features Channel concatenation is performed to obtain concatenated features. These features are then input into a 3×3 convolutional layer to restore the target number of channels, outputting fused features. These fused features are then input into the C3k2 (Cross Stage Partial Kernel 2) module for convolution to extract more discriminative features, outputting first-level visual fusion features. .

[0063] First-level visual fusion features Upsampling is performed to improve spatial resolution, resulting in a second upsampled visual feature U3. The above steps are repeated for the second upsampled visual feature U3 and the shallow visual feature F3 to calculate the second-level visual fusion feature. .

[0064] Secondary visual fusion features Upsampling is performed to improve spatial resolution, resulting in a second upsampled visual feature U4; the above steps are repeated for the second upsampled visual feature U4 to output the third-level visual fusion feature. By fusing primary visual features Secondary visual fusion features and third-level visual fusion features Multi-level visual fusion features are obtained through integration.

[0065] This embodiment adds a temperature information fusion branch to the multimodal detection model, which is used to incorporate the physical characteristics of road surface temperature into multi-level visual fusion features, enabling the multimodal detection model to distinguish between road surfaces that are "similar in appearance but different in physical properties." For example... Figure 6 As shown: First, a non-contact road temperature sensor acquires the road surface temperature signal, which is then synchronized with the camera's frame rate using a timestamp, ensuring a correspondence between the road surface temperature signal and the road surface image signal. The road surface temperature signal is then normalized as follows:

[0066] In the formula: This is the road surface temperature signal; The minimum temperature value within the normalized temperature range is taken as -30. ℃ ; The maximum temperature value within the temperature normalization range is set to 50. ℃ ; This is the normalized temperature value, ranging from 0 to 1.

[0067] The normalized temperature values ​​are input into a two-layer fully connected network to extract semantic features of temperature. The weight matrix of the first fully connected network is set to a dimension of 64×32, mapping the normalized temperature values ​​to 32-dimensional primary temperature features, achieving initial nonlinear compression. The weight matrix of the second fully connected network is set to a dimension of 32×16, further extracting the semantic features of the primary temperature features to obtain low-dimensional temperature features. .

[0068] Low-dimensional temperature characteristics The input is fed into a dimension-expanding fully connected layer to convert low-dimensional temperature features. Mapped to the same feature dimension (1024-dimensional) as the multi-level visual fusion features, and scaled using ReLU (Rectified Linear Unit) and LayerNorm (Layer Normalization) functions to generate high-dimensional temperature features. .

[0069] High-dimensional temperature characteristics The fused features are weighted and summed with the multi-level visual features according to fixed weights to obtain the fused feature vector, as shown in the following formula:

[0070] In the formula: This is a multi-level visual fusion feature; To fuse feature vectors.

[0071] It should be noted that the weight of multi-level visual fusion features is greater than that of high-dimensional temperature features. This not only highlights the dominant role of multi-level visual fusion features, but also supplements physical attribute information through high-dimensional temperature features, thereby improving classification accuracy.

[0072] The temperature information fusion branch in this embodiment can not only improve the stability of the multimodal detection model in reflective, shadow, and low-light nighttime scenarios, but also distinguish between ice surfaces, water films, and slippery asphalt based on temperature differences. Through weighted fusion operations, the consistency of classification confidence can be significantly improved and the probability of false detection can be reduced without destroying the original structure of the multimodal detection model.

[0073] The multimodal detection model in this embodiment can simultaneously perform two tasks: 1. Obtain road surface category and road surface region segmentation information; 2. Obtain road surface thickness estimation.

[0074] Therefore, while retaining the original detection and segmentation heads in the multimodal detection model, a new thickness estimation regression head is added for road thickness estimation.

[0075] like Figure 7As shown, in this embodiment, the thickness estimation regression head utilizes multi-level visual fusion features output by the feature fusion module. These multi-level visual fusion features consist of a single-level visual fusion feature. Secondary visual fusion features and third-level visual fusion features Composition; due to the different spatial dimensions of each visual fusion feature, specifically as follows: , , Therefore, it is necessary to first align the spatial dimensions of each visual fusion feature.

[0076] Specifically, firstly, the secondary visual fusion features are... and third-level visual fusion features The spatial dimensions are enlarged respectively, for example, by fusing secondary visual features. Upsampling by 2x, three-level visual fusion features Upsampled by 4 times, both are fused with primary visual features. Align to size 80×80.

[0077] First-level visual fusion features after spatial dimension alignment Secondary visual fusion features and third-level visual fusion features By concatenating along the channel dimension, the fused features are obtained. :

[0078] Fusion features The input is fed into a 1×1 convolutional layer, compressing the number of channels from 1792 to 256 to reduce the channel dimension, reduce the amount of subsequent computation, and obtain compressed features.

[0079] The compressed features are input into two depthwise separable convolutional layers. Each depthwise separable convolutional layer first performs a channel-wise 3×3 spatial convolution, followed by a 1×1 pointwise convolution to extract local thickness gradients, edge variations, and water film / snow layer texture variations related to thickness. The extracted features are then input into a 1×1 convolutional layer, mapping the channel dimension to 1 to generate a road surface thickness estimate. .

[0080] To ensure the output conforms to physical meaning, numerical constraints are applied to the road surface thickness estimation, limiting all pixel values ​​in the estimation to a preset reasonable range. In this embodiment, a preferred range is 0mm to 50mm. .

[0081] In this embodiment, the detection and segmentation head receives fused feature vectors from the temperature fusion branch and multi-level visual fusion features from the feature fusion module, and performs road surface category classification and road surface region segmentation in parallel.

[0082] Specifically, the fused feature vector is input into the fully connected layer, the information is integrated and mapped to the category space, and the probability of belonging to each preset road surface category, such as dry, wet, waterlogged, snowy, and icy, is calculated by the Softmax (normalization exponent) function. The category with the highest probability is taken as the road surface category.

[0083] Multi-level visual fusion features are input into the detection and segmentation head. By sequentially performing convolution and upsampling on the multi-level visual fusion features, a size segmentation map is obtained. The Softmax function is applied to each pixel in the size segmentation map to calculate the probability of each pixel belonging to each road surface category. The category with the highest probability is selected as the predicted road surface category for that pixel, and finally, road surface region segmentation information is generated.

[0084] By integrating information on pavement thickness estimation, pavement type, and pavement area segmentation, the perception results are obtained.

[0085] Step 3: This step receives the perception results from Step 2 and combines them with the obtained vehicle speed signal and roadside information to determine the road surface condition of the section of road the vehicle is traveling on. Specifically, the perception results are checked for consistency with the road surface condition to obtain a prediction result. If the prediction result conforms to the preset logic, the prediction result is retained.

[0086] For example, when the road surface temperature is greater than 0℃, if the predicted road surface is waterlogged, the prediction result conforms to the preset logic and is retained; if the predicted road surface is icy, it is considered a low-probability event and therefore does not conform to the preset logic. In this case, the prediction result of the current frame is compared with the prediction results of the previous five frames. If the prediction result of the current frame is consistent with the prediction results of the previous five frames, it is considered that the non-contact road temperature sensor has malfunctioned and an alarm is triggered; if the prediction result of the current frame is inconsistent with the prediction results of the previous five frames, the prediction result of the current frame is considered an abnormal frame.

[0087] When an abnormal frame occurs, the prediction results of the frame before and the frame after the abnormal frame are averaged or fused, and this result is used as the prediction result after the abnormal frame is corrected, so as to eliminate the influence of the abnormal frame on the overall prediction result and thus obtain the corrected prediction result.

[0088] After obtaining the prediction results, the prediction results corresponding to each frame are arranged in the order of timestamps, and the vehicle travel distance between adjacent prediction frames is calculated by combining the vehicle's current driving speed and the time interval of road image signal acquisition, so as to determine the spatial position of each prediction result in the road in front of the vehicle; then the prediction results corresponding to each position are arranged according to the vehicle's driving direction to generate a road surface slipperiness level distribution map within a certain distance range in front of the vehicle.

[0089] After obtaining the road surface slipperiness level distribution map, the road surface adhesion coefficient at each distance in front of the vehicle is estimated based on the mapping relationship between the road surface slipperiness level and the road surface adhesion coefficient. The road surface adhesion coefficient is then arranged in spatial order to form a curve showing the change of the road surface adhesion coefficient in front of the vehicle with distance, i.e., the forward adhesion coefficient prediction curve.

[0090] Subsequently, based on the road surface slipperiness level distribution map and the forward adhesion coefficient prediction curve, a comprehensive judgment is made on the slipperiness state of the road ahead of the vehicle. When the slipperiness level in a certain interval is higher than a set threshold or the adhesion coefficient is lower than a safe adhesion coefficient threshold, the interval is determined to be a potential slippery risk area. In this embodiment, the safe adhesion coefficient threshold is 0.4. Combining the acquired roadside information, the road surface condition information uploaded by past vehicles on the same road segment is compared and analyzed. When the vehicle's perception result is consistent with the historical vehicle information, indicating that the road segment is slippery or has low adhesion, the road segment is determined to be a risk interval, thus realizing the prediction of forward risk intervals for the road segment ahead of the vehicle and determining the forward risk interval.

[0091] Based on vehicle speed signals, forward adhesion coefficient prediction curves, and forward risk zones, the comprehensive road-vehicle safety margin index is calculated using the following formula:

[0092] In the formula: For safety margin index; This represents the predicted road surface adhesion coefficient; min is the minimum value. The minimum adhesion coefficient required for the forward risk zone corresponding to the vehicle's current speed, where Let g be the vehicle speed, and g be the acceleration due to gravity. Forward risk zone; Safe braking distance (m); This represents the rate of change of the average adhesion coefficient over three consecutive frames, reflecting the dynamic evolution of the road surface adhesion coefficient. This represents the rate of change of the adhesion coefficient of the image in frame t. This represents the rate of change of the adhesion coefficient of the (t-1)th frame. This represents the rate of change of the adhesion coefficient of the image in frame t-2; in , This represents the prediction attachment coefficient of the image in frame t. This represents the prediction adhesion coefficient of the (t-1)th frame image. The time interval between two adjacent frames; The adhesion change rate threshold is set at 0.07, 0.06, 0.05, 0.04, and 0.02 for dry, wet, waterlogged, snow-covered, and icy road surfaces, respectively.

[0093] Step 4: In this embodiment, the risk level is determined based on the calculated road-vehicle safety margin index, and corresponding driving guidance instructions are generated according to the risk level. Specifically, when IVRSI ≥ 1.0, it is considered safe and no warning is given; when 0.7 ≤ IVRSI < 1.0, it is considered a minor risk and the human-machine interface prompts "Pay attention to road changes"; when 0.4 ≤ IVRSI < 0.7, it is considered a moderate risk and the human-machine interface prompts "It is recommended to reduce speed"; when IVRSI < 0.4, it is considered a high risk and the human-machine interface prompts "You must reduce speed," and the AEB (Automatic Emergency Braking) system is activated to intervene.

[0094] The method in this embodiment also includes a security monitoring step, which is performed concurrently with steps one, two, and three, and the specific process is as follows: By analyzing the road surface image signals captured by the camera, the quality of the road surface image signals (such as image clarity and lighting conditions) is analyzed to determine whether it is normal; if occlusion or abnormality is detected, the camera priority can be adjusted.

[0095] Monitor the fluctuation range and update frequency of the road surface temperature signal output by the non-contact road temperature sensor. If the data remains unchanged for a long time, the jump amplitude is abnormal, or the sampling frequency is lower than 1Hz, the non-contact road temperature sensor is determined to be faulty.

[0096] If the vehicle speed signal received via the CAN bus is lost, remains at zero (while the vehicle is in motion), or exceeds the vehicle's limits, the vehicle speed signal is considered abnormal.

[0097] Monitor the frequency, validity, and consistency of roadside information acquired by V2X. If no roadside information is received for several consecutive cycles, or if the received roadside information is in the wrong format, the V2X is determined to be in "low trust" mode or interrupted.

[0098] If the camera cannot provide a valid road surface image signal, the non-contact road temperature sensor malfunctions, the vehicle speed signal is invalid, or roadside information is lost, an alarm will be triggered and reliance on the sensor data will be reduced.

[0099] This embodiment performs cross-modal consistency verification on the sensing results to obtain the verification results; the specific process is as follows: Camera and V2X consistency: The perception results output in step two are compared with the roadside information acquired by V2X. If the roadside information reports water accumulation in a certain road section, but the road surface image signal identifies the road surface as dry in that section, and there is no change for several consecutive frames, a consistency alarm is triggered, indicating that there is a data contradiction.

[0100] Consistency between non-contact road temperature sensor and camera: Consistency verification is performed based on road surface image signal and road surface temperature signal. Specifically, for example, if the road surface temperature signal indicates the possibility of icing (temperature below 0℃), it is compared with the sensing result output in step two. If the temperature is below 0℃, but the identified road surface image signal shows icing or signs of icing, it indicates that the results of the camera and non-contact road temperature sensor are consistent. If the road surface image signal identified by the camera shows water accumulation or dryness (non-icing), it is determined that there is a contradiction. In this case, abnormal frames are eliminated by verifying multiple consecutive frames of sensing results.

[0101] Consistency between vehicle speed signal and roadside information: The driving status is determined based on the vehicle speed signal and compared with the perception results; for example, if the vehicle speed is too high and the perception results identify a wet or icy road surface, a prompt will be made on the human-machine interface to remind the driver to slow down.

[0102] The degradation strategy in this embodiment is used to reduce reliance on the driver assistance system. Specifically, if the quality of the acquired road surface image signal is too low, the weight of the road surface image signal in the multimodal detection model in step two is reduced. If the non-contact road temperature sensor fails, the temperature information fusion branch is disabled in the multimodal detection model in step two, and the perception result is obtained solely by relying on the road surface image signal. If V2X communication is interrupted, the roadside information is not used in step three, and the risk range is determined solely by the forward adhesion coefficient prediction curve.

[0103] The embodiment also displays the system status to the driver in real time through a human-machine interface; for example, when the camera malfunctions, the screen displays "Camera malfunction, please drive with caution" and is accompanied by a voice prompt; when the non-contact road temperature sensor is unstable, it prompts "Non-contact road temperature sensor is abnormal, the risk of road icing may be misjudged"; in addition, when the downgrade strategy takes effect, the driver is simultaneously informed that the current system is in a downgraded operation state, and it is recommended to reduce the vehicle speed and increase attention.

[0104] Based on the same inventive concept, this invention also provides a safe driving guidance system based on road surface slipperiness detection. Since the principle of this safe driving guidance system based on road surface slipperiness detection is similar to that of the aforementioned safe driving guidance method based on road surface slipperiness detection, the implementation of this safe driving guidance system based on road surface slipperiness detection can refer to the implementation of the safe driving guidance method based on road surface slipperiness detection, and the repeated parts will not be described again.

[0105] In specific implementation, the safe driving guidance system based on road surface slipperiness detection provided in this embodiment of the invention specifically includes: The sensing module is used to collect real-time images of the road surface, road surface temperature, vehicle speed, and roadside information in front of the vehicle. The state reasoning module is used to input road surface image signals and road surface temperature signals into a pre-trained multimodal detection model, and output perception results including road surface category, road surface region segmentation information and road surface thickness estimation. The decision output module is used to determine the road surface slippage level distribution map of the forward road surface based on the perception results; calculate the forward adhesion coefficient prediction curve based on the road surface slippage level distribution map; determine the forward risk zone based on the forward adhesion coefficient prediction curve and the road surface slippage level distribution map, combined with roadside information; and calculate the comprehensive road surface-vehicle safety margin index based on the vehicle speed signal, the forward adhesion coefficient prediction curve and the forward risk zone. The driving guidance instruction output module is used to compare the safety margin index with a preset threshold and output the risk level; based on the risk level, it generates corresponding driving guidance instructions.

[0106] Accordingly, this embodiment of the invention also provides a safe driving guidance device based on road surface slipperiness detection, including a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the safe driving guidance method based on road surface slipperiness detection as provided in this embodiment of the invention.

[0107] For more detailed information on the above methods, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0108] Accordingly, embodiments of the present invention also provide a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described safe driving guidance method based on road surface slipperiness detection provided in embodiments of the present invention.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems, devices, and storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0110] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0112] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] The above provides a detailed description of the safe driving guidance method, system, equipment, and storage medium based on road surface slipperiness detection provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A safe driving guidance method based on road surface slipperiness detection, characterized in that, Includes the following steps: Real-time acquisition of road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information in front of the vehicle; The road surface image signal and road surface temperature signal are input into a pre-trained multimodal detection model, and the output includes the perception results containing road surface category, road surface region segmentation information and road surface thickness estimation. Based on the perception results, a distribution map of the road surface slipperiness level of the road ahead is determined; The forward adhesion coefficient prediction curve was calculated based on the road surface slipperiness level distribution map; Based on the forward adhesion coefficient prediction curve and the road surface slipperiness level distribution map, combined with roadside information, the forward risk zone is determined. Based on vehicle speed signal, forward adhesion coefficient prediction curve and forward risk zone, the comprehensive road-vehicle safety margin index is calculated. The safety margin index is compared with a preset threshold to output the risk level; Based on the risk level, generate corresponding driving instructions.

2. The safe driving guidance method based on road surface slipperiness detection according to claim 1, characterized in that, The road surface image signal and road surface temperature signal are input into a pre-trained multimodal detection model, which outputs perception results including road surface category, road surface region segmentation information, and road surface thickness estimation, including: The multimodal detection model is an improvement on the YOLOv11 network and includes a backbone network, a feature aggregation module, a feature fusion module, a temperature information fusion branch, a thickness estimation regression head, and a detection and segmentation head. The backbone network sequentially performs convolution and attention processing on the road surface image signal to obtain multi-scale visual features. The feature aggregation module performs multi-scale pooling on the multi-scale visual features to obtain deep visual features. The feature fusion module sequentially performs cross-scale fusion and semantic enhancement processing on deep visual features and other levels of visual features in multi-scale visual features to obtain multi-level visual fusion features. The temperature information fusion branch encodes and expands the dimensions of the road surface temperature signal to generate high-dimensional temperature features; and then weights and fuses the high-dimensional temperature features with the multi-level visual fusion features to obtain a fusion feature vector. The thickness estimation regression head sequentially performs upsampling, feature stitching and convolutional regression processing on the multi-level visual fusion features to output a road surface thickness estimate. The detection and segmentation heads respectively perform road condition classification processing on the fused feature vector and segmentation processing on the multi-level visual fusion features to obtain road category and road area segmentation information; The system integrates road surface type, road surface area segmentation information, and road surface thickness estimation to output perception results.

3. The safe driving guidance method based on road surface slipperiness detection according to claim 2, characterized in that, The road surface categories in the perception results include dry road surface, wet road surface, waterlogged road surface, snow-covered road surface, and icy road surface; The road surface thickness is estimated to be either the depth of water accumulation or the thickness of snow accumulation, with a value ranging from 0 mm to 50 mm.

4. The safe driving guidance method based on road surface slipperiness detection according to claim 1, characterized in that, The forward adhesion coefficient prediction curve is calculated based on the road surface slipperiness level distribution map, including: Based on the mapping relationship between the road surface slipperiness level and the road surface adhesion coefficient in each region of the road surface slipperiness level distribution map, the road surface adhesion coefficient at each distance in front of the vehicle is estimated and arranged in spatial order to obtain the forward adhesion coefficient prediction curve.

5. The safe driving guidance method based on road surface slipperiness detection according to claim 1, characterized in that, The formula for calculating the comprehensive road-vehicle safety margin index is as follows: In the formula: For safety margin index; min is the minimum value; The predicted road surface adhesion coefficient; The minimum adhesion coefficient required for the forward risk zone corresponding to the vehicle's current speed; Forward risk zone; For safe braking distance; This represents the average rate of change of the adhesion coefficient over three consecutive adjacent frames. This is the threshold for the rate of change of the adhesion coefficient.

6. The safe driving guidance method based on road surface slipperiness detection according to claim 1, characterized in that, The safety margin index is compared with a preset threshold to output the risk level; Based on the risk level, generate corresponding driving instructions, including: When the safety margin index is ≥1.0, the risk level is safe, and no driving guidance instructions are generated; When 0.7 ≤ safety margin index < 1.0, the risk level is mild, and a warning instruction to pay attention to changes in the road surface is generated; When 0.4 ≤ safety margin index < 0.7, the risk level is moderate, and a prompt instruction to reduce speed is generated. When the safety margin index is less than 0.4, the risk level is high risk, generating a warning instruction that the vehicle must slow down and triggering the vehicle's AEB system to intervene with braking.

7. The safe driving guidance method based on road surface slipperiness detection according to claim 1, characterized in that, It also includes a security monitoring step, which includes: Effectiveness of real-time monitoring of road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information; The cross-modal consistency of the perception results is verified based on road surface image signals, road surface temperature signals, vehicle speed signals, and roadside information to obtain the verification results. When at least one of the road surface image signal, road surface temperature signal, vehicle speed signal, and roadside information is detected to be faulty, or when the verification results indicate that the perception result is inconsistent with any one of the road surface image signal, road surface temperature signal, vehicle speed signal, and roadside information, a downgrade strategy is executed.

8. A safe driving guidance system based on road surface slipperiness detection, characterized in that, include: The sensing module is used to collect real-time images of the road surface, road surface temperature, vehicle speed, and roadside information in front of the vehicle. The state reasoning module is used to input road surface image signals and road surface temperature signals into a pre-trained multimodal detection model, and output perception results including road surface category, road surface region segmentation information and road surface thickness estimation. The decision output module is used to determine the distribution map of road surface slippage level of the road surface ahead based on the perception results; The forward adhesion coefficient prediction curve was calculated based on the road surface slipperiness level distribution map; Based on the forward adhesion coefficient prediction curve and the road surface slipperiness level distribution map, combined with roadside information, the forward risk zone is determined. Based on vehicle speed signal, forward adhesion coefficient prediction curve and forward risk zone, the comprehensive road-vehicle safety margin index is calculated. The driving guidance instruction output module is used to compare the safety margin index with a preset threshold and output the risk level; Based on the risk level, generate corresponding driving instructions.

9. A safe driving guidance device based on road surface slipperiness detection, characterized in that, It includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the safe driving guidance method based on road surface slipperiness detection as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the safe driving guidance method based on road surface slipperiness detection as described in any one of claims 1 to 7.

Images