Road adhesion coefficient estimation method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of vehicles, and provides a road adhesion coefficient estimation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an image in front of a vehicle based on a vehicle-mounted camera, carrying out the pixel-level classification of a key road region in the image through employing a semantic segmentation network, and obtaining a road type and visual confidence, a visual attachment coefficient is obtained based on pavement type matching; real-time dynamic data of the vehicle are obtained, and a dynamic attachment coefficient and a dynamic confidence coefficient are calculated through the vehicle dynamic model and the state observer based on the dynamic data; and predicting the time when the vehicle is about to drive into the key road surface area based on the dynamic data, and when the vehicle drives into the key road surface area, carrying out weighted fusion on the visual adhesion coefficient and the dynamic adhesion coefficient based on the visual confidence coefficient and the dynamic confidence coefficient to obtain a final road surface adhesion coefficient. According to the method, the device, the electronic equipment and the storage medium provided by the invention, the accuracy and the robustness of an estimation result are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to a method, apparatus, electronic device, and storage medium for estimating road surface adhesion coefficient. Background Technology

[0002] As Advanced Driver Assistance Systems (ADAS) evolve towards higher levels of autonomous driving, unprecedented demands are placed on the completeness and accuracy of environmental perception. Vehicles not only need to identify obstacles and lane markings, but also need to accurately perceive the interaction between the tires and the road surface. Among these, the road adhesion coefficient, as a key parameter, directly determines the vehicle's longitudinal braking safety distance and lateral handling stability boundaries.

[0003] In existing technologies, the sources of perception for road surface adhesion coefficient are too singular and modular. Mainstream solutions heavily rely on wheel-end signals such as wheel speed sensors in vehicle electronic stability systems for indirect estimation. These methods are based on the relationship between tire slip ratio and adhesion coefficient during vehicle braking or driving, resulting in inherent lag and inaccuracy. First, it is essentially a "post-hoc estimation," only starting calculation after significant slippage occurs on the drive or braking wheels, failing to provide proactive warnings about the road surface the vehicle is about to enter, and thus failing to meet the prediction time requirements of active safety systems. Second, the estimation accuracy is highly susceptible to vehicle handling conditions. Under conditions of extremely low tire slip ratio, such as constant speed cruising, the estimation model often fails or has extremely low confidence due to the lack of sufficient dynamic excitation. This lack of perception capability makes it difficult for ADAS to make advance perception and adaptive decisions like human drivers when facing complex or severe weather conditions such as localized water accumulation, black ice, or compacted snow, which cause a sudden drop in adhesion coefficient. This significantly limits the reliability and robustness of the system in real-world complex scenarios. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for estimating road surface adhesion coefficient, in order to overcome the shortcomings of existing technologies in estimating road surface adhesion coefficient, such as lag, poor accuracy, and lack of foresight and self-learning capabilities.

[0005] This invention provides a method for estimating the road surface adhesion coefficient, comprising: The vehicle acquires images of the front of the vehicle using an in-vehicle camera, and uses a semantic segmentation network to perform pixel-level classification of key road surface areas in the images to obtain road surface type and visual confidence. The visual adhesion coefficient is then obtained based on the road surface type matching. Real-time dynamic data of the vehicle is acquired, and the dynamic adhesion coefficient and dynamic confidence level are calculated based on the dynamic data through the vehicle dynamics model and state observer. Based on the dynamic data, the time when the vehicle is about to enter the critical road surface area is predicted. When the vehicle enters the critical road surface area, the visual adhesion coefficient and the dynamic adhesion coefficient are weighted and fused based on the visual confidence and the dynamic confidence to obtain the final road surface adhesion coefficient.

[0006] According to the road surface adhesion coefficient estimation method provided by the present invention, the step of weightedly fusing the visual adhesion coefficient and the dynamic adhesion coefficient based on the visual confidence score and the dynamic confidence score to obtain the final road surface adhesion coefficient includes: The following formula is used for weighted fusion: μ _ 融合的 =( C _ 视觉 μ _ 视觉 + C _ 动力学 μ _ 动力学 ) / ( C _ 视觉 + C _ 动力学 ) In the formula, μ _ 融合的 The final road surface adhesion coefficient, C _ 视觉 For visual confidence, μ _ 视觉 The visual adhesion coefficient, C _ 动力学 For the confidence level of the dynamics, μ _ 动力学 This is the dynamic adhesion coefficient.

[0007] According to the road surface adhesion coefficient estimation method provided by the present invention, the visual confidence level is determined based on the image sharpness and / or road surface type classification probability of the image.

[0008] According to the road adhesion coefficient estimation method provided by the present invention, the step of determining the dynamic confidence level includes: The first confidence level is determined based on the covariance of the state estimation error output by the state observer; The second confidence level is determined based on the sufficiency of the vehicle's longitudinal and lateral dynamic excitations. The first confidence level and the second confidence level are combined to obtain the dynamic confidence level.

[0009] According to the road surface adhesion coefficient estimation method provided by the present invention, the method further includes: The change in the dynamic adhesion coefficient is monitored when the vehicle moves from one road surface type to another. If the dynamic adhesion coefficient changes in a direction that approaches the visual adhesion coefficient, then the visual confidence level is increased and / or the road surface type-adhesion coefficient mapping table is fine-tuned.

[0010] According to the road surface adhesion coefficient estimation method provided by the present invention, the method further includes: If the dynamic adhesion coefficient changes in a direction away from the visual adhesion coefficient, radar perception information from the vehicle-mounted millimeter-wave radar is acquired, and road conditions are assessed based on the radar perception information. If the dynamic adhesion coefficient and the visual adhesion coefficient still deviate continuously, fault diagnosis is triggered.

[0011] The present invention also provides a road surface adhesion coefficient estimation device, comprising: The visual estimation unit is used to acquire images of the front of the vehicle based on the vehicle-mounted camera, use a semantic segmentation network to perform pixel-level classification of key road surface areas in the image, obtain road surface type and visual confidence, and obtain visual adhesion coefficient based on the road surface type. The dynamic estimation unit is used to acquire real-time dynamic data of the vehicle, and calculate the dynamic adhesion coefficient and dynamic confidence level based on the dynamic data through the vehicle dynamic model and state observer. The fusion unit is used to predict the time when the vehicle will enter the critical road surface area based on the dynamic data. When the vehicle enters the critical road surface area, the visual adhesion coefficient and the dynamic adhesion coefficient are weighted and fused based on the visual confidence and the dynamic confidence to obtain the final road surface adhesion coefficient.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the road surface adhesion coefficient estimation method as described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the road surface adhesion coefficient estimation method as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the road surface adhesion coefficient estimation method as described above.

[0015] The road surface adhesion coefficient estimation method, device, electronic equipment, and storage medium provided by this invention accurately identify key road surface areas ahead through semantic segmentation technology, overcoming the lag of traditional solutions that can only perceive the current road surface under the wheels. This endows the system with forward-looking warning capabilities, reserving valuable decision-making time for advanced driver assistance systems. Secondly, through the collaborative work of the vehicle dynamics model and the state observer, the reliability of dynamic estimation under various driving conditions is ensured, and a dynamic confidence score reflecting its own credibility is dynamically output. Finally, based on a spatiotemporally aligned confidence score weighted fusion strategy, visual and dynamic information sources can be intelligently balanced: when the vehicle is driving smoothly and the dynamic excitation is insufficient, the system automatically relies on visual feedforward information; while during aggressive driving and when the dynamic signal is sufficient, the dynamic feedback result is given priority. This dynamic weight allocation mechanism effectively overcomes the limitations of a single sensor and significantly improves the accuracy and robustness of the estimation results. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts illustrating the road surface adhesion coefficient estimation method provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the road surface adhesion coefficient estimation device provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] To address the problems of lagging, poor accuracy, and lack of foresight and self-learning capabilities in the estimation of road surface adhesion coefficient in existing technologies, this invention proposes a road surface adhesion coefficient estimation method. In this method, an image of the vehicle's front is acquired using an onboard camera. A semantic segmentation network is used to perform pixel-level classification of key road surface areas in the image to obtain road surface type and visual confidence score. The visual adhesion coefficient is then obtained based on road surface type matching. Real-time vehicle dynamics data is acquired, and the dynamic adhesion coefficient and dynamic confidence score are calculated based on the dynamics data using a vehicle dynamics model and a state observer. The time when the vehicle is about to enter a key road surface area is predicted based on the dynamics data. When the vehicle enters the key road surface area, the visual adhesion coefficient and dynamic adhesion coefficient are weighted and fused based on the visual confidence score and dynamic confidence score to obtain the final road surface adhesion coefficient.

[0022] The method provided in this invention significantly improves the timeliness, accuracy, and intelligence of road surface adhesion coefficient estimation by deeply integrating the predictive power of visual feedforward with the precision of dynamic feedback. First, this method utilizes semantic segmentation technology to accurately identify key road surface areas ahead, overcoming the lag of traditional solutions that can only perceive the current under-wheel surface, thus endowing the system with forward-looking warning capabilities and reserving valuable decision-making time for advanced driver assistance systems. Second, through the collaborative work of the vehicle dynamics model and the state observer, the reliability of dynamic estimation under various driving conditions is ensured, and a dynamic confidence score reflecting its own credibility is dynamically output. Finally, a spatiotemporally aligned confidence score weighted fusion strategy intelligently balances visual and dynamic information sources: when the vehicle is driving smoothly and dynamic excitation is insufficient, the system automatically relies on visual feedforward information; while during aggressive driving and when dynamic signals are sufficient, the dynamic feedback results are given priority. This dynamic weight allocation mechanism effectively overcomes the limitations of a single sensor and significantly improves the accuracy and robustness of the estimation results.

[0023] The embodiments of this invention can be applied to scenarios requiring road surface adhesion coefficient estimation, such as emergency braking and active lane changing. The execution entity of this method can be an electronic device such as a vehicle terminal, computer, server, server cluster, or a specially designed road surface adhesion coefficient estimation device, or a road surface adhesion coefficient estimation device installed in such an electronic device. This device can be implemented through software, hardware, or a combination of both.

[0024] Figure 1 This is one of the flowcharts illustrating the road surface adhesion coefficient estimation method provided by the present invention, such as... Figure 1 As shown, the method includes the following steps 110-130.

[0025] Step 110: Acquire images of the front of the vehicle based on the vehicle-mounted camera, use a semantic segmentation network to perform pixel-level classification of key road surface areas in the image, obtain road surface type and visual confidence, and obtain visual adhesion coefficient based on road surface type matching.

[0026] Specifically, by continuously capturing video streams through a camera, preprocessing operations such as denoising, enhancement, and correction can be performed on the captured images to improve subsequent recognition accuracy. The key road surface region here can be a region of interest, typically focusing on the area the vehicle is about to enter, such as the road surface area 10-50 meters in front of the vehicle. A semantic segmentation network is a deep learning model that can classify each pixel in an image, labeling it as belonging to a specific category (such as road, sky, vehicle, pedestrian, etc.). In this step, the semantic segmentation network is specifically used to identify road surface regions in the image and further distinguish the type of road surface.

[0027] The preprocessed image is input into a pre-trained semantic segmentation network (such as DeepLabV3, UNet, etc.). This network first accurately segments key road surface regions from the image, and then performs finer-grained classification on the pixels within those regions, outputting the dominant road surface type. Road surface types may include, for example, dry asphalt, wet asphalt, snow, ice, mud, etc.

[0028] The system calculates the visual confidence score for this identification based on the maximum probability value of the classification result and image features. Visual confidence score is a quantitative indicator that reflects the reliability of the semantic segmentation network's classification result for the current road surface type.

[0029] Simultaneously, based on the identified road surface type, a pre-configured "Road Surface Type - Nominal Adhesion Coefficient" lookup table is used to obtain a predictive adhesion coefficient matching that road surface type, called the visual adhesion coefficient. The road surface type - nominal adhesion coefficient lookup table is built based on extensive experimental data and prior knowledge. For example: dry asphalt: μ≈ 0.8 - 1.0; wet asphalt: μ≈ 0.5 - 0.7; compacted snow: μ≈ 0.2 - 0.3; ice surface: μ≈ 0.1 - 0.15. Where μ is the adhesion coefficient.

[0030] The core objective of this step is to achieve forward-looking perception. Before the vehicle enters the target area, it uses the predictive power of vision to provide a priori, predictive estimate of the adhesion coefficient of the road surface it will enter in the future.

[0031] Preferably, visual confidence is determined based on the image sharpness and / or road surface type classification probability of the image.

[0032] Specifically, image sharpness is fundamental to assessing the reliability of visual perception. Sharpness can be obtained through gradient energy calculation or frequency domain analysis. The edge and texture intensity of an image is measured by calculating the gradient of pixel values. Sharp images contain rich edge information and high gradient values; conversely, blurry images show a significant decrease in gradient energy. Alternatively, converting the image to the frequency domain reveals that sharp images have more energy distribution in the high-frequency components, while the high-frequency components of blurry images are attenuated. An image quality score positively correlated with sharpness is calculated; when this score is below a preset threshold, it indicates poor image quality and significantly reduces visual confidence.

[0033] Road surface type classification probability refers to the degree of certainty that a semantic segmentation network can identify the road surface type given image features. A higher classification probability means the model is very confident, and a high visual confidence level should be assigned; a lower classification probability means the model is hesitant, and the visual confidence level should be lowered.

[0034] Step 120: Obtain real-time dynamic data of the vehicle, and calculate the dynamic adhesion coefficient and dynamic confidence level based on the dynamic data using the vehicle dynamics model and state observer.

[0035] Specifically, real-time dynamics data is data reflecting the vehicle's motion state acquired in real time via the vehicle's CAN bus. This typically includes wheel speed pulses, yaw rate, longitudinal / lateral acceleration, steering wheel angle, engine torque / brake pressure, etc. A vehicle dynamics model is a set of mathematical equations describing the vehicle's motion, such as a "bicycle model" or a more complex "three-degree-of-freedom model," establishing the relationship between the vehicle's inputs, states, and outputs. A state observer is an estimation algorithm, such as the Extended Kalman Filter (EKF) or the Unscented Kalman Filter (UKF).

[0036] The system acquires raw sensor data from the CAN bus and performs signal processing such as filtering and noise reduction. A vehicle dynamics model and a tire model are established, incorporating the road surface adhesion coefficient as a state variable. Preprocessed dynamic data (such as steering wheel angle and wheel speed) are input to the state observer. The observer continuously runs a "predict-update" loop to optimally estimate the system state, including the dynamic adhesion coefficient. State estimation includes longitudinal and lateral estimation. Longitudinal estimation, during vehicle acceleration or braking, utilizes the wheel speed difference between the drive and non-drive wheels, or the relationship between engine torque / braking force and actual deceleration, to estimate the longitudinal adhesion coefficient. Lateral estimation, during vehicle steering, compares the actual yaw rate with the expected yaw rate based on a "bicycle model" or "steering geometry model." The deviation between these two values ​​is highly correlated with the road surface adhesion capability.

[0037] Simultaneously, the dynamic confidence level is calculated based on the estimation error covariance output by the observer and the current vehicle dynamic excitation level.

[0038] Understandably, the purpose of this step is to provide high-precision real-time verification by obtaining a high-confidence measurement of the adhesion coefficient based on the vehicle's current actual behavior, namely the dynamic adhesion coefficient. Although the dynamic adhesion coefficient reflects the condition of the road surface that has already been traversed, it is highly accurate because it originates from physical laws and can be used as the ground truth for verifying visual predictions and for fusion.

[0039] The dynamic confidence score reflects the reliability of the dynamic adhesion coefficient estimate. Its calculation typically considers two aspects: first, the estimation uncertainty of the state observer's algorithm output; and second, whether the current vehicle motion state is sufficient to effectively excite the tire-road model—for example, the confidence score is lower during constant speed driving and higher during emergency braking. In some embodiments, the steps for determining the dynamic confidence score include: The first confidence level is determined based on the covariance of the state estimation error output by the state observer; The second confidence level is determined based on the sufficiency of the vehicle's longitudinal and lateral dynamic excitations. The first confidence level and the second confidence level are combined to obtain the dynamic confidence level.

[0040] Specifically, when using state observers such as extended Kalman filters or unscented Kalman filters, the algorithm itself outputs a key metric—the covariance matrix of the state estimation error. A larger variance indicates higher uncertainty and lower confidence in the estimate. The estimation variances of the four wheels can be combined, for example, by taking the maximum or average value, to obtain a scalar representing the overall dynamic estimation uncertainty. The first confidence level should be negatively correlated with the uncertainty; for example, an exponential decay function can be used. When the uncertainty is very small, the first confidence level is close to 1; when the uncertainty is large, the first confidence level approaches 0.

[0041] Even if the observer algorithm converges, the estimation results may still be inaccurate if the vehicle's motion is insufficient to excite the tire-road nonlinear characteristics. In this case, a second confidence level needs to be calculated. First, a scalar reflecting the strength of the current longitudinal and lateral dynamic excitation is calculated. An excitation threshold is set; when the scalar is below this threshold, the vehicle is considered to be in a quasi-steady state, indicating insufficient dynamic excitation. This scalar is then mapped to a second confidence level. If the excitation is insufficient, the second confidence level decreases linearly with the excitation level.

[0042] Based on the first and second confidence levels, the first and second confidence levels are merged to obtain the final dynamic confidence level.

[0043] Step 130: Based on dynamic data, predict the time when the vehicle will enter the critical road surface area. When the vehicle enters the critical road surface area, the visual adhesion coefficient and the dynamic adhesion coefficient are weighted and fused based on visual confidence and dynamic confidence to obtain the final road surface adhesion coefficient.

[0044] Specifically, considering the temporal and spatial mismatch between visual feedforward information and vehicle dynamics feedback information, spatiotemporal alignment is required before adhesion coefficient fusion. When the vision module identifies a specific road surface type and outputs the predicted visual adhesion coefficient, the system immediately records the current vehicle position (obtained via GPS, odometer, etc.). Combining the camera's intrinsic and extrinsic parameters with the vision algorithm, the distance and orientation of the key road surface area relative to the vehicle can be calculated.

[0045] The system continuously predicts the estimated time for a vehicle to reach a critical road surface area based on its real-time speed and trajectory. For example, at the current speed, it will enter the road surface identified as wet asphalt in 2.5 seconds.

[0046] The system continuously monitors the vehicle's actual position. When the system determines that the vehicle's front wheels have just entered or are about to fully enter the previously marked critical road surface area, it immediately triggers the fusion process. At this point, the critical road surface area pointed to by the visual prediction spatially becomes the current road surface to be measured by the dynamic feedback.

[0047] At the moment of fusion triggering, the system reads the visual adhesion coefficient and visual confidence level previously predicted for the area by the vision module. It also reads the dynamic adhesion coefficient and dynamic confidence level estimated in real time by the dynamics module based on the current tire-road contact state. The visual adhesion coefficient and dynamic adhesion coefficient are then weighted and fused to obtain the final road adhesion coefficient.

[0048] In some embodiments, the following formula is used for weighted fusion: μ _ 融合的 =( C _ 视觉 μ _ 视觉 + C _ 动力学 μ _ 动力学 ) / ( C _ 视觉 + C _ 动力学 ) In the formula, μ _ 融合的 The final road surface adhesion coefficient, C _ 视觉 For visual confidence,μ _ 视觉 The visual adhesion coefficient, C _ 动力学 For the confidence level of the dynamics, μ _ 动力学 This is the dynamic adhesion coefficient.

[0049] It should be noted that visual confidence and dynamic confidence are dynamic. For example, if the vehicle is cruising smoothly with insufficient dynamic excitation, the dynamic confidence is low, and the system will trust the visual prediction more. If the vehicle is braking hard or making a sharp turn with sufficient dynamic excitation and high dynamic confidence, the system will trust the dynamic feedback value more.

[0050] This step intelligently combines the advantages of visual feedforward and dynamic feedback, overcoming the limitations of a single information source and obtaining a fast and reliable final road adhesion coefficient estimation result.

[0051] In some embodiments, the method further includes: Monitor changes in the dynamic adhesion coefficient as the vehicle moves from one road surface type to another. If the dynamic adhesion coefficient changes in a direction that approaches the visual adhesion coefficient, then increase the visual confidence level and / or fine-tune the road type-adhesion coefficient mapping table.

[0052] Specifically, this method also has human-like self-learning and adaptive calibration capabilities. When a vehicle enters the visual prediction area, the accuracy of visual recognition is verified by comparing the changing trend of dynamic feedback. Based on this, the visual confidence level is updated online or the road surface type-adhesion coefficient mapping table is fine-tuned, enabling the system to continuously optimize its performance and adapt to different environments and road conditions.

[0053] The vehicle moves from its current road surface, such as dry asphalt with a high coefficient of adhesion, into a different type of road surface previously identified by the vision module, such as an icy or snowy area with a lower coefficient of adhesion predicted by vision. At this point, the system initiates a high-priority monitoring procedure. This involves using vehicle dynamics feedback as the actual ground condition to verify the accuracy of the visual feedforward prediction. While the vision system can predict in advance, its judgment may be uncertain due to factors such as light, shadows, and reflections; while the dynamics system is accurate, its perception is lagging. This embodiment combines the advantages of both to achieve closed-loop optimization.

[0054] When the system detects that the changing trend of the dynamic adhesion coefficient is close to the predicted value of the visual adhesion coefficient, the system will perform the following operations: Increasing visual confidence aims to immediately reward accurate visual predictions, enhancing the system's trust in similar visual information in the future. The visual confidence parameters are dynamically adjusted for this type of road surface, such as "ice and snow." When the vision system identifies similar ice and snow surfaces again, its predicted output will carry a higher weight in the fusion decision, thus influencing the final fusion result and vehicle control strategy earlier and more significantly.

[0055] Fine-tuning the pavement type-adhesion coefficient mapping table aims to achieve long-term, adaptive performance optimization, making the system's prior knowledge more consistent with actual road environments. This fine-tuning can be achieved using incremental learning algorithms. For example, a first-order low-pass filter or a recursive least-squares-based learning algorithm can be used to fine-tune the original nominal values ​​in the mapping table. This fine-tuning mechanism allows the system to adapt to pavements in different regions, seasons, or with varying degrees of wear. For instance, it can gradually learn that in specific cold regions, the actual adhesion coefficient of compacted snow is typically higher than the standard manual value.

[0056] The method provided in this invention automatically compensates for the performance degradation of the visual sensor due to aging and dirt by learning online and adaptively calibrating from each driving experience across different road surfaces, and adapts to diverse real road environments. This reduces the reliance on manual calibration and greatly improves the reliability and practicality of the entire road surface adhesion coefficient estimation method.

[0057] In other embodiments, if the dynamic adhesion coefficient changes in a direction away from the visual adhesion coefficient, radar perception information from the vehicle-mounted millimeter-wave radar is acquired, and road conditions are assessed based on the radar perception information. If there is a persistent deviation between the dynamic adhesion coefficient and the visual adhesion coefficient, fault diagnosis will be triggered.

[0058] Specifically, when the vehicle enters a critical road surface area, the system detects that the dynamic adhesion coefficient does not change in the direction predicted by the visual adhesion coefficient, but either remains unchanged or changes in the opposite direction, indicating a deviation between the two. This deviation suggests an internal contradiction within the system, making it impossible to make a correct judgment solely based on visual and dynamic information. Therefore, the system introduces data from a third sensor independent of optical and dynamic principles: radar perception information from the vehicle's millimeter-wave radar. This radar perception information can include target status, road boundary and geometric information, and road surface material detection.

[0059] For example, radar can detect indirect evidence such as abnormal lateral swaying or abnormal deceleration of other vehicles on suspected low-friction surfaces ahead. If radar detects a vehicle skidding in that area, it supports visual predictions of low-friction surfaces.

[0060] Visual recognition of water accumulation may cause lane lines to become distorted and deformed. Radar detection results of lane boundaries or curbs can be used to verify whether the visually perceived road model has become abnormal.

[0061] Advanced millimeter-wave radar can provide direct evidence of the presence of coverings, as the reflective properties of materials with a certain volume, such as snow or water, differ from those of dry roads.

[0062] Based on this, and using radar perception information, the system assists in determining the reliability of the discrepancy between visual and dynamic information: if the radar information supports the visual judgment, such as detecting snow accumulation, it suspects that the dynamic model is inaccurate or that the current vehicle's dynamic excitation is insufficient, resulting in a failure to accurately reflect the actual road conditions. The system will then temporarily lower the dynamic confidence level.

[0063] If radar information does not support visual judgment, such as detecting a dry road surface and a vehicle driving smoothly ahead, the system will suspect a visual recognition error, such as misinterpreting a shadow or reflection as ice. The system will then significantly reduce the confidence level of this visual recognition.

[0064] If, after supplementary verification with radar data, a persistent deviation exists between the dynamic adhesion coefficient and the visual adhesion coefficient, a fault diagnosis is triggered. The fault event is recorded, including the time, location, relevant sensor data (images, radar point clouds, dynamic data), and estimated values. Based on the radar arbitration results, the fault is initially classified as either "suspected visual sensor misidentification" or "suspected dynamic model inaccuracy." In the fault state, safety is prioritized. For example, a lower adhesion coefficient value is used by default for safe distance calculation, or the driver is prompted to take over. An alert is issued to the driver or fleet management system requiring sensor inspection or system calibration.

[0065] The method provided in this invention, by introducing millimeter-wave radar, constructs a triple information cross-verification system of vision, dynamics, and radar, avoiding the predicament of information conflict in single or dual-sensor systems. It can not only detect anomalies but also preliminarily determine the source of the fault through information fusion, providing a clear direction for subsequent maintenance and debugging, and improving the maintainability of the system.

[0066] The road surface adhesion coefficient estimation device provided by the present invention is described below. The road surface adhesion coefficient estimation device described below can be referred to in correspondence with the road surface adhesion coefficient estimation method described above.

[0067] Based on the above embodiments, Figure 2 This is a schematic diagram of the road surface adhesion coefficient estimation device provided by the present invention, as shown below. Figure 2 As shown, the device includes: The visual estimation unit 210 is used to acquire images of the front of the vehicle based on the vehicle-mounted camera, use a semantic segmentation network to perform pixel-level classification of key road surface areas in the image, obtain road surface type and visual confidence, and obtain visual adhesion coefficient based on the road surface type. The dynamic estimation unit 220 is used to acquire real-time dynamic data of the vehicle, and calculate the dynamic adhesion coefficient and dynamic confidence level based on the dynamic data through the vehicle dynamic model and state observer. The fusion unit 230 is used to predict the time when the vehicle will enter the critical road surface area based on the dynamic data. When the vehicle enters the critical road surface area, the visual adhesion coefficient and the dynamic adhesion coefficient are weighted and fused based on the visual confidence and the dynamic confidence to obtain the final road surface adhesion coefficient.

[0068] Based on the above embodiments, the fusion unit is specifically used for: The following formula is used for weighted fusion: μ _ 融合的 =( C _ 视觉 μ _ 视觉 + C _ 动力学 μ _ 动力学 ) / ( C _ 视觉 + C _ 动力学 ) In the formula, μ _ 融合的 The final road surface adhesion coefficient, C _ 视觉 For visual confidence, μ _ 视觉 The visual adhesion coefficient, C _ 动力学 For the confidence level of the dynamics, μ _ 动力学 This is the dynamic adhesion coefficient.

[0069] Based on the above embodiments, the visual confidence level is determined based on the image sharpness and / or road surface type classification probability of the image.

[0070] Based on the above embodiments, a dynamic confidence determination unit is also included, used for: The first confidence level is determined based on the covariance of the state estimation error output by the state observer; The second confidence level is determined based on the sufficiency of the vehicle's longitudinal and lateral dynamic excitations. The first confidence level and the second confidence level are combined to obtain the dynamic confidence level.

[0071] Based on the above embodiments, the device further includes an adaptive calibration unit, used for: The change in the dynamic adhesion coefficient is monitored when the vehicle moves from one road surface type to another. If the dynamic adhesion coefficient changes in a direction that approaches the visual adhesion coefficient, then the visual confidence level is increased and / or the road surface type-adhesion coefficient mapping table is fine-tuned.

[0072] Based on the above embodiments, the adaptive calibration unit is further configured to: If the dynamic adhesion coefficient changes in a direction away from the visual adhesion coefficient, radar perception information from the vehicle-mounted millimeter-wave radar is acquired, and road conditions are assessed based on the radar perception information. If the dynamic adhesion coefficient and the visual adhesion coefficient still deviate continuously, fault diagnosis is triggered.

[0073] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a road surface adhesion coefficient estimation method. This method includes: acquiring an image of the vehicle's front based on an onboard camera; performing pixel-level classification of key road surface areas in the image using a semantic segmentation network to obtain road surface type and visual confidence score; and obtaining a visual adhesion coefficient based on the road surface type matching; acquiring real-time dynamic data of the vehicle; calculating a dynamic adhesion coefficient and dynamic confidence score based on the dynamic data using a vehicle dynamics model and a state observer; predicting the time when the vehicle will enter the key road surface area based on the dynamic data; and when the vehicle enters the key road surface area, weightedly fusing the visual adhesion coefficient and the dynamic adhesion coefficient based on the visual confidence score and the dynamic confidence score to obtain the final road surface adhesion coefficient.

[0074] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the road adhesion coefficient estimation method provided by the above methods. The method includes: acquiring an image of the front of the vehicle based on an onboard camera; performing pixel-level classification of key road surface areas in the image using a semantic segmentation network to obtain road surface type and visual confidence score; and obtaining a visual adhesion coefficient based on the road surface type matching; acquiring real-time dynamic data of the vehicle; calculating a dynamic adhesion coefficient and dynamic confidence score based on the dynamic data using a vehicle dynamics model and a state observer; predicting the time when the vehicle will enter the key road surface area based on the dynamic data; and when the vehicle enters the key road surface area, weightedly fusing the visual adhesion coefficient and the dynamic adhesion coefficient based on the visual confidence score and the dynamic confidence score to obtain the final road adhesion coefficient.

[0076] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a road surface adhesion coefficient estimation method provided by the above methods. The method includes: acquiring an image of the front of the vehicle based on an onboard camera; performing pixel-level classification of key road surface areas in the image using a semantic segmentation network to obtain road surface type and visual confidence score; and obtaining a visual adhesion coefficient based on the road surface type matching; acquiring real-time dynamic data of the vehicle; calculating a dynamic adhesion coefficient and dynamic confidence score based on the dynamic data using a vehicle dynamics model and a state observer; predicting the time when the vehicle will enter the key road surface area based on the dynamic data; and when the vehicle enters the key road surface area, weightedly fusing the visual adhesion coefficient and the dynamic adhesion coefficient based on the visual confidence score and the dynamic confidence score to obtain the final road surface adhesion coefficient.

[0077] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of estimating a road adhesion coefficient, characterized by, The method comprises: obtaining an image in front of the vehicle based on a vehicle-mounted camera, performing pixel-level classification on a key road surface area in the image using a semantic segmentation network to obtain a road surface type and a visual confidence, and matching the road surface type to obtain a visual adhesion coefficient; obtaining real-time dynamics data of the vehicle, and calculating a dynamics adhesion coefficient and a dynamics confidence based on the dynamics data through a vehicle dynamics model and a state observer; predicting a time when the vehicle is about to enter the key road surface area based on the dynamics data, and when the vehicle enters the key road surface area, weighting and fusing the visual adhesion coefficient and the dynamics adhesion coefficient based on the visual confidence and the dynamics confidence to obtain a final road surface adhesion coefficient.

2. The road adhesion coefficient estimation method according to claim 1, characterized by, The weighting and fusing of the visual adhesion coefficient and the dynamics adhesion coefficient based on the visual confidence and the dynamics confidence to obtain the final road surface adhesion coefficient comprises: using the following formula for weighting and fusing: μ _ fused =( C _ vision μ _ vision + C _ dynamics μ _ dynamics ) / ( C _ vision + C _ dynamics ) wherein μ fused is the final road surface adhesion coefficient, C vision is the visual confidence, μ vision is the visual adhesion coefficient, C dynamics is the dynamic confidence, μ dynamics is the dynamic adhesion coefficient.​​​​​ 3. The road adhesion coefficient estimation method according to claim 2, characterized by, The visual confidence is determined based on image definition and / or road surface type classification probability of the image.

4. The road adhesion coefficient estimation method according to claim 2, characterized by, The determination of the dynamics confidence comprises: determining a first confidence based on a covariance of a state estimation error output by the state observer; determining a second confidence based on a sufficient degree of longitudinal and lateral dynamics excitation of the vehicle; fusing the first confidence and the second confidence to obtain the dynamics confidence.

5. The road adhesion coefficient estimation method according to any one of claims 1 to 4, characterized by, The method further comprises: when the vehicle enters from one road surface type to another road surface type, monitoring a change in the dynamics adhesion coefficient; if the dynamics adhesion coefficient changes in a direction approaching the visual adhesion coefficient, then increasing the visual confidence and / or fine-tuning a road surface type-adhesion coefficient mapping table.

6. The road adhesion coefficient estimation method according to claim 5, characterized by, The method further comprises: if the dynamics adhesion coefficient changes in a direction away from the visual adhesion coefficient, then obtaining radar perception information of a vehicle-mounted millimeter wave radar, and assisting in judging a road surface condition based on the radar perception information; if there is a persistent deviation between the dynamics adhesion coefficient and the visual adhesion coefficient, then triggering a fault diagnosis.

7. A road adhesion coefficient estimation device characterized by comprising: The method comprises: a visual estimation unit configured to obtain an image in front of the vehicle based on a vehicle-mounted camera, perform pixel-level classification on a key road surface area in the image using a semantic segmentation network to obtain a road surface type and a visual confidence, and match the road surface type to obtain a visual adhesion coefficient; a dynamics estimation unit configured to obtain real-time dynamics data of the vehicle, and calculate a dynamics adhesion coefficient and a dynamics confidence based on the dynamics data through a vehicle dynamics model and a state observer; a fusion unit configured to predict a time when the vehicle is about to enter the key road surface area based on the dynamics data, and when the vehicle enters the key road surface area, weight and fuse the visual adhesion coefficient and the dynamics adhesion coefficient based on the visual confidence and the dynamics confidence to obtain a final road surface adhesion coefficient.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The computer program is executed by the processor to implement the road adhesion coefficient estimation method according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the road adhesion coefficient estimation method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the road adhesion coefficient estimation method according to any one of claims 1 to 6.

Images