A method and system for estimating road adhesion coefficient by fusing image recognition and dynamics

Abstract

This invention discloses a method and system for estimating road surface adhesion coefficient by fusing image recognition and dynamics. The method includes: determining the road surface type using a semantic segmentation network and a convolutional neural network; estimating the tire longitudinal force using a Kalman filter based on a single-wheel dynamics model and vehicle state information; establishing a road surface adhesion coefficient estimator based on the magic tire formula and a particle filter algorithm using the tire longitudinal force to estimate a first road surface adhesion coefficient; determining all image recognition results corresponding to the road surface with the first road surface adhesion coefficient through a spatiotemporal synchronization module, and determining the final image recognition result and image recognition confidence level based on all image recognition results; and establishing an image recognition and dynamics fusion strategy based on the final image recognition result, image recognition confidence level, the first road surface adhesion coefficient, and the difference between the estimated and true values ​​of the vehicle's longitudinal acceleration to determine the final road surface adhesion coefficient. This invention can improve the accuracy of road surface adhesion coefficient estimation.

Description

Technical Field

[0001] This invention relates to the field of automotive control, and in particular to a method and system for estimating road surface adhesion coefficient by fusing image recognition and dynamics. Background Technology

[0002] With the development of automotive intelligence, active safety control systems are gradually becoming widespread in passenger vehicles, such as Electronic Stability Program (ESP) and Automatic Emergency Braking (AEB), effectively improving vehicle active safety performance. The road surface adhesion coefficient (BPCC) is a key parameter of active safety control systems, and accurate and rapid estimation of BPCC is crucial for ensuring vehicle active safety control. Existing research on BPCC estimation mainly falls into two categories: cause-based methods and effect-based methods. Cause-based methods utilize sensors such as sound waves, infrared, cameras, and radar to identify road conditions and further estimate the BPCC. These methods can predict the road surface conditions ahead, but can only determine the road type and cannot accurately obtain the BPCC value; the estimation results are easily affected by the environment. Effect-based methods establish a vehicle dynamics model related to the BPCC and combine it with a state observer design to estimate the BPCC. Existing research mainly focuses on estimating the BPCC through tire or vehicle dynamic response characteristics.

[0003] The existing technologies mainly include:

[0004] Scheme (1) Constructing a fusion estimator to estimate the road surface adhesion coefficient

[0005] Patent CN 110765909 A proposes a road surface estimation method for distributed-drive electric vehicles based on onboard cameras. Its basic principle is as follows: Onboard cameras capture images of dry and wet road surfaces in front of the vehicle; road surface image features are extracted using the color moment method and gray-level co-occurrence matrix method; support vector machines are used to classify and identify the features of each road surface image; a perturbation estimator is designed based on tire longitudinal force to estimate the road surface adhesion coefficient; a fusion estimator is constructed considering the difference between the empirical value and the true value of the adhesion coefficient mapped by road type to obtain the fusion estimation result.

[0006] Scheme (2) Estimation of road surface adhesion coefficient by combining image recognition method and dynamic method

[0007] Patent CN 111688707 A proposes a method for estimating road surface adhesion coefficient by fusing vision and dynamics. Its basic principle is as follows: First, an image of the road surface ahead of the vehicle is acquired during vehicle movement. This image is then input into a trained deep neural network model for road surface classification to obtain the road surface type. Based on the mapping relationship between the road surface type and the road surface adhesion coefficient, a visual estimate of the road surface adhesion coefficient is obtained. Next, the dynamic information of the tires during vehicle movement is acquired, and a road surface adhesion coefficient-tire longitudinal force estimator is used to obtain an estimate of the current peak road surface adhesion coefficient. Finally, combining the visual estimate of the road surface adhesion coefficient and the peak road surface adhesion coefficient estimate based on tire dynamics, fuzzy inference rules are used to obtain the final estimate of the road surface adhesion coefficient.

[0008] Scheme (3) integrates three methods: camera, radar, and vehicle dynamics.

[0009] Patent CN 111845709 B proposes a method and system for estimating road surface adhesion coefficient based on multi-information fusion. Its basic principle is as follows: First, a camera captures an image of the road surface ahead, which is input into a trained road surface classifier to identify the road surface type, obtaining an empirical value of the road surface adhesion coefficient corresponding to the road surface type, denoted as the first road surface adhesion coefficient. Second, a second road surface adhesion coefficient is obtained from the ground point cloud reflection intensity map obtained by lidar scanning. During vehicle movement, a third road surface adhesion coefficient at the vehicle tire position is estimated using the longitudinal dynamic response of the tires. Finally, the first and second road surface adhesion coefficients are corrected based on the third road surface adhesion coefficient to obtain the final road surface adhesion coefficient.

[0010] Scheme (4) selects the road adhesion coefficient estimate based on image recognition or dynamics based on the fusion mechanism.

[0011] Patent CN 113361121 B proposes a method for estimating road surface adhesion coefficient based on spatiotemporal synchronization and information fusion. Its basic principle is as follows: First, sensors acquire image information of the road surface ahead of the vehicle and vehicle dynamic response information. Second, a trained semantic segmentation network extracts the road surface region from the acquired image of the road surface ahead of the vehicle, and then feeds it into a trained road surface type recognition network to obtain the road surface type recognition result, which is mapped to obtain the range of road surface adhesion coefficient. Based on the acquired vehicle dynamic response information, an unscented Kalman filter estimation method is used to obtain the estimated value of the road surface adhesion coefficient. A spatiotemporal synchronization method is used to filter out the road surface type recognition result and the dynamic road surface adhesion coefficient estimate that meet the fusion conditions. Finally, the accuracy of the road surface type recognition result is judged based on the confidence threshold and weighted probability value. The road surface adhesion coefficient range based on image information and the dynamic road surface adhesion coefficient estimate are fused to output the final estimated value.

[0012] Scheme (5) uses the image estimation results as constraints for the dynamics estimator.

[0013] Patent CN 115195751 A proposes a method and system for estimating the road surface adhesion coefficient based on multi-source information fusion. Its basic principle is as follows: First, vehicle status information and images of the road surface ahead are acquired; the vehicle's center of gravity sideslip angle is estimated based on the vehicle status information and the lateral distance between the camera's pre-aiming point and the lane lines; the vehicle's lateral force is estimated based on the vehicle status information; the road surface type is identified from the road surface image and mapped to the range of road surface adhesion coefficients; the road surface adhesion coefficient is estimated by combining the above estimation and identification results with the vehicle status information.

[0014] Regarding solution (1): In the image recognition part, only the dry and wet states of the road surface are identified. The problem of low recognition accuracy is caused by manually extracting image features. Using the image recognition result as one of the parameters of the dynamic estimator does not give full play to the advantage of image recognition method in predicting the state of the road surface ahead.

[0015] Regarding scheme (2): using fuzzy inference rules to fuse the visual estimate of the road surface adhesion coefficient with the estimate of the peak road surface adhesion coefficient based on tire dynamics to obtain the final estimate of the road surface adhesion coefficient, it also fails to leverage the advantage of image recognition in predicting the road surface condition ahead.

[0016] Regarding scheme (3): Three methods are used to estimate the road surface adhesion coefficient. The confidence level of camera image features is compared with the lowest confidence threshold, the confidence level of lidar image features is compared with the low confidence threshold, and the difference between the road surface adhesion coefficient values ​​identified by different methods is compared with the difference threshold to select a reliable estimation result. This scheme does not avoid the disadvantage that the road surface adhesion coefficient value determined by image features is inaccurate. Instead, it uses the most accurate dynamic estimate as the basis for judgment. The dynamic estimate is only selected when the image features and lidar image features are not accurately identified. The judgment logic has defects.

[0017] Regarding scheme (4): A judgment mechanism is used to integrate the road adhesion coefficient estimation results based on vision and the road adhesion coefficient estimation based on dynamics. The judgment mechanism does not consider the case where the accuracy of the road adhesion coefficient estimated based on dynamics is poor when the wheel is in a small slip / small side deviation condition.

[0018] Regarding scheme (5): the road surface adhesion coefficient range is determined by the road surface information obtained by the camera. This range serves as a constraint for the dynamic road surface adhesion coefficient estimator. The road surface information collected by the camera is greatly affected by the lighting. This scheme does not consider the situation where the road surface information collected by the camera is inaccurate.

[0019] Existing methods for identifying road adhesion coefficients by fusing road image information and vehicle dynamics information can be summarized into two main categories: estimator-based fusion and rule-based fusion. Estimator-based fusion uses the road adhesion coefficient determined by image information as one of the parameters of a dynamics-based road adhesion coefficient estimator. This type of method can obtain relatively accurate estimates, but the estimator is still primarily based on dynamics, and inaccuracies still exist in cases of small tire slippage / side bias. Rule-based fusion establishes a set of judgment logic to select between the two estimation results. This type of method covers a wide range of operating conditions, but its simple logic makes it difficult to fully leverage the advantages of both image-based and dynamics-based methods. Meanwhile, methods for road surface identification based on image information include both machine learning and deep learning. Machine learning involves a complex process of manually extracting image features, and the recognition accuracy is related to the feature selection. Deep learning can automatically learn image features, but high-precision networks have complex structures and numerous model parameters.

[0020] Therefore, in order to solve the above problems, there is an urgent need to provide a new method or system for determining the road surface adhesion coefficient, so as to improve the accuracy of the determination of the road surface adhesion coefficient. Summary of the Invention

[0021] The purpose of this invention is to provide a method and system for estimating road surface adhesion coefficient by fusing image recognition and dynamics, which can improve the accuracy of determining the road surface adhesion coefficient.

[0022] To achieve the above objectives, the present invention provides the following solution:

[0023] A method for estimating road surface adhesion coefficient by fusing image recognition and dynamics includes:

[0024] Use an onboard camera to obtain images of the road surface in front of the vehicle;

[0025] The DeeplabV3+ semantic segmentation network was used to segment the road surface image in front of the vehicle; and the MobileNetV2 lightweight convolutional neural network was used to classify the road surface type of the segmented road surface image in front of the vehicle.

[0026] Vehicle status information is acquired using onboard sensors; the vehicle status information includes: front wheel steering angle, vehicle acceleration, wheel speed, and torque;

[0027] Based on vehicle status information and a single-wheel dynamics model, Kalman filtering is used to estimate the longitudinal force of the tires.

[0028] Based on the longitudinal force of the tire and the magic tire formula, a particle filter algorithm is used to establish a road adhesion coefficient estimator to estimate the first road adhesion coefficient.

[0029] The spatiotemporal synchronization module determines all image recognition results corresponding to the road surface with the first road surface adhesion coefficient, and determines the final image recognition result and image recognition confidence based on all image recognition results; the spatiotemporal synchronization module is used to find the image recognition results of all captured images of the road surface in front of the vehicle that contain the road surface with the first road surface adhesion coefficient, and takes the road surface type with the highest frequency of all image recognition results as the final image recognition result;

[0030] Based on the final image recognition results, image recognition confidence, first road surface adhesion coefficient, and the difference between the estimated and true values ​​of the vehicle's longitudinal acceleration, an image recognition and dynamics fusion strategy is established.

[0031] The final road surface adhesion coefficient is determined based on an image recognition and dynamics fusion strategy.

[0032] Optionally, the step of estimating the tire longitudinal force using Kalman filtering based on a single-wheel dynamics model and vehicle state information specifically includes:

[0033] use Determine the dynamic model of a single wheel;

[0034] Using formula Determine the discrete state-space equations corresponding to the single-wheel dynamics model;

[0035] in, The moment of inertia of the wheel; The angular velocity of the wheel rotation; The angular acceleration of the wheel's rotation. The driving or braking torque acting on the wheels; The radius of the wheel's rolling radius; This refers to the longitudinal force of the tire; For discrete time intervals; for The state variable at time t, for The state variable at any given time; = For observation variables; for The observed variables at time; = To control the quantity, for The amount of control at any given moment; for The time follows a normal distribution (0, Process noise; for The time follows a normal distribution (0, ) observation noise; For process noise The covariance matrix; To observe noise The covariance matrix; The state matrix, For the control matrix, This is the observation matrix.

[0036] Optionally, the step of establishing a road surface adhesion coefficient estimator based on the tire longitudinal force, the magic tire formula, and the particle filter algorithm to estimate the first road surface adhesion coefficient specifically includes the following formula:

[0037] ;

[0038] in, For the longitudinal force of the tire, This is the actual road surface adhesion coefficient. For tire slip ratio, , , and These are the stiffness factor, shape factor, peak factor, and curvature factor of the tire's mechanical property curve, respectively. and This represents the horizontal and longitudinal offsets of the tire force curve relative to the origin.

[0039] Optionally, the step of establishing an image recognition and dynamics fusion strategy based on the final image recognition result, image recognition confidence level, first road surface adhesion coefficient, and the difference between the estimated and true values ​​of the vehicle's longitudinal acceleration specifically includes:

[0040] When the image recognition confidence level is greater than or equal to the image method confidence threshold, the corresponding range of road surface adhesion coefficient is determined based on the final image recognition result.

[0041] The second road surface adhesion coefficient is determined based on the range of road surface adhesion coefficients;

[0042] When the first road surface adhesion coefficient is within the range of road surface adhesion coefficients, the first road surface adhesion coefficient is taken as the final road surface adhesion coefficient.

[0043] When the first road surface adhesion coefficient is not within the range of road surface adhesion coefficients, the second road surface adhesion coefficient shall be used as the final road surface adhesion coefficient.

[0044] When the image recognition confidence level is less than the image method confidence level threshold, determine whether the difference between the estimated value and the true value of the vehicle's longitudinal acceleration is greater than the error threshold.

[0045] If it is greater than the previous time, the road adhesion coefficient determined by the road adhesion coefficient estimator will be used as the final road adhesion coefficient.

[0046] If it is less than or equal to, then the first road surface adhesion coefficient shall be used as the final road surface adhesion coefficient.

[0047] Optionally, when the image recognition confidence level is greater than or equal to the image method confidence threshold, determining the corresponding road surface adhesion coefficient range based on the final image recognition result further includes:

[0048] When the final image recognition result indicates a low-friction road surface, a warning signal is sent to the vehicle.

[0049] A road surface adhesion coefficient estimation system that integrates image recognition and dynamics includes:

[0050] The road surface image acquisition unit is used to acquire images of the road surface in front of the vehicle using an onboard camera.

[0051] The road surface classification unit is used to segment the road surface image in front of the vehicle using the DeeplabV3+ semantic segmentation network; and to classify the road surface type of the segmented road surface image in front of the vehicle using the MobileNetV2 lightweight convolutional neural network.

[0052] The vehicle status information acquisition unit is used to acquire vehicle status information using on-board sensors; the vehicle status information includes: front wheel steering angle, vehicle acceleration, wheel speed and torque.

[0053] The tire longitudinal force estimation unit is used to estimate the tire longitudinal force based on the vehicle state information and a single-wheel dynamics model using Kalman filtering.

[0054] The first road surface adhesion coefficient estimation unit is used to establish a road surface adhesion coefficient estimator based on the tire longitudinal force, the magic tire formula, and the particle filter algorithm to estimate the first road surface adhesion coefficient.

[0055] The final image recognition result determination unit is used to determine all image recognition results corresponding to the road surface with the first road surface adhesion coefficient through the spatiotemporal synchronization module, and to determine the final image recognition result and image recognition confidence based on all image recognition results; the spatiotemporal synchronization module is used to search for the image recognition results of all captured images of the road surface in front of the vehicle that contain the road surface with the first road surface adhesion coefficient, and to take the road surface type with the highest frequency of all image recognition results as the final image recognition result;

[0056] The image recognition and dynamics fusion strategy establishment unit is used to establish an image recognition and dynamics fusion strategy based on the final image recognition result, image recognition confidence, first road surface adhesion coefficient, and the difference between the estimated and true values ​​of the vehicle's longitudinal acceleration.

[0057] The final road surface adhesion coefficient determination unit is used to determine the final road surface adhesion coefficient based on the image recognition and dynamics fusion strategy.

[0058] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the image recognition and dynamics fusion method for estimating road surface adhesion coefficient.

[0059] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0060] This invention provides a method and system for estimating road adhesion coefficient by fusing image recognition and dynamics. It employs a lightweight DeeplabV3+ semantic segmentation network and a MobilenetV2 classification network, reducing model parameters, increasing computation speed, and achieving high accuracy in road type recognition, thus meeting vehicle-mounted requirements. The camera captures images of the road surface ahead, while dynamic information reflects the current road surface state. The proposed spatiotemporal synchronization module filters all image recognition results that match the dynamic response points, matching the image recognition results with the dynamic response information, laying the foundation for the overall fusion estimation method. Since image recognition is greatly affected by the external environment, a confidence calculation method based on image recognition is proposed, with the confidence calculation automatically adjusted according to vehicle driving conditions. This invention estimates tire longitudinal force based on a single-wheel dynamic model, then constructs a road adhesion coefficient dynamic estimator based on particle filtering; secondly, it establishes a road adhesion coefficient estimation method based on image recognition. For three states of structured roads—dry, wet, and icy / snowy—the road is segmented using a DeeplabV3+ semantic segmentation network, and then classified using a MobileNetV2 lightweight convolutional neural network. The forward road adhesion coefficient is obtained by looking up a table. Finally, a spatiotemporal synchronization method and fusion strategy for image recognition and dynamic estimation are established, which realizes effective correlation and reliable fusion of the two types of adhesion coefficient estimation results. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A schematic diagram of the process for estimating road surface adhesion coefficient by fusing image recognition and dynamics, provided by the present invention;

[0063] Figure 2 This is a schematic diagram of the lightweight DeeplabV3+ network structure;

[0064] Figure 3 This is a schematic diagram of spatiotemporal synchronization;

[0065] Figure 4 This is a schematic diagram of the image recognition and dynamics fusion strategy process. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] The purpose of this invention is to provide a method and system for estimating road surface adhesion coefficient by fusing image recognition and dynamics, which can improve the accuracy of determining the road surface adhesion coefficient.

[0068] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] like Figure 1 As shown, the road surface adhesion coefficient estimation method based on image recognition and dynamics fusion provided by the present invention includes:

[0070] S101 uses an onboard camera to acquire images of the road surface in front of the vehicle.

[0071] S102, use DeeplabV3+ semantic segmentation network to segment the road surface image in front of the vehicle; and use MobileNetV2 lightweight convolutional neural network to classify the road surface type of the segmented road surface image in front of the vehicle.

[0072] DeeplabV3+ Semantic Segmentation Network: DeeplabV3+ is currently the most advanced semantic segmentation network, capable of road surface region segmentation. DeeplabV3+ employs an encoder-decoder structure. The encoder first extracts low-dimensional feature maps through a backbone network, then uses Atrous Spatial Pyramid Pooling (ASPP) to extract multi-scale feature information. 1×1 convolutions extract detailed feature information, while 3×3 dilated convolutions with different dilation rates extract information from receptive fields of different sizes. Global average pooling extracts global information. The decoder cascades low-dimensional feature maps with high-dimensional feature maps upsampled by four times, fusing spatial and channel information to improve semantic segmentation performance. Finally, 3×3 convolutions and four-fold upsampling yield the final segmentation result.

[0073] MobileNetV2 Convolutional Neural Network: MobileNetV2 employs depthwise separable convolution. First, each channel of the feature map is convolved with only one kernel, resulting in a feature map with the same number of channels, C. Then, the new feature map is convolved with N kernels of size 1×1×C. Finally, N feature maps are obtained, matching the number of kernels. Depthwise separable convolution effectively reduces parameter computation. MobileNetv2 further incorporates an inverse residual structure and a linear bottleneck module. The inverse residual structure adjusts the previous dimensionality reduction-up approach to a dimensionality increase-down approach, increasing the number of features to improve model accuracy. Since the ReLU activation function sets non-positive input features to zero, resulting in significant information loss, the bottleneck module uses a linear activation function to minimize this loss.

[0074] This invention employs the state-of-the-art DeeplabV3+ semantic segmentation network for road surface region division. Considering the computational capabilities of the onboard computing unit, the original backbone network Xception is replaced with MobilenetV2, which has fewer parameters. The network structure is as follows: Figure 2 As shown.

[0075] The semantic segmentation result labels the road surface area in purple and other background areas in black. The resulting image is then converted to grayscale, producing a grayscale image with only two values: 0 for the background and 90 for the road surface. Pixels with a value of 90 are assigned a value of 1, resulting in a matrix where the background is 0 and the road surface is 1. This matrix is ​​then compared with the original image matrix to obtain an image containing only the road surface area. This image is then input into a convolutional neural network to identify the road surface type.

[0076] This invention employs the lightweight convolutional neural network MobileNetV2. The MobileNetV2 network structure is shown in Table 1. In Table 1, Bottleneck represents the linear bottleneck module of the inverse residual; Conv2d represents two-dimensional convolution, and all convolutions without special annotation are 3×3 convolutions; Avgpool represents average pooling; and k represents the number of classes.

[0077] Table 1 MobileNetV2 Network Structure

[0078]

[0079] The road surface type is obtained from the image recognition, and the range of the road surface adhesion coefficient is determined by the mapping relationship between the road surface type and the road surface adhesion coefficient. The estimated value of the road surface adhesion coefficient is (upper limit of the range + lower limit of the range) / 2. The mapping relationship is shown in Table 2.

[0080] Table 2 Mapping Relationship between Road Type and Road Adhesion Coefficient

[0081]

[0082] S103, acquire vehicle status information using onboard sensors; the vehicle status information includes: front wheel steering angle, vehicle acceleration, wheel speed and torque.

[0083] S104. Based on vehicle state information and a single-wheel dynamics model, Kalman filtering is used to estimate the longitudinal force of the tire.

[0084] Establish a single-wheel dynamics model:

[0085] (1)

[0086] In the formula: The moment of inertia of the wheel; The angular velocity of the wheel rotation; The angular acceleration of the wheel's rotation. The driving or braking torque acting on the wheels; The radius of the wheel's rolling radius; This refers to the longitudinal force of the tire.

[0087] The discrete state-space equation of formula (1) is:

[0088] (2)

[0089] In the formula: For discrete time intervals; for The state variable at time t, =[ , ] T , for The state variable at any given time; = For observation variables; for The observed variables at time; = To control the quantity, for The amount of control at any given moment; for The time follows a normal distribution (0, Process noise; for The time follows a normal distribution (0, ) observation noise; For process noise The covariance matrix; The covariance matrix of the observation noise v; the state matrix. Control matrix With observation matrix They are respectively:

[0090] (3)

[0091] In the formula: For discrete sampling time.

[0092] Formula (2) is a linear system. The Kalman filter algorithm is used, and the basic process is as follows:

[0093] Step 4.1: Calculate the prior estimate based on the state variables and process noise from the previous time step, which can be expressed as:

[0094] (4)

[0095] in, This is the transpose of the state matrix. The prior estimate of the state variables at time k. for Estimated state variables at time points. Let k be the prior covariance matrix. for Time-varying covariance matrix For process noise The covariance matrix.

[0096] Step 4.2: Calculate the Kalman gain, which can be expressed as:

[0097] (5)

[0098] in, for Moment-time Kalman gain, For the transpose of the observation matrix, To observe noise The covariance matrix.

[0099] Step 4.3: Correct the prior estimate, which can be expressed as:

[0100] (6)

[0101] in, It is an identity matrix.

[0102] Step 4.4: Obtain from Time's up Optimal estimate of the state quantity at time step Repeat Step 4.1-Step 4.4.

[0103] S105. Based on the longitudinal force of the tire and the magic tire formula, a particle filter algorithm is used to establish a road surface adhesion coefficient estimator to estimate the first road surface adhesion coefficient.

[0104] Assuming that the road surface adhesion coefficient does not change drastically in a short period of time, the state-space equation can be expressed as:

[0105] (7)

[0106] In the formula: = Let be the state vector, where the subscript is . These represent the left front wheel, right front wheel, left rear wheel, and right rear wheel, respectively. = For observation vectors; Here is the state transition equation; The observation equation; for Time-matter noise; for Observe noise at all times.

[0107] State transition equation Represented as:

[0108] (8)

[0109] in, Indicates wheels, for Time of the first Estimated value of the road surface adhesion coefficient of the wheel. for Time of the first Estimated value of the road surface adhesion coefficient of the wheel. For the first wheel Time-matter noise.

[0110] Observation equations Composed of the vehicle's longitudinal motion equations and a magic tire model under longitudinal operating conditions, the magic tire formula model is modified using the principle of friction similarity to accurately characterize the tire's mechanical properties. The modified formula... Represented as:

[0111] (9)

[0112] In the formula: This refers to the longitudinal force of the tire; This is the actual road surface adhesion coefficient; This refers to the tire slip ratio.

[0113] Observation equations Represented as:

[0114] (10)

[0115] In the formula: For the overall vehicle weight; The longitudinal acceleration of the vehicle is measured by an inertial element; For the front wheel steering angle, for Time of the first The longitudinal force of the wheel's tire, for The longitudinal force on the left front tire at all times. for The longitudinal force on the right front tire at all times. for The longitudinal force on the left rear wheel at all times. for The longitudinal force on the right rear tire at all times. for Continuously monitor noise. , , , These are the stiffness factor, shape factor, peak factor, and curvature factor of the tire's mechanical property curve. for Time of the first Wheel slip ratio.

[0116] As can be seen, the road adhesion coefficient estimator is a nonlinear system; therefore, the particle filter algorithm, suitable for nonlinear non-Gaussian systems, is adopted. Taking a single wheel as an example, the particle filter algorithm flow is as follows:

[0117] Step 5.1: Initialize the number of particles Process noise Observation noise The initial particle set is determined based on the road surface adhesion coefficient value predicted by image recognition method. .

[0118] Step 5.2: According to The set of particles at time Each particle in the equation is calculated using the state transition equation (9). The prior estimate of the road surface adhesion coefficient at time t is:

[0119] (11)

[0120] Subscript Indicates the first One particle, =1, 2, 3, ... .

[0121] Step 5.3: Substitute the prior estimate of the state into formula (10) to calculate the prior observation. Error between prior observations and sensor measurements for:

[0122] (12)

[0123] Posterior probability density function With importance function The ratio is used as the particle weight. ,Right now:

[0124] (13)

[0125] in, for Time observation is State quantities under certain conditions The particle.

[0126] Particle weights after normalization It can be represented as:

[0127] (14)

[0128] Posterior estimate of road surface adhesion coefficient at time step for:

[0129] (15)

[0130] Step 5.4: Resample the particle set. First, generate random numbers from the uniform distribution U(0,1]. Based on this, a random array is generated as follows:

[0131] (16)

[0132] when When the index is within the cumulative interval of a certain particle weight, the sequence number The generated sequence number set is:

[0133] (17)

[0134] in, A function for calculating the new particle index. For serial numbers.

[0135] The new particle set is obtained by redistributing the weights of the new particles on an equal basis:

[0136] (18)

[0137] The aforementioned new set of particles is in = , = and = New number after resampling under the condition particle and the new number after resampling Particle weight The value of .

[0138] in, for After resampling the state quantity at time step, the new first time step particle, for New time after resampling Particle weights for The first state quantity at time moment particle.

[0139] Obtain the new set of particles and their weights, substitute them into the next loop, and repeat Step 5.2 - Step 5.4.

[0140] S106, the spatiotemporal synchronization module determines all image recognition results corresponding to the road surface with the first road surface adhesion coefficient, and determines the final image recognition result and image recognition confidence based on all image recognition results; the spatiotemporal synchronization module is used to find the image recognition results of all captured images of the road surface in front of the vehicle that contain the road surface with the first road surface adhesion coefficient, and takes the road surface type with the highest frequency of all image recognition results as the final image recognition result.

[0141] The working process of the time-space synchronization module is as follows: Figure 3 As shown, the vehicle-mounted front camera can predict road surface adhesion by capturing images of the road ahead of the vehicle, while the vehicle dynamics estimation method can only estimate the current road surface adhesion coefficient. Therefore, the estimation results from the two methods need to be matched in time and space. The camera sampling frequency is 30Hz, the GPS sampling frequency is 20Hz, and the inertial navigation sampling frequency is 100Hz; sampling is performed at a frequency of 20Hz.

[0142] Since image recognition-based road adhesion coefficient prediction identifies the road conditions in the future driving area of ​​the entire vehicle, rather than focusing on individual wheels, the vehicle can be approximated as a point mass for analysis during spatiotemporal synchronization. For example... Figure 3 As shown, The camera's field of view. This is the minimum distance from the vehicle's center of gravity to the shooting range. ( =1, 2, ..., ,at this time, For the first Each interval (Total number of intervals) Point cameras obtain road information ahead; In Get vehicle dynamics response information. The road surface information acquired by the cameras within the section all include Point. Images captured by the camera need to undergo semantic segmentation and convolutional neural network processing. Assume the computation time is... Then after The following vehicle was Move to It is necessary to ensure No more than Therefore, it is possible to... The camera capture points used in the vehicle dynamics algorithm fusion should meet the following requirements:

[0143] Condition (1)

[0144] Condition (2)

[0145] Select The road surface type that appears most frequently in all image recognition results within the interval is taken as the final image recognition result and input into the dynamic response processor. Record the number of times this type appears. The classification probability is then... Set the confidence level for image recognition. When a certain type of road surface occurs more frequently than Only then is the road surface type used as the final result of image recognition.

[0146] S107. Based on the final image recognition result, image recognition confidence level, first road surface adhesion coefficient, and the difference between the estimated and true values ​​of the vehicle's longitudinal acceleration, an image recognition and dynamics fusion strategy is established.

[0147] S108, the final road surface adhesion coefficient is determined based on the image recognition and dynamics fusion strategy.

[0148] S107 specifically includes:

[0149] When the image recognition confidence level is greater than or equal to the image method confidence threshold, the corresponding range of road surface adhesion coefficient is determined based on the final image recognition result.

[0150] The second road surface adhesion coefficient is determined based on the range of road surface adhesion coefficients.

[0151] When the first road surface adhesion coefficient is within the range of road surface adhesion coefficients, the first road surface adhesion coefficient is taken as the final road surface adhesion coefficient.

[0152] When the first road surface adhesion coefficient is not within the range of road surface adhesion coefficients, the second road surface adhesion coefficient is used as the final road surface adhesion coefficient.

[0153] When the image recognition confidence level is less than the image method confidence level threshold, determine whether the difference between the estimated value and the true value of the vehicle's longitudinal acceleration is greater than the error threshold.

[0154] If it is greater than the previous value, the road adhesion coefficient determined by the road adhesion coefficient estimator will be used as the final road adhesion coefficient.

[0155] If it is less than or equal to, then the first road surface adhesion coefficient shall be used as the final road surface adhesion coefficient.

[0156] When the image recognition confidence level is greater than or equal to the image method confidence threshold, the corresponding road surface adhesion coefficient range is determined based on the final image recognition result, and then the process further includes:

[0157] When the final image recognition result indicates a low-friction road surface, a warning signal is sent to the vehicle.

[0158] As a specific example, such as Figure 4 As shown, there are three possible scenarios:

[0159] (1) When the image recognition result is a low-adhesion road surface, a warning signal is sent to the vehicle.

[0160] Case (2) when When the image recognition results are reliable, the range of road adhesion coefficients corresponding to the road surface type can be obtained by looking up the table. The predicted value is , as the initial value for the dynamics-based particle filter estimator . judge Is it within the range? Between; if Located in the interval If the dynamic estimation is accurate, then the dynamic estimate is used. As the final Estimates; conversely, predicting results using graphical methods. As the final Estimated value.

[0161] Case (3) when If the image prediction result is unreliable, it is necessary to determine whether the dynamic-based estimation result is reliable. This is calculated as the difference between the observed longitudinal acceleration and the sensor measurement. As a basis for judgment, if The estimation based on dynamics is accurate, and the dynamic estimate is used. As the final The estimated value is used; otherwise, the estimated value from the previous time step remains unchanged.

[0162] The present invention also provides a road surface adhesion coefficient estimation system that integrates image recognition and dynamics, comprising:

[0163] The road surface image acquisition unit is used to acquire images of the road surface in front of the vehicle using an onboard camera.

[0164] The road surface classification unit is used to segment the road surface image in front of the vehicle using the DeeplabV3+ semantic segmentation network; and to classify the road surface type of the segmented road surface image in front of the vehicle using the MobileNetV2 lightweight convolutional neural network.

[0165] The vehicle status information acquisition unit is used to acquire vehicle status information using on-board sensors; the vehicle status information includes: front wheel steering angle, vehicle acceleration, wheel speed and torque.

[0166] The tire longitudinal force estimation unit is used to estimate the tire longitudinal force based on the vehicle state information and a single-wheel dynamics model using Kalman filtering.

[0167] The first road surface adhesion coefficient estimation unit is used to establish a road surface adhesion coefficient estimator based on the tire longitudinal force, the magic tire formula, and the particle filtering algorithm, and to estimate the first road surface adhesion coefficient.

[0168] The final image recognition result determination unit is used to determine all image recognition results corresponding to the road surface with the first road surface adhesion coefficient through the spatiotemporal synchronization module, and to determine the final image recognition result based on all image recognition results; the spatiotemporal synchronization module is used to search for all captured images of the road surface in front of the vehicle that contain the road surface with the first road surface adhesion coefficient, and to take the road surface type with the highest frequency of all image recognition results as the final image recognition result.

[0169] The image recognition and dynamics fusion strategy establishment unit is used to establish an image recognition and dynamics fusion strategy based on the final image recognition result, image recognition confidence, first road surface adhesion coefficient, and the difference between the estimated and true values ​​of the vehicle's longitudinal acceleration.

[0170] The final road surface adhesion coefficient determination unit is used to determine the final road surface adhesion coefficient based on the image recognition and dynamics fusion strategy.

[0171] Corresponding to the above method, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the aforementioned method for estimating road surface adhesion coefficient by image recognition and dynamic fusion.

[0172] Based on the above description, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned computer storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0173] This invention, in image recognition, comprehensively considers network recognition accuracy and computational speed, employing a lightweight DeeplabV3+ semantic segmentation network and a MobilenetV2 classification network. This effectively reduces model parameters, improves computational efficiency, and achieves high accuracy in road surface type recognition. The proposed spatiotemporal synchronization module filters all image recognition results that match the dynamic response points, effectively linking image recognition results with dynamic response information. A confidence calculation method for the image recognition method is proposed, with the confidence calculation automatically adjusted according to vehicle driving conditions. A low-adhesion road surface warning module is proposed, sending a warning signal to the vehicle when the road surface ahead is identified as low-adhesion. The road surface adhesion coefficient estimation method based on image recognition and dynamics proposed in this invention sets image recognition confidence thresholds and dynamic estimation error thresholds, fully considering situations where image recognition or dynamics recognition results are inaccurate, and can broadly cover vehicle driving conditions. To fully utilize the predictive capability of the image recognition method, the proposed low-adhesion road surface warning module can send a warning signal to the vehicle when the road surface ahead is identified as low-adhesion. Current research rarely considers the inaccuracy of image recognition and dynamic estimation of road adhesion coefficient. The fusion method of this invention fully considers the inaccuracy of the estimation results of the two methods, and sets image recognition confidence threshold and dynamic estimation error threshold, thus covering a wider range of working conditions.

[0174] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0175] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for estimating road surface adhesion coefficient by fusing image recognition and dynamics, characterized in that, include: Use an onboard camera to obtain images of the road surface in front of the vehicle; The DeeplabV3+ semantic segmentation network was used to segment the road surface image in front of the vehicle; The MobileNetV2 lightweight convolutional neural network was used to classify the road surface type of the segmented road surface image in front of the vehicle. Vehicle status information is obtained using onboard sensors; The vehicle status information includes: front wheel steering angle, vehicle acceleration, wheel speed, and torque; Based on vehicle status information and a single-wheel dynamics model, Kalman filtering is used to estimate the longitudinal force of the tires. Based on the longitudinal force of the tire and the magic tire formula, a particle filter algorithm is used to establish a road adhesion coefficient estimator to estimate the first road adhesion coefficient. The spatiotemporal synchronization module determines all image recognition results corresponding to the road surface with the first road surface adhesion coefficient, and determines the final image recognition result and image recognition confidence based on all image recognition results; the spatiotemporal synchronization module is used to find the image recognition results of all captured images of the road surface in front of the vehicle that contain the road surface with the first road surface adhesion coefficient, and takes the road surface type with the highest frequency of all image recognition results as the final image recognition result; Based on the final image recognition results, image recognition confidence, first road surface adhesion coefficient, and the difference between the estimated and true values ​​of the vehicle's longitudinal acceleration, an image recognition and dynamics fusion strategy is established. The final road surface adhesion coefficient is determined based on an image recognition and dynamics fusion strategy; The step of estimating tire longitudinal force based on vehicle state information, using a single-wheel dynamics model, and employing Kalman filtering specifically includes: use Determine the dynamic model of a single wheel; Using formula Determine the discrete state-space equations corresponding to the single-wheel dynamics model; Among them, J w ω is the moment of inertia of the wheel. w The angular velocity of the wheel rotation; T is the angular acceleration of the wheel. m R is the driving or braking torque acting on the wheels; w F is the rolling radius of the wheel; x For the longitudinal force of the tire; k is the discrete time step; x k-1 Let x be the state variable at time k-1. k Let z be the state variable at time k; z = ω w z is the observed variable; k Let u be the observed variable at time k; u = T m To control the quantity, u k-1 w is the control quantity at time k-1; k-1 The process noise at time k-1 follows a normal distribution N(0, Q); v k Let be the observation noise at time k that follows a normal distribution N(0, R); Q be the covariance matrix of the process noise w; R be the covariance matrix of the observation noise v; A be the state matrix; B be the control matrix; and H be the observation matrix. Based on the longitudinal force of the tire and the Magic Tire Formula, a particle filter algorithm is used to establish a road surface adhesion coefficient estimator to estimate the first road surface adhesion coefficient. Specifically, this includes the following formula: Among them, F x (s) represents the tire longitudinal force, μ is the actual road adhesion coefficient, s is the tire slip ratio, B, C, D, and E are the stiffness factor, shape factor, peak factor, and curvature factor of the tire mechanical characteristic curve, respectively, and S h and S v This represents the horizontal and longitudinal offsets of the tire force curve relative to the origin.

2. The method for estimating road surface adhesion coefficient by image recognition and dynamics fusion according to claim 1, characterized in that, The step of establishing an image recognition and dynamics fusion strategy based on the final image recognition result, image recognition confidence level, first road surface adhesion coefficient, and the difference between the estimated and true values ​​of vehicle longitudinal acceleration specifically includes: When the image recognition confidence level is greater than or equal to the image method confidence threshold, the corresponding range of road surface adhesion coefficient is determined based on the final image recognition result. The second road surface adhesion coefficient is determined based on the range of road surface adhesion coefficients; When the first road surface adhesion coefficient is within the range of road surface adhesion coefficients, the first road surface adhesion coefficient is taken as the final road surface adhesion coefficient. When the first road surface adhesion coefficient is not within the range of road surface adhesion coefficients, the second road surface adhesion coefficient shall be used as the final road surface adhesion coefficient. When the image recognition confidence level is less than the image method confidence level threshold, determine whether the difference between the estimated value and the true value of the vehicle's longitudinal acceleration is greater than the error threshold. If it is greater than the previous time, the road adhesion coefficient determined by the road adhesion coefficient estimator will be used as the final road adhesion coefficient. If it is less than or equal to, then the first road surface adhesion coefficient shall be used as the final road surface adhesion coefficient.

3. The method for estimating road surface adhesion coefficient by fusing image recognition and dynamics according to claim 2, characterized in that, When the image recognition confidence level is greater than or equal to the image method confidence threshold, the corresponding road surface adhesion coefficient range is determined based on the final image recognition result, and then the process further includes: When the final image recognition result indicates a low-friction road surface, a warning signal is sent to the vehicle.

4. A road surface adhesion coefficient estimation system integrating image recognition and dynamics, used to implement the road surface adhesion coefficient estimation method integrating image recognition and dynamics as described in any one of claims 1-3, characterized in that, include: The road surface image acquisition unit is used to acquire images of the road surface in front of the vehicle using an onboard camera. The road surface classification unit is used to segment images of the road surface in front of a vehicle using the DeeplabV3+ semantic segmentation network. The MobileNetV2 lightweight convolutional neural network was used to classify the road surface type of the segmented road surface image in front of the vehicle. The vehicle status information acquisition unit is used to acquire vehicle status information using on-board sensors. The vehicle status information includes: front wheel steering angle, vehicle acceleration, wheel speed, and torque; The tire longitudinal force estimation unit is used to estimate the tire longitudinal force based on the vehicle state information and a single-wheel dynamics model using Kalman filtering. The first road surface adhesion coefficient estimation unit is used to establish a road surface adhesion coefficient estimator based on the tire longitudinal force, the magic tire formula, and the particle filter algorithm to estimate the first road surface adhesion coefficient. The final image recognition result determination unit is used to determine all image recognition results corresponding to the road surface with the first road surface adhesion coefficient through the spatiotemporal synchronization module, and to determine the final image recognition result and image recognition confidence based on all image recognition results; the spatiotemporal synchronization module is used to search for the image recognition results of all captured images of the road surface in front of the vehicle that contain the road surface with the first road surface adhesion coefficient, and to take the road surface type with the highest frequency of all image recognition results as the final image recognition result; The image recognition and dynamics fusion strategy establishment unit is used to establish an image recognition and dynamics fusion strategy based on the final image recognition result, image recognition confidence, first road surface adhesion coefficient, and the difference between the estimated and true values ​​of the vehicle's longitudinal acceleration. The final road surface adhesion coefficient determination unit is used to determine the final road surface adhesion coefficient based on the image recognition and dynamics fusion strategy.

5. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to enable the electronic device to perform a method for estimating road surface adhesion coefficient by image recognition and dynamic fusion according to any one of claims 1 to 3.

Images