Vision and dynamics information fused road adhesion coefficient estimation device and method

Abstract

The invention provides a road adhesion coefficient estimation device and method based on vision and dynamics information fusion. The method comprises the steps of vehicle dynamics estimation based on extended Kalman filtering, road type identification based on a lightweight convolutional neural network and a fusion strategy of vision and dynamics information. According to the pavement type identification method based on the lightweight convolutional neural network, efficient and accurate visual information processing is realized; in combination with dynamic response information, space-time synchronization and fusion estimation of vision and dynamic data are realized, and predictability and accuracy of road adhesion coefficient estimation are improved; in addition, real-time and reliable adhesion coefficient input is provided for an active safety system, and the vehicle safety and stability in a complex scene are remarkably improved.

Description

Technical Field

[0001] The invention relates to a method for estimating a road adhesion coefficient of an intelligent vehicle, and in particular to a method for estimating a road adhesion coefficient by fusing visual and dynamic information. Background Art

[0002] There are few studies on the existing technologies for the identification of road adhesion coefficient under the vehicle's lateral and longitudinal coupled motion, and the identification results are lagging, with large estimation errors and insufficient real-time performance. In addition, the road type has a significant impact on the adhesion coefficient. The existing estimation methods perform poorly under extreme conditions and are difficult to cope with complex and changing driving environments. In addition, most of the existing methods are designed based on the measurement information of traditional on-board sensors, and fail to make full use of the new sensor information brought about by the development of automobile intelligence, such as video images. Therefore, it is a technical problem that urgently needs to be solved to study how to combine visual information and vehicle dynamics response to improve the accuracy and real-time performance of road adhesion coefficient estimation and overcome complex road influencing factors in order to achieve predictive and high-precision estimation of intelligent vehicles. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a road adhesion coefficient estimation method by integrating visual and dynamic information, comprising the following steps:

[0004] Step 1: Vehicle dynamics estimation based on extended Kalman filter.

[0005] Firstly, a vehicle dynamics model is constructed to describe the longitudinal, lateral and yaw motion characteristics of the vehicle. The longitudinal and lateral forces of the tires are modeled to describe the tire characteristics under different working conditions. Then, an extended Kalman filter is used to construct a friction coefficient estimation model based on dynamic information to estimate the road adhesion coefficient.

[0006] Step 2: Road surface type recognition based on lightweight convolutional neural network.

[0007] First, the on-board camera is used to collect road images and screen out different road types. Then, the convolutional neural network model is trained offline so that the model can accurately identify the road type. Finally, the correspondence between different road types and adhesion coefficient ranges at different vehicle speeds is established, and the identified road type is matched with the road adhesion coefficient.

[0008] Step 3: Strategy for integrating visual and dynamic information.

[0009] Firstly, a spatiotemporal synchronization scheme of visual information and dynamic information is designed to ensure that the estimation results of different methods are matched at the same time. Then, fusion rules are formulated to assign judgment logic to the estimation results of visual method and dynamic method, and the estimation results after the fusion of visual information and dynamic information are obtained.

[0010] Specifically, in step 1, the constructed vehicle dynamics equation is characterized as:

[0011] ma x =(F xfl +F xfr )cosδ-(F yfl +F yfr )sinδ+F xrl +F xrr

[0012] ma y =(F xfl +F xfr )sinδ+(F yfl +F yfr )cosδ+F yrl +F yrr

[0013]

[0014] Among them, a x is the longitudinal acceleration of the vehicle, a y is the vehicle lateral acceleration, is the yaw angular acceleration, I z is the moment of inertia of the vehicle around the z-axis, F xij is the longitudinal force in the wheel coordinate system, F yij is the lateral force in the wheel coordinate system. The subscripts fl, fr, rl, and rr represent the front left wheel, front right wheel, rear left wheel, and rear right wheel of the vehicle, respectively. f is the distance from the center of mass to the front axle, l r is the distance from the center of mass to the rear axle, δ is the front wheel turning angle, m is the total mass of the vehicle, T f is the front wheel track, T r is the rear wheel track;

[0015] Considering the axle load transfer during vehicle driving, the vertical load formula of each wheel of the vehicle is:

[0016]

[0017] Among them, F zij is the vertical load of each wheel, h is the height of the vehicle's center of mass, l is the distance from the front axle to the rear axle, l = l f +l r , g is the acceleration due to gravity;

[0018] For single wheel force analysis, the formulas for tire longitudinal force and lateral force in the tire coordinate system are:

[0019]

[0020] Where μ is the road adhesion coefficient, is the normalized tire longitudinal force, is the normalized tire lateral force, C x is the tire longitudinal slip coefficient, C y is the tire cornering stiffness, λ is the wheel slip rate, α is the wheel slip angle, and L is the boundary correction value.

[0021] Furthermore, by extending the Kalman filter, a friction coefficient estimation model based on dynamic information is constructed. The steps are as follows:

[0022] The vehicle dynamics system is modeled as follows:

[0023] x k+1 =f(x k ,u k )+w k

[0024] y k =h(x k ,u k )+v k

[0025] In the formula, x k is the state variable, u k is the input variable, y k is the observed variable, f(x k ,u k ) is the nonlinear state transfer equation, h(x k ,u k ) is the nonlinear observation equation, w k and v k are process noise and measurement noise, respectively, which are assumed to be Gaussian distributed;

[0026] Among them, the state variable The friction coefficients of the front left wheel, front right wheel, rear left wheel, and rear right wheel of the car are input variables u k =[δ], observed variable

[0027] The specific calculation formula of the observation equation is:

[0028]

[0029] For the above observation equation, the Jacobian matrix of the partial derivative of the state vector is solved to obtain the observation equation for estimating the road adhesion coefficient:

[0030]

[0031] in:

[0032]

[0033]

[0034] Furthermore, the iterative process of the extended Kalman filter is as follows:

[0035] a. Initialization: Initialize state variables and set state variables x 0 and the covariance matrix P 0 , define the noise covariance matrices Q and R to represent process noise and measurement noise respectively;

[0036] b. Time update: Generate Sigma points of state variables through untraceable transformation:

[0037]

[0038] Where n is the state variable dimension, ξ is the scaling parameter;

[0039] Secondly, substitute each Sigma point into the nonlinear state transfer equation f(x k ,u k )have to:

[0040]

[0041] Then, the predicted state mean is calculated based on the propagated Sigma point and the predicted covariance P k+1∣k have to:

[0042]

[0043] in, and are the weights of the mean and covariance respectively;

[0044] c. Measurement update: First, through the observation equation h(x k ,u k ), calculate the observed value of each Sigma point:

[0045]

[0046] Compute the mean of the predicted observations:

[0047]

[0048] Next, calculate the observation covariance:

[0049]

[0050] Compute the cross-covariance of states and observations:

[0051]

[0052] Finally, the state estimate is updated using the Kalman gain K k Update state mean and covariance P k+1 :

[0053]

[0054] After iterating through steps a to c, the road adhesion coefficient μ based on the extended Kalman filter is estimated. D .

[0055] Specifically, in step 2, the vehicle-mounted camera is installed on the front air intake grille of the vehicle, the viewing angle of the vehicle-mounted camera is adjusted so that the captured road image occupies most of the entire video image area, the image is read at a fixed frequency, and the acquired road image is collected and stored through the IPC as data used for offline training of the convolutional neural network model; the road surface area of ​​the original road surface image is cropped into image blocks of a certain size, and the data types are divided according to the attributes of the road, including: dry asphalt road, wet asphalt road, dry cement road, wet cement road, dry soil road, wet soil road, dry gravel road, wet gravel road, loose snow road, melting snow road and ice film road; the divided data sets are divided into one or more of the training set, the validation set and the test set.

[0056] Furthermore, the convolutional neural network model architecture mainly includes three parts: convolution layer, pooling layer and fully connected layer; the activation function adopts a linear rectification function to improve the learning speed and classification ability; the features extracted from the convolution layer and the pooling layer are mapped to the sample label space through two fully connected layers, so as to classify and predict various types of road surfaces; a combination of a normalized exponential function and a cross entropy loss function is used to evaluate the performance of the model by quantifying the difference between the predicted and true label probability distributions; the cross entropy loss function is expressed as follows:

[0057]

[0058] Among them, x[m] represents the model output of the mth category, and x[n] represents the model output of the true category label.

[0059] The above convolutional neural network model is trained offline using the divided data set, so that the model can accurately identify the road surface type;

[0060] Establish a mapping relationship between different vehicle speeds and different road surface types and adhesion coefficient ranges, match the identified road surface type with the road surface adhesion coefficient according to the mapping relationship, and output the adhesion coefficient μ corresponding to the identified road surface type p .

[0061] Specifically, in step 3, the goal of the spatiotemporal synchronization scheme is to find the time point t k , when the vehicle completes the image classification, the vehicle has moved from position P i Move to position P i ', the spatial position of the visual information needs to be corrected so that both can reflect the road surface characteristics at the same position;

[0062] If the vehicle is traveling at a non-constant speed, at t k At time x, the vehicle is k Position, vehicle dynamics response information is D(t k ), the front road image collected by the on-board camera covers a distance of 5 to 50 meters in front of the vehicle, ensuring that the effective area of ​​the collected image includes the position:

[0063] P i +5≤x k ≤P i +50

[0064] Furthermore, for different situations, the time-space synchronization scheme is as follows:

[0065] (1) Image processing is delayed, but contains the vehicle’s current position x k :If the image area covers the current position of the vehicle, the dynamic information D(x k ) synchronized with visual information;

[0066] (2) Image processing is delayed and does not include the vehicle's current position x k : Synchronization is unavailable at this time, marking the image and the corresponding dynamics information as invalid;

[0067] (3) Fusion of multiple frames: Select the image close to the current position x k The multiple frames of images that are closest in time are weighted averaged to improve the robustness of the synchronization results.

[0068] Furthermore, the fusion rule comprises the following steps:

[0069] First, according to step 1, vehicle dynamics information is collected to estimate the road adhesion coefficient μ D ; According to step 2, the road image is used to identify the road type, and the road type information is output as [Index ab μ P ], where a and b represent the lower and upper limits of the corresponding road type, respectively;

[0070] When the vehicle starts running, the friction coefficient estimation model based on dynamic information initializes the parameters, the neural network model starts running, and the time and mileage information are recorded;

[0071] In t kAt this moment, find t according to the time-space synchronization scheme k The nearest several road image sample points before time, the position is P' i , i=(1,2,…,n), the road type recognition result is Index i , where the larger the value of i, the closer the sample is to t k The shorter the time interval, the farther away from x the image is when it is taken. k The closer it is, the more reliable the image recognition result is;

[0072] Preset weight coefficient w P,i , the larger i is, the larger w is P,i The larger it is, the formula is satisfied:

[0073]

[0074] The weighted sum probability value P(Index=j) of detecting the same Index value in n sample points:

[0075]

[0076] Among them, j = (0, 1, ..., m), indicating m types of roads;

[0077] By comparing, find the maximum probability value P max and its corresponding Index value, indicating the most reliable road type recognition result; finally, t k The image recognition results used for the fusion algorithm at this moment [Index k a k b k ], image recognition confidence threshold P CF , establish the following fusion rules:

[0078] If P max ≤P CF , indicating that the road surface features are not obvious, or such image sample sets have not been established in the road surface image database. In this case, the final adhesion coefficient output is based on the dynamic estimation, that is, μ = μ D ; Save a set of road images and prior values ​​μ at this moment P =μ D , update the image sample library for the post-training process of the classification network;

[0079] If (P max ≥P CF )∩(a k ≤μ D ≤ b k ), indicating that the road surface image features are obvious, the road type recognition result has a high confidence level, and is consistent with the dynamic estimation result. At this time, the final adhesion coefficient output is μD , and update the prior value μ of the image recognition result P =μ D ;

[0080] If (P max ≥P CF )∩(μ D ≤a k )∪(μ D >b k ), indicating that the road type recognition result has a high confidence level, but is inconsistent with the dynamic estimation result. At this time, the final adhesion coefficient output is based on image recognition, that is, μ = μ D .

[0081] The present invention also provides a road adhesion coefficient estimation device that integrates visual and dynamic information, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned road adhesion coefficient estimation method that integrates visual and dynamic information is implemented.

[0082] Beneficial effects of the present invention:

[0083] The present invention provides a road adhesion coefficient estimation method that integrates visual and dynamic information, proposes a road type recognition method based on a lightweight convolutional neural network, and realizes efficient and accurate visual information processing; combined with dynamic response information, it realizes the spatiotemporal synchronization and fusion estimation of visual and dynamic data, and improves the predictability and accuracy of road adhesion coefficient estimation; in addition, the present invention provides real-time and reliable adhesion coefficient input for active safety systems, significantly improving vehicle safety and stability in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 Schematic diagram of the process of estimating the road adhesion coefficient of the present invention;

[0085] Figure 2 A schematic diagram of the vehicle deployment of the vehicle-mounted camera of the present invention;

[0086] Figure 3 This is a schematic diagram of the offline training structure of the convolutional neural network of the present invention. DETAILED DESCRIPTION

[0087] The present invention provides a road adhesion coefficient estimation method by integrating visual and dynamic information, such as Figure 1 As shown, the following steps are included:

[0088] Step 1: Vehicle dynamics estimation based on extended Kalman filter.

[0089] Firstly, a vehicle dynamics model is constructed to describe the longitudinal, lateral and yaw motion characteristics of the vehicle. The longitudinal and lateral forces of the tires are modeled to describe the tire characteristics under different working conditions. Then, an extended Kalman filter is used to construct a friction coefficient estimation model based on dynamic information to estimate the road adhesion coefficient.

[0090] Specifically, in step 1, the constructed vehicle dynamics equation is characterized as:

[0091] ma x =(F xfl +F xfr )cosδ-(F yfl +F yfr )sinδ+F xrl +F xrr

[0092] ma y =(F xfl +F xfr )sinδ+(F yfl +F yfr )cosδ+F yrl +F yrr

[0093]

[0094] Among them, a x is the longitudinal acceleration of the vehicle, a y is the vehicle lateral acceleration, is the yaw angular acceleration, I z is the moment of inertia of the vehicle around the z-axis, F xij is the longitudinal force in the wheel coordinate system, F yij is the lateral force in the wheel coordinate system. The subscripts fl, fr, rl, and rr represent the front left wheel, front right wheel, rear left wheel, and rear right wheel of the vehicle, respectively. f is the distance from the center of mass to the front axle, l r is the distance from the center of mass to the rear axle, δ is the front wheel turning angle, m is the total mass of the vehicle, T f is the front wheel track, T r is the rear wheel track;

[0095] Considering the axle load transfer during vehicle driving, the vertical load formula of each wheel of the vehicle is:

[0096]

[0097] Among them, F zij is the vertical load on each wheel, h is the height of the vehicle's center of mass, l is the distance from the front axle to the rear axle, l = a + b, and g is the acceleration due to gravity;

[0098] For single wheel force analysis, the formulas for tire longitudinal force and lateral force in the tire coordinate system are:

[0099]

[0100] Where μ is the road adhesion coefficient, is the normalized tire longitudinal force, is the normalized tire lateral force, C x is the tire longitudinal slip coefficient, C y is the tire cornering stiffness, λ is the wheel slip rate, α is the wheel slip angle, and L is the boundary correction value.

[0101] By extending Kalman filtering, a friction coefficient estimation model based on dynamic information is constructed. The steps are as follows:

[0102] The vehicle dynamics system is modeled as follows:

[0103] x k+1 =f(x k ,u k )+w k

[0104] y k =h(x k ,μ k )+v k

[0105] In the formula, x k is the state variable, u k is the input variable, y k is the observed variable, f(x k ,u k ) is the nonlinear state transfer equation, h(x k ,u k ) is the nonlinear observation equation, w k and v k are process noise and measurement noise, respectively, which are assumed to be Gaussian distributed;

[0106] Among them, the state variable The friction coefficients of the front left wheel, front right wheel, rear left wheel, and rear right wheel of the car are input variables u k =[δ], observed variable

[0107] The specific calculation formula of the observation equation is:

[0108]

[0109] For the above observation equation, the Jacobian matrix of the partial derivative of the state vector is solved to obtain the observation equation for estimating the road adhesion coefficient:

[0110]

[0111] in:

[0112]

[0113] The iterative process of the extended Kalman filter is as follows:

[0114] a. Initialization: Initialize state variables and set state variables x 0 and the covariance matrix P 0 , define the noise covariance matrices Q and R to represent process noise and measurement noise respectively;

[0115] b. Time update: Generate Sigma points of state variables through untraceable transformation:

[0116]

[0117] Where n is the state variable dimension, ξ is the scaling parameter;

[0118] Secondly, substitute each Sigma point into the nonlinear state transfer equation f(x k ,u k )have to:

[0119]

[0120] Then, the predicted state mean is calculated based on the propagated Sigma point and the predicted covariance P k+1∣k have to:

[0121]

[0122] in, and are the weights of the mean and covariance respectively;

[0123] c. Measurement update: First, through the observation equation h(x k ,u k ), calculate the observed value of each Sigma point:

[0124]

[0125] Compute the mean of the predicted observations:

[0126]

[0127] Next, calculate the observation covariance:

[0128]

[0129] Compute the cross-covariance of states and observations:

[0130]

[0131] Finally, the state estimate is updated using the Kalman gain K k Update state mean and covariance P k+1 :

[0132]

[0133] After iterating through steps a to c, the road adhesion coefficient μ based on the extended Kalman filter is estimated. D , the estimation method can be configured in the vehicle controller.

[0134] Step 2: Road surface type recognition based on lightweight convolutional neural network.

[0135] First, the on-board camera is used to collect road images and screen out different road types. Then, the convolutional neural network model is trained offline so that the model can accurately identify the road type. Finally, the correspondence between different road types and adhesion coefficient ranges at different vehicle speeds is established, and the identified road type is matched with the road adhesion coefficient.

[0136] Specifically, in step 2, if Figure 2 As shown, the vehicle-mounted camera is installed on the front air intake grille of the vehicle, the viewing angle of the vehicle-mounted camera is adjusted so that the photographed road image occupies more than 95% of the entire video image area, the image is read at a fixed frequency, and the acquired road image is collected and stored through the IPC as data used for offline training of the convolutional neural network model; the road surface area of ​​the original road surface image is cropped into image blocks of size 360*240, and the data types are divided according to the attributes of the road, including: dry asphalt road, wet asphalt road, dry cement road, wet cement road, dry soil road, wet soil road, dry gravel road, wet gravel road, loose snow road, melting snow road and ice film road; the divided data set is divided into a training set, a validation set and a test set.

[0137] like Figure 3 As shown in the figure, the convolutional neural network model architecture mainly includes three parts: convolution layer, pooling layer and fully connected layer; the activation function adopts a linear rectification function to improve the learning speed and classification ability; the features extracted from the convolution layer and the pooling layer are mapped to the sample label space through two fully connected layers, so as to classify and predict various types of road surfaces; the combination of the normalized exponential function and the cross entropy loss function is used to evaluate the performance of the model by quantifying the difference between the predicted and true label probability distributions; the cross entropy loss function is expressed as follows:

[0138]

[0139] Among them, x[m] represents the model output of the mth category, and x[n] represents the model output of the true category label.

[0140] The above convolutional neural network model is trained offline using the divided data set, so that the model can accurately identify the road surface type;

[0141] Establish a mapping relationship between different vehicle speeds and different road surface types and adhesion coefficient ranges, match the identified road surface type with the road surface adhesion coefficient according to the mapping relationship, and output the adhesion coefficient μ corresponding to the identified road surface type p The mapping relationship is shown in Table 1.

[0142] Step 3: Strategy for integrating visual and dynamic information.

[0143] Firstly, a spatiotemporal synchronization scheme of visual information and dynamic information is designed to ensure that the estimation results of different methods are matched at the same time. Then, fusion rules are formulated to assign judgment logic to the estimation results of visual method and dynamic method, and the estimation results after the fusion of visual information and dynamic information are obtained.

[0144] Specifically, in step 3, the goal of the spatiotemporal synchronization scheme is to find the time point t k , when the vehicle completes the image classification, the vehicle has moved from position P i Move to position P i ', the spatial position of the visual information needs to be corrected so that both can reflect the road surface characteristics at the same position;

[0145] If the vehicle is traveling at a non-constant speed, at t k At time x, the vehicle is t Position, vehicle dynamics response information is D(t k ), the front road image collected by the on-board camera covers a distance of 5 to 50 meters in front of the vehicle, ensuring that the effective area of ​​the collected image includes the position:

[0146] P i +5≤x k ≤P i +50

[0147] According to different situations, the time-space synchronization scheme is as follows:

[0148] (1) Image processing is delayed, but contains the vehicle’s current position x k :If the image area covers the current position of the vehicle, the dynamic information D(t k ) synchronized with visual information;

[0149] (2) Image processing is delayed and does not include the vehicle's current position xk : Synchronization is unavailable at this time, marking the image and the corresponding dynamics information as invalid;

[0150] (3) Fusion of multiple frames: Select the image close to the current position x k The multiple frames of images that are closest in time are weighted averaged to improve the robustness of the synchronization results.

[0151] The fusion rule comprises the following steps:

[0152] First, according to step 1, vehicle dynamics information is collected to estimate the road adhesion coefficient μ D ; According to step 2, the road image is used to identify the road type, and the road type information is output as [Index ab μ P ], where a and d represent the lower and upper limits of the corresponding road type, respectively;

[0153] When the vehicle starts running, the friction coefficient estimation model based on dynamic information initializes the parameters, the neural network model starts running, and the time and mileage information are recorded;

[0154] In t k At this moment, find t according to the time-space synchronization scheme k The 10 nearest road image sample points before time, position is P' i , i=(1,2,…,10), the road type recognition result is Index i , where the larger the value of i, the closer the sample is to t k The shorter the time interval, the farther away from x the image is when it is taken. k The closer it is, the more reliable the image recognition result is;

[0155] Preset weight coefficient w P,i , the larger i is, the larger w is P,i The larger it is, the formula is satisfied:

[0156]

[0157] Detect the weighted sum probability value P(Index=j) of the same Index value in 10 sample points:

[0158]

[0159] Among them, j = (0, 1, ..., 11), indicating 11 road types;

[0160] By comparing, find the maximum probability value P max and its corresponding Index value, indicating the most reliable road type recognition result; finally, t k The image recognition results used for the fusion algorithm at this moment [Indexk a k b k ], image recognition confidence threshold P CF , establish the following fusion rules:

[0161] If P max ≤P CF , indicating that the road surface features are not obvious, or such image sample sets have not been established in the road surface image database. In this case, the final adhesion coefficient output is based on the dynamic estimation, that is, μ = μ D ; Save a set of road images and prior values ​​μ at this moment P =μ D , update the image sample library for the post-training process of the classification network;

[0162] If (P max ≥P CF )∩(a k ≤μ D ≤ b k ), indicating that the road surface image features are obvious, the road type recognition result has a high confidence level, and is consistent with the dynamic estimation result. At this time, the final adhesion coefficient output is μ D , and update the prior value μ of the image recognition result P =μ D ;

[0163] If (P max ≥P CF )∩(μ D ≤a k )∪(μ D >b k ), indicating that the road type recognition result has a high confidence level, but is inconsistent with the dynamic estimation result. At this time, the final adhesion coefficient output is based on image recognition, that is, μ = μ D .

[0164] Table 1 Comparison table of different road surface types and adhesion coefficient range values

[0165]

Claims

1. A road adhesion coefficient estimation method based on the fusion of visual and dynamic information, characterized in that: The following steps are involved: Step 1: Vehicle dynamics estimation based on extended Kalman filter: First, a vehicle dynamics model is constructed to describe the longitudinal, lateral and yaw motion characteristics of the vehicle; the longitudinal and lateral forces of the tires are modeled to describe the tire characteristics under different working conditions; then, an extended Kalman filter is used to construct a friction coefficient estimation model based on dynamic information to estimate the road adhesion coefficient; Step 2: Road surface type recognition based on lightweight convolutional neural network: First, the vehicle-mounted camera is used to collect road images and screen out different road types. Then, the convolutional neural network model is trained offline so that the model can accurately identify the road type. Finally, the corresponding relationship between different road types and adhesion coefficient ranges at different vehicle speeds is established, and the identified road type is matched with the road adhesion coefficient. Step 3: Fusion strategy of visual and dynamic information: Firstly, a spatiotemporal synchronization scheme of visual information and dynamic information is designed to ensure that the estimation results of different methods are matched at the same time. Then, fusion rules are formulated to assign judgment logic to the estimation results of visual method and dynamic method, and the estimation results after the fusion of visual information and dynamic information are obtained.

2. The method for estimating road adhesion coefficient by integrating visual and dynamic information according to claim 1, characterized in that: In step 1, the constructed vehicle dynamics equation is represented as: ma x =(F xfl +F xfr )cosδ-(F yfl +F yfr )sinδ+F xrl +F xrr ma y =(F xfl +F xfr )sinδ+(F yfl +F yfr )cosδ+F yrl +F yrr Among them, a x is the longitudinal acceleration of the vehicle, a y is the vehicle lateral acceleration, is the yaw angular acceleration, I z is the moment of inertia of the vehicle around the z-axis, F xij is the longitudinal force in the wheel coordinate system, F yij is the lateral force in the wheel coordinate system. The subscripts fl, fr, rl, and r represent the front left wheel, front right wheel, rear left wheel, and rear right wheel of the vehicle, respectively. f is the distance from the center of mass to the front axle, l r is the distance from the center of mass to the rear axle, δ is the front wheel turning angle, m is the total mass of the vehicle, T f is the front wheel track, T r is the rear wheel track; Considering the axle load transfer during vehicle driving, the vertical load formula of each wheel of the vehicle is: Among them, F zij is the vertical load of each wheel, h is the height of the vehicle's center of mass, l is the distance from the front axle to the rear axle, l = l f +l r , g is the acceleration due to gravity; For single wheel force analysis, the formulas for tire longitudinal force and lateral force in the tire coordinate system are: Where μ is the road adhesion coefficient, is the normalized tire longitudinal force, is the normalized tire lateral force, C x is the tire longitudinal slip coefficient, C y is the tire cornering stiffness, λ is the wheel slip rate, α is the wheel slip angle, and L is the boundary correction value.

3. The method for estimating road adhesion coefficient by integrating visual and dynamic information according to claim 1, characterized in that: In step 1, the friction coefficient estimation model based on dynamic information is constructed by extending the Kalman filter. The steps are as follows: The vehicle dynamics system is modeled as follows: x k+1 =f(x k ,u k )+w k y k =h(x k ,u k )+v k In the formula, x k is the state variable, u k is the input variable, y k is the observed variable, f(x k ,u k ) is the nonlinear state transfer equation, h(x k ,u k ) is the nonlinear observation equation, w k and v k are process noise and measurement noise, respectively, which are assumed to be Gaussian distributed; Among them, the state variable The friction coefficients of the front left wheel, front right wheel, rear left wheel, and rear right wheel of the car are input variables u k =[δ], observed variable The calculation formula of the observation equation is: For the above observation equation, the Jacobian matrix of the partial derivative of the state vector is solved to obtain the observation equation for estimating the road adhesion coefficient: in:

4. The method for estimating road adhesion coefficient by integrating visual and dynamic information according to claim 3 is characterized in that: The iterative process of the extended Kalman filter is as follows: a. Initialization: Initialize the state variables, set the state variable x0 and the covariance matrix P0, and define the noise covariance matrices Q and R to represent the process noise and measurement noise respectively; b. Time update: Generate Sigma points of state variables through untraceable transformation: Where n is the state variable dimension, ξ is the scaling parameter; Secondly, substitute each Sigma point into the nonlinear state transfer equation f(x k ,u k )have to: Then, the predicted state mean is calculated based on the propagated Sigma point and the predicted covariance P k+1∣k have to: in, and are the weights of the mean and covariance respectively; c. Measurement update: First, through the observation equation h(x k ,u k ), calculate the observed value of each Sigma point: Compute the mean of the predicted observations: Next, calculate the observation covariance: Compute the cross-covariance of states and observations: Finally, the state estimate is updated using the Kalman gain K k Update state mean and covariance P k+1 : After iterating through steps a to c, the road adhesion coefficient μ based on the extended Kalman filter is estimated. D .

5. The method for estimating road adhesion coefficient by integrating visual and dynamic information according to claim 1, characterized in that: In step 2, road images are collected as data for offline training of the convolutional neural network model; the road surface area of ​​the original road image is cropped into image blocks of a certain size, and the data types are divided according to the attributes of the road, including: dry asphalt road, wet asphalt road, dry cement road, wet cement road, dry soil road, wet soil road, dry gravel road, wet gravel road, loose snow road, melting snow road and ice film road; the divided data sets are divided into one or more of the following: training set, validation set and test set.

6. The method for estimating road adhesion coefficient by integrating visual and dynamic information according to claim 1, characterized in that: In step 2, the convolutional neural network model architecture mainly includes three parts: convolution layer, pooling layer and fully connected layer; the activation function adopts a linear rectification function; the features extracted from the convolution layer and the pooling layer are mapped to the sample label space through two fully connected layers, so as to classify and predict various types of road surfaces; A combination of the normalized exponential function and the cross entropy loss function is used to evaluate the performance of the model by quantifying the difference between the predicted and true label probability distributions; the cross entropy loss function is expressed as follows: Among them, x[m] represents the model output of the mth category, and x[n] represents the model output of the true category label; The above convolutional neural network model is trained offline using the divided data set, so that the model can accurately identify the road surface type; Establish a mapping relationship between different vehicle speeds and different road surface types and adhesion coefficient ranges, match the identified road surface type with the road surface adhesion coefficient according to the mapping relationship, and output the adhesion coefficient μ corresponding to the identified road surface type p .

7. The method for estimating road adhesion coefficient by integrating visual and dynamic information according to claim 1, characterized in that: In step 3, the time-space synchronization scheme is as follows: (1) Image processing is delayed, but contains the vehicle’s current position x k :If the image area covers the current position of the vehicle, the dynamic information D(t k ) synchronized with visual information; (2) Image processing is delayed and does not include the vehicle's current position x k : Synchronization is unavailable at this time, marking the image and the corresponding dynamics information as invalid; (3) Fusion of multiple frames: Select the image close to the current position x k The multiple frames of images that are closest in time are weighted averaged to improve the robustness of the synchronization results.

8. The method for estimating road adhesion coefficient by integrating visual and dynamic information according to claim 1, characterized in that: In step 3, the fusion rule includes the following steps: First, according to step 1, vehicle dynamics information is collected to estimate the road adhesion coefficient μ D ; According to step 2, the road image is used to identify the road type, and the road type information is output as [Index a bμ P ], where a and b represent the lower and upper limits of the corresponding road type, respectively; When the vehicle starts running, the friction coefficient estimation model based on dynamic information initializes the parameters, the neural network model starts running, and the time and mileage information are recorded; In t k At this moment, find t according to the time-space synchronization scheme k The nearest several road image sample points before time, position is P′ i , i = (1, 2, ..., n), the road type recognition result is Index i , where the larger the value of i, the closer the sample is to t k The shorter the time interval, the farther away from x the image is when it is taken. k The closer it is, the more reliable the image recognition result is; Preset weight coefficient w P,i , the larger i is, the larger w is P，i The larger it is, the formula is satisfied: The weighted sum probability value P(Index=j) of detecting the same Index value in n sample points: Where, j = (0, 1, ..., m), indicating m types of roads; By comparing, find the maximum probability value P max and its corresponding Index value, indicating the most reliable road type recognition result; finally, t k The image recognition results used for the fusion algorithm at this moment [Index k a k b k ], image recognition confidence threshold P CF , establish the following fusion rules: If P max ≤P CF , indicating that the road surface features are not obvious, or such image sample sets have not been established in the road surface image database. In this case, the final adhesion coefficient output is based on the dynamic estimation, that is, μ = μ D ; Save a set of road images and prior values ​​μ at this moment P =μ D , update the image sample library for the post-training process of the classification network; If (P max ≥P CF )∩(a k ≤μ D ≤ b k ), indicating that the road surface image features are obvious, the road type recognition result has a high confidence level, and is consistent with the dynamic estimation result. At this time, the final adhesion coefficient output is μ D , and update the prior value μ of the image recognition result P =μ D ; If (P max ≥P CF )∩(μ D ≤a k )∪(μ D >b k ), indicating that the confidence of the road type recognition result is high, but it is inconsistent with the dynamic estimation result. At this time, the final adhesion coefficient output is based on image recognition, that is, μ = μ D .

9. A road adhesion coefficient estimation device integrating visual and dynamic information, comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the processor executes the computer program, it implements the road adhesion coefficient estimation method of integrating visual and dynamic information as described in any one of claims 1-8.

Images