CNN-LSTM-based multi-factor-fused adaptive foggy day driving speed guiding method

Abstract

The invention relates to a CNN-LSTM-based multi-factor-fused adaptive foggy day driving speed guiding method. The method comprises the following steps: S1, collecting traffic flow operation data in a highway network connection environment; s2, predicting a speed limit value of safe passage in the highway fog area, and predicting an optimal speed limit value of the vehicle in real time based on a CNN-LSTM deep learning model in combination with the highway fog environment data and the current vehicle state; s3, formulating early warning information content; s4, transmitting early warning information to the driver by using a vehicle-mounted HMI and a voice prompt function; s5, determining the position where the fog area information is published; and S6, early warning information pushing is carried out for the intelligent network connection vehicle end. The method can effectively cope with a complex foggy day road environment, has high scene adaptability, provides all-around safety support for a driver, and remarkably improves the driving safety and traffic efficiency in foggy days.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic safety, and more specifically to an adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors. Background Art

[0002] With the development of artificial intelligence, big data and new generation information technology, vehicle-road-cloud integration technology based on intelligent connected vehicles has made continuous breakthroughs, deeply empowering the transportation system to achieve intelligent safety management and intelligent transportation services, and promoting the coordinated evolution of transportation system management and control. However, adverse weather conditions are still a "long tail" problem facing the safe operation of intelligent connected vehicles. Among them, foggy days are one of the typical "long tail" scenarios. In this special environment, perception uncertainty and driving instability increase significantly, differences in vehicle information acquisition at different automation levels are highlighted, and the safety risks of transportation system operation increase significantly. How to effectively improve driving safety in foggy conditions has become a key issue that needs to be solved urgently.

[0003] At present, the warning method for driving in fog mainly adopts a non-real-time broadcast warning information push strategy, that is, obtaining fog information through the meteorological platform, and then issuing fog warnings and speed limit reminders through roadside facilities or vehicle-mounted equipment. For example, intelligent transportation facilities (such as variable information boards) issue fog driving reminders in the form of text and images at specific points; or rely on the network environment to push the adverse weather information released by the meteorological platform to the vehicle terminal (usually the navigation system) to assist the driver in making decisions. However, the update frequency of traditional roadside facilities such as variable information boards is low, resulting in delayed information updates and difficulty in timely reflecting current road conditions; while the warning system based on vehicle-mounted equipment relies on the accuracy of meteorological data and monitoring systems, which may lead to inaccurate information due to data deviations. More importantly, existing fog warnings mostly use a fixed speed limit recommendation strategy. This warning strategy cannot dynamically adjust to the actual vehicle movement status (such as speed control) and road traffic environment status (such as visibility, traffic flow, distance between front and rear vehicles, etc.) under foggy conditions. It is difficult to meet the driver's driving control expectations, resulting in the warning information being difficult to comply with, and thus it is impossible to truly guide vehicle movement and control traffic operations, ultimately affecting road capacity.

[0004] Existing foggy speed guidance methods are usually based on simple linear models, mainly relying on the linear relationship between visibility and vehicle speed. However, in actual foggy driving environments, the factors that affect vehicle speed are extremely complex and changeable. Visibility is only one of the variables. Environmental factors such as the distance to the front vehicle, traffic flow, and the driver's operating behavior will jointly affect the optimal driving speed. Therefore, a single linear model based on visibility cannot fully adapt to the complex foggy driving environment, and cannot effectively and comprehensively consider these nonlinear and dynamically changing factors, resulting in insufficient accuracy and adaptability of vehicle speed guidance, which in turn affects driving safety. In addition, existing foggy warnings generally lack an effective dynamic human-computer interactive feedback mechanism. Most of them are based on fixed warning signals, prompt sounds or visual warnings, and mainly rely on one-way information transmission. This method cannot provide timely adjustment suggestions based on the driver's actual operation and reaction in a dynamic and changing driving environment. In complex foggy driving scenarios, drivers need to obtain real-time safe driving guidance based on their behavior status and changes in the external environment. Existing foggy warnings fail to fully consider this demand and lack an adaptive and real-time feedback mechanism, resulting in drivers being unable to obtain sufficient safety guidance or adjust their driving behavior in a timely manner, increasing the risk of accidents. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors, which can greatly improve the accuracy of vehicle speed prediction and the timeliness of warning, ensure that warning information can be quickly and accurately conveyed to the driver, improve user experience, and effectively reduce the incidence of traffic accidents.

[0006] The technical solution adopted by the present invention to solve the technical problem is to construct an adaptive fog driving speed guidance method based on CNN-LSTM integrating multiple factors, including the following steps:

[0007] S1. Collect traffic flow operation data in the highway network environment;

[0008] S2: Predict the speed limit for safe passage in foggy areas on highways. Based on the CNN-LSTM deep learning model, combined with highway foggy environment data and current vehicle status, the optimal speed limit for vehicles is predicted in real time.

[0009] S3. Formulate the content of early warning information;

[0010] S4. Use the vehicle HMI and voice prompt function to convey warning information to the driver;

[0011] S5. Determine the location for fog area information release, and determine the release node according to the warning information release standard to ensure that drivers receive the warning information and respond at the best time;

[0012] S6. Push warning information to intelligent connected vehicles. The on-board equipment will push speed suggestions, distance reminders to the foggy area ahead, and current speed reminders in real time, and dynamically adjust the reminder frequency and intensity based on the driver's feedback on the guidance speed.

[0013] According to the above scheme, in step S1, the traffic flow operation data includes image data, time series data and traffic density data;

[0014] The image data is obtained by taking images of the foggy area through a vehicle-mounted camera of a preceding vehicle approaching the foggy area, and transmitting the images to the backend through a network. The image data includes basic road condition information such as road scenes, visibility, and lane lines;

[0015] The time series data is collected through a GPS module, including timestamp, vehicle speed, acceleration, and distance to the preceding vehicle;

[0016] The traffic density data is acquired through traffic management.

[0017] According to the above scheme, in step S2, the image data and time series data in the CNN-LSTM model are fused at the same time scale, and the timestamps of the time series data are expanded using a linear interpolation method for data with different sampling frequencies;

[0018] For each image data point, the timestamps of the lidar data and GPS / OBD-II data are interpolated to the timestamp of the image data using the linear interpolation formula and the interpolation step is repeated to ensure that the lidar and GPS / OBD-II data have corresponding interpolated values ​​at each image timestamp.

[0019] According to the above scheme, in step S2, after processing the data using the interpolation method, the data is cleaned and preprocessed using the Kalman filter algorithm before inputting the data into the CNN-LSTM model.

[0020] According to the above scheme, in step S2, the image data is normalized.

[0021] According to the above scheme, in step S3, the warning information content includes speed limit suggestions, distance reminders for the foggy area ahead, and distance warnings for the vehicles ahead.

[0022] According to the above scheme, step S4 specifically includes the following steps:

[0023] S401, providing the driver with a real-time optimal driving route through the route planning function of the navigation software;

[0024] S402. Through secondary development with JavaScript, the prediction results based on the CNN-LSTM model are embedded into the Amap. Multimodal information release is adopted, and in combination with the vehicle HMI and voice prompt functions, information such as speed limit suggestions, fog zone distance prompts, and vehicle ahead distance warnings are released to the driver, thus realizing personalized early warning information push.

[0025] According to the above scheme, in step S5, speed guidance information is issued in advance within a range of 1-2 kilometers before the vehicle enters the fog area, providing the driver with sufficient time to slow down and adjust the driving strategy when the visibility is not yet limited, and adjusting the reminder frequency according to the current driver's operation.

[0026] According to the above scheme, in step S5, if the driver does not decelerate according to the reminder, the reminder frequency is increased; if the driver decelerates according to the reminder, the reminder frequency is reduced;

[0027] At the edge of the fog zone, clear information is issued to remind drivers of the special driving conditions in the fog zone, and re-emphasize the need to slow down and maintain a safe distance, with clear reminders to drivers that they are about to enter a low-visibility area;

[0028] In foggy areas, multiple information release points are set up at intervals of 500 meters to 1 kilometer to ensure that drivers continue to receive effective driving guidance, continuously push speed and distance warning information to help drivers maintain appropriate driving speed and safe distance;

[0029] Within 500 meters to 1 kilometer before the exit of the fog area, release points are set up to remind drivers that they are about to exit the fog area and gradually resume normal driving.

[0030] According to the above scheme, in step S6, the warning information is conveyed in a multi-modal manner through the vehicle-mounted HMI and voice prompts.

[0031] The implementation of the adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors of the present invention has the following beneficial effects:

[0032] 1. The present invention effectively solves the problem that the current fog warning fails to fully consider the influence of multiple factors. In addition to traditional time series data such as speed and acceleration, the present invention introduces road traffic environment data (such as the distance to the vehicle in front and traffic flow) and image data, so that the model can more comprehensively reflect the actual road conditions, thereby improving the accuracy of speed limit recommendations. By combining convolutional neural networks (CNN) with long short-term memory networks, the advantages of both in spatial feature extraction and time series modeling are fully utilized; CNN extracts visibility features from image data, while LSTM captures the changes in visibility, speed, acceleration, distance to the vehicle in front and traffic flow in the time dimension, accurately predicts the complex nonlinear interaction between multiple factors, and thus significantly improves the optimal speed prediction and warning capabilities in foggy driving;

[0033] 2. The speed guidance method of the present invention realizes adaptive adjustment. The proposed speed guidance strategy can dynamically adjust the warning content and frequency according to the real-time driver feedback on the warning information (such as failure to slow down in time or failure to slow down to the guidance speed) and changes in environmental conditions (such as changes in visibility and vehicle distance); this adaptive mechanism ensures that the driver can receive timely and accurate safety prompts in various complex situations, avoiding the inefficiency and redundant information caused by fixed-mode warnings;

[0034] 3. The present invention fills the gap that most existing navigation systems lack a special warning function under foggy conditions. Through secondary development with Amap, combined with route planning, foggy speed guidance and navigation services, it provides an innovative beyond-visual-range risk warning solution; it can not only provide speed limits in foggy areas, distances to vehicles in front and prompts for exiting foggy areas, but also optimize route planning in real time; in the complex environment of foggy days, drivers can receive safety warnings in advance, thereby improving driving safety and enhancing the ability to adapt to complex road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0036] Figure 1 It is the technical roadmap of the present invention;

[0037] Figure 2 It is a flowchart of the speed guidance push for highway foggy weather of the present invention;

[0038] Figure 3 It is a schematic diagram of the structure of the CNN-LSTM model of the present invention;

[0039] Figure 4 This is a schematic diagram of the early warning interface using Amap in the present invention. DETAILED DESCRIPTION

[0040] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0041] like Figure 1-4 As shown, the adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors of the present invention comprises the following steps:

[0042] S1. Collect traffic flow operation data in the networked environment of highways. Image data is collected through on-board cameras, including basic road conditions such as road scenes, visibility, and lane lines. In order to ensure that the vehicle accurately captures key environmental information such as current visibility, front vehicles, and road conditions during driving, image data is obtained by the front vehicle approaching the fog area using the on-board camera to capture fog area images, and the images are transmitted to the background through the network. The sampling frequency is set to 10Hz-30Hz. The roadside sensing device is used to obtain the distance between the current vehicle and the front vehicle, and its sampling frequency is set to 1Hz. Timestamp, vehicle speed and acceleration information are collected through the GPS module, and the sampling frequency is set to 5-10Hz. Traffic density data is obtained through the traffic management platform. The update frequency of the warning information is 1Hz-5Hz to ensure timely response to the driver's operation and environmental changes.

[0043] The traffic flow operation data collection solution for foggy highway scenarios is designed to provide speed dynamic warning information push services for vehicles traveling on foggy highways, so as to optimize their driving speed and improve road traffic safety.

[0044] S2. Predict the speed limit for safe passage in foggy areas on highways. Based on the CNN-LSTM deep learning model, combined with highway foggy environmental data and current vehicle status, the optimal speed limit for vehicles is predicted in real time to ensure accurate speed guidance under different road conditions. Based on the CNN-LSTM deep learning model, additional variables in foggy environments are introduced into the model input variables, including the distance to the vehicle in front and traffic density. Compared with the traditional CNN-LSTM model that does not consider these environmental variables, the optimization model can more comprehensively describe the complex feature relationship under foggy conditions, thereby providing more accurate speed guidance under different road conditions, effectively improving the safety and adaptability of driving in foggy conditions.

[0045] Since the image data and time series data in the CNN-LSTM model need to be fused at the same time scale, for data with different sampling frequencies, a simple and computationally efficient method called linear interpolation will be used to expand the timestamps of the time series data to align with the sampling frequency of the image data, thereby ensuring the temporal consistency of the data. The linear interpolation formula is:

[0046]

[0047] Where y1 and y2 are the values ​​corresponding to the known data points, i.e., the distance data of the preceding vehicle collected by the lidar or the speed and acceleration information collected by the GPS device, t1 and t2 are the corresponding timestamps, t is the target time point to be interpolated, corresponding to the timestamp of the image data, and y(t) is the value of the target time point obtained after interpolation.

[0048] For each image data point (i.e., the timestamp of each image), the timestamps of the LiDAR data and GPS / OBD-II data are interpolated to the timestamp of the image data using the linear interpolation formula above. Repeat the interpolation step for each image data point (timestamp) to ensure that the LiDAR and GPS / OBD-II data have corresponding interpolated values ​​at each image timestamp. Ultimately, all data of different frequencies are aligned according to the timestamp of the image data to ensure that the data input into the CNN-LSTM model has a consistent time scale.

[0049] After processing the data using interpolation, data cleaning and preprocessing are important steps to ensure data quality before inputting the data into the CNN-LSTM model. The main purpose of data cleaning and preprocessing is to remove noise, fill missing values, standardize data, and ensure data consistency, reliability, and adaptability to model requirements. The present invention uses the Kalman filter algorithm to reduce measurement noise and provide optimal state estimation. The Kalman filter state equation describes how the system transfers from one state at one moment to the next. The state equation is:

[0050] x k =F k x k-1 +B k u k +w k

[0051] In the formula, x k is the system state at time k (vehicle position, speed, etc.); F k B is the state transfer matrix, which represents the process from the battle at time k-1 to time k; k is the control input matrix, which is used to describe the impact of external control (such as acceleration) on the system; u k is the control input (such as acceleration, etc.); w k is the process noise, which is set to Gaussian white noise.

[0052] The observation equation describes how to obtain the state of the system from the observation data. The observation equation is:

[0053] z k =H k x k +v k

[0054] In the formula, z k is the observed value at time k (such as sensor measurement data); H k is the observation matrix, which represents the relationship between the actual state of the system and the observed value; v k is the observation noise, set to Gaussian white noise. Before starting filtering, it is necessary to initialize the parameters of the Kalman filter, including the initial state and covariance matrix of the system. The initial state is estimated as The initial covariance matrix is ​​P0, which represents the confidence of the initial state estimate, usually the unit matrix. The goal of prediction is to predict the state estimate formula based on the system state and covariance of the previous step according to the dynamic model of the system:

[0055]

[0056] In the formula, is the predicted state estimate. The prediction covariance matrix formula is:

[0057]

[0058] In the formula, is the predicted covariance matrix, which represents the uncertainty of the current state; Q k is the process noise covariance matrix, which represents the uncertainty of the model prediction. Then it is updated to combine the actual observation data with the prediction results to obtain a more accurate state estimate. The Kalman gain determines the weighted average ratio of the prediction results to the actual observation results. The calculation formula is:

[0059]

[0060] In the formula, K k is the Kalman gain, is the predicted covariance matrix, H k is the observation matrix, P k is the observation noise covariance matrix, which represents the uncertainty of the observation data. Based on the Kalman gain, the state estimation and covariance matrix at the current moment are updated, and the formula is:

[0061]

[0062]

[0063] In the formula, is the updated state estimate; z k is the actual observed value; P k is the updated covariance matrix, which represents the uncertainty of the current state estimate. Whenever there is new observation data z kAs input is fed into the system, prediction and update steps are performed to continuously refine the state estimate. After each iteration, the state estimate and covariance matrix are updated to provide a more accurate estimate of the system.

[0064] After obtaining the smoothed and denoised data, it is necessary to standardize the data so that each feature has the same scale. The standardization formula is:

[0065]

[0066] In the formula, X ij is the jth eigenvalue of the i-th sample, μ j is the mean of the jth feature, σ j is the standard deviation of the jth feature.

[0067] The collected image data and time series data sets are divided into a training set (70%), a validation set (15%), and a test set (15%). The image data needs to be extracted through a convolutional neural network (CNN) to identify image features of areas with limited visibility, and the visibility is marked as six categories from low to high, 0-5, according to the extracted features. Since the sizes of images are different, all images need to be adjusted to a unified format. The present invention sets the image pixel value size to 224x224, and the image format to RGB format (each image has 3 channels: red, green, and blue), that is, the input image size is: 224×224×3. Normalizing the image and scaling the pixel values ​​to the range of [0-1] can improve the model training efficiency and improve the stability of the model. The formula is:

[0068]

[0069] After the image data is preprocessed, it is convolved through CNN to extract local features. The convolution layer uses Sobel edge detection to extract spatial features in the image. The convolution formula is:

[0070]

[0071] In the formula, I is the input image; K is the convolution kernel; (i, j) is the position coordinate of the output image. The pooling layer reduces the image size and the amount of calculation. The maximum pooling formula is:

[0072]

[0073] Where P(i,j) is the image feature after pooling; I(i+m,j+n) is the maximum value of the pooling area. After multiple layers of convolution and pooling operations, CNN generates a set of feature maps. The features extracted by the convolution layer and the pooling layer are combined through the fully connected layer as the output of CNN. The output layer uses the Softmax activation function to output the final visibility level prediction result. The Softmax activation function converts the score of each category into a probability. The formula is:

[0074]

[0075] In the formula, z i is the score for each category, k is the number of categories, P(y=c i |x) is the image belonging to category c i The time series data features input into the LSTM model include vehicle speed v(t), acceleration a(t), vehicle distance D(t), traffic flow density ρ(t) and visibility grade S(t). The input data of each time step t constitutes a feature vector. For each time step t, the input of LSTM is:

[0076] X(t)=[v(t),a(t),D(t),ρ(t),S(t)]

[0077] In order to enable the model to learn dependencies over longer time windows, a sliding window feature can be constructed. The data sequence for each time step can be obtained through the sliding window. The step size of the window is selected as 1 second, the size of the sliding window is 5 seconds, and the features within each window will be used as the input of the LSTM. For each time step t, the sliding window includes the historical data of the past 5 seconds, that is:

[0078]

[0079] In the formula, T is the total duration of the data set, W is the sliding window size, and each X input (t) is a sequence containing the data of the past 5 time steps (5 seconds), with a shape of (5,5), representing the data features of each time step (5 features, namely speed, acceleration, vehicle distance, traffic volume, and visibility level). The shape of LSTM input data is (N,5,5), where N is the number of samples generated after sliding the window, i.e. T-W+1, the first 5 is the window size in the time window, and the second 5 is the number of data features for each time step. The core of the LSTM network is to capture time dependencies through the gating mechanism (input gate, forget gate, output gate), and the formula is as follows:

[0080] Input gate: i t =σ(W i ·[h t-1 ,X t]+b i )

[0081] Forget gate: f t =σ(W f ·[h t-1 ,X t ]+b f )

[0082] Candidate memory cells:

[0083] Memory unit status:

[0084] Output gate: o t =σ(W o ·[h t-1 ,X t ]+b o )

[0085] Hidden state: h t =o t tanh(C t )

[0086] Where W f , W i , W C and W o is the weight matrix, b i , b f , b C and b o is the corresponding bias vector, tanh is the hyperbolic tangent function, h t-1 is the output at the previous moment, f t is the retention value, C t-1 is the memory state of the previous moment, i t is the added degree value of the current state, is the intermediate state, C t is the current state, o t is the output degree value, h t is the output at the current moment, X t The input at the current moment.

[0087] Considering various factors, the model output guides the speed, and the formula is:

[0088]

[0089] In the formula, v s(t) is the guidance speed, v(t) is the current speed, α is the visibility adjustment factor, which is 0.1-0.6. As the visibility level decreases, the speed adjustment factor increases. When the visibility is level 0, the value is 0.1, and the adjustment factor increases by 0.1 for each level increase. β is the vehicle distance adjustment factor, which is 0.01-0.1 and increases as the distance to the vehicle ahead decreases. γ is the acceleration adjustment factor, which is 0.2. δ is the traffic density adjustment factor, which is 0.1-0.3 and increases as the traffic density increases. D min is the minimum safe distance, which is the distance required to travel at the current speed for 3 seconds, m; ρ max is the maximum traffic density, and the maximum traffic density of the month in the current scenario is taken as vehicles / km. The calculated guide speed is the target value of the LSTM model. That is, each input sequence (consisting of 5 seconds of historical data) will correspond to a target value h(t), which is the recommended speed that the LSTM model should predict at this time step. The recommended speed calculated in the above steps is paired with the input data (that is, the data of the past 5 seconds) as a label to form a training data set. Each input sequence:

[0090] X input (t)=[X(t-4),X(t-3),X(t-2),X(t-1),X(t)]

[0091] The corresponding target values ​​are:

[0092] h(t)=v l (t), which is the guide speed output by LSTM. The output of the LSTM layer is passed to the fully connected layer, and the output of the fully connected layer is the optimal driving speed recommendation v at the current moment p (t). The fully connected layer linearly transforms the output of LSTM through the weight matrix and translates it through the bias term to obtain the final guidance speed

[0093] v p (t) = W out ·h t +b out

[0094] Where W out is the weight matrix of the fully connected layer, with a dimension of 1×1, and needs to be randomly initialized. The model can be adjusted through the training process, and the method used is Xavier initialization; b out is the bias term of the fully connected layer, and the bias term is initialized to a small positive number, such as 0.1 here. The loss function is used to measure the error between the predicted speed and the actual speed. The mean square error (MSE) formula is:

[0095]

[0096] In the formula, vp,i is the guidance speed of the model output, v true,i is the actual optimal boot speed. MSE measures the average square error between the model prediction value and the true value. The smaller the value, the better the model effect. Next, optimize the model, that is, adjust the parameters (weights and biases) of the model by minimizing the loss function. The Adam (Adaptive MomentEstimation) optimizer is selected for optimization. This is an optimization method based on momentum and adaptive learning rate. It has the ability to adaptively adjust the learning rate and can automatically adjust the learning rate of each parameter during training. The formula is:

[0097]

[0098]

[0099] Where β1 and β2 are the decay factors of momentum and adaptive learning rate, η is the learning rate, and ò is a small constant to prevent division by zero. The learning rate is an important hyperparameter that controls the step size of the model parameters at each update. Adam is used to dynamically adjust the learning rate. After calculating the error of the model through the loss function, the gradient is calculated according to the result of the loss function and the weights and biases are updated through the back propagation algorithm. The weights and bias terms are updated using the Adam optimizer, and the above steps are repeated until the model converges. During and after the training process, the validation set is used to evaluate the performance of the model, and the MSE value is calculated on the validation set to ensure that the model is not overfitting or underfitting. After the training is completed, the test set is used to evaluate the final effect of the model and test its generalization ability.

[0100] S3. Formulate the content of warning information, including speed limit suggestions, distance reminders for foggy areas ahead, and warnings for the distance between vehicles ahead. The information design follows the principles of clarity, specificity, and simplicity to ensure that drivers can quickly understand and take appropriate actions, reducing cognitive burden. Speed ​​limit suggestions are dynamically adjusted based on current road conditions, visibility in foggy environments, and traffic flow conditions to ensure that drivers receive the most appropriate driving speed reminders. Distance reminders for foggy areas ahead and warnings for the distance between vehicles ahead are updated in real time so that drivers can take countermeasures in advance. Based on the driver's feedback on warning information, adjust the content of warning information release in real time.

[0101] Taking the example of a situation where the distance to the fog area is 1km, the distance to the vehicle in front is 20m, and the guide speed is 65km / h, the warning information content is constructed as: "The fog area is 1km ahead, the distance to the vehicle in front is 20m, and the recommended speed is 65." It should not only be concise and clear, but also be delivered to the driver in a timely and accurate manner, so as to help him respond quickly in the complex environment of foggy days and ensure driving safety.

[0102] S4. Determine the method of issuing warning information. Figure 2 The second development uses the vehicle HMI and voice prompt function to convey warning information to the driver. First, the route planning function of Amap provides the driver with the best real-time driving route, which not only takes into account the current traffic flow, road conditions, construction information, etc., but also dynamically adjusts the route to help the driver avoid congested sections or accident-prone areas. In foggy days and other conditions with low visibility, the system automatically selects roads with high visibility and low traffic flow for the driver to avoid the driver from entering areas with high traffic risks. The real-time positioning function provided by Amap can continuously track dynamic information such as the real-time position and direction of the vehicle, provide real-time feedback on the vehicle status, and judge the distance of its driving path from the foggy area based on the current position of the vehicle. When the vehicle approaches an area with low visibility, relevant warning information will be pushed in advance. On this basis, secondary development is carried out through JavaScript to embed the prediction results based on the CNN-LSTM model into Amap, thereby realizing personalized warning information push. To ensure that drivers can receive warning information comprehensively and promptly, a multimodal information release method is adopted, combining the on-board HMI and voice prompt functions to issue speed limit suggestions, fog area distance prompts, front vehicle distance warnings and other information to drivers, ensuring that drivers can quickly understand and respond.

[0103] Specifically: Through Gaode Figure 2 Secondary development combines the mobile phone display and voice prompt function to convey warning information to the driver. Using the route planning and real-time positioning API of AutoNavi Map, personalized warning information push is realized through secondary development. Combined with the route planning function of AutoNavi Map, it can provide the driver with the best driving route, and combined with real-time positioning information, continuously track the current position and driving status of the vehicle. On this basis, the prediction results of the CNN-LSTM model are embedded in AutoNavi Map, and warning information is conveyed to the driver through the mobile phone display and voice function, including speed limit suggestions, distance prompts in the foggy area ahead, and warning information about the distance between the front vehicle and the front vehicle. The release of warning information adopts a multimodal method combining display and voice and is displayed on the in-vehicle HMI of the intelligent connected car, ensuring that the driver can receive effective safety reminders clearly and promptly during driving, thereby improving the safety and user experience of driving in foggy weather.

[0104] S5. Determine the location for fog zone information release, comprehensively consider the current distance from the fog zone, vehicle speed and traffic flow ahead, accurately determine the release node according to the warning information release standard, ensure that the driver can receive the warning information and respond at the best time, and the driver can receive relevant information outside the visual range of the fog zone, take safety measures in time, and achieve over-visual-range warning, thereby significantly improving driving safety. Release speed guidance information in advance within the range of 1-2 kilometers before the vehicle enters the fog zone, so that the driver can provide enough time to slow down and adjust the driving strategy before the visual range is limited. And adjust the reminder frequency according to the current driver's operation. If the driver does not slow down according to the reminder, increase the reminder frequency; if the driver slows down according to the reminder, reduce the reminder frequency. At the edge entrance of the fog zone, issue a clear reminder to inform the driver that he is about to enter a low-visibility area. The information released here clearly reminds the driver of the special driving conditions in the fog zone, and once again emphasizes the importance of slowing down and maintaining a safe distance. In the foggy area, to ensure that drivers continue to receive effective driving guidance, multiple information release points are set up at intervals of 500 meters to 1 kilometer to continuously push vehicle speed and distance warning information to help drivers maintain an appropriate driving speed and safe distance. Release points are set up within 500 meters to 1 kilometer before the exit of the foggy area to remind drivers that they are about to leave the foggy area and gradually resume normal driving.

[0105] Specifically: The release locations of highway fog warning information are set in several key areas. A warning is issued 1-2 kilometers before the vehicle enters the fog area to help drivers slow down and prepare in advance. A reminder is issued at the entrance to the edge of the fog area to inform that you are about to enter a low-visibility area. Multiple release points are set up every 500 meters to 1 kilometer inside the fog area to continuously push vehicle speed and distance warning information. Release points are set up within 500 meters to 1 kilometer before the exit of the fog area to remind drivers that they are about to leave the fog area and gradually resume normal driving.

[0106] S6. Push warning information to the intelligent connected vehicle. The on-board equipment pushes speed suggestions, distance reminders to the foggy area ahead, and current speed reminders in real time, and dynamically adjusts the reminder frequency and intensity based on the driver's feedback on the guidance speed. For example, when the driver does not slow down in time, the reminder frequency is increased to ensure that the driver receives effective reminders in time during driving. The warning information is conveyed in a multi-modal way through the on-board HMI and voice prompts to provide speed guidance for the driver and improve the safety of driving in foggy weather.

[0107] Specifically, it pushes warning information to the vehicle side, and pushes speed limit suggestions, distance reminders for the foggy area ahead, and distance warnings for the vehicle ahead in real time through mobile devices to ensure that drivers receive effective reminders in a timely manner during driving. Warning information provides comprehensive guidance to drivers through dual feedback methods of mobile phone display and voice prompts, improving the safety of driving in foggy weather. The vehicle driving status and environmental data are obtained in real time through the on-board equipment, and the speed limit suggestions are dynamically adjusted and optimized in combination with the CNN-LSTM deep learning model. Figure 2 The system is developed for the first time to transmit warning information to the driver in a multi-modal form of voice and vision through the on-board HMI, and adjust the prompt frequency and intensity in real time according to the driver's feedback on speed guidance, ensuring the timeliness and effectiveness of information transmission, thereby improving driving safety in foggy conditions.

[0108] In a preferred embodiment of the present invention, the visibility information is collected in real time by the on-board camera of the vehicle close to the fog area and uploaded to the background. The traffic density provided by the traffic control platform, the distance to the preceding vehicle detected by the roadside sensing facilities, and the speed and acceleration data of the vehicle are combined to output the optimal guidance speed using the CNN-LSTM model. Figure 2 The system was developed for the first time, embedding a trained CNN-LSTM model, integrating path planning and real-time positioning functions, and conveying fog area warning information to drivers in advance in a multi-modal way of vision and voice through on-board equipment, ensuring that relevant information can be obtained outside the visible range and safety measures can be taken in time, realizing beyond-visual-range warning, and dynamically adjusting the prompt frequency and intensity according to the driver's feedback on the guidance speed. This not only enhances the driver's ability to cope with complex road conditions in foggy days, but also significantly improves the safety and convenience of driving.

[0109] In a preferred embodiment of the present invention, a deep learning model is constructed by combining a convolutional neural network (CNN) and a long short-term memory network (LSTM). Among them, CNN is mainly used to process image data, and can extract visibility-related features from road images in foggy environments, effectively identify different levels of fog, and provide important image information basis for subsequent vehicle speed prediction and early warning. LSTM is used to process time series data, and can analyze the time series data of vehicle speed data, vehicle distance change data with the preceding vehicle, and traffic flow, capture the long-term dependency and dynamic change trend therein, and better consider the influence of time factors on driving speed. The combination of CNN and LSTM gives full play to the advantages of the two networks and realizes efficient processing of different types of data. By fusing image and time series data, the interactive relationship of multiple factors can be accurately captured, which can greatly improve the accuracy of vehicle speed prediction and the timeliness of early warning, and effectively reduce the incidence of traffic accidents. Comprehensively consider the multi-dimensional factors of speed, acceleration, visibility, vehicle distance with the preceding vehicle, and traffic flow, dynamically adjust the feature weights, make full use of historical data and real-time feedback, and realize accurate vehicle speed prediction and personalized early warning. The system dynamically adjusts the warning content and frequency based on the driver's real-time operation (such as speeding or ignoring warnings), changes in visibility, and changes in the distance to the vehicle in front. It can adapt to different degrees of foggy environments and various complex road conditions, and provide effective speed guidance for drivers. Figure 2 The development of the transmission of warning information, with the help of Amap's extensive user base and powerful navigation function, ensures that warning information can be quickly and accurately conveyed to drivers, improving user experience. The specific content is as follows:

[0110] 1) Calculation of highway speed limit in foggy weather based on CNN-LSTM model

[0111] The CNN-LSTM model is a deep learning model that combines convolutional neural networks and long short-term memory networks. It can process sequence data that contains both spatial features and temporal dynamic characteristics. It can fully combine the image understanding ability of convolutional neural networks and the temporal modeling ability of long short-term memory networks, making it perform well in processing data with spatial and temporal characteristics, and has significant advantages in complex environmental perception applications. Traditional models are usually difficult to handle the interactive relationship between multiple factors, and need to be modeled through linear combinations or logical relationships. This method is insufficient when facing complex nonlinear relationships and is prone to underfitting problems. The advantage of the CNN-LSTM model is that it can learn and represent complex nonlinear interactive relationships between multiple factors, learn complex interactions from large amounts of data, and significantly improve decision quality and accuracy.

[0112] In order to solve the problem of calculating the speed limit value in the foggy scene of the highway, considering the advantages of the CNN-LSTM model in processing images and time series data, the present invention adopts the CNN-LSTM model to calculate the vehicle speed limit value. At the same time, the road traffic environment data, including the distance from the front vehicle and the traffic flow, are innovatively introduced into the model input variables to more accurately reflect the complex road conditions. The front vehicle close to the foggy area takes the foggy area image through the on-board camera and transmits the image to the background through the network. Based on the collected highway foggy image data, the CNN-LSTM model first takes the image data as input, extracts the features in the image through the CNN layer, and grades the visibility of the current road based on the visibility classification model. Subsequently, the visibility classification result output by the CNN layer is used as input, combined with the collected vehicle time series data (including timestamp, vehicle speed, acceleration, distance from the front vehicle and road traffic density) far from the foggy area, and the data is transmitted to the background through the network and input into the LSTM layer. The LSTM layer models these time series data and visibility classification in the time dimension to capture the dynamic changes and mutual influence of these factors in the time series. Finally, the model outputs the optimal driving speed recommendation under the current environment.

[0113] 2) Based on Gaode Figure 2 The method of pushing early warning information developed by the

[0114] In order to solve the adaptability problem of vehicle-mounted equipment in speed guidance, based on Amap Figure 2 This is the first time that we have developed a beyond-visual-range risk warning method. By combining the path planning and real-time positioning functions of Amap, it can provide drivers with the best path planning. When the vehicle approaches a foggy area, it will issue an early warning, providing distance prompts and speed limit information for the foggy area; when the vehicle enters the foggy area, it will continue to provide distance prompts and vehicle speed limit suggestions for exiting the foggy area, and dynamically adjust the prompt frequency and intensity based on the driver's feedback on the guidance speed. This beyond-visual-range warning provides drivers with more comprehensive path planning, speed guidance and safety warnings in foggy conditions, effectively improving the driver's ability to cope with complex environments, thereby significantly enhancing driving safety and convenience.

[0115] At present, intelligent networked vehicles are generally equipped with on-board display screens, which are used as information release terminals. By combining display information and voice prompts, timely warning services can be provided to drivers. Not only does it realize the path planning and navigation functions, but it also expands the application scenarios, providing a new solution for the push of warning information on foggy highways. At present, Amap has not yet provided a warning function specifically for foggy conditions. The present invention provides drivers with a safer travel experience by innovatively combining path planning, foggy risk warnings and navigation services.

[0116] In the preferred embodiment of the present invention, the CNN-LSTM model is combined with Amap Figure 2 The second development proposed a solution for foggy highway driving assistance for intelligent connected vehicles. The CNN-LSTM model extracts features from images and time series data, dynamically calculates speed limit recommendations, and improves the safety of driving in foggy weather. Through the secondary development of Amap, the vehicle-mounted equipment is used as an early warning carrier, combined with path planning and real-time positioning, to provide distance prompts, speed limit recommendations, and vehicle distance information, making up for the shortcomings of current foggy early warnings and providing drivers with more intelligent and personalized driving guidance and safety warnings.

[0117] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. An adaptive foggy driving speed guidance method based on CNN-LSTM integrating multiple factors, characterized in that: The following steps are involved: S1. Collect traffic flow operation data in the highway network environment; S2: Predict the speed limit for safe passage in foggy areas on highways. Based on the CNN-LSTM deep learning model, combined with highway foggy environment data and current vehicle status, the optimal speed limit for vehicles is predicted in real time. S3. Formulate the content of early warning information; S4. Use the vehicle HMI and voice prompt function to convey warning information to the driver; S5. Determine the location for fog area information release, and determine the release node according to the warning information release standard to ensure that drivers receive the warning information and respond at the best time; S6. Push warning information to intelligent connected vehicles. The on-board equipment will push speed suggestions, distance reminders to the foggy area ahead, and current speed reminders in real time, and dynamically adjust the reminder frequency and intensity based on the driver's feedback on the guidance speed.

2. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 1 is characterized in that: In step S1, the traffic flow operation data includes image data, time series data and traffic density data; The image data is obtained by taking images of the foggy area through a vehicle-mounted camera of a preceding vehicle approaching the foggy area, and transmitting the images to the backend through a network. The image data includes basic road condition information such as road scenes, visibility, and lane lines; The time series data is collected through a GPS module, including timestamp, vehicle speed, acceleration, and distance to the preceding vehicle; The traffic density data is acquired through traffic management.

3. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 2 is characterized in that: In step S2, the image data and the time series data in the CNN-LSTM model are fused at the same time scale, and the timestamps of the time series data are extended using a linear interpolation method for data with different sampling frequencies; For each image data point, the timestamps of the lidar data and GPS / OBD-II data are interpolated to the timestamp of the image data using the linear interpolation formula and the interpolation step is repeated to ensure that the lidar and GPS / OBD-II data have corresponding interpolated values ​​at each image timestamp.

4. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 3 is characterized in that: In step S2, after the data is processed using the interpolation method, the data is cleaned and preprocessed using the Kalman filter algorithm before the data is input into the CNN-LSTM model.

5. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 4 is characterized in that: In the step S2, the image data is normalized.

6. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 1 is characterized in that: In step S3, the warning information includes speed limit suggestions, distance reminders to the foggy area ahead, and distance warnings to the vehicles ahead.

7. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 1 is characterized in that: The step S4 specifically comprises the following steps: S401, providing the driver with a real-time optimal driving route through the route planning function of the navigation software; S402. Through secondary development with JavaScript, the prediction results based on the CNN-LSTM model are embedded into the Amap. Multimodal information release is adopted, and in combination with the vehicle HMI and voice prompt functions, information such as speed limit suggestions, fog zone distance prompts, and vehicle ahead distance warnings are released to the driver, thus realizing personalized early warning information push.

8. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 1 is characterized in that: In step S5, speed guidance information is issued in advance within a range of 1-2 kilometers before the vehicle enters the fog area, providing the driver with sufficient time to slow down and adjust the driving strategy when the visibility is not limited, and adjusting the reminder frequency according to the current driver's operation.

9. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 8 is characterized in that: In step S5, if the driver does not decelerate according to the reminder, the reminder frequency is increased; if the driver decelerates according to the reminder, the reminder frequency is reduced; At the edge of the fog zone, clear information is issued to remind drivers of the special driving conditions in the fog zone, and re-emphasize the need to slow down and maintain a safe distance, with clear reminders to drivers that they are about to enter a low-visibility area; In foggy areas, multiple information release points are set up at intervals of 500 meters to 1 kilometer to ensure that drivers continue to receive effective driving guidance, continuously push speed and distance warning information to help drivers maintain appropriate driving speed and safe distance; Within 500 meters to 1 kilometer before the exit of the fog area, release points are set up to remind drivers that they are about to exit the fog area and gradually resume normal driving.

10. The adaptive foggy driving speed guidance method based on CNN-LSTM and integrating multiple factors according to claim 1, characterized in that: In step S6, the warning information is conveyed in a multi-modal manner through the vehicle-mounted HMI and voice prompts.

Images