A big data driven highway traffic incident duration prediction method
By combining big data technology and text mining methods, a highway traffic event duration prediction model based on random forests is established, which solves the shortcomings of the existing models in adaptability and text information fusion, and achieves more accurate and efficient traffic event duration prediction.
Patent Information
- Application Number
- CN202210152038.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-18
- Publication Date
- 2025-04-11
- Estimated Expiration
- 2042-02-18
AI Technical Summary
The existing highway traffic event duration prediction model is insufficient in the face of temporary maintenance construction, special vehicle passage, cargo drop, road damage and other traffic events, and fails to effectively integrate traffic event text information to improve prediction accuracy.
Big data technology and text mining methods are adopted to combine structured data with text data to establish a highway traffic event duration prediction model based on random forests, and improve the accuracy and adaptability of prediction through the fusion of text feature vectors.
It realizes more accurate and efficient prediction of the duration of highway traffic events, can effectively adapt to different types of traffic events, reduce driving time losses, and provide reliable road services to the public.
Smart Images
Figure CN114626684B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent transportation, and in particular to a method for predicting the duration of highway traffic events driven by big data. Background Art
[0002] Highway traffic incidents refer to events that have an impact on traffic safety, road operation status, traffic capacity, travel time, etc., and are of concern to traffic operation management departments. With the continuous improvement of intelligent traffic monitoring systems, the safety of people and property is necessarily guaranteed, but congestion and delays caused by traffic incidents are becoming more and more normalized. Traffic incidents are still the main cause of highway traffic congestion, and the time delays caused by them have become the focus of attention within the industry and the public. Reasonable prediction of the duration of highway traffic incidents can achieve induced diversion control of the road network, avoid huge driving time losses, and provide decision-making basis and reliable road services for public travel. For these incidents, if they cannot be handled in a timely manner, they are likely to cause secondary traffic accidents, which means that the severity of the traffic incidents will increase and the duration of the traffic incidents will be further prolonged. Therefore, the task of predicting the duration of highway traffic incidents has become particularly critical.
[0003] At present, the research on the duration of traffic events focuses on the duration prediction of events such as traffic accidents, vehicle breakdowns, and natural disasters. However, temporary maintenance and construction, the passage of special vehicles, cargo drops, road damage, and other traffic events also occur from time to time, which can also cause long-term traffic congestion. In the face of such traffic events, the adaptability of existing duration prediction models is slightly insufficient. At the same time, the flow of traffic event information between different operating management units and different business handling roles contains text data descriptions in natural language. Compared with structured attribute features, the information contained in text data has a certain richness and diversity. Therefore, more information that is helpful for the study of traffic event duration can be obtained from it. The integration of traffic event text information has positive significance for the establishment of prediction models. Summary of the invention
[0004] The purpose of the present invention is to provide an accurate and reliable method for predicting the duration of highway traffic incidents based on big data technology and text mining. Based on a massive, multi-source highway traffic incident data set, the present invention combines structured data mining with text analysis, takes traffic incidents that affect road capacity as the research object, and establishes a highway traffic incident duration prediction model to avoid huge driving time losses and provide decision-making basis and reliable road services for public travel.
[0005] The present invention is achieved through the following technical solutions:
[0006] The present invention relates to a method for predicting the duration of a highway traffic event driven by big data, comprising the following steps:
[0007] Step 1: Establish a highway traffic incident dataset and extract the fields required for the study;
[0008] Step 2, preprocessing the extracted data fields, including preprocessing of structured data and preprocessing of text data;
[0009] Step 3: Based on data preprocessing, a random forest model is established using traffic accidents, vehicle failures, road spills, weather conditions, casualties, locations, and alarm types to predict the duration of highway traffic events.
[0010] Step 4: Based on data preprocessing, a traffic incident duration prediction model based on random forest is established by using the attributes of traffic accidents, vehicle failures, road spills, weather conditions, number of casualties, location, and alarm type, combined with the feature vector of event text information;
[0011] In step 5, the mean absolute error (MAE) and mean relative error (MAPE) are used to compare and analyze the accuracy of the two prediction models. Through the comparison results, it can be seen that the highway traffic event duration prediction model based on text mining and random forest algorithm has better performance indicators and shows better applicability in actual data.
[0012] Preferably, the specific steps in step 1 are: combining the structured data in the highway traffic event data with the text data to form highway traffic event big data, and extracting the fields required for the research therefrom.
[0013] (1) Extract the required fields from the structured data, including: event number, direction, alarm personnel, alarm time, number of casualties, number of damaged vehicles, impact range, weather conditions, event processing completion time, etc.
[0014] (2) From the text data, the fields extracted include: the time of the incident, the specific road section where the incident occurred, the specific circumstances of the collision (single vehicle hitting the guardrail, two or more vehicles colliding, etc.), whether there was a fire, the type of vehicle involved (van, semi-trailer, etc.), the lane occupied (1 lane, 2 lanes, 3 lanes), road capacity (congestion level), road spillage, temporary maintenance and construction, special vehicle passage, heavy traffic flow, disposal measures, disposal effect, disposal department, response progress, etc.
[0015] Preferably, in step 2, the four main tasks of the preprocessing are: data cleaning, data integration, data transformation and data reduction; this step designs the database table structure and its fields in combination with the research objectives to ensure the efficiency of data query and analysis under massive sample conditions. The data required for data mining usually come from different sources, and data integration refers to the operation of merging multiple different data sources into the same data storage. Data transformation refers to converting data into a specific, standardized form in combination with the needs of the mining task or mining algorithm. New attributes can be constructed based on the existing attribute set. Usually, data mining of a complete large data set will inevitably take a long time or perform complex analytical calculations. Data reduction refers to the generation of a smaller new data set while ensuring data integrity.
[0016] In step 2, the preprocessing of the structured data is specifically as follows:
[0017] (1) Data cleaning is to filter and delete duplicate data, noise data and other data that are irrelevant to the research objectives in the original data set. Abnormal data is eliminated, including missing data, erroneous data, etc. Abnormal data mainly includes: missing traffic incident alarm time or event processing completion time, abnormal time data records, and abnormal location data records.
[0018] (2) According to the two fields of alarm time and traffic event processing completion time in the data record after the highway traffic event occurs, a new field is constructed to calculate the duration of the highway traffic event. Expression (5) is as follows:
[0019] t duration =t processdone -t alarm (5)
[0020] In formula (1), t duration Traffic incident duration, t processdone is the time to complete the traffic incident processing, t alarm The alarm time.
[0021] (3) Calculate the upper and lower quartiles of the traffic incident duration sample data. The upper and lower quartile values are the upper and lower limits of the valid data interval. Data outside this range is considered to be noise data. Expressions (6) and (7) are as follows:
[0022] t limit-down =t 25％ -1.5×(t 75％ -t 25％ ) (6)
[0023] t limit-up =t 25％ +1.5×(t 75％ -t25％ ) (7)
[0024] In formula (6), t limit-down is the lower limit of the valid data interval G; in formula (7), t limit-up is the upper limit of the valid data interval G; t 25％ and t 75％ Represent the 25% and 75% quantiles of the sample data respectively.
[0025] (4) Digitally encode the type data in the research fields, such as weather conditions, traffic accident types, weather conditions, and vehicle types involved, to facilitate subsequent calculations.
[0026] In step 2, the preprocessing of the text data is specifically as follows:
[0027] When performing word segmentation on text data, the main information of the text must be ensured. There are a large number of interjections, modal particles, function words, and other words that are meaningless to the study. Therefore, it is necessary to remove stop words, thereby greatly reducing the number of redundant features in the text and avoiding the interference of useless features on the subsequent model construction. The processing process is as follows:
[0028] For a set D containing n texts, D = {d1, d1, ..., d n}, and the set of m words that appear in all texts w = {w1,w2,…,w m}. The data of the words appearing in the text is represented by a word-text matrix, denoted as X, as follows:
[0029]
[0030] This is an m×n matrix, with element x ij Represents word w i In text d j The weights appearing in , the weights are usually expressed by word frequency-inverse document frequency (TF-IDF), and its expression (8) is as follows:
[0031]
[0032] Where tf ij It is the word w i Appears in text d j The frequency in tf .j is text j The sum of the frequencies of all words that appear in , df i Contains the word w i df is the number of texts in the text set D.
[0033] The specific analysis process of step 3 is as follows:
[0034] (1) Based on step 2, perform S random samplings from the structured data, each time randomly extracting (with replacement) m features and n data samples to form the feature vector space C S and dataset D S (where S = 1, 2, 3, ...), construct S decision trees, and the algorithm used to construct the decision tree is C4.5, which uses information gain ratio to select features. Its information gain ratio expression (9) is as follows:
[0035]
[0036] In formula (5), A represents a feature and D represents a data set.
[0037] in
[0038] n represents the number of subsets into which the dataset D is divided according to feature A, namely D1, D2, …D n , i = 1, 2, 3…, n, θ(D, A) is the information gain of feature A for the training data set.
[0039] (2) Pruning the decision tree
[0040] In order to prevent overfitting after building a decision tree, that is, the prediction effect of the training data set is very strong, but the prediction effect of the test data set is greatly reduced. Prune each decision tree. The specific steps are as follows:
[0041] Decision tree pruning is often achieved by minimizing the overall loss function of the decision tree. Let the number of leaf nodes of tree T be |T|, t is a leaf node of tree T, and the leaf node has N t sample points, of which class k has N sample points tk , k=1,2,…,K,H t (T) is the empirical entropy on leaf node t, α≥0 is the parameter, then the loss function of decision tree learning can be defined as
[0042]
[0043] Among them, the empirical entropy is
[0044]
[0045] Substitute equation (2) into equation (1), and write the first term on the right side of the final expression as
[0046]
[0047] Based on the above, we can get
[0048] Cα (T) = C (T) + α | T | (4)
[0049] In formula (3), C(T) represents the prediction error of the model for the training set, and |T| represents the complexity of the model. In formula (4), the parameter α≥0 controls the influence between the two.
[0050] For the decision trees before and after pruning, use the above formula (4) to calculate the prediction errors of the two trees respectively. If the following conditions are met:
[0051] C α (T 后 )≤C α (T 前 )
[0052] Then prune is performed, that is, the parent node is changed into a new leaf node.
[0053] (3) Constructing a random forest model
[0054] Based on the decision tree generated above, a random forest model is constructed with the following construction rules:
[0055] Each decision tree needs to be used to predict the real-time duration of a traffic event, and finally the final traffic event duration prediction value is determined based on the principle of "the minority obeys the majority".
[0056] The specific analysis process of step 4 is as follows:
[0057] The results of structured data processing and text data mining are combined to form a new traffic event data set. The feature quantities and samples in the data set are used to build a text data feature prediction model based on random forest. The construction process is similar to the prediction model construction in step 3. The difference is that the data sets used by the two are different, mainly reflected in the difference in the number of features.
[0058] The specific analysis process of step 5 is as follows:
[0059] Mean absolute error E MAE And the average relative error E MAPE The calculation formula is as follows.
[0060]
[0061]
[0062] Where N is the number of samples, t p (i) represents the predicted value of the i-th sample, t a (i) represents the actual value of the i-th sample.
[0063] The mean absolute error (MAE) and mean relative error (MAPE) evaluation indicators are used to compare and analyze the accuracy of the two prediction models. The experimental results show that the prediction model based on random forest and text mining has significantly reduced mean absolute error and mean relative error compared with the simple random forest prediction model. The highway traffic incident prediction model based on random forest and text mining not only has high prediction accuracy, but also has certain advantages in adaptability to different types of traffic incidents.
[0064] The existing duration prediction models focus on sudden traffic accidents such as rear-end collisions and collisions, and the data sources are mainly statistical data from public security departments or traffic police departments. However, traffic incidents are still the main cause of highway traffic congestion, and the duration of occasional traffic incidents such as road spillage, temporary maintenance and construction, special vehicle traffic, and heavy traffic leading to slow traffic has not been fully discussed. Most of the data sources of existing studies are structured attribute data such as accident type, vehicle type involved, number of casualties, number of affected lanes, accident location, weather conditions, and time period. Highway traffic event records also contain text information described in natural language, which describes the basic information of the event, response measures, and implementation effects.
[0065] The duration of a traffic incident is related to many factors, such as time characteristics, event characteristics, road characteristics, traffic characteristics, and weather conditions. Existing technologies use structured data to analyze the causal mechanism and extract significant influencing factors to predict the duration of a traffic incident.
[0066] The present invention has the following advantages:
[0067] (1) Based on a massive, multi-source highway traffic event data set, the present invention combines structured data mining with text analysis, takes traffic events that affect road capacity as the object, and establishes a highway traffic event duration prediction model to avoid huge driving time losses and provide decision-making basis and reliable road services for public travel; the prediction model involved can not only predict the duration of sudden traffic accidents and natural disasters, but also predict typical occasional traffic events such as road spillage, temporary maintenance and construction, special vehicle passage, and slow traffic caused by heavy traffic flow.
[0068] (2) The present invention not only considers significant influencing factors, but also incorporates semi-structured data in the data set into the prediction method, performs text analysis on the natural language description of traffic event information reporting, and enriches the feature vector dimension of traffic events to achieve more accurate duration prediction; The present invention incorporates traffic event text information into the research scope and uses natural language text analysis to extract feature vectors. The present invention integrates text mining and data-driven algorithms to establish a highway traffic event duration prediction model to improve the accuracy and adaptability of the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] Figure 1 It is a flow chart of the calculation process of the method of the present invention. DETAILED DESCRIPTION
[0070] The present invention is described in detail below in conjunction with specific embodiments. It should be noted that the following embodiments are only further descriptions of the present invention, but the protection scope of the present invention is not limited to the following embodiments.
[0071] Example
[0072] This embodiment relates to a method for predicting the duration of highway traffic events driven by big data. Figure 1 As shown, the following steps are included:
[0073] Step 1: Establish a highway traffic incident dataset and extract the fields required for the study;
[0074] Step 2, preprocessing the extracted data fields, including preprocessing of structured data and preprocessing of text data;
[0075] Step 3: Based on data preprocessing, a random forest model is established using traffic accidents, vehicle failures, road spills, weather conditions, casualties, locations, and alarm types to predict the duration of highway traffic events.
[0076] Step 4: Based on data preprocessing, a traffic incident duration prediction model based on random forest is established by using the attributes of traffic accidents, vehicle failures, road spills, weather conditions, number of casualties, location, and alarm type, combined with the feature vector of event text information;
[0077] In step 5, the mean absolute error (MAE) and mean relative error (MAPE) are used to compare and analyze the accuracy of the two prediction models. Through the comparison results, it can be seen that the highway traffic event duration prediction model based on text mining and random forest algorithm has better performance indicators and shows better applicability in actual data.
[0078] The specific steps in step 1 are: combining the structured data in the highway traffic event data with the text data to form highway traffic event big data, and extracting the fields required for the study from it.
[0079] (1) Extract the required fields from the structured data, including the event number, direction, alarm personnel, alarm time, number of casualties, number of damaged vehicles, impact range, weather conditions, and event processing completion time.
[0080] (2) From the text data, the fields extracted include the time of the incident, the specific road section where the incident occurred, the specific circumstances of the collision (single vehicle hitting the guardrail, two or more vehicles colliding, etc.), whether there was a fire, the type of vehicle involved (van, semi-trailer, etc.), the lane occupied (1 lane, 2 lanes, 3 lanes), road capacity (congestion level), road spillage, temporary maintenance and construction, special vehicle passage, heavy traffic flow, disposal measures, disposal effect, disposal department, response progress, etc.
[0081] In step 2, the four main tasks of the preprocessing are: data cleaning, data integration, data transformation and data reduction; this step is combined with the research objectives to design the database table structure and its fields to ensure the efficiency of data query and analysis under massive sample conditions. The data required for data mining usually come from different sources. Data integration refers to the operation of merging multiple different data sources into the same data storage. Data transformation refers to converting data into a specific and standardized form in combination with the needs of mining tasks or mining algorithms. New attributes can be constructed based on existing attribute sets. Usually, data mining of a complete large data set will inevitably take a long time or perform complex analytical calculations. Data reduction refers to the generation of a smaller new data set while ensuring data integrity.
[0082] The preprocessing of the structured data is specifically as follows:
[0083] (1) Data cleaning is to filter and delete duplicate data, noise data and other data that are irrelevant to the research objectives in the original data set. Abnormal data is eliminated, including missing data, erroneous data, etc. Abnormal data mainly includes: missing traffic incident alarm time or event processing completion time, abnormal time data records, and abnormal location data records.
[0084] (2) According to the two fields of alarm time and traffic event processing completion time in the data record after the highway traffic event occurs, a new field is constructed to calculate the duration of the highway traffic event. Expression (5) is as follows:
[0085] t duration =t processdone -t alarm (5)
[0086] In formula (1), t duration Traffic incident duration, t processdone is the time to complete the traffic incident processing, t alarm The alarm time.
[0087] (3) Calculate the upper and lower quartiles of the traffic incident duration sample data. The upper and lower quartile values are the upper and lower limits of the valid data interval. Data outside this range is considered to be noise data. Expressions (6) and (7) are as follows:
[0088] t limit-down =t 25％ -1.5×(t 75％ -t 25％ ) (6)
[0089] t limit-up =t 25％ +1.5×(t 75％ -t 25％ ) (7)
[0090] In formula (6), t limit-down is the lower limit of the valid data interval G; in formula (7), t limit-up is the upper limit of the valid data interval G; t 25％ and t 75％ Represent the 25% and 75% quantiles of the sample data respectively.
[0091] (4) Digitally encode the type data in the research fields, such as weather conditions, traffic accident types, weather conditions, and vehicle types involved, to facilitate subsequent calculations.
[0092] The preprocessing of the text data is specifically as follows:
[0093] When performing word segmentation on text data, the main information of the text must be ensured. There are a large number of interjections, modal particles, function words, and other words that are meaningless to the study. Therefore, it is necessary to remove stop words, thereby greatly reducing the number of redundant features in the text and avoiding the interference of useless features on the subsequent model construction. The processing process is as follows:
[0094] For a set D containing n texts, D = {d1, d1, ..., d n}, and the set of m words that appear in all texts w = {w1,w2,…,w m}. The data of the words appearing in the text is represented by a word-text matrix, denoted as X, as follows:
[0095]
[0096] This is an m×n matrix, with element xij Represents word w i In text d j The weights appearing in , the weights are usually expressed by word frequency-inverse document frequency (TF-IDF), and its expression (8) is as follows:
[0097]
[0098] Where tf ij It is the word w i Appears in text d j The frequency in tf .j is text j The sum of the frequencies of all words that appear in , df i Contains the word w i df is the number of texts in the text set D.
[0099] The specific analysis process of step 3 is as follows:
[0100] (1) Based on step 2, perform S random samplings from the structured data, each time randomly extracting (with replacement) m features and n data samples to form the feature vector space C S and dataset D S (where S = 1, 2, 3, ...), construct S decision trees, and the algorithm used to construct the decision tree is C4.5, which uses information gain ratio to select features. Its information gain ratio expression (9) is as follows:
[0101]
[0102] In formula (5), A represents a feature and D represents a data set.
[0103] in
[0104] n represents the number of subsets into which the dataset D is divided according to feature A, namely D1, D2, …D n , i = 1, 2, 3…, n, θ(D, A) is the information gain of feature A for the training data set.
[0105] (2) Pruning the decision tree
[0106] In order to prevent overfitting after building a decision tree, that is, the prediction effect of the training data set is very strong, but the prediction effect of the test data set is greatly reduced. Prune each decision tree. The specific steps are as follows:
[0107] Decision tree pruning is often achieved by minimizing the overall loss function of the decision tree. Let the number of leaf nodes of tree T be |T|, t is a leaf node of tree T, and the leaf node has Nt sample points, of which class k has N sample points tk , k=1,2,…,K,H t (T) is the empirical entropy on leaf node t, α≥0 is the parameter, then the loss function of decision tree learning can be defined as
[0108]
[0109] Among them, the empirical entropy is
[0110]
[0111] Substitute equation (2) into equation (1), and write the first term on the right side of the final expression as
[0112]
[0113] Based on the above, we can get
[0114] C α (T) = C (T) + α | T | (4)
[0115] In formula (3), C(T) represents the prediction error of the model for the training set, and |T| represents the complexity of the model. In formula (4), the parameter α≥0 controls the influence between the two.
[0116] For the decision trees before and after pruning, use the above formula (4) to calculate the prediction errors of the two trees respectively. If the following conditions are met:
[0117] C α (T 后 )≤C α (T 前 )
[0118] Then prune is performed, that is, the parent node is changed into a new leaf node.
[0119] (3) Constructing a random forest model
[0120] Based on the decision tree generated above, a random forest model is constructed with the following construction rules:
[0121] Each decision tree needs to be used to predict the real-time duration of a traffic event, and finally the final traffic event duration prediction value is determined based on the principle of "the minority obeys the majority".
[0122] The specific analysis process of step 4 is as follows:
[0123] The results of structured data processing and text data mining are combined to form a new traffic event data set. The feature quantities and samples in the data set are used to build a text data feature prediction model based on random forest. The construction process is similar to the prediction model construction in step 3. The difference is that the data sets used by the two are different, mainly reflected in the difference in the number of features.
[0124] The specific analysis process of step 5 is as follows:
[0125] Mean absolute error E MAE And the average relative error E MAPE The calculation formula is as follows.
[0126]
[0127]
[0128] Where N is the number of samples, t p (i) represents the predicted value of the i-th sample, t a (i) represents the actual value of the i-th sample.
[0129] The mean absolute error (MAE) and mean relative error (MAPE) evaluation indicators are used to compare and analyze the accuracy of the two prediction models. The experimental results show that the prediction model based on random forest and text mining has significantly reduced mean absolute error and mean relative error compared with the simple random forest prediction model. The highway traffic incident prediction model based on random forest and text mining not only has high prediction accuracy, but also has certain advantages in adaptability to different types of traffic incidents.
[0130] Based on the previous research on the duration of traffic events, this invention combines text data mining, integrates structured attribute features and semi-structured text data, and forms a new traffic event data set. On this basis, a highway traffic event duration prediction model based on big data technology and text mining is constructed to improve the accuracy and adaptability of the prediction model. For traffic managers, emergency command and dispatch can be carried out reasonably according to the predicted time; for road users, it is possible to arrange travel plans reasonably according to the predicted time. In short, the efficiency of road traffic use can be better improved.
[0131] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.
Claims
1. A big data driven highway traffic event duration prediction method, characterized in that: The steps include: Step 1: Establish a highway traffic incident data set and extract the fields required for the study: Specifically, combine the structured data and text data in the highway traffic incident data to form highway traffic incident big data, and extract the fields required for the study from it; Step 2: preprocess the extracted data fields, including preprocessing of structured data and preprocessing of text data; the four tasks of preprocessing are: data cleaning, data integration, data transformation and data reduction; Step 3: Based on data preprocessing, a random forest model is established using structured data attributes to predict the duration of highway traffic events; the attributes are: traffic accident, vehicle breakdown, road spillage, weather conditions, number of casualties, location, and alarm type; Step 4: Based on data preprocessing, a duration prediction model based on random forest is established by using structured data attributes and combining feature vectors of event text information; wherein the attributes are: traffic accident, vehicle breakdown, road spillage, weather conditions, number of casualties, location, alarm type, and text feature vectors extracted from event information described in natural language; Step 5: Use the mean absolute error and mean relative error as two evaluation indicators to compare and analyze the accuracy of the two prediction models.