Traffic flow abnormity detection method and system

Abstract

The invention discloses a traffic flow anomaly detection method and system, and relates to the technical field of data anomaly detection, and the method comprises the steps: enabling the output end of a long short-term memory network LSTM to be connected with the input end of a deep Q network DQN, so as to construct a traffic flow prediction model; using a kernel density estimation method KDE to obtain a probability density estimation value of the characteristics of the traffic flow data as a normality score; determining whether the data is normal according to the normality score, and obtaining a determination result; taking a condition that a prediction result of the characteristics of the traffic flow data by the DQN is consistent with a determination result as a positive reward, taking a condition that the prediction result is inconsistent with the determination result as a negative reward to construct a data-driven reward function, and replacing a reward function of the DQN in the traffic flow prediction model with the data-driven reward function; and acquiring real-time traffic flow data, and predicting real-time traffic flow abnormity according to the trained traffic flow prediction model. The method can get rid of set real labels and threshold values to carry out anomaly detection of autonomous learning.

Description

Technical Field

[0001] The present invention relates to the technical field of data anomaly detection, and in particular to a method and system for detecting traffic flow anomalies. Background Art

[0002] Sequential anomalies in traffic flow data not only reflect the continuous changes in traffic flow data over time, but also reveal the overall abnormal patterns of traffic conditions within a specific time period. Accurately detecting sequential anomalies in traffic flow data can provide a deeper understanding of the changing patterns of traffic flow in mountainous areas, providing stronger support for traffic management and planning.

[0003] Prior art methods, such as statistical and prediction-based methods, have been proposed for time series anomaly detection. Statistical methods typically assume that data is generated by a specific statistical model. The goal is to construct a statistical model based on the given data and then check whether new data fits the model. Prediction-based methods are similar to statistical methods, first estimating the next value based on the previous subsequence and then performing anomaly detection based on specific rules. While these methods reduce the reliance on large amounts of labeled data to some extent, they still cannot completely eliminate the need for labeled data.

[0004] In summary, the conventional operating mechanism of anomaly detection models is highly dependent on labels for model training and cannot be separated from the setting of thresholds to accurately determine anomalies. How to break away from the set real labels and thresholds for autonomous learning of anomaly detection is an important issue that needs to be solved urgently. Summary of the Invention

[0005] The embodiments of the present invention provide a method and system for detecting traffic flow anomalies, which can solve the problem in the prior art of how to perform autonomous learning anomaly detection without setting real labels and thresholds.

[0006] An embodiment of the present invention provides a method for detecting traffic flow anomalies, comprising the following steps: Connect the output of the feature extraction model to the input of the deep Q-network (DQN) to build a traffic flow prediction model. The kernel density estimation method (KDE) is used to obtain the probability density estimate of the characteristics of the traffic flow data as a normality score. The normality of the data is determined based on the normality score to obtain a positive result. The results of the DQN prediction of the characteristics of the traffic flow data are consistent with the determined results as a positive reward, and the results of the prediction and the determined results are inconsistent as a negative reward, so as to construct a data-driven reward function for autonomous learning. Use a data-driven reward function to replace the DQN reward function, and use historical traffic flow data to train the traffic flow prediction model to obtain a trained traffic flow prediction model; Obtain real-time traffic flow data and predict real-time traffic flow anomalies based on the trained traffic flow prediction model.

[0007] Furthermore, the method for detecting traffic flow anomalies further includes: Historical traffic flow data was divided by season and by weekdays and non-workdays to obtain seasonal variation characteristics of traffic flow and the difference characteristics between weekdays and non-workdays. Data on external factors affecting traffic flow were divided by the severity of weather, the impact of accidents, and the impact of construction areas to obtain the characteristics of severe weather, the severity of accidents, and the impact of construction areas. The weighted score of each feature is obtained according to the product between the eigenvalue corresponding to each feature and the weight corresponding to each feature; The degree of influence of each feature on traffic flow is determined based on the weighted score of each feature.

[0008] Furthermore, the construction of a data-driven reward function for autonomous learning comprises the following specific steps: Based on the traffic flow data used in training , and historical traffic flow data Historical traffic flow data for the same period , get the set ; right Perform the KDE clustering operation using the kernel density estimation method to obtain Cluster ; Get normality score , the formula is: in, yes The number of clusters in , is the average cluster size; According to the normality score, we can obtain the data-driven reward function r t+1 , the formula is: in, Predict results for the model; t is the time step.

[0009] Furthermore, the use of historical traffic flow data to train a traffic flow prediction model specifically includes the following steps: Based on historical traffic flow data, LSTM is used as a function approximator to obtain an intermediate vector representation containing time series features, and the intermediate vector representation is input into DQN; Set the state space to ;in, is the time step t Observations of and including current traffic flow , historical average traffic and selected actions The triplet of Set the action space to A= , represents whether the traffic flow is considered abnormal; where 1 is abnormal and 0 is normal; According to the data-driven reward function, as well as the state space and action space, we adopt - Greedy strategy training DQN obtains traffic flow anomalies by intermediate vector representation and obtains trained DQN; A bidirectional sliding window smoothing operation is used on the trained DQN to eliminate the - False positives caused by random actions generated by the greedy strategy; DQN trained using the Adam optimizer; Obtain the trained traffic flow prediction model.

[0010] Furthermore, the real-time traffic flow data is obtained, specifically comprising the steps of: removing outliers and processing missing values ​​from the real-time traffic flow data and the data of external factors affecting the traffic flow, completing data cleaning; and normalizing the data.

[0011] Furthermore, the feature extraction model is a long short-term memory network LSTM.

[0012] An embodiment of the present invention provides a traffic flow anomaly detection system, comprising: A model building module is used to connect the output of the feature extraction model to the input of the deep Q network (DQN) to build a traffic flow prediction model; The model training module is used to use the kernel density estimation method (KDE) to obtain a probability density estimate of the characteristics of the traffic flow data as a normality score; determine whether the data is normal based on the normality score and obtain a confirmation result; use the case where the DQN's prediction result of the traffic flow data characteristics is consistent with the confirmation result as a positive reward, and use the case where the prediction result is inconsistent with the confirmation result as a negative reward to construct a data-driven reward function that achieves autonomous learning; use the data-driven reward function to replace the DQN reward function, and use historical traffic flow data to train the traffic flow prediction model to obtain a trained traffic flow prediction model; The anomaly prediction module is used to obtain real-time traffic flow data and predict real-time traffic flow anomalies based on the trained traffic flow prediction model.

[0013] The embodiments of the present invention provide a method and system for detecting traffic flow anomalies. Compared with the prior art, the methods and systems have the following advantages: The output of the feature extraction model is connected to the input of the deep Q network (DQN) to construct a traffic flow prediction model; the kernel density estimation method (KDE) is used to obtain the probability density estimate of the features of the traffic flow data as a normality score; whether the data is normal is determined based on the normality score to obtain a determination result; the case where the DQN prediction result of the traffic flow data features is consistent with the determination result is used as a positive reward, and the case where the prediction result is inconsistent with the determination result is used as a negative reward to construct a data-driven reward function for autonomous learning; the data-driven reward function is used to replace the DQN reward function, and the traffic flow prediction model is trained using historical traffic flow data to obtain a trained traffic flow prediction model; real-time traffic flow data is obtained, and real-time traffic flow anomalies are predicted based on the trained traffic flow prediction model.

[0014] Among them, the kernel density estimation method KDE is used to obtain the probability density estimate of the characteristics of traffic flow data as the normality score. The normality score changes with the changes in traffic flow data. Then, whether the data is normal is determined based on the normality score. The DQN's prediction result is compared with the determination result as the DQN's reward function, allowing DQN to learn based on the normality score, breaking away from the supervised learning method of setting true labels and thresholds, and realizing unsupervised autonomous learning anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of a method for detecting traffic flow anomalies provided by an embodiment of the present invention; Figure 2 An example of the state and operation of an RL-based anomaly detector for a method for detecting traffic flow anomalies provided by an embodiment of the present invention; Figure 3 An example of the state and operation of an RL-based anomaly detector for a method for detecting traffic flow anomalies provided by an embodiment of the present invention; Figure 4 Comparison of anomaly detection performance of Case 1 of a traffic flow anomaly detection method provided by an embodiment of the present invention, wherein a is a schematic diagram of Case 1 of Model 1, b is a schematic diagram of Case 1 of Model 2, c is a schematic diagram of Case 1 of Model 3, and d is a schematic diagram of Case 1 of the model of the present invention; Figure 5 Comparison of anomaly detection performance in Case 2 of a traffic flow anomaly detection method provided by an embodiment of the present invention, wherein a is a schematic diagram of Case 2 of Model 1, b is a schematic diagram of Case 2 of Model 2, c is a schematic diagram of Case 2 of Model 3, and d is a schematic diagram of Case 2 of the model of the present invention; Figure 6A schematic diagram of normal and abnormal data of a method for detecting traffic flow anomalies provided by an embodiment of the present invention, wherein a is a schematic diagram of a synthesized normal traffic time series, and b is a schematic diagram of a synthesized abnormal traffic time series; Figure 7 Performance comparison of different anomaly densities for a traffic flow anomaly detection method provided by an embodiment of the present invention, where a is a diagram of precision, b is a diagram of recall, and c is a diagram of F1 Score results. Figure 8 Performance comparison of different error rates for a traffic flow anomaly detection method provided by an embodiment of the present invention, where a is a schematic diagram of precision, b is a schematic diagram of recall, and c is a schematic diagram of F1 Score results. Figure 9 Performance comparison of different anomaly sequence lengths for a traffic flow anomaly detection method provided by an embodiment of the present invention, where a is a diagram of precision, b is a diagram of recall, and c is a diagram of F1 Score results. Figure 10 Performance comparison of different data sizes for a traffic flow anomaly detection method provided by an embodiment of the present invention, where a is a diagram of precision, b is a diagram of recall, and c is a diagram of F1 Score results. Figure 11 The performance of a model of a traffic flow anomaly detection method provided by an embodiment of the present invention at different time steps under different states, where a is a schematic diagram of precision, b is a schematic diagram of recall, and c is a schematic diagram of F1 Score results. DETAILED DESCRIPTION

[0016] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0017] See also Figure 1 , an embodiment of the present invention provides a method for detecting traffic flow anomalies, comprising the following steps: Step 1: Connect the output of the feature extraction model to the input of the deep Q network (DQN) to build a traffic flow prediction model.

[0018] Step 2: Use the kernel density estimation method KDE to obtain the probability density estimate of the characteristics of the traffic flow data as a normality score; determine whether the data is normal based on the normality score and obtain a confirmation result; use the case where the DQN's prediction result of the traffic flow data characteristics is consistent with the confirmation result as a positive reward, and the case where the prediction result is inconsistent with the confirmation result as a negative reward to construct a data-driven reward function that realizes autonomous learning; use the data-driven reward function to replace the DQN reward function, and use historical traffic flow data to train the traffic flow prediction model to obtain the trained traffic flow prediction model.

[0019] Step 3: Obtain real-time traffic flow data and predict real-time traffic flow anomalies based on the trained traffic flow prediction model.

[0020] 1. Data collection and preprocessing.

[0021] First, data collection was performed. Historical data on traffic volume (number of vehicles per unit time, e.g., veh / h) and speed were collected from loop detectors in the mountainous traffic network. Data on external factors that could affect traffic flow, such as weather conditions (temperature, precipitation, wind speed), accidents, and construction zone information, were also collected.

[0022] Traffic flow and speed data are key indicators that directly reflect traffic flow conditions. External factor data is crucial for accurately analyzing traffic flow anomalies. Weather conditions significantly impact traffic flow. Severe weather conditions (such as heavy rain, snow, and dense fog) can reduce visibility and road friction, leading to slower vehicle speeds and even accidents, resulting in traffic congestion or disruption. Collecting these external factor data and combining them with traffic flow data can provide a more comprehensive understanding of the potential causes of traffic flow anomalies and improve the accuracy and reliability of anomaly detection.

[0023] Remove outliers and handle missing values ​​in the data, perform data cleaning, ensure data quality, and provide a reliable basis for subsequent analysis. Outliers may be inaccurate measurements caused by sensor failure, data transmission errors, or special events (such as temporary traffic control, large vehicle failure, etc.).

[0024] Outlier handling method: Statistical analysis-based approach: Calculating the mean of traffic flow and speed data and standard deviation , which will usually exceed ( The value can be selected based on the data distribution and actual situation. Data within a certain range (e.g., the mean, median, or interpolation of adjacent data) are considered outliers. After identifying these outliers, you can choose to directly delete the outlier records or replace them with reasonable estimates (such as the mean, median, or interpolation of adjacent data).

[0025] Data distribution-based method: By analyzing the distribution of data, such as normal distribution, skew distribution, etc., determine the reasonable range of data. Data points that do not conform to the overall distribution of data are considered outliers.

[0026] Method for handling missing values: Mean filling method: For missing values in traffic flow and speed data, the historical average flow or speed of the same time period (such as the same working day, the same time period in the same season) of the road segment can be used to fill. For external factor data, such as missing temperature values in weather data, the historical average temperature of the region in the season, the time period can be filled.

[0027] After data cleaning, the data quality can be improved, and more accurate and reliable data basis can be provided for subsequent model training.

[0028] Normalize the data, the formula is as follows: Where, x is the original data, μ is the mean of the data, σ is the standard deviation of the data, x′ is the normalized data. Take the traffic flow data as an example, suppose we collect the hourly traffic flow data of a mountainous road segment for a period of time .

[0029] First, calculate the mean .

[0030] Then calculate the standard deviation .

[0031] Map the data to a specific interval, for example or . Take the mapping to the interval as an example, the calculation process is as follows: first, calculate the mean μ and standard deviation σ of the data, for each original data , calculate the normalized value x′ by the above formula.

[0032] For each original flow data , calculate its normalized value by the formula. If you want to map the data to the interval , you need to further perform linear transformation (if the original normalized result is not in the interval), for example , where and are the minimum and maximum values of the normalized data, respectively.

[0033] Mapping can accelerate the convergence of model training: In unnormalized data, the value ranges of different features (such as traffic flow and speed, as well as features of different external factors) may vary greatly. After normalization, all features are mapped to a similar scale (such as or ), so that during the gradient descent process, the update step size of each feature is relatively balanced, and it can converge towards the optimal solution more stably and quickly, thereby improving the efficiency of model training.

[0034] 2. Feature engineering extraction.

[0035] First, traffic flow feature extraction is performed: traffic flow data is analyzed to extract key features such as seasonal changes in traffic flow and differences between weekdays and non-workdays, in order to better capture the inherent laws of traffic flow. Seasonal change characteristics of traffic flow: By dividing and analyzing historical traffic flow data by season, it can be observed that traffic flow in different seasons presents different patterns. These seasonal change characteristics help the model understand the laws of traffic flow on a long-term time scale, predict changes in traffic demand in advance, and provide a basis for traffic planning and resource allocation. Difference characteristics between weekdays and non-workdays: During weekdays, traffic flow usually presents a clear pattern of morning and evening peaks, which is closely related to residents' commuting behavior. By extracting this feature, the model can better distinguish the traffic flow status under different date types.

[0036] Extract features influencing traffic flow from collected data on external factors, such as weather severity and accident severity. Weather severity features: Quantify the impact of weather on traffic flow based on weather conditions (such as temperature, precipitation, and wind speed). In traffic flow prediction models, weather severity serves as a key feature, helping the model more accurately predict changes in traffic flow under specific weather conditions. Accident severity features: Quantify accident severity by considering factors such as the number of vehicles involved, accident type (such as rear-end collision, collision, rollover), whether casualties were caused, and the impact on road capacity. Incorporating accident severity as a feature into the model enables the model to more promptly and accurately identify traffic flow anomalies caused by accidents. Construction zone impact features: Factors such as the location, size, and duration of a construction zone can significantly influence traffic flow. By converting relevant information about the construction zone into features (such as the distance from the construction zone to the road, the proportion of the construction area to the total road area, and the overlap between construction time and peak traffic hours), the model can predict the scope and extent of the construction zone's impact on surrounding traffic flow.

[0037] The feature weight is calculated according to the following formula.

[0038] .

[0039] in, is the weighted score of the feature, is the weight of the feature, is the value of the feature. First determine the feature Value For traffic flow characteristics, the traffic flow characteristic value can be the actual traffic flow value in a certain period of time. For external factor characteristics, the weather severity characteristic value can be a quantitative value corresponding to the set weather level (such as 3 for heavy rain, 1 for light rain, etc.). The accident severity characteristic value can be a value calculated based on the accident type, number of vehicles involved, casualties, etc. The construction area impact characteristic value can be a quantitative value calculated based on the construction scale, location, progress, etc.

[0040] Then determine the characteristics Weight There are many ways to determine the weights. One is to set them based on expert experience. For example, based on long-term observation and research on mountain traffic, traffic management experts believe that traffic flow characteristics are the most important in judging traffic flow anomalies, and can be given a higher weight (such as 0.5), while the severity of weather is relatively less important and is given a lower weight (such as 0.2). Another method is to determine it through data analysis, such as using methods such as principal component analysis (PCA) to analyze the correlation between features and traffic flow anomalies, and features with high correlation are given greater weights. The present invention selects the first method to determine the weights.

[0041] Finally, the eigenvalue With weight Multiply to get the feature Weighted score During model training and anomaly detection, the model comprehensively considers the impact of each feature based on these weighted scores, allowing for more accurate judgments on traffic flow anomalies. For example, when determining whether traffic flow is abnormal at a given moment, if the weighted score for traffic flow is high (indicating an abnormal change in traffic flow) and the weighted score for accident severity is also high (indicating an accident has occurred), the model is more likely to determine that traffic flow is abnormal at that moment, even if the weighted scores for other features are lower.

[0042] 3. Model construction.

[0043] Build a deep learning model based on long short-term memory network (LSTM), such as Figure 2 As shown in the figure, LSTM can process long-term data dependencies. Traffic flow data is a typical time series data. In the complex traffic environment of mountainous areas, changes in traffic flow are affected by multiple factors, and they have long-term temporal dependencies.

[0044] In the traffic flow anomaly detection method of this invention, LSTM is a core component of the data processing phase. After data collection, preprocessing, and feature engineering, traffic flow data is fed into the LSTM network, which performs deep processing on the data, converting the raw traffic flow data into an intermediate vector representation containing rich time series features. This intermediate vector serves as a key input to the subsequent Deep Q Network (DQN) reinforcement learning framework, providing the DQN with high-quality feature information. This enables the DQN to make better decisions and learn strategies based on these features, thereby accurately detecting traffic flow anomalies.

[0045] LSTM and DQN work together to achieve the traffic flow anomaly detection goal of the present invention. LSTM focuses on learning the inherent patterns and characteristics of traffic flow data, providing the model with an accurate understanding of the traffic flow state; DQN uses the feature information output by LSTM, and under the framework of reinforcement learning, continuously optimizes the decision-making strategy through interaction with the environment (selecting actions based on the current state and receiving reward feedback) to determine whether the traffic flow is abnormal. The two work together to enable the model of the present invention to automatically learn abnormal patterns from traffic flow data in the case of unsupervised learning, thereby improving the accuracy and efficiency of traffic flow anomaly detection in mountainous areas. The LSTM structure is as follows Figure 2 shown.

[0046] Learning the time series characteristics of traffic flow data can effectively handle long-term dependencies in long-sequence data.

[0047] Applications of LSTM networks include: Input: Time series traffic flow data, including historical traffic volume and speed data, as well as relevant feature data after feature engineering. Processing: Utilizing the LSTM memory cells and gate structure, it learns long-term dependencies in traffic flow data, models traffic flow trends, and extracts key features from the data. Output: An intermediate vector representation captures the temporal characteristics of the traffic flow data, serving as input to the subsequent DQN network for further decision-making and policy learning.

[0048] Then, the DQN reinforcement learning framework is introduced to optimize the decision-making process of the traffic flow prediction model. The DQN combines deep learning (approximating the Q-value function through a neural network) with reinforcement learning. In the present invention, LSTM is used as a function approximator to represent the Q function, which can fully utilize the ability of LSTM to process long sequence data and integrate historical information of traffic flow into the decision-making process. At the same time, the powerful expression ability of deep neural networks can better handle the complex relationship between state space and action space. The amount of mountainous traffic flow data is large, the characteristics are complex, and the patterns of traffic flow anomalies are diverse, making it difficult for traditional reinforcement learning methods to effectively handle them. DQN can learn the complex mapping relationship between traffic flow state and optimal action through the nonlinear mapping ability of neural networks, thereby improving the accuracy of decision-making.

[0049] DQN uses experience replay and target network techniques to stabilize training and learn the optimal policy by maximizing cumulative rewards. The state space and action space are defined as follows: .

[0050] wherein, is the observation value at time step i, and is a triple containing the current traffic flow , historical average flow , and the selected action , A={0,1} represents the action space, =1 represents that the current traffic flow is considered abnormal, =0 represents normal, as shown in Figure 3 .

[0051] The DQN network application includes: input: the state representation composed of the feature vector output by the LSTM network and the action, reward, and other information under the current state. Processing: approximating the Q-value function through a deep neural network, using experience replay and target network techniques to stabilize the training process, learning the policy to select the optimal action under different states to maximize cumulative rewards. Output: outputting the Q-value of each action (judging whether the traffic flow is abnormal), selecting the action according to the Q-value, and using the -greedy strategy for action selection, balancing between exploring new actions and utilizing existing experience.

[0052] 4. Establish an unsupervised reward learning mechanism.

[0053] Defining the reward function is the most critical step in RL (reinforcement learning), as the learning of the agent largely depends on the reward feedback for the operations it selects. In the anomaly detection problem, the agent should receive a positive reward for correctly detecting anomalies, and a penalty (e.g., "negative reward") for incorrect detection.

[0054] Since there are no true labels available, it is challenging to determine whether the agent has made the correct decision. Therefore, a data-driven reward function is developed that automatically sets the reward value for a given state-action pair based on the data.

[0055] Introducing a metric, the normal score, to quantify the current traffic The comparison between the “normal” of the current time period and the typical “traffic value” is defined based on this normal score, so that the agent will Low normality scores are considered abnormal and receive positive rewards, while A high normal score is considered an anomaly and is penalized.

[0056] To determine a "typical" flow value for a given period, the present invention uses historical flow values ​​collected in a mountainous area between June and September.

[0057] The normality score and reward value are calculated as follows: The kernel density estimation (KDE) technique was used to calculate the “normality” score of the data.

[0058] Kernel Density Estimation (KDE) is a nonparametric estimation method used to estimate the probability density function of a random variable. The basic idea is to Place a kernel function , and then estimate the overall probability density function by weighted summation of all kernel functions For a given dataset , the estimation formula of the probability density function is:

[0059] .

[0060] in is the bandwidth, which controls the smoothness of the kernel function. is the number of data points. Kernel function Usually a symmetric probability density function. Bandwidth The choice of has a significant impact on the estimation results.

[0061] The advantages of using KDE in this invention are: Adapt to the characteristics of traffic flow data: Traffic flow data has high complexity and dynamic variability, influenced by various factors such as time, geographical location, external events. KDE, as a non-parametric method, does not require prior assumptions about the distribution of data, and can well adapt to the complex distribution of traffic flow data. It can automatically estimate the probability density according to the actual distribution of data, whether it is unimodal, multimodal or asymmetric distribution, and can effectively handle it.

[0062] Given traffic value , extract historical traffic data similar to the current time from the reference traffic database D , including historical traffic data of two time steps before and after the current time step, to form set. The specific reasons for forming the set are as follows:

[0063] Combine the current traffic value with the historical traffic data similar to the current time extracted from the reference traffic database D , in order to evaluate the "normality" of in a comprehensive data environment. is the object to be judged for abnormality, while represents the traffic flow conditions in the past under similar time conditions. By combining the two, the information of historical data can be used to provide reference for judging .

[0064] Reflect the dynamic changes of traffic flow: Traffic flow is not static, its pattern will change with time, external factors, etc. By forming the set , we can capture the dynamic changes to some extent. Historical traffic data reflects the range and pattern of traffic flow under different conditions in the past, and when combined with the current traffic value , it can more comprehensively consider the possible changes of traffic flow. For example, during the tourist season, the traffic of roads around a scenic spot may increase significantly than usual, if only considering the current traffic , it may be misjudged as abnormal, but combined with the traffic pattern of the tourist season in the historical data (included in ), it can more accurately judge whether it is really abnormal.

[0065] KDE clustering is performed on , which is a non-parametric method for estimating the probability density function (PDF) of a random variable. KDE generates a smooth curve of PDF based on kernel smoothing, and is used to cluster one-dimensional data by identifying different peaks (local maxima) in the PDF.

[0066] Find Cluster , calculate the "normality" score: in, yes The number of clusters in , is the average cluster size.

[0067] Score based on "normality" Define the reward function as follows: like Figure 3 As shown, the model prediction is rewarded or punished. When the model predicts ( ) and the “normality” of the data ( ) are consistent, give positive rewards, for example, when (Data may be abnormal) and (Model judgment is abnormal), or (The data may be normal) and (The model judges it to be normal); conversely, when the prediction is inconsistent with the data, a negative reward (penalty) is given. This guides the model to learn anomaly detection and prediction capabilities, as shown in Table 1.

[0068] Table 1 Model prediction summary Compared to other traditional methods, the unsupervised reward learning method of the present invention offers significant improvements. It can accurately determine whether current traffic changes are abnormal through reward learning, thereby improving the accuracy of anomaly detection. It also enhances the model's adaptability, enabling it to adapt to diverse traffic flow data distributions. Unlike threshold-based methods, many existing anomaly detection methods rely on thresholds to determine anomalies. These thresholds may be predefined or adaptively adjusted based on the data distribution. The method of the present invention is threshold-free. It automatically determines anomalies through data-driven reward learning, avoiding the difficulty of threshold selection. This allows for more flexible adaptation to diverse traffic conditions and locations, improving the accuracy and versatility of anomaly detection. Furthermore, it differs from other RL-based methods: Some reinforcement learning (RL)-based methods rely partially or completely on labeled data for model training. For example, some methods require known anomaly labels to define rewards or evaluate model performance, which is often difficult to obtain in real-world traffic data. However, the unsupervised reward learning algorithm of the present invention does not require ground truth labels and can learn anomaly detection capabilities in an unsupervised manner, reducing the requirements for data annotation and improving the model's practicality and scalability.

[0069] 5. Real-time prediction and dynamic adjustment.

[0070] With real-time data stream input, the model processes traffic flow data and updates predictions in real time. Based on real-time external factors (such as sudden accidents or weather changes), the model's prediction strategy is dynamically adjusted to improve prediction accuracy and adaptability.

[0071] The model constructed by this invention connects to the data collection system in the mountainous transportation network to realize the input of real-time data stream. Traffic flow data and external factor data can be transmitted to the model in real time. The long short-term memory network (LSTM) in the model is the core component, which continuously updates its internal state and learns the real-time changing pattern of traffic flow data. For example, at each time step , new traffic flow and speed and other data are input into LSTM, and LSTM is based on the previous state and input (including current flow, speed and possible external factors information), by calculating the forget gate Decide which historical information to forget, input gate: ; Determine how new information is updated, unit status update , output gate: , calculate the new output . Thus, the dynamic changes of traffic flow can be tracked in real time.

[0072] 6. Model training and optimization.

[0073] A deep learning model is trained using historical traffic flow data, and the prediction strategy is continuously optimized through a reinforcement learning framework. Cross-validation and other techniques are used to evaluate model performance, ensuring good prediction results under diverse mountain traffic flow conditions.

[0074] The training process is as follows: The model is trained using a reinforcement learning framework, where the LSTM network is used as a function approximator to process traffic flow data, converting it into an intermediate vector representation containing rich time series features, which is then input into the Deep Q Network (DQN). DQN uses experience replay and target network techniques to stabilize training and learn the optimal strategy by maximizing cumulative rewards. During the training process, the state space and action space are defined. The state space is ,in is the time step The observation value of , historical average traffic and selected actions triples; action space A= , represents whether the traffic flow is considered abnormal (1 is abnormal, 0 is normal). - Greedy strategy, which makes a trade-off between exploring new actions and leveraging existing experience to better optimize the model. As training progresses, Gradually reduce, so that the model gradually shifts from more exploration to more use of learned experience. In order to eliminate the RL module based on To avoid false positives caused by random actions generated by the greedy strategy, the model was post-processed using a bidirectional sliding window smoothing operation, including forward smoothing, backward smoothing, and logical AND operations.

[0075] The logical AND operation involves performing a logical AND operation on the results of forward and backward smoothing. For each action, the value is set to 1 only if both smoothed values ​​are 1, otherwise it is set to 0. This eliminates random false positives where a single or a few traffic values ​​are mistakenly identified as abnormal, as their adjacent traffic is mostly labeled as normal. The resulting model's detection results indicate that each traffic value is labeled as 1 (abnormal) or 0 (normal).

[0076] Optimization model method: The Adam optimizer is used to optimize the model to minimize the loss function. The loss function is defined by measuring the mean square error between the Q value predicted by the Q target network and the Q value calculated based on the Q learning update rule. The formula is:

[0077] = .

[0078] Where N is the training batch size, is the discount factor, are network parameters.

[0079] By continuously adjusting the model parameters, the model can gradually learn the optimal strategy during the training process to accurately determine whether the traffic flow is abnormal.

[0080] The evaluation indicators used are accuracy, recall and F1 score.

[0081] Based on the present invention, the following evaluations are made: To evaluate the model's performance on real-world traffic data, two link flow sequences were selected from different loop detectors. Since no ground-truth labels were available for real-world traffic data, quantitative performance measurements were impossible. Instead, the anomaly detection results of each model were visually investigated. Specifically, three months of traffic data from two different links in a specific location were used, where hourly traffic volume (veh / h) was measured based on traffic counts collected at 3-minute aggregation intervals. Since flow patterns typically differ between weekdays and weekends, only weekday flow data was considered.

[0082] respectively Figure 4 (referred to as Case 1) and Figure 5The sample flow time series and detection results for two selected links are shown in Figure 2 (referred to as Case 2). For each case, the sample flow time series covers a two-week period with some visible abnormal data points, where each time step in the x-axis represents a 3-minute interval. Case 1 shows an overall drop and gradual decrease in traffic between time steps 3000 and 3500, and Case 2 shows two consecutive days of very low traffic between time steps 3500 and 4500. The performance of the three baseline models and the proposed RL model in detecting these abnormal periods is compared, where the anomalies detected by the models are shown as red curves in the blue shaded area. Overall, DeepAnt and Telemon perform poorly. From Figure 4 (a) and Figure 5 In (a), it is observed that DeepAnt generates a large number of false positives by identifying normal flows as anomalies. DeepAnt also misses detections in Case 2, on the two days when the traffic is very low. Figure 4 (b) and Figure 5 (b) shows that Telemon fails to detect the entire abnormal cycle in Case 1 and only detects part of the abnormal cycle in Case 2. Telemon also has a false positive on the third day of the time series. BreakoutDetection and the proposed model show good performance in both cases, correctly detecting the abnormal sequence with only a few missed detections. However, BreakoutDetection generates multiple false positives throughout the normal period, and Model 3 in Case 1 is shown in the schematic diagram. Figure 4 (c) shows the schematic diagram of model 3 in case 2. Figure 5 (c) In both case studies, the model did not have any false positives and demonstrated its ability to detect the entire sequence of abnormal cycles in a completely unsupervised manner. Figure 4 (d) shows the schematic diagram of the model in Case 2. Figure 5 (d) shown.

[0083] DeepAnt (model 1), Telemon (model 2), and BreakoutDetection (model 3) are the three benchmark models used for comparison in this paper. Their details are as follows: DeepAnt: uses a one-dimensional convolutional neural network (CNN) to The subsequence of step flow is used as input to predict the next The L2 norm distance between the predicted value and the observed flow is measured, and the observed flow is determined to be abnormal based on a manually selected threshold. In the experiment, its parameters are set as follows: kernel size = 4, pool size = 2×2, batch size = 70, hidden layer = 80, dropout rate = 0.5, =20, = 1. Telemon: Apply LSTM to the previous traffic subsequence of length As input, predict the next The prediction error is calculated by comparing the predicted flow with the observed flow, and the threshold is automatically calculated based on the distribution of the recent observed flow sequence to determine whether the observed flow is abnormal. In the experiment, its parameters are set as follows: batch size = 50, hidden layer = 80, dropout rate = 0.3, =120, = 20. BreakoutDetection (BD): This open-source R package is based on the seasonal hybrid ESD (SH-ESD) algorithm, which is derived from the generalized extreme studentized deviation (ESD) test. It measures the value with the largest deviation from the flow rate at the same time point and marks it as an anomaly based on a robust statistical metric. In the experiments presented here, this algorithm was used to evaluate the performance of the proposed model.

[0084] We simulate normal traffic time series by analyzing real data and incorporating random fluctuations. The specific steps are as follows: First, we extract three months of weekday traffic data from a specific link. This data is grouped into 3-minute time periods. Then, we record the traffic observations within each time period and calculate their mean and standard deviation. Using these mean and standard deviation, we draw traffic values ​​from a Gaussian distribution to generate a normal traffic time series. Figure 6 (a) shows an example of a synthetic normal traffic time series. Figure 6 (b) shows an example of synthetic data with anomalies, where the anomalous subsequences are shown as red curves in the blue shaded region.

[0085] Considering the anomaly density parameter, the performance of the model on all three metrics remains stable when increasing the density, e.g. Figure 7As shown, this demonstrates that the model is robust to increasing anomalies in the dataset. BreakoutDetection performs well when the anomaly density is within a certain level (e.g., around 30%), achieving better recall at lower densities and better precision at higher densities. Telemon performs best when the density is around 20%, while all three metrics decline in other cases. DeepAnt shows improved performance with increasing density, but its overall performance is relatively poor, with an F1 score of only 22% in the best case. In contrast, models with varying densities maintain a stable precision of around 90% and a recall above 80%, resulting in an F1 score of around 85% across all density settings. Figure 7 (a) is a schematic diagram of accuracy, Figure 7 (b) is a diagram of recall rate. Figure 7 (c) is a schematic diagram of the F1 Score results.

[0086] As for the error rate parameter, Figure 8 As can be seen in the figure, in the model, the precision is stable when the error rate is different. The recall rate is also stable overall, but when the error rate is lower than 50%, the recall rate drops. This is because a smaller error rate will produce anomalies with smaller deviations from normal data, resulting in a higher level of missed detection. When the error rate is greater than or equal to 50%, the model performs well, with an F1 score higher than 85%. The recall rate of BreakoutDetection is better than that of the model of the present invention, but the precision is much worse. The performance of Telemon fluctuates as the error rate increases, while DeepAnt shows a downward trend. The reason may be that when the difference becomes larger, the model itself captures such changes and makes predictions more accurately. Figure 8 (a) is a schematic diagram of accuracy, Figure 8 (b) is a diagram of recall rate. Figure 8 (c) is a schematic diagram of the F1 Score results.

[0087] As for the length of the abnormal sequence, the performance of the proposed model increases with the length of the sequence, e.g. Figure 9 As shown in the figure. Notably, when the sequence length is within [200, 250] flow points, the model achieves an F1 score of approximately 95%, demonstrating that our model can capture long-range anomalies. BreakoutDetection and Telemon both exhibit fluctuating performance depending on the length of the anomaly sequence, while DeepAnt's performance increases slightly with increasing sequence length. BreakoutDetection achieves better recall in some cases, while simultaneously exhibiting significantly lower precision, demonstrating the model's superiority. Figure 9 (a) is a schematic diagram of accuracy, Figure 9 (b) is a diagram of recall rate.Figure 9 (c) is a schematic diagram of the F1 Score results.

[0088] Next, we analyze the performance of different data sizes, such as Figure 10 As shown in the figure, the model performs slightly better with more input data. However, the performance difference for 3 months of data is minimal compared to 12 months of data, which is computationally much more expensive because the runtime scales linearly with the data size. Similarly, Breakout Detection shows a slight increase in performance with increasing data size, while the model outperforms it in F1 score by approximately 10%. Compared to the other two systems, Telemon and DeepAnt achieve F1 scores below 50% and 20%, respectively, indicating that their performance is worse. Figure 10 (a) is a schematic diagram of accuracy, Figure 10 (b) is a diagram of recall rate. Figure 10 (c) is a schematic diagram of the F1 Score results.

[0089] Performance analysis of the RL model of the present invention: In this set of experiments, we only focus on the performance of the model of the present invention under different parameter settings. We mainly study the effects of changing two parameters: m and I, which are state The number of time steps and the size of the sliding window used for smoothing in post-processing.

[0090] First, to study how the number of time steps in the state affects the performance of the model, we change the parameter from 10 to 50. Since each time step refers to a 3-minute stream aggregation, for the state sequence, we basically consider the past 30, 60, 90, 120, and 150 minutes of streams. Figure 11 The results of the model's precision, recall, and F1 score are shown. Figure 11 (a) is a schematic diagram of accuracy, Figure 11 (b) is a diagram of recall rate. Figure 11 (c) is a diagram of the F1 Score results. All three metrics show a slight decrease in performance as the state includes more time steps. m =10 and m = 20, the performance difference is not significant, with the Fl score around 87% in both cases. It was observed that traffic trends up to 60 minutes long are sufficient to indicate whether the current traffic is abnormal, while longer traffic series may add irrelevant information, resulting in poor performance. Therefore, the default value of the time step is set to 20.

[0091] Results show that threshold-based methods perform very poorly, producing low F1 scores ranging from 24.91% to 31.02%, while the RL model produces an F1 score of 87.23%. Larger thresholds lead to lower recall and therefore more missed detections, while smaller thresholds lead to lower precision and therefore more false positives. This highlights the challenge of choosing an appropriate threshold. The scale of normality scores varies significantly across datasets, making it even more difficult to determine the correct threshold. This experiment reaffirms the motivation for pursuing threshold-free anomaly detection models. RL models address this need by learning to set a reward signal based on the data, without requiring any external threshold setting.

[0092] This paper proposes to develop an artificial intelligence (AI) intelligent system for anomaly detection based on reinforcement learning (RL). The system can accurately identify the starting and ending points of abnormal intervals in mountain traffic time series data (starting point: the time point when abnormal traffic begins to appear; ending point: the time point when abnormal traffic ends) by learning a series of decision-making processes, without relying on defined rules or ground truth labels.

[0093] Specifically, the present invention contributes as follows: 1. Unsupervised Reward Learning: We develop a novel data-driven reward function that automatically learns rewards based on the distribution of the data. This differs from existing RL-based anomaly detection models, which define rewards based on known anomalies in the streaming data and require ground truth labels. By implementing this unsupervised reward learning approach, our RL model does not require ground truth labels. 2. Threshold-Free: Most existing anomaly detection models require predefined or self-tuning thresholds to identify anomalies. In contrast, our model is threshold-free, making it more flexible and adaptable. 3. Generalization: Our model is location-independent and can be easily applied to different locations without requiring much human intervention. To train the model for new detector locations, we simply feed the new streaming time series data into our RL model. The RL agent then interacts with the data to find the optimal anomaly detection strategy for the new data pattern. 4. Results: We conduct a comprehensive series of experiments on both real-world and synthetic data and compare our model with three state-of-the-art models. The results demonstrate that our model significantly outperforms the other models across a variety of parameter settings. Overall, the model proposed in this paper can achieve 90% precision, 80% recall and 85% F1 score.

[0094] An embodiment of the present invention provides a traffic flow anomaly detection system, comprising: The model building module is used to connect the output of the feature extraction model with the input of the deep Q network DQN to build a traffic flow prediction model.

[0095] The model training module is used to use the kernel density estimation method KDE to obtain the probability density estimate of the characteristics of traffic flow data as a normality score; determine whether the data is normal based on the normality score and obtain a determination result; use the case where the DQN prediction result of the characteristics of the traffic flow data is consistent with the determination result as a positive reward, and use the case where the prediction result is inconsistent with the determination result as a negative reward to construct a data-driven reward function for autonomous learning; use the data-driven reward function to replace the DQN reward function, and use historical traffic flow data to train the traffic flow prediction model to obtain a trained traffic flow prediction model.

[0096] The anomaly prediction module is used to obtain real-time traffic flow data and predict real-time traffic flow anomalies based on the trained traffic flow prediction model.

[0097] A specific embodiment is as follows: The present invention discloses a method for detecting traffic flow anomalies, which specifically comprises the following steps: S1. Traffic flow data includes data related to traffic flow and data on external factors that affect traffic flow.

[0098] S2. Perform feature extraction engineering, including: traffic flow feature extraction, and extracting features that affect traffic flow from the collected external factor data.

[0099] S3. Build a traffic flow prediction model.

[0100] S4. Use unsupervised reward learning mechanism during model training.

[0101] S5. Real-time prediction and dynamic adjustment.

[0102] S6. Model training and optimization.

[0103] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A method for detecting traffic flow anomalies, characterized in that: The following steps are involved: Connect the output of the feature extraction model to the input of the deep Q-network (DQN) to build a traffic flow prediction model. The kernel density estimation method KDE is used to obtain the probability density estimate of the characteristics of traffic flow data as the normality score; Determine whether the data is normal based on the normality score and obtain a confirmed result; use the case where the DQN's prediction result of the traffic flow data characteristics is consistent with the confirmed result as a positive reward, and the case where the prediction result is inconsistent with the confirmed result as a negative reward, to build a data-driven reward function that achieves autonomous learning; Use a data-driven reward function to replace the DQN reward function, and use historical traffic flow data to train the traffic flow prediction model to obtain a trained traffic flow prediction model; Obtain real-time traffic flow data and predict real-time traffic flow anomalies based on the trained traffic flow prediction model.

2. The method for detecting traffic flow anomalies according to claim 1, wherein: The method for detecting traffic flow anomalies further includes: Historical traffic flow data was divided by season and by weekdays and non-workdays to obtain seasonal variation characteristics of traffic flow and the difference characteristics between weekdays and non-workdays. Data on external factors affecting traffic flow were divided by the severity of weather, the impact of accidents, and the impact of construction areas to obtain the characteristics of severe weather, the severity of accidents, and the impact of construction areas. The weighted score of each feature is obtained according to the product between the eigenvalue corresponding to each feature and the weight corresponding to each feature; The degree of influence of each feature on traffic flow is determined based on the weighted score of each feature.

3. The method for detecting traffic flow anomalies according to claim 1, wherein: The data-driven reward function for autonomous learning is constructed, and the specific steps include: Based on the traffic flow data used in training , and historical traffic flow data Historical traffic flow data for the same period , get the set ; right Perform the KDE clustering operation using the kernel density estimation method to obtain Cluster ; Get normality score , the formula is: in, yes The number of clusters in , is the average cluster size; According to the normality score, we can obtain the data-driven reward function r t+1 , the formula is: in, Predict results for the model; t is the time step.

4. The method for detecting traffic flow anomalies according to claim 1, wherein: The specific steps of using historical traffic flow data to train the traffic flow prediction model include: Based on historical traffic flow data, LSTM is used as a function approximator to obtain an intermediate vector representation containing time series features, and the intermediate vector representation is input into DQN; Set the state space to ;in, is the time step t Observations of and including current traffic flow , historical average traffic and selected actions The triplet of Set the action space to A= , represents whether the traffic flow is considered abnormal; where 1 is abnormal and 0 is normal; According to the data-driven reward function, as well as the state space and action space, we adopt - Greedy strategy training DQN obtains traffic flow anomalies by intermediate vector representation and obtains trained DQN; A bidirectional sliding window smoothing operation is used on the trained DQN to eliminate the - False positives caused by random actions generated by the greedy strategy; DQN trained using the Adam optimizer; Obtain the trained traffic flow prediction model.

5. The method for detecting traffic flow anomaly according to claim 1, wherein: The specific steps of obtaining real-time traffic flow data include: Remove outliers and process missing values ​​from real-time traffic flow data and data on external factors affecting traffic flow to complete data cleaning; Normalize the data.

6. The method for detecting traffic flow anomaly according to claim 1, wherein: The feature extraction model is a long short-term memory network LSTM.

7. A traffic flow anomaly detection system, characterized in that: include: A model building module is used to connect the output of the feature extraction model to the input of the deep Q network (DQN) to build a traffic flow prediction model; A model training module is used to obtain the probability density estimate of the characteristics of traffic flow data as the normality score using the kernel density estimation method KDE; Determine whether the data is normal based on the normality score and obtain a confirmed result; use the case where the DQN's prediction result of the traffic flow data characteristics is consistent with the confirmed result as a positive reward, and the case where the prediction result is inconsistent with the confirmed result as a negative reward, to construct a data-driven reward function that achieves autonomous learning; use the data-driven reward function to replace the DQN reward function, and use historical traffic flow data to train the traffic flow prediction model to obtain the trained traffic flow prediction model; The anomaly prediction module is used to obtain real-time traffic flow data and predict real-time traffic flow anomalies based on the trained traffic flow prediction model.

Images