Missing traffic flow data processing method and system based on energy diffusion model

Abstract

The present invention discloses a method and system for processing missing traffic flow data based on an energy diffusion model, which belongs to the field of intelligent traffic data processing technology. The method preprocesses the input sensor data, locates and interpolates the missing flow position based on the energy flow diffusion relationship between road network nodes, simulates the real energy flow changes between road network nodes through the energy flow diffusion relationship between road network nodes, constructs an energy diffusion model, and outputs the interpolated synthetic data set. The present invention uses the energy flow principle and combines the road network node flow characteristics to interpolate the missing traffic flow trend, solving the problem of missing information in energy flow sampling.

Description

Technical Field

[0001] The present invention proposes a missing traffic flow data processing method and system based on an energy diffusion model, and relates to the technical field of intelligent traffic data information detection. Background Art

[0002] With the development of intelligent transportation systems, high-precision prediction technologies have become increasingly important. The development of these technologies has an important impact on urban computing, planning, and urban development planning, including population migration. They play a decisive role in the development of the economy, cultural environment, and educational resources of a region or city. However, in actual operations, due to factors such as road reconstruction in urban planning, failures in urban power systems, or failures in sensors, time series data are often missing or abnormal, increasing traffic delays during travel, leading to frequent traffic accidents, and inefficient traffic operations also lead to energy waste and environmental pollution, reducing the happiness index of urban residents. In this case, traffic analysis tasks based on incomplete data sets cannot fully demonstrate the potential performance of the network, and may even lead to misjudgment of the model. In reality, since collecting traffic data is usually costly, it becomes extremely difficult to obtain high-quality data sets or valuable traffic information. Therefore, how to effectively use existing incomplete data sets, interpolate missing traffic data by learning the complex spatiotemporal distribution inherent in the data, and generate high-quality traffic information for downstream task analysis has become a major challenge in scientific research progress.

[0003] In recent years, there has been a lot of work on designing targeted models for traffic networks in different scenarios, and these models have shown good performance. The RNN model can effectively extract temporal features from spatiotemporal traffic data. The GNN network-based model can effectively learn non-Euclidean topological structures and can learn spatially unique data features. These excellent variants have proven their unique advantages in current research. Although the above methods can achieve significantly advanced performance in the field of traffic prediction, these prediction methods inevitably have a common limitation. They all rely on complete traffic data, which means that the model's learning generalization ability is extremely severely limited. Similarly, in real life, the data itself may be missing or abnormal, which inevitably causes obstacles to model learning, which also poses a challenge to the robustness of the model itself.

[0004] Traffic data interpolation methods focus more on spatiotemporal modeling and can be roughly divided into four categories. The first category is traditional methods, which are generally based on zero filling, mean or interpolation methods to fill in. The existing EM algorithm estimates missing values ​​based on historical time series. Although statistical methods provide rigorous mathematical proofs, due to the unique spatiotemporal data characteristics of the road network itself, traditional methods often cannot express its own complex spatiotemporal dependency characteristics, which intuitively does not express the deep representation of the network. And because the data itself changes with the time dimension, traditional methods cannot capture the characteristics of these changes well. The second category is machine learning methods. Methods such as VAE and KNN often determine missing values ​​based on traditional Euclidean distance, which limits the application of the model and is not suitable for processing large-scale spatiotemporal data tasks. The third is based on matrix feature interpretation. Existing work believes that the original high-dimensional data contains many low-rank matrices. By decomposing high-dimensional matrices into low-rank matrices, the underlying spatiotemporal dependencies are extracted through low-rank matrices. Although this indicates deep spatiotemporal dependencies to some extent, the application of the model is often limited by the sparsity of the data itself and the high computational complexity of time series tasks. Moreover, the method based on matrix feature decomposition does not consider the spatial topological relationship among nodes in the real road network, which leads to its lack of generalization and the lack of robustness of the network in complex missing conditions. Finally, there are deep learning methods that have emerged in recent years. For example, the GMHANN model effectively processes the spatial and temporal characteristics of traffic flow data by combining the multi-head attention mechanism and the gated recurrent unit (AGRU), improving the accuracy and long-term stability of traffic flow prediction. However, its RNN architecture itself often causes cumulative errors during training when the data distribution is not obvious or the data itself has serious missing values.

[0005] Existing studies have shown that it is not only difficult to capture the dynamic temporal correlation and complex spatial dependency of traffic prediction tasks with missing values, but also causes the problem of error accumulation during training. The existing paper TimeGrad designs a denoising module based on the RNN structure and encodes the hidden state obtained by encoding the historical sequence and external variables as the input of the module, thereby guiding the process of the diffusion model to generate the sequence. The existing paper IAGCN proposes a solution for spatiotemporal traffic data kriging based on an inductive adaptive graph convolutional network, which can effectively model the unknown nodes in the latent space in time and space. Although these technologies have shown good performance, the complex task design and specialized model structure limit their robustness. The recent success of the diffusion model has led scholars to apply it to the field of time series. The existing paper CSDI uses the diffusion model to model the time series filling task. First, the missing sequence is filled with noise, and then the diffusion model is used to gradually predict the noise to achieve denoising. After multiple steps, the filling result is finally obtained. The existing paper PriSTI constructs priors and interpolates spatiotemporal data based on the extracted conditional features and geographic features. Although existing studies have focused on specific tasks and achieved some achievements, they still face some inevitable problems.

[0006] First, most of these studies are designed based on specific scenarios and lack sufficient scalability and generalization capabilities. Although they have achieved success in specific fields, the robustness of the model is not ideal when the data distribution is not obvious or there is data missing.

[0007] Secondly, these studies have been successful based on static graphs and some assumptions, which limits the ability of the model to capture the dynamic changes of the road network. The network changes dynamically due to maintenance or other reasons, but it is difficult for static graphs to effectively capture such topological changes. Although some studies have analyzed the dynamics of roads and designed high-performance task frameworks, these are usually based on known node data, which limits the generalization ability of the model. In addition, existing studies mainly focus on capturing the spatial characteristics between nodes, but fail to explain how these characteristics express the dynamic correlation between nodes.

[0008] Finally, many studies rely on accurate network node information and have extremely high requirements on data quality. For example, when node information is unavailable or lost, these methods are difficult to reconstruct accurate spatiotemporal data characteristics of road networks. Summary of the invention

[0009] In order to solve the above technical problems, the present invention proposes a method for processing missing traffic flow data based on an energy diffusion model, which is characterized by comprising the following steps:

[0010] S1, preprocessing the input sensor data, locating and interpolating the missing flow positions based on the energy flow diffusion relationship between road network nodes;

[0011] S2, through the energy flow diffusion relationship between road network nodes, simulates the real energy flow changes between road network nodes, builds an energy diffusion model, and outputs the interpolated synthetic data set.

[0012] Furthermore, in step S1, the position missing positioning matrix Locate the missing point, the missing position is for:

[0013] ;

[0014] In the formula, When , it means that the data point is not missing, otherwise it is missing. i and j represent the row and column numbers of the position missing positioning matrix.

[0015] Furthermore, the position missing positioning moment is combined with the original data matrix X to update and obtain the target sequence to be interpolated :

[0016] ;

[0017] In the formula, represents an all-ones matrix.

[0018] Furthermore, the expression of energy flow diffusion relationship between road network nodes is:

[0019] ;

[0020] ;

[0021] In the formula, For the original data Data with missing values ​​generated cumulatively at T moments The forward diffusion process, represents the sensor data at the current time t, represents the sensor data at time t-1, represents the forward diffusion cumulative data; N represents Gaussian distribution, Describes a state variable In the previous state The surrounding is normally distributed. represents the scaling constant factor for adding noise intensity, Indicates how much the variance is expected to vary around the mean.

[0022] Furthermore, in step S2, the original data matrix X and the interpolated target sequence The corresponding embedding features are obtained through the fully connected layer , using the energy flow relationship, the position embedding information P1 between road network nodes is expressed as , and its reverse process is expressed as , where A represents the adjacency matrix composed of road network nodes.

[0023] Furthermore, the process of aggregating the initially collected road network nodes is as follows:

[0024] ;

[0025] In the formula, ET is the data after aggregation, represents the number of clustering layers, represents trainable parameters;

[0026] By continuously collecting the collected road network node data, the global energy information is collected. for:

[0027] ;

[0028] In the formula, express Activation function, Represents trainable weight and bias matrices;

[0029] Output comprehensive energy information characteristics The expression is:

[0030] ;

[0031] In the formula, Indicates the aggregation operation between road network nodes. represents a learnable parameter.

[0032] Furthermore, the collected local road network node data and comprehensive energy information features are combined to obtain heterogeneous energy information for:

[0033] ;

[0034] In the formula, represents the nonlinear function of the energy reverse conductor, Represented as the temporal and spatial characteristics of the road network nodes themselves.

[0035] Furthermore, the final target interpolation feature for:

[0036] ;

[0037] Will As a priori condition for generating interpolation values, the formula is expressed as:

[0038] ;

[0039] ;

[0040] In the formula, represents the reverse conduction process of the diffusion model, represents the road time series data at the current time t, Represents the data with missing values ​​generated cumulatively at T moments, represents the noise sequence for reverse reasoning, Represents a trainable parameter.

[0041] The present invention also proposes a missing traffic flow data processing system based on an energy diffusion model, which implements the missing traffic flow data processing method based on the energy diffusion model. The system comprises:

[0042] A data preprocessing module is used to preprocess the input sensor data and locate and interpolate the missing flow positions based on the energy flow diffusion relationship between road network nodes;

[0043] The energy diffusion model construction module simulates the actual energy flow changes between network nodes through the energy flow diffusion relationship between road network nodes, constructs an energy diffusion model, and outputs the interpolated synthetic data set.

[0044] Furthermore, the energy diffusion model construction module includes: a road energy information collector, an information completer and an energy reverse conductor; the road energy information collector is used to collect data features of the original data matrix X, the information completer is used to locate and interpolate the missing flow positions in the road energy information collector, and the energy reverse conductor is used to generate an interpolated synthetic data set that conforms to the characteristics of the original data.

[0045] Compared with the prior art, the present invention has the following beneficial technical effects:

[0046] The present invention combines the diffusion generation model and proposes for the first time a missing traffic flow data processing strategy based on the energy diffusion model. By integrating the road energy information collector and utilizing the characteristics of the energy flow diffusion relationship of the traffic density between road nodes in the road network, the real energy flow changes between nodes are simulated, and the missing positions in the spatiotemporal data set of the road nodes are located, thereby solving the problem that the robustness of the model is not ideal when the data distribution is not obvious or there is data missing. The information completer collects the energy information of the global nodes of the road network, and combines the energy information characteristics of the nodes themselves. The preliminary energy collection layer, the global information aggregation layer, and the energy entropy output layer in the information completer are used for global energy collection, respectively, and the temporal and spatial heterogeneous information of the overall road node information is captured in combination with the energy flow relationship between individual nodes, so that the generalization ability of the model is further improved, and it itself has significant adaptability and robustness. The energy diffusion network is trained to approximate the real data distribution through the energy reverse conductor, and the accurate spatiotemporal data characteristics of the road network are reconstructed.

[0047] Compared with the existing technology, the present invention further improves the accuracy and applicability of traffic time series prediction, greatly enhances the prediction ability of existing models in the case of missing data, and provides an explainable solution from a physical perspective: the high accuracy of the E-diffusion model is used to improve the accuracy of traffic time series prediction, thereby providing a more accurate and personalized prediction solution for smart transportation.

[0048] This paper proposes a missing traffic flow data processing (E-Diffusion) strategy based on the energy diffusion model. The samples generated by E-Diffusion approximate the original road node data distribution. At the same time, the model energy utilization diffusion model solves the problem that the robustness of the model is not ideal when the data distribution is not obvious or there is data missing. Experiments were conducted on two data sets, METR-LA and PEMS-BAY. The experimental results show that the strategy has good performance indicators in overall performance and good versatility.

[0049] In view of the fact that the existing models are designed for specific scenarios and lack sufficient scalability and generalization capabilities, an energy diffusion model is proposed to effectively expand the generalization ability and poor robustness of the existing prediction models.

[0050] In view of the fact that data distribution is not obvious or there is data missing in the existing data sets, it is proposed that the road energy information collector uses the energy flow diffusion relationship of traffic density between road nodes in the road network to collect the nonlinear characteristics between spatiotemporal data, which effectively solves the problem of missing data interpolation in the data set.

[0051] In response to the problems of long calculation time, poor effect and accumulated error in existing interpolation algorithms, a model based on energy diffusion model is proposed to train the network model to approach the actual distribution of missing data, which solves the problems of poor performance of prediction model due to large-scale missing data and long time and poor quality of interpolation data output by interpolation model.

[0052] In response to the data missing problem in existing public data sets, which leads to poor robustness of research model algorithms, this paper proposes an innovative deep learning model - "Energy Diffusion Neural Network" (E-Diffusion), which is specially designed to effectively fill in traffic data sets with missing values. The model consists of a road energy information collector, an information completer and an energy reverse conductor, which can fill in the complex nonlinear spatiotemporal traffic network characteristics in the real world to the greatest extent and improve the prediction accuracy of downstream prediction tasks.

[0053] In the test on METR-LA and PEMS-BAY datasets, E-Diffusion showed excellent interpolation performance. The present invention conducted interpolation experiments on METR-LA and PEMS-BAY datasets. The experimental results showed that the MSE reached 7.35 / 0.95 and the MAE reached 1.65 / 0.49 respectively, and the model effect was better than all the existing model effects. After generating the interpolation dataset, we performed a prediction task on METR-LA. The experimental results showed that the RMSE reached 7.45 and the MAE reached 5.42, and the model effect was better than all the existing model effects. These data indicate the significant ability of E-Diffusion in generating high-quality interpolation data. Compared with other existing interpolation network methods, E-Diffusion demonstrates synthesis quality and diversity that surpass all models.

[0054] The high interpolation quality and generalization of the energy diffusion model of the present invention have great potential in the field of time series data, especially in traffic data sets with missing values. Due to its unique model architecture, it is more applicable to intelligent transportation systems and a wide range of scenarios, and also highlights the important role of technological innovation in improving the quality of life of urban residents. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0056] Figure 1is a flow chart of a method for processing missing traffic flow data based on an energy diffusion model provided by an embodiment of the present invention;

[0057] Figure 2 is a flow chart of the preprocessing stage provided by an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of an energy diffusion model provided by an embodiment of the present invention;

[0059] Figure 4 This is a rendering of a synthetic data set of an energy diffusion model provided by an embodiment of the present invention;

[0060] Figure 5 It is an experimental prediction effect diagram provided by an embodiment of the present invention;

[0061] Figure 6 It is a schematic diagram comparing the effect of the energy diffusion model of the present invention and the prediction result of the IAGCN model. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] In the drawings of the specific embodiments of the present invention, in order to better and more clearly describe the working principles of the various components in the system, the connection relationship of the various parts in the device is shown, which only clearly distinguishes the relative position relationship between the various components, and cannot constitute a limitation on the signal transmission direction, connection sequence and structural size, size and shape of each part within the component or structure.

[0064] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0065] like Figure 1 As shown, the missing traffic flow data processing method based on the energy diffusion model provided by the embodiment of the present invention includes:

[0066] S1. Preprocess the input sensor data and locate and interpolate the missing flow positions based on the energy flow diffusion relationship between road network nodes.

[0067] In this step, the METR-LA dataset is first collected. The METR-LA dataset contains a total of 207 target road nodes. The dataset records a total of 34,727 timestamp traffic features from March to June 2012. The missing data in the dataset accounts for about 25% of the total data. The dataset has been proven to be effective and is used as a typical dataset in the field of traffic interpolation and prediction.

[0068] In order to effectively determine the missing positions in the data set, the position missing positioning matrix is ​​defined as , by position missing positioning matrix Locate the missing points.

[0069] Position missing location matrix for:

[0070] ;

[0071] In the formula, When , it means that the data point is not missing, otherwise it is missing. i and j are the row and column numbers of the position missing positioning matrix, representing the position (i, j) of the road network node.

[0072] In a preferred embodiment, for missing nodes in the original data matrix X, the position missing positioning matrix is ​​combined with the original data matrix X to update the target sequence to be interpolated , expressed as:

[0073] ;

[0074] In the formula, Represents an all-ones matrix, that is, a matrix with the same shape as the original matrix but with all matrix elements set to 1.

[0075] Based on the above analysis, the expression of the energy flow diffusion relationship between road network nodes in this embodiment is:

[0076] ;

[0077] In the formula, For the original data Data with missing values ​​generated cumulatively at T moments The forward diffusion process, represents the sensor data at the current time t, represents the sensor data at time t-1, Represents the forward diffusion cumulative data.

[0078] In order to meet the general form of the road network system, Decompose it and its characteristics satisfy the Gaussian distribution. The calculation formula for each time is expressed as:

[0079] ;

[0080] Where N represents Gaussian distribution, Describes a state variable In the previous state The surrounding is normally distributed. represents the scaling constant factor for adding noise intensity, Indicates how much the variance is expected to vary around the mean.

[0081] After comprehensive verification of the system, the present invention confirms that the selected missing traffic flow data processing system based on the energy diffusion model can effectively describe the energy flow relationship between the nodes of the traffic system, and through the energy diffusion effect, it can effectively capture the missing conditions between systems and the actual spatiotemporal characteristics of traffic flow.

[0082] S2, through the energy flow diffusion relationship between road network nodes, simulates the real energy flow changes between network nodes, builds an energy diffusion model, and outputs the interpolated synthetic data set.

[0083] like Figure 2 As shown, the energy diffusion model building module is used to solve the road flow prediction task problem with missing values ​​in the data set.

[0084] The energy diffusion model construction module includes: a road energy information collector, an information completer and an energy reverse conductor, wherein the road energy information collector is used to collect data features of the original data matrix X, the information completer is used to supplement the missing information in the road energy information collector, and finally the energy reverse conductor is used to use the collected guidance information as the guidance condition information for reverse denoising of the model, so as to generate an interpolated synthetic data set that meets the characteristics of the original data.

[0085] For the road energy information collector, this component converts the original data matrix X and the interpolated target sequence The corresponding embedding features are obtained through the fully connected layer , the energy flow relationship is used to further express the location embedding information between road network nodes as , and its reverse process is expressed as ,in Represents the adjacency matrix of road network nodes;

[0086] The information completer includes: a preliminary energy collection layer, a global information aggregation layer, and an energy entropy output layer, which are used for global energy collection and capture the temporal and spatial heterogeneous information of the overall road node information in combination with the energy flow relationship between individual nodes. The process of aggregating the preliminary collected road network nodes is as follows:

[0087] ;

[0088] In the formula, The information completer initially collects the road network node data and aggregates them. K represents the number of aggregation layers. Represents a trainable parameter.

[0089] The global information aggregation layer of the information completer collects global energy information by continuously collecting the collected road network node data. for:

[0090] ;

[0091] In the formula, is the global energy information obtained by the information completer by collecting the global node energy flow information, express Activation function, represents the trainable weight and bias matrices, It means that the location information between road network nodes is embedded by using the energy flow relationship.

[0092] The energy entropy output layer of the information completer can output the comprehensive energy information characteristics containing the node itself and the global energy information. The expression is:

[0093] ;

[0094] In the formula, Indicates the aggregation operation between road network nodes. Represents the learnable parameters. During the training process, the preliminary energy collection layer, the global information aggregation layer, and the energy entropy output layer of the information completer are trained together, and the iterative parameters are updated at the same time.

[0095] The energy reverse conductor is used to combine the collected local and global energy relationship information to obtain more comprehensive heterogeneous energy information of the road network, and locate and interpolate the missing information according to the flow of energy, effectively filling the missing sensor information and fully restoring the original heterogeneous spatiotemporal data.

[0096] Combine the collected local road network node data and comprehensive energy information features to obtain more comprehensive heterogeneous energy information of the road network The expression is:

[0097] ;

[0098] In the formula, Indicates the output state of the energy reverse conductor, represents the nonlinear function of the energy reverse conductor, It is represented by the time and space characteristics of the road node itself, and its expressions are:

[0099] ;

[0100] In the formula, for Activation function, is the normalization parameter, is the information value, is the true value of the tth sample The importance of the time features in the original time series, is the query matrix, For content that requires attention, is a click operation; calculate exist The attention weights on Scaling, similarly, ,in, , , , represents the trainable information parameter matrix, , , Represents a trainable query parameter matrix.

[0101] Therefore, the target feature Combine them to get the final target interpolation features:

[0102] ;

[0103] Different from the general diffusion reverse process, here As a priori condition for generating interpolation values, the formula is expressed as:

[0104] ;

[0105] ;

[0106] In the formula represents the reverse conduction process of the diffusion model, Represents the road time series data at the current moment, Represents the data with missing values ​​generated cumulatively at T moments, Represents the original data features and target interpolation information features. Among them, represents the noise sequence for reverse reasoning, represents the trainable parameters, which can be expressed by the following formula:

[0107] ;

[0108] ;

[0109] In the formula, represents the conditional mean function, which represents the conditional mean function under given historical information. The expected value of represents the scaling constant factor of the added noise intensity, represents the functional form of the denoising network, represents the variance of the random variable, Represents a trainable parameter.

[0110] The mean square error (MSE) function is used to evaluate the performance of the road energy information collector, the quality of the interpolation data generated by the information completer and the energy reverse conductor. The smaller the MSE, the higher the quality of the generated interpolation data. The MSE calculation formula is:

[0111] ;

[0112] In the formula, is the sample size, For the The true value of the samples, is the interpolated sample value of the tth sample obtained through training of the road energy information collector, information completer and energy reverse conductor.

[0113] In a preferred embodiment, the energy diffusion model needs to be trained, and the training process of outputting the interpolated synthetic data set includes:

[0114] A1, input road node sensor information with missing values , the maximum number of iterations is , ; Train the road energy information collector and information completer to minimize the MSE value;

[0115] A2, training the road energy information collector and information completer, by collecting energy information features between nodes to provide information guidance for the missing data generated by the energy reverse conductor, so as to minimize the RMSE value;

[0116] A3, alternately train the road energy information collector and the information completer to make the energy reverse conductor converge to the correct imitation direction. During the training process, the energy reverse conductor minimizes the RMSE and iterates times, until the maximum number of iterations ;

[0117] A4, generate interpolated synthetic datasets based on conditional constraints , where the conditional constraints include:

[0118] ;

[0119] In the formula, It represents the minimization difference function of the interpolation data between the road energy information collector and the information completer and the energy reverse conductor. represents the mean square error loss function, Represents the true value and the predicted value.

[0120] Finally, output the interpolated synthetic dataset .

[0121] A5, judging whether the interpolated data meets the original data distribution result based on the obtained interpolated synthetic data set, and saving the judgment result.

[0122] The training process ends when the error interval between the actual value and the predicted value gradually approaches a stable state and gradually approaches a mean value:

[0123] The root mean square error RMSE calculation formula is:

[0124] ;

[0125] The mean absolute error MAE calculation formula is:

[0126] ;

[0127] In the formula, RMSE is the root mean square error, MAE is the mean absolute error, is the total number of samples, is the true value, is the predicted value.

[0128] Example 2

[0129] like Figure 4 As shown, the missing traffic flow data processing system based on the energy diffusion model provided by the embodiment of the present invention includes:

[0130] A data preprocessing module is used to preprocess the input sensor data and locate and interpolate the missing flow positions based on the energy flow diffusion relationship between road network nodes;

[0131] The energy diffusion model construction module simulates the actual energy flow changes between network nodes through the energy flow diffusion relationship between road network nodes, constructs an energy diffusion model, and outputs the interpolated synthetic data set.

[0132] The energy diffusion model construction module includes: a road energy information collector, an information completer and an energy reverse conductor; the road energy information collector is used to collect data features of the original data matrix X, the information completer is used to locate and interpolate the missing flow positions in the road energy information collector, and the energy reverse conductor is used to generate an interpolated synthetic data set that conforms to the characteristics of the original data.

[0133] It can be seen from the above embodiments that, firstly, the present invention is committed to solving the problem of missing values ​​in existing data sets. After the data set is interpolated using the present invention, all subsequent downstream tasks will obtain more realistic and appropriate indicator evaluations.

[0134] Example 3: Random missing and mixed missing experiments were performed on the METR-LA and PEMS-BAY datasets with a missing rate of 20%.

[0135] The experimental design includes two situations: one is the random loss of sensor node information in the data set, and the other is the mixed loss of sensor node information in the data set. The experiment sets the missing rate in two modes, and the missing rate ranges from 10% to 90%. The experimental results show that as the missing rate of the data set increases, the present invention can still fill in the missing data and maintain excellent evaluation index parameters.

[0136] As shown in Tables 1 and 2, the present invention conducts random missing and mixed missing experiments on the METR-LA and PEMS-BAY datasets with a missing rate of 20%. By comparing with the most advanced interpolation algorithm, the experimental results show that the performance of the network model proposed by the present invention is better than other model indicators. In general, the MSE reaches 7.35 / 0.95 and the MAE reaches 1.65 / 0.49 in the case of random missing, and the MSE reaches 10.28 / 3.10 and the MAE reaches 1.75 / 0.75 in the case of mixed missing.

[0137] Table 1 Interpolation result table (Random):

[0138]

[0139] Table 2 Interpolation results table (Hybrid):

[0140]

[0141] At the same time, the present invention uses dimensionality reduction technology to visualize the interpolation samples. In order to address the impact of excessive missing rate on the performance degradation of the model, it is found in the experiment that when the missing rate exceeds half, the model training will lead to poor results, while the low missing rate does not meet the actual missing situation. Therefore, the results are trained under the conditions of 20% random missing and mixed missing, and examples of interpolation samples in both cases are generated. At the same time, they are compared with the real data. Figure 5 It can be seen intuitively that the synthetic data and the real data visually present similar signal distributions, which verifies the effectiveness of the model proposed in this invention in generating high-quality data sets from data sets with missing values ​​in the traffic field.

[0142] At the same time, the present invention compares the prediction results of the energy diffusion model with those of the same IAGCN model. Figure 6 The prediction effect comparison of node traffic signal in the METR-LA data set (compared with node 23) is composed of a road energy information collector, an information completer, and an energy reverse conductor. It can effectively solve the problem that the robustness of the model is not ideal when the data distribution of the existing model is not obvious or there is data missing.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0144] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing missing traffic flow data based on an energy diffusion model, characterized in that: The steps include: S1, preprocessing the input sensor data, locating and interpolating the missing flow positions based on the energy flow diffusion relationship between road network nodes; Positioning matrix by position missing Locate the missing point: ； In the formula, When , it means that there is no missing data point, otherwise it is missing. i and j represent the row and column numbers of the missing location matrix; Target sequence to be interpolated for: ； In the formula, represents an all-one matrix, X is the original data matrix; S2, through the energy flow diffusion relationship between road network nodes, simulates the real energy flow changes between road network nodes, builds an energy diffusion model, and outputs the interpolated synthetic data set; The original data matrix X and the target sequence to be interpolated The corresponding embedding features are obtained through the fully connected layer , , using the energy flow relationship, the position embedding information P1 between road network nodes is expressed as , and its reverse process is expressed as , where A represents the adjacency matrix composed of road network nodes; The process of aggregating the initially collected road network nodes is as follows: ； In the formula, ET is the data after aggregation, represents the number of clustering layers, , represents trainable parameters; By continuously collecting the collected road network node data, the global energy information is collected. for: ； In the formula, express Activation function, , Represents trainable weight and bias matrices; Output comprehensive energy information characteristics The expression is: ； In the formula, Indicates the aggregation operation between road network nodes. represents a learnable parameter; The local road network node data and comprehensive energy information features Combination to obtain heterogeneous energy information of road networks : ； In the formula, Indicates the output state of the energy reverse conductor, represents the nonlinear function of the energy reverse conductor, , Represents the temporal and spatial characteristics of the road node itself; Final target interpolation feature for: ； in, , is the embedded feature.

2. The missing traffic flow data processing method based on the energy diffusion model according to claim 1 is characterized in that: The expression of energy flow diffusion relationship between road network nodes is: ； In the formula, For the original data Data with missing values ​​generated cumulatively at T moments The forward diffusion process, represents the sensor data at the current time t, represents the sensor data at time t-1, represents the forward diffusion cumulative data; N represents Gaussian distribution, Describes a state variable In the previous state The surrounding is normally distributed. represents the scaling constant factor for adding noise intensity, Indicates how much the variance is expected to vary around the mean.

3. The missing traffic flow data processing method based on the energy diffusion model according to claim 1 is characterized in that: Will , As a priori condition for generating interpolation values, the formula is expressed as: ； In the formula, represents the reverse conduction process of the diffusion model, represents the road time series data at the current time t, Represents the data with missing values ​​generated cumulatively at T moments, represents the noise sequence for reverse reasoning, Represents a trainable parameter.

4. The missing traffic flow data processing system based on energy diffusion model is characterized by: The system implements the missing traffic flow data processing method based on the energy diffusion model as claimed in any one of claims 1 to 3, and the system comprises: A data preprocessing module is used to preprocess the input sensor data and locate and interpolate the missing flow positions based on the energy flow diffusion relationship between road network nodes; The energy diffusion model construction module simulates the actual energy flow changes between network nodes through the energy flow diffusion relationship between road network nodes, constructs an energy diffusion model, and outputs an interpolated synthetic data set; the energy diffusion model construction module includes: a road energy information collector, an information completer and an energy reverse conductor; the road energy information collector is used to collect data features of the original data matrix X, the information completer is used to locate and interpolate the missing flow positions in the road energy information collector, and the energy reverse conductor is used to generate an interpolated synthetic data set that meets the features of the original data.

Images