Flight delay prediction method, device and medium based on spatiotemporal multi-modal fusion

Abstract

The application relates to the field of computer technology, and provides a flight delay prediction method, equipment and medium based on space-time multi-modal fusion, which comprises the following steps: firstly, flight delay correlation data and airport basic information of multiple airports within a preset time window are acquired; then, a node embedding matrix is generated; and three space dependence matrices are constructed based on route physical connection, historical collaborative delay rate and space-time relationship. Time feature extraction is performed on the pretreated flight data through a time feature extraction module, and a layer-by-layer graph learning module is used to fuse the node embedding and the multi-source space dependence matrix, so that hierarchical learning of space features is realized. Finally, the time and space features are integrated through a multi-modal feature fusion module, and the flight delay time prediction result is obtained by inputting the delay prediction module. Through joint modeling of the multi-modal space-time features, the method can effectively capture the space-time propagation law of flight delays in the airport network, and improve the flight delay prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer technology application, and in particular to a flight delay prediction method, device and medium based on spatiotemporal multi-modal fusion. BACKGROUND

[0002] In the aviation operation system, flight delay problem has been a key factor affecting the quality of aviation service and operation efficiency. The airport network as a complex system has a high degree of correlation in time and space dimensions, and the generation and propagation of flight delay are deeply influenced by this spatiotemporal correlation characteristic. To cope with the flight delay prediction problem, relevant research has derived complex spatiotemporal graph convolution network models, such as STGCN and STSGCN. These models attempt to use spatiotemporal graph convolution network to mine the rules of flight delay in spatiotemporal distribution, so as to realize the prediction of delay situation. However, the existing graph convolution model has obvious shortcomings. On the one hand, it mainly focuses on the static correlation between airport nodes, and does not fully consider the dynamic change characteristic of the influence degree of different airports in the airport network over time. In actual aviation operation, in different time periods, the intensity and mode of mutual influence of airports will dynamically change due to factors such as flight peak, weather change, air space control adjustment, and the static correlation assumption is difficult to accurately describe this complex change. On the other hand, the mutual influence between airports is the result of comprehensive action of multiple factors. Although the existing model involves some factors, it does not consider them comprehensively. The complexity of airport network determines that the mutual influence is not only restricted by geographical distance - the airports with short geographical distance may have more direct influence due to flight flow interaction and air space overlap; it also involves historical delay time, and the airports with frequent and interrelated historical delays are also easily affected by each other due to chain reaction in subsequent operation; at the same time, flight flow is also a key factor, and the flight scheduling and resource competition between high-flow airports will significantly change the degree of mutual influence, and the existing model has shortcomings in mining and application of the synergistic effect of these multiple factors. Therefore, the existing flight delay prediction technology has shortcomings in describing the spatiotemporal dynamic correlation and multiple factor comprehensive influence of airport network, and it is difficult to meet the actual demand of accurately predicting the spatiotemporal distribution of flight delay, and an innovative technical solution is needed to break through the bottleneck. SUMMARY

[0003] In view of the above technical problems, the technical scheme adopted by the present application is as follows:

[0004] The embodiment of the present application provides a flight delay prediction method based on spatiotemporal multi-modal fusion, which is realized based on a trained spatiotemporal prediction model of layer-by-layer graph convolution. The spatiotemporal prediction model based on layer-by-layer graph convolution includes a time feature extraction module, a layer-by-layer graph learning module, a multi-modal feature fusion module and a delay prediction module. The method comprises the following steps:

[0005] S100, acquire a prediction data set and airport basic information; the prediction data set includes flight delay correlation data of multiple airports in w time windows in the future.

[0006] S200, based on the airport basic information, acquire an airport correlation network, and acquire three kinds of space dependent matrices based on physical routes, historical collaborative delay rates and space relations, respectively denoted as a physical connection matrix, a time sequence correlation matrix and a space interaction matrix.

[0007] S300, pre-process the prediction data set to obtain a pre-processed data set, and input into a time feature extraction module to extract time features and send to the multi-modal feature fusion module.

[0008] S400, input the node embedding matrix of the airport correlation network, the physical connection matrix, the time sequence correlation matrix and the space interaction matrix into the layer-by-layer graph learning module to acquire space features through layer-by-layer graph feature learning, and send to the multi-modal feature fusion module.

[0009] S500, use the multi-modal feature fusion module to fuse the time features and the space features to obtain multi-modal fusion features, and send to the delay prediction module to acquire a prediction target corresponding to the prediction data set, the prediction target being a flight delay time.

[0010] The present application has at least the following beneficial effects:

[0011] The flight delay prediction method based on spatio-temporal multi-modal fusion provided by the embodiment of the present application is realized based on a trained layer-by-layer graph convolution spatio-temporal prediction model, and the model includes a time feature extraction module, a layer-by-layer graph learning module, a multi-modal feature fusion module and a delay prediction module. The method first acquires flight delay correlation data of multiple airports in a preset time window and airport basic information, then generates a node embedding matrix, and constructs three kinds of space dependent matrices based on route physical connection, historical collaborative delay rate and space-time relation. The time feature extraction module mines time features from the pre-processed flight data, and the layer-by-layer graph learning module fuses node embedding and multi-source space dependent matrices to realize layered learning of space features. Finally, the multi-modal feature fusion module integrates time and space features, and inputs into the delay prediction module to obtain a flight delay time prediction result. The method can effectively capture the spatio-temporal propagation law of flight delays in the airport network through joint modeling of multi-modal spatio-temporal features, and improve the flight delay prediction accuracy.

[0012] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0014] Figure 1 A flowchart of a flight delay prediction method based on spatiotemporal multi-modal fusion provided by the embodiments of the present application. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the scope of protection of the present application.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this description, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0017] It should be noted that some of the example embodiments are described as processes that are depicted as a flowchart or a method. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations can be re-arranged. A process can be terminated when its operations are completed, but could also occur under some other condition or event. The processes might correspond to methods, functions, procedures, subroutines, subprograms, etc. When processes correspond to functions, the functions can be the basis of computer programs, software, and / or the like.

[0018] The embodiments of the present application provide a flight delay prediction method based on spatiotemporal multi-modal fusion, which is realized based on a trained spatiotemporal prediction model of layer-by-layer graph convolution. The spatiotemporal prediction model based on layer-by-layer graph convolution includes a time feature extraction module, a layer-by-layer graph learning module, a multi-modal feature fusion module and a delay prediction module.

[0019] In the embodiments of the present application, the time feature extraction module includes a time sequence convolution network and an encoder connected in sequence.

[0020] In the embodiment of the present application, the time sequence convolution network is used to extract the time sequence features in the input sequence data and capture long and short term dependencies. The time sequence convolution network can include a first convolution module, a second convolution module, a one-dimensional convolution layer and a splicing module, wherein the first convolution module and the second convolution module are connected in sequence and have the same structure. Each convolution block includes a dilated causal convolution layer, a weight normalization layer and a batch normalization layer connected in sequence, an activation function layer and a dropout layer. The dropout layer of the first convolution block is connected with the dilated causal convolution layer of the second convolution block, that is, the output of the dropout layer of the first convolution block is taken as the input of the dilated causal convolution layer of the second convolution block, and the one-dimensional convolution layer and the dropout layer of the second convolution block are connected with the splicing module.

[0021] In the embodiment of the present application, the activation function layer uses the ReLU activation function for processing. The splicing module is used to splice the outputs of the one-dimensional convolution layer and the dropout layer of the second convolution block. The sequence data input into the time sequence convolution network will be divided into two paths: one path expands the receptive field through the dilated causal convolution layer to capture long-time dependence, and after weight normalization and batch normalization, the ReLU activation function and the dropout layer are used to reduce overfitting; the other path performs residual connection operation. Finally, the outputs of the two paths are added to obtain the final output of the time sequence convolution network.

[0022] In the embodiment of the present application, the encoder is a transform architecture, which is used to further model the complex dependencies in the time dimension through the self-attention mechanism. The encoder can include a position encoding module, a multi-head attention mechanism layer, a first combination module, a feedforward neural network layer and a second combination module connected in sequence.

[0023] The position encoding module is used to position encode the output features of the time sequence convolution network to determine the relative position relationship of each element in the sequence. In the embodiment of the present application, the sine-cosine position encoding can be used to position encode the output features of the time sequence convolution network. This encoding provides continuous time information input while maintaining sensitivity to different time scales, which is an efficient method for processing complex flight data.

[0024] In the embodiment of the present application, the multi-head attention mechanism layer realizes the information capture of different semantic subspaces by mapping the input features to multiple subspaces (heads) for parallel attention calculation and splicing the output results. The first combination module and the second combination module have the same structure and are combinations of residual connection and layer normalization. The first combination module is connected with the position encoding module and the multi-head attention mechanism layer, and is used to perform layer normalization processing after residual connection of the output of the position encoding module and the output of the multi-head attention mechanism layer. The second combination module is connected with the first combination module and the feedforward neural network, and is used to perform layer normalization processing after residual connection of the output of the first combination module and the output of the feedforward neural network.

[0025] Those skilled in the art should understand that the working principle of the multi-head attention mechanism, residual connection and layer normalization combination (Add&Norm) and the feedforward neural network layer belongs to the prior art.

[0026] In the embodiment of the application, the layer-by-layer graph learning module can include a first layer-by-layer learning network and a second layer-by-layer learning network, the first layer-by-layer learning network includes a first heterogeneous graph convolution network, a second heterogeneous graph convolution network and a third heterogeneous graph convolution network, and the second layer-by-layer learning network includes a first heterogeneous graph feature enhancement network, a second heterogeneous graph feature enhancement network and a third heterogeneous graph feature enhancement network.

[0027] Further, the delay prediction module can include a bidirectional long short-term memory network and a fully connected layer connected in sequence. The bidirectional long short-term memory network is composed of two independent long short-term memory network layers, which are a forward long short-term memory network layer and a reverse long short-term memory network layer. The forward long short-term memory network layer processes the forward sequence to generate a forward hidden state, and the reverse long short-term memory network layer processes the reverse sequence to generate a reverse hidden state. The output of the bidirectional long short-term memory network is a spliced feature obtained by splicing the forward hidden state and the reverse hidden state.

[0028] Further, as shown in Figure 1 The method can include the following steps:

[0029] S100, acquiring a prediction data set and airport basic information; the prediction data set includes flight delay correlation data of a plurality of airports in w future time windows.

[0030] In the embodiment of the application, the flight delay correlation data can include basic attributes of the flight, time-related features, space-related features, running states and environmental conditions. The basic attributes include flight number, aircraft tail number, starting airport, destination airport, etc. The time-related features include scheduled departure time, departure delay time, aircraft taxi-out time, aircraft wheel closing time, aircraft wheel opening time, aircraft taxi-in time, scheduled arrival time, scheduled flight time, actual flight time, in-flight flight time, etc. The space-related features include control flight distance. The environmental conditions include meteorological parameters of the starting airport and meteorological parameters of the destination airport.

[0031] Those skilled in the art should understand that the flight delay correlation data in the w future time windows can be obtained based on the flight schedule. The length of each time window can be one day, and the w future time windows are w days in the future. w can be set based on actual needs, and can be an integer greater than or equal to 1, for example, w can be set to 1, 3 or 7, etc.

[0032] In the embodiment of the application, the airport basic information can include the geographical position and route information of the airport, etc.

[0033] S200, based on the airport basic information, an airport correlation network is obtained, and three kinds of space dependent matrices based on physical routes, historical collaborative delay rates and spatial relationships are obtained, respectively denoted as a physical connection matrix, a time sequence correlation matrix and a spatial interaction matrix.

[0034] In the embodiment of the application, the nodes of the airport correlation network represent an airport, and the number of nodes is the number of airports. If there is a route between two airports, there is an edge between the corresponding two nodes. The characteristics of each node can include the average departure delay time, the average arrival delay time, the average aircraft sliding-in time, the average aircraft sliding-out time, the average wheel opening time, the average wheel closing time, the average temperature, the average humidity, the average wind speed, the precipitation, the visibility and the weather phenomenon of the airport in the future w time windows.

[0035] In the embodiment of the application, the weather phenomenon can be a historically occurring weather phenomenon, which can include fog, heavy rain, moderate rain, light rain, heavy snow, light snow, tornado, funnel cloud, etc. The weather phenomenon is represented in the form of a vector, and the length of the vector is the number of weather phenomena. If a certain weather phenomenon occurs, the corresponding position is 1, otherwise it is 0. The characteristics corresponding to the weather phenomenon can be the weighted sum of all weather phenomena, and the weight of each weather phenomenon can be determined based on actual conditions.

[0036] Further, the node embedding matrix of the airport correlation network satisfies the following conditions:

[0037] X={X1,X2,……,X r ,……,X n}。

[0038] Wherein, X is the node embedding matrix, X r is the feature vector of the rth airport node in the airport correlation network, r takes a value of 1 to n, and n is the number of airports; X r ={X r1 ,X r2 ,……,X rj ,……,X rm},X rj is the jth feature in X r , X rj =∑ w k=1 w rj (k)•x rj (k),x rj (k) is the value of the jth feature in the kth time window in the future w time windows, j takes a value of 1 to m, and m is the number of node characteristics. k takes a value of 1 to w, w rj (k) is the weight corresponding to x rj (k)

[0039] Further, w rj (k) = x rj (k) / å w k=1 x rp (k). x rp (k) is the value of the pth feature in X r in the kth time window in the future w time windows, and p is 1 to m.

[0040] In the embodiment of the application, the physical connection matrix captures the flight delay propagation by representing the connection relationship between airports.

[0041] Further, the physical connection matrix satisfies the following condition: Ag ;

[0042] Wherein, Ag is the physical connection matrix, and a ig is the flight route identifier between the ith airport and the gth airport, and a ig = 1 if there is a flight route between the ith airport and the gth airport, otherwise a ig = 0, i is 1 to n, g is 1 to n, and n is the number of airports.

[0043] The airport historical delay time reflects the propagation characteristics of flight delays between airports. If there is a large delay time on the historical flight route between two airports, the propagation possibility of flight delays between the two airports is also high. Based on this, a time sequence correlation matrix based on historical collaborative delay rate is constructed.

[0044] Further, the time sequence correlation matrix satisfies the following condition: At ;

[0045] Wherein, At is the time sequence correlation matrix, and b ig is the flight delay influence strength of the ith airport on the gth airport in the future w time windows. b ig = (F ig -minF ig ) / (maxF ig -minF ig ), F ig is the average value of the w average arrival delay times corresponding to the w time windows in the future w time windows of the flight from the ith airport to the gth airport, minF ig is the minimum value of the w average arrival delay times corresponding to the w time windows in the future w time windows of the flight from the ith airport to the gth airport, and maxF ig is the maximum value of the w average arrival delay times corresponding to the w time windows in the future w time windows of the flight from the ith airport to the gth airport.

[0046] Flight delays often do not occur in isolation, and delays can spread from one airport to other connected airports, especially when flight traffic is large or the distance between airports is close, the speed of delay propagation will be accelerated. By combining traffic relations and distance relations, a spatial interaction matrix based on spatial relations is constructed to comprehensively consider multiple influencing factors between airports.

[0047] Further, the spatial interaction matrix satisfies the following condition: Af= ;

[0048] Wherein, Af is the spatial interaction matrix, c ig is the normalized flight traffic of the i th airport to the g th airport in the future w time window, c ig = (c ig -minc) / (maxc-minc), wherein minc is the minimum value in matrix B, maxc is the maximum value in matrix B, B= , is the actual flight traffic of the i th airport to the g th airport in the future w time window. d ig is the normalized route distance between the i th airport and the g th airport, d ig = (d ig -mind) / (maxd-mind), wherein mind is the minimum value in matrix C, maxd is the maximum value in matrix C, C= , is the actual route distance between the i th airport and the g th airport.

[0049] In the embodiment of the application, the flight traffic refers to the number of flights. Those skilled in the art should understand that if there is no route between two airports, the corresponding normalized flight traffic is 0. If there are multiple routes between two airports, the route distance between the two airports can be the average of the route distances of the multiple routes. S300, the prediction data set is preprocessed to obtain a preprocessed data set, and is input into a time feature extraction module to extract time features and sent to the multi-modal feature fusion module.

[0050] In the embodiment of the application, preprocessing can include data cleaning, encoding conversion, normalization and the like to convert the prediction data set into a feature vector recognizable by a computer.

[0051] Those skilled in the art should understand that the method of using a time series convolution network and an encoder to extract features from input time series data belongs to the prior art.

[0052] S400, input the node embedding matrix, the physical connection matrix, the time sequence correlation matrix and the space interaction matrix of the airport correlation network into the layer-by-layer graph learning module to obtain the space feature by layer-by-layer graph feature learning and send to the multi-modal feature fusion module.

[0053] In the embodiment of the application, the layer-by-layer graph learning module combines a graph convolution network (GCN) and a graph attention mechanism (GAT) to mine the topological correlation and dynamic influence between airports in an alternating manner. The graph convolution network captures the spatial dependency in the airport network, and the graph attention mechanism dynamically learns the interaction between different airports to optimize feature learning.

[0054] Further, S400 specifically comprises:

[0055] S401, input the node embedding matrix and the physical connection matrix into a first heterogeneous graph convolution network to obtain a first graph convolution feature, and input the node embedding matrix and the physical connection matrix into a first heterogeneous graph feature enhancement network to obtain a first graph attention feature.

[0056] S402, fuse the first graph convolution feature and the first graph attention feature to obtain a first fusion feature, input the first fusion feature and the time sequence correlation matrix into a second heterogeneous graph convolution network to obtain a second graph convolution feature, and input the first fusion feature and the time sequence correlation matrix into a second heterogeneous graph feature enhancement network to obtain a second graph attention feature.

[0057] S403, fuse the second graph convolution feature and the second graph attention feature to obtain a second fusion feature, input the second fusion feature and the space interaction matrix into a third heterogeneous graph convolution network to obtain a third graph convolution feature, and input the second fusion feature and the space interaction matrix into a third heterogeneous graph feature enhancement network to obtain a third graph attention feature.

[0058] S404, fuse the third graph convolution feature and the third graph attention feature to obtain a third fusion feature as the space feature.

[0059] In the embodiment of the present application, the output feature HG1 of the first heterogeneous graph convolution network is: HG1 = σ (Ag•X•WG1). Wherein, WG1 is the weight matrix of the first heterogeneous graph convolution network, σ () is the ReLU activation function, and • represents dot multiplication. The output feature HG2 of the second heterogeneous graph convolution network is: HG2 = σ (At•HG1•WG2). Wherein, WG2 is the weight matrix of the second heterogeneous graph convolution network. The output feature HG3 of the third heterogeneous graph convolution network is: HG3 = σ (Af•HG2•WG3). Wherein, WG3 is the weight matrix of the third heterogeneous graph convolution network.

[0060] In the embodiment of the present application, the output feature HT1 of the first heterogeneous graph feature enhancement network is: HT1 = (HT11, HT12, …, HT1 p , …, HT1 n ), HT1 p is the feature of the pth airport node obtained by the first heterogeneous graph feature enhancement network, p is 1 to n, HT1 p = σ (∑ N1（p） q1=1 f1 pq1 •WT1•h1 q1 ), f1 pq1 is the attention coefficient between the pth airport node and the q1th adjacent node of the pth airport node obtained by the first heterogeneous graph feature enhancement network, q1 is 1 to N1 (p), N1 (p) is the number of neighbor nodes of the pth airport node based on the physical connection matrix, wherein, in the physical connection matrix, if α pe = 1, it means that the e th airport node is the neighbor node of the pth airport node, otherwise, it is not the neighbor node of the pth airport node, e is 1 to n. Wherein, . Wherein, exp () is the natural exponential function, LeakyReLU () is the LeakyReLU activation function, f1 T is the attention vector of the first heterogeneous graph feature enhancement network, WT1 is the weight matrix of the first heterogeneous graph feature enhancement network, h1 p is the feature of the pth airport node based on X, h1 q1 is the feature of the q1th airport node based on X, h1 u1 is the feature of the u1th airport node based on X, u1 is 1 to N1 (p), and || represents splicing.

[0061] In the embodiment of the present application, the output feature HT2 of the second heterogeneous graph feature enhancement network is: HT2 = (HT21, HT22, …, HT2 p , …, HT2 n ), HT2p is the feature of the pth airport node obtained by the second heterogeneous graph feature enhancement network. HT2 p =σ(∑ N2（p） q2=1 f2 pq2 •WT2•h2 q2 ), f2 pq2 is the attention coefficient between the pth airport node and the q2th adjacent node of the pth airport node obtained by the second heterogeneous graph feature enhancement network, q2 is 1 to N2(p), N2(p) is the number of neighbor nodes of the pth airport node based on the time correlation matrix, wherein, in the time correlation matrix, if β pe ≠0, it indicates that the e th airport node is a neighbor node of the pth airport node, otherwise, it is not a neighbor node of the pth airport node.

[0062] wherein, , wherein f2 T is the attention vector of the second heterogeneous graph feature enhancement network, WT2 is the weight matrix of the second heterogeneous graph feature enhancement network, h2 p is the feature of the pth airport node based on HT1, h2 q2 is the feature of the q2th airport node based on HT1, h2 u2 is the feature of the u2th airport node based on HT1, u2 is 1 to N2(p).

[0063] In the embodiment of the application, the output feature HT3 of the third heterogeneous graph feature enhancement network is: HT3=(HT31, HT32, …, HT3 p , …, HT3 n ), HT3 p is the feature of the pth airport node obtained by the third heterogeneous graph feature enhancement network. HT3 p =σ(∑ N3（p） q3=1 f3 pq3 •WT3•h3 q3 ), f3 pq3 is the attention coefficient between the pth airport node and the q3th adjacent node of the pth airport node obtained by the third heterogeneous graph feature enhancement network, q3 is 1 to N3(p), N3(p) is the number of neighbor nodes of the pth airport node based on the space interaction matrix, wherein, in the space interaction matrix, if c pe •d pe ≠0, it indicates that the e th airport node is a neighbor node of the pth airport node, otherwise, it is not a neighbor node of the pth airport node.

[0064] wherein, wherein f3 T is an attention vector of the third heterogeneous graph feature enhancement network, WT3 is a weight matrix of the third heterogeneous graph feature enhancement network, h3 p is a feature of the pth airport node based on HT2, h3 q3 is a feature of the q3th airport node based on HT2, h3 u3 is a feature of the u3th airport node based on HT2, u3 is an integer from 1 to N3 (p).

[0065] In the embodiment of the present application, the output features of each heterogeneous graph convolution network and each heterogeneous graph feature enhancement network can be fused by a fusion module. The fusion module can include a fusion layer, a fully connected layer and a ReLU activation function layer. The fusion features obtained after the output features of each heterogeneous graph convolution network and each heterogeneous graph feature enhancement network are fused by the fusion module satisfy the following condition: HM = ReLU (Wf•contact (HG, HT) + bf).

[0066] wherein HM is the fusion feature, HG is the output feature of each heterogeneous graph convolution network, HT is the output feature of each heterogeneous graph feature enhancement network, Wf is the weight matrix of the fully connected layer, bf is the bias vector of the fully connected layer, and contact () represents the concatenation operation.

[0067] S500, the time feature and the space feature are fused by the multi-modal feature fusion module to obtain a multi-modal fusion feature, and the multi-modal fusion feature is sent to a delay prediction module to obtain a predicted target corresponding to the prediction data set, the predicted target being a flight delay time, specifically a flight arrival delay time.

[0068] In the embodiment of the present application, the multi-modal fusion feature obtained after the time feature and the space feature are fused by the multi-modal feature fusion module satisfies the following condition: HMS = (Ws•contact (HM, HS) + bs). HMS is the multi-modal fusion feature, HS is the time feature, Ws is the weight matrix of the multi-modal feature fusion module, and bs is the bias vector of the multi-modal feature fusion module.

[0069] Those skilled in the art should understand that the method of using a bidirectional long short-term memory network and a fully connected layer to process the input multi-modal fusion feature to obtain the flight delay time belongs to the prior art.

[0070] In summary, the flight delay prediction method based on spatio-temporal multi-modal fusion provided by the embodiment of the application can significantly enhance the capturing ability of time sequence features in flight delay prediction by combining a time sequence convolution network and a Transformer mechanism. In addition, the model introduces a heterogeneous graph convolution network and a graph attention network, can better simulate the mutual relationship between airports and its influence by constructing a multi-type airport graph structure, and uses a specific aggregation method for node embedding to deeply mine the topological relationship and dynamic influence between airports in a layer-by-layer alternating manner. Through this layer-by-layer alternating mechanism, the model can capture more fine-grained dynamic change information at different levels. Finally, the model can improve the flight delay prediction accuracy by fusing the extracted spatio-temporal features into a unified high-dimensional feature representation and inputting it into a bidirectional long short-term memory network for flight delay prediction.

[0071] In the embodiment of the application, the trained layer-by-layer graph convolution spatio-temporal prediction model can be trained using a sample data set to obtain an initial layer-by-layer graph convolution spatio-temporal prediction model. In the embodiment of the application, the sample data set can be a public data set. The flight-related data is flight punctuality data provided by the United States Department of Transportation, and the selected data is historical flight data for the whole year of 2023, totaling more than 6 million flight records. The weather history data is from the United States National Oceanic and Atmospheric Administration. Since the data set itself contains a large amount of feature missing and contains abnormal value data, for the row of missing weather data, the mean value of the previous three days is used to fill in, if there is no data for the previous three days, the data is deleted, and for the flight data, the row containing abnormal value data is directly deleted.

[0072] In the training process, the input time sequence window length can be set to 7 days, that is, the sample data of each batch is 7 days of data, and the time window length of the prediction target can be set to 1 day, 3 days or 5 days, etc.

[0073] Those skilled in the art should understand that the method of training the initial layer-by-layer graph convolution spatio-temporal prediction model using the sample data set belongs to the category of prior art.

[0074] The flight delay prediction method based on spatio-temporal multi-modal fusion provided by the embodiment of the application takes the sample data set as an example, and the prediction error is reduced by about 3% compared with the existing prediction model.

[0075] The embodiment of the application also provides an electronic device, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the application.

[0076] The embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used for executing the method described in the embodiment of the present application.

[0077] It should be understood that the steps shown above can be reordered, added to, or deleted from, using various forms of flow. For example, the steps described in the present application can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present application can be achieved, which is not limited herein.

[0078] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A flight delay prediction method based on spatio-temporal multi-modal fusion, characterized in that, The method is realized based on a trained layer-by-layer graph convolutional spatiotemporal prediction model, the layer-by-layer graph convolutional spatiotemporal prediction model comprises a time feature extraction module, a layer-by-layer graph learning module, a multi-modal feature fusion module and a delay prediction module, and the method comprises the following steps: S100, acquiring a prediction data set and airport basic information; the prediction data set comprises flight delay correlation data of multiple airports within w time windows in the future; S200, acquiring an airport correlation network based on the airport basic information, and acquiring three kinds of spatial dependence matrices based on a physical route, a historical collaborative delay rate and a spatial relationship, which are respectively denoted as a physical connection matrix, a time sequence correlation matrix and a spatial interaction matrix; S300, preprocessing the prediction data set to obtain a preprocessed data set, and inputting the preprocessed data set into the time feature extraction module to extract time features and send the time features to the multi-modal feature fusion module; S400, inputting a node embedding matrix of the airport correlation network, the physical connection matrix, the time sequence correlation matrix and the spatial interaction matrix into the layer-by-layer graph learning module to acquire spatial features through layer-by-layer graph feature learning and send the spatial features to the multi-modal feature fusion module; S500, using the multi-modal feature fusion module to perform feature fusion on the time features and the spatial features to obtain multi-modal fusion features, and sending the multi-modal fusion features to the delay prediction module to acquire a prediction target corresponding to the prediction data set, the prediction target being a flight delay time.

2. The method of claim 1, wherein, The layer-by-layer graph learning module comprises a first layer-by-layer learning network and a second layer-by-layer learning network, the first layer-by-layer learning network comprises a first heterogeneous graph convolutional network, a second heterogeneous graph convolutional network and a third heterogeneous graph convolutional network, and the second layer-by-layer learning network comprises a first heterogeneous graph feature enhancement network, a second heterogeneous graph feature enhancement network and a third heterogeneous graph feature enhancement network.

3. The method of claim 2, wherein, S400 specifically comprises: S401, inputting the node embedding matrix and the physical connection matrix into the first heterogeneous graph convolutional network to acquire first graph convolutional features, and inputting the node embedding matrix and the physical connection matrix into the first heterogeneous graph feature enhancement network to acquire first graph attention features; S402, performing feature fusion on the first graph convolutional features and the first graph attention features to obtain first fusion features, inputting the first fusion features and the time sequence correlation matrix into the second heterogeneous graph convolutional network to acquire second graph convolutional features, and inputting the first fusion features and the time sequence correlation matrix into the second heterogeneous graph feature enhancement network to acquire second graph attention features; S403, performing feature fusion on the second graph convolutional features and the second graph attention features to obtain second fusion features, inputting the second fusion features and the spatial interaction matrix into the third heterogeneous graph convolutional network to acquire third graph convolutional features, and inputting the second fusion features and the spatial interaction matrix into the third heterogeneous graph feature enhancement network to acquire third graph attention features; S404, performing feature fusion on the third graph convolution feature and the third graph attention feature to obtain a third fusion feature as the spatial feature.

4. The method of claim 1, wherein, The time feature extraction module comprises a time sequence convolution network and an encoder connected in sequence.

5. The method of claim 1, wherein, The delay prediction module comprises a bidirectional long short-term memory network and a full connection layer connected in sequence.

6. An electronic device, comprising: comprise a processor and a memory; The processor is configured to execute the steps of the method according to any one of claims 1 to 5 by invoking programs or instructions stored in the memory.

7. A computer readable storage medium characterized in that, The computer readable storage medium is configured to store programs or instructions, which enable a computer to execute the steps of the method according to any one of claims 1 to 5.

Images