A multimodal transportation recommendation method based on heterogeneous graph neural network

Abstract

The present invention discloses a multimodal traffic recommendation method based on a heterogeneous graph neural network, including obtaining historical travel data and processing to obtain a training data set; constructing a start-destination multi-bipartite graph embedding scheme; constructing a heterogeneous traffic map; designing a hierarchical attention mechanism with a residual network; constructing a multimodal traffic recommendation initial model and training to obtain a multimodal traffic recommendation model based on a heterogeneous graph neural network; and using the multimodal traffic recommendation model based on a heterogeneous graph neural network to perform real-time multimodal traffic recommendation. The method of the present invention can explicitly capture the association between entities and capture the potential connections between different entities in a hierarchical manner from multiple perspectives, while solving the problem that the model cannot distinguish node features, and can effectively improve the accuracy of the multimodal traffic recommendation problem, with high recommendation accuracy and good reliability.

Description

Technical Field

[0001] The present invention belongs to the field of digital information processing, and in particular relates to a multimodal traffic recommendation method based on a heterogeneous graph neural network. Background Art

[0002] In recent years, with the development of economy and technology, traffic recommendation (navigation application) has been widely used in people's production and life, bringing endless convenience to people's production and life. In the field of traffic recommendation technology, multimodal traffic recommendation is the core component of many navigation applications; research on multimodal traffic recommendation has received extensive attention from transportation departments, academia and industry.

[0003] In navigation planning, unimodal transportation recommendation, which only considers one mode of transportation (such as bicycle, bus, taxi or subway), has been widely studied. In contrast, multimodal transportation recommendation research is still in its early stages; this type of transportation recommendation generally considers both a single mode of transportation (such as bicycle, bus, taxi and subway) and mixed modes of transportation (such as taxi-bus, taxi-subway) in the road network, and unimodal transportation recommendation schemes cannot be directly applied to multimodal transportation recommendation scenarios.

[0004] In response to the above problems, researchers have proposed personalized route recommendation method FAVOUR, multimodal transportation mode recommendation method Trans2vec, multimodal transportation recommendation solution Hydra and other technical solutions; moreover, solutions such as Trans2vec and Hydra have been commercialized. However, the current technical solutions all have the problem of over-smoothing: when using graph neural networks, as the number of network layers and iterations increase, all nodes eventually converge to the same value, resulting in a decrease in the model's representation ability. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal traffic recommendation method based on heterogeneous graph neural network with high recommendation accuracy and good reliability.

[0006] The multimodal traffic recommendation method based on heterogeneous graph neural network provided by the present invention comprises the following steps:

[0007] S1. Obtain historical travel data and process it to obtain a training data set;

[0008] S2. Construct an embedding scheme for the origin-destination multi-bipartite graph;

[0009] S3. construct a heterogeneous traffic graph according to the constructed origin-destination multi-bipartite graph embedding scheme;

[0010] S4. Design a hierarchical attention mechanism with residual networks to address the over-smoothing problem;

[0011] S5. Aggregate and process the results of step S4 to obtain the final multimodal transportation recommendation results and construct an initial model for multimodal transportation recommendation;

[0012] S6. Using the training data set obtained in step S1, the multimodal traffic recommendation initial model constructed in step S5 is trained to obtain a multimodal traffic recommendation model based on a heterogeneous graph neural network;

[0013] S7. Use the multimodal traffic recommendation model based on heterogeneous graph neural network obtained in step S6 to perform real-time multimodal traffic recommendation.

[0014] The acquisition of historical travel data in step S1 and processing to obtain a training data set specifically includes the following steps:

[0015] The historical travel data includes user attribute data, starting point data, destination data, situational awareness data and transportation mode data, and is saved in a time series format;

[0016] According to historical travel data, the corresponding starting points and destinations are combined into starting point-destination pairs (OD pairs); traffic data sets are collected and converted into graph structure data; the graph structure data includes users, transportation modes, OD pairs, starting points, destination points, edges with relationships and driving semantics; node data is formally represented using the following formula:

[0017]

[0018]

[0019] Where X u is user attribute data, and N is the total data volume of the navigation data set, q is the dimension of the user attribute data; X o is the starting point data, and p is the dimension of the origin and destination; X d is the destination data, and X con is context-aware data, and k is the dimension of context data; X od is the OD pair data, and n is the dimension of OD pairs, and n=p+p.

[0020] The scheme for constructing the origin-destination multi-bipartite graph embedding described in step S2 specifically includes the following steps:

[0021] Extract the features of the origin o, destination d, OD pair od, and user u;

[0022] Based on the OD pairs, a bipartite graph of origin and destination is constructed, and information from neighboring nodes is aggregated in the GCN layer to represent the points of the bipartite graph. Based on the data information of the user's travel on the OD pairs, a bipartite graph of user-OD pairs is constructed, and the node information is updated through the GAT layer.

[0023] The GCN layer specifically updates the features of the origin and destination respectively through the following GCN model after obtaining the origin-destination bipartite graph:

[0024]

[0025]

[0026] In the formula As the starting point i The embedding of the l+1th layer; N(o i ) is the starting point o i degree; N(d j ) is the destination d j degree; Destination j The embedding of the lth layer of Destination i The embedding of the l+1th layer; N(d i ) is the destination d i degree; N(o j ) is the starting point o j degree; As the starting point j The embedding of the lth layer; when the OD pair exists, e od =e o ||e d ,|| is a merge operation;

[0027] The GAT layer is specifically that after obtaining the user-OD bipartite graph, the initialization representation matrix E of the user-OD bipartite graph is

[0028]

[0029] Where M is the number of OD pairs; Q is the number of users; is the initialization embedding of the jth OD pair, j = 1, 2, ..., M; is the initialization embedding of the i-th user, i=1,2,...,Q;

[0030] In order to obtain the user's characteristic information, the information propagated by the user u's neighbors is aggregated to update the representation of user u; the following aggregation function is defined

[0031]

[0032] In the formula is user u at layer l+1 i Embedding of; N(u i ) is u i Neighbor aggregation; W 1 (l) is the weight matrix obtained through training; For user u i andod j The attention value between

[0033] is user u in layer l i andod j The attention value between Softmax() is the normalized exponential function softmax, is the importance of the i-th user to the j-th OD pair in the l-th layer, and ρ() is the activation function LeakyReLU, a is the attention weight vector, and is the weight parameter obtained through training, for u i In the representation at level l, for j Representation at level l;

[0034] Through the GAT layer based on the user-OD bipartite graph, the user's characteristics are e u and e od ; The context information is normalized by standard to e m .

[0035] The step S3 described in which a heterogeneous traffic graph is constructed according to the constructed origin-destination multi-bipartite graph embedding scheme specifically includes the following steps:

[0036] According to the semantic structure (u, m, od) of navigation data, there is a user u using transportation mode m to travel on OD pair od to build a heterogeneous traffic graph. The initialization embedding vector of each node on the heterogeneous traffic graph is represented as e init =e u ||e od ||e m ,|| is a merge operation; e u is the initialization embedding vector of the user node, e od is the initialization embedding vector of the OD pair, e m Initialize embedding vectors for transportation modes;

[0037] The final embedding of the transportation mode is represented as Where E u is the user node embedding, and E od is the OD pair node embedding, and E m is the transportation mode node embedding, and f 1 is the size of the user node embedding dimension; f 2 is the size of the node embedding dimension of OD; f 3 is the size of the embedding dimension of the transportation mode node; e' m ∈ε m , e' m Multimodal embedding vectors for each mode of transportation.

[0038] Step S4 describes designing a hierarchical attention mechanism with a residual network to solve the problem of over-smoothing, which specifically includes the following steps:

[0039] The hierarchical attention mechanism with residual network includes residual network layer, node attention layer and weight attention layer;

[0040] The residual network layer is used to maintain the stability of network training; the value of the residual network layer Directly affected by the l+1th layer hx perspective traffic mode node embedding and the layer l hx perspective traffic mode node embedding Impact:

[0041]

[0042] Where hx is the node type, including ho and he, where ho represents a homogeneous node and he represents a heterogeneous node; ψ s is a given set of meta-paths; σ() is the ReLU activation function; is the trainable weight matrix of the hx perspective of the l+2th layer; b is the embedding of the hx perspective transportation mode node at the l+1th layer; b hx(l+2) is the bias vector of the hx perspective of the l+2th layer;

[0043] Node attention layer: The following formula is used to calculate the i-th transportation mode node embedding from the perspective of the l+1th layer hx:

[0044]

[0045] Where m j is the i-th transportation mode node m i Neighbor nodes; N(m i ) is the i-th transportation mode node m i The set of neighbor nodes of is m i and m j The attention value between is the trainable weight matrix of the lth layer;

[0046] Weighted attention layer: The final embedding of each transportation mode m is calculated by the following formula:

[0047]

[0048] Where softmax() is the normalized exponential function softmax.

[0049] The step S5 described in step S4 is aggregating and processing the results to obtain the final multimodal transportation recommendation results and construct the initial model of multimodal transportation recommendation, which specifically includes the following steps:

[0050] Design a fusion neural layer to aggregate the results of step S4 and input the results into the fully connected layer to achieve multimodal transportation recommendation;

[0051] Multimodal embedding e' with several fully connected layers m Used to predict the most suitable transportation mode m for user u to travel on od:

[0052]

[0053]

[0054] In the formula Embedding of transportation mode nodes for homogeneous perspective; is the node embedding of transportation modes from heterogeneous perspectives; || is the merging operation; softmax() is the softmax function; θ() is the activation function ReLU; is the weight matrix obtained through training; is the final representation of the transportation mode m at the lth level; b (l) is the bias vector; y' m Score the final prediction;

[0055] Sort the final scores of several modes of transportation used by the user to make multi-modal transportation recommendations for the user;

[0056] During the model training process, supervised training is performed, and the following cross entropy loss is used as the objective function Loss:

[0057]

[0058] Where C is the number of categories; is the real label; is the predicted label;

[0059] Under the guidance of labeled data, the proposed initial model of multimodal transportation recommendation is optimized through back propagation.

[0060] The multimodal traffic recommendation method based on heterogeneous graph neural network provided by the present invention converts the existing traffic data set into graph structure data to characterize the explicit associations between different entities such as users, traffic modes and OD pairs from the data level; at the same time, from the perspective of model architecture, the proposed hierarchical attention mechanism realizes the comprehensive capture of the potential relationships between different types of nodes and edges from heterogeneous and homogeneous perspectives; in addition, the residual network is designed to participate in model training, which alleviates the problem of node features converging to the same value due to the increase in the number of iterations; therefore, the method of the present invention can explicitly capture the associations between entities and capture the potential connections between different entities in a hierarchical manner from multiple perspectives, while solving the problem that the model cannot distinguish node features, and can effectively improve the accuracy of multimodal traffic recommendation problems, with high recommendation accuracy and good reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The figure is a schematic diagram of the method flow of the present invention.

[0062] Figure 2 Schematic diagram of the performance comparison results of the method of the present invention and the existing recommended method on F1, Pre, Rec and NDCG.

[0063] Figure 3 It is a schematic diagram of the grouping robustness verification effect of the method of the present invention.

[0064] Figure 4 Schematic diagram of the ablation experiment effect of the method of the present invention.

[0065] Figure 5 It is a schematic diagram of the rationality verification effect of multi-bipartite graph embedding of the method of the present invention. DETAILED DESCRIPTION

[0066] like Figure 1 The method flow diagram of the method of the present invention is shown as follows: The multimodal traffic recommendation method based on heterogeneous graph neural network provided by the present invention comprises the following steps:

[0067] S1. Obtain historical travel data and process it to obtain a training data set; specifically, the steps include:

[0068] The historical travel data includes user attribute data (including age, gender, occupation, etc.), starting point data (latitude and longitude coordinate data), destination data (latitude and longitude coordinate data), situational awareness data (weather data, etc.) and transportation mode data (transportation mode type, etc.), and is saved in a time series format;

[0069] According to historical travel data, the corresponding starting points and destinations are combined into starting point-destination pairs (OD pairs); traffic data sets are collected and converted into graph structure data; the graph structure data includes users, transportation modes, OD pairs, starting points, destination points, edges with relationships (with relationships such as "Take" and "Travel") and driving semantics; node data is formally represented using the following formula:

[0070]

[0071]

[0072] Where X u is user attribute data, and N is the total data volume of the navigation data set, q is the dimension of the user attribute data; X o is the starting point data, and p is the dimension of the origin and destination; X d is the destination data, and X con is context-aware data, and k is the dimension of context data; X od is the OD pair data, and n is the dimension of OD pairs, and n=p+p;

[0073] S2. Construct a multi-bipartite graph embedding scheme for the origin-destination region; specifically, the following steps are included:

[0074] Extract the features of the origin o, destination d, OD pair od, and user u;

[0075] Based on the OD pairs, a bipartite graph of origin and destination is constructed to mine the relationship between the origin and destination and the relationship between the OD pairs and the users. Information from neighbor nodes is aggregated in the GCN layer to represent the points of the bipartite graph. Based on the data information of the user traveling in the OD pairs, a bipartite graph of user-OD pairs is constructed, and the node information is updated through the GAT layer.

[0076] The GCN layer constructs an origin-destination bipartite graph based on historical data. Data analysis shows that the combination of origin and destination is highly random, and the representations of origin and destination are similar. The importance of distinguishing neighbors is no longer important, so GCN is suitable for extracting the embedding of origin and destination. Specifically, after obtaining the origin-destination bipartite graph, the features of the origin and destination are updated respectively through the following GCN model:

[0077]

[0078]

[0079] In the formula As the starting point i The embedding of the l+1th layer; N(o i ) is the starting point o i degree; N(d j ) is the destination d j degree; Destination j The embedding of the lth layer; Destination i The embedding of the l+1th layer; N(d i ) is the destination d i degree; N(o j ) is the starting point o j degree; As the starting point j The embedding of the lth layer; when the OD pair exists, e od =e o ||e d ,|| is a merge operation;

[0080] The GAT layer learns the embedding of users (or OD pairs) based on the user-OD bipartite graph. In the user-OD bipartite graph, the feature space and dimension of user nodes and OD pair nodes are very different, and it is found from the data that there is a certain regularity in the user's selection of OD. It is critical to introduce GAT to distinguish the importance of neighbors. Specifically, after obtaining the user-OD bipartite graph, the initialization representation matrix E of the user-OD bipartite graph is

[0081]

[0082] Where M is the number of OD pairs; Q is the number of users; is the initialization embedding of the jth OD pair, j = 1, 2, ..., M; is the initial embedding of the i-th user, i=1,2,...,Q;

[0083] In order to obtain the user's characteristic information, the information propagated by the user u's neighbors is aggregated to update the representation of user u; the following aggregation function is defined

[0084]

[0085] In the formula is user u at layer l+1 i Embedding of; N(u i ) is u i Neighbor aggregation; W 1 (l) is the weight matrix obtained through training; For user ui andod j The attention value between

[0086] is user u in layer l i andod j The attention value between Softmax() is the normalized exponential function softmax, is the importance of the i-th user to the j-th OD pair in the l-th layer, and ρ() is the activation function LeakyReLU, a is the attention weight vector, and is the weight parameter obtained through training, for u i In the representation at level l, for j Representation at level l;

[0087] Through the GAT layer based on the user-OD bipartite graph, the user's characteristics are e u and e od ; The context information is normalized by standard to e m ;

[0088] S3. Construct a heterogeneous traffic map according to the constructed origin-destination multi-bipartite graph embedding scheme; specifically, the steps include:

[0089] According to the semantic structure (u, m, od) of navigation data, there is a user u using transportation mode m to travel on OD pair od to build a heterogeneous traffic graph. The initialization embedding vector of each node on the heterogeneous traffic graph is represented as e init =e u ||e od ||e m ,|| is a merge operation; e u is the initialization embedding vector of the user node, e od is the initialization embedding vector of the OD pair, e m Initialize embedding vectors for transportation modes;

[0090] The final embedding of the transportation mode is represented as Where E u is the user node embedding, and E od is the OD pair node embedding, and E m is the transportation mode node embedding, and f 1 is the size of the user node embedding dimension; f 2 is the size of the node embedding dimension of OD; f 3is the size of the embedding dimension of the transportation mode node; e' m ∈ε m , e' m Multimodal embedding vectors for each mode of transportation;

[0091] S4. Design a hierarchical attention mechanism with residual networks to address the over-smoothing problem;

[0092] In order to make full use of the rich semantics of heterogeneous traffic graphs, this application designs a hierarchical attention mechanism with residual blocks, which includes ResNet layers, node attention layers and weight attention layers; this mechanism can mine potential features from homogeneous and heterogeneous perspectives and ensure stable network training; the representation of traffic mode nodes is affected by multiple types of neighbor nodes and the degree of influence is different; therefore, it is necessary to use a node attention mechanism to distinguish the importance of neighbor nodes; it is worth noting that the importance of edges (i.e., meta-paths) between neighbors will also affect the representation of traffic mode m; different meta-paths have different effects on the representation of nodes; therefore, this application introduces a weight attention mechanism to distinguish the importance of different meta-paths to nodes; for deep neural networks, as the number of network layers increases, they should be trained better and better; however, as the number of network layers and the number of iterations increases, the problem of over-smoothing becomes more and more obvious; the ResNet module in this application makes it possible to ensure good performance when training deeper networks;

[0093] The specific steps include:

[0094] The hierarchical attention mechanism with residual network includes residual network layer, node attention layer and weight attention layer;

[0095] The residual network layer is used to maintain the stability of network training; the value of the residual network layer Directly affected by the l+1th layer hx perspective traffic mode node embedding and the layer l hx perspective traffic mode node embedding Impact:

[0096]

[0097] Where hx is the node type, including ho and he, where ho represents a homogeneous node and he represents a heterogeneous node; ψ s is a given set of meta-paths; σ() is the ReLU activation function; is the trainable weight matrix of the hx perspective of the l+2th layer; b is the embedding of the hx perspective transportation mode node at the l+1th layer; b hx(l+2) is the bias vector of the hx perspective of the l+2th layer;

[0098] Node attention layer: The following formula is used to calculate the i-th transportation mode node embedding from the perspective of the l+1th layer hx:

[0099]

[0100] Where m j is the i-th transportation mode node m i Neighbor nodes; N(m i ) is the i-th transportation mode node m i The set of neighbor nodes of is m i and m j The attention value between is the trainable weight matrix of the lth layer;

[0101] Weighted attention layer: The final embedding of each transportation mode m is calculated by the following formula:

[0102]

[0103] Where softmax() is the normalized exponential function softmax; researchers can apply the final embedding to specific tasks, such as node classification, node prediction, and recommendation systems;

[0104] S5. Aggregate and process the results of step S4 to obtain the final multimodal transportation recommendation results and construct an initial model for multimodal transportation recommendation; specifically, the steps include:

[0105] Design a fusion neural layer to aggregate the results of step S4 and input the results into the fully connected layer to achieve multimodal transportation recommendation;

[0106] Multimodal embedding e' with several fully connected layers m Used to predict the most suitable transportation mode m for user u to travel on od:

[0107]

[0108]

[0109] In the formula Embedding of transportation mode nodes for homogeneous perspective; is the node embedding of transportation modes from heterogeneous perspectives; || is the merging operation; softmax() is the softmax function; θ() is the activation function ReLU; is the weight matrix obtained through training; is the final representation of the transportation mode m at the lth level; b (l) is the bias vector; y' mScore the final prediction;

[0110] Sort the final scores of several modes of transportation used by the user to make multi-modal transportation recommendations for the user;

[0111] During the model training process, supervised training is performed, and the following cross entropy loss is used as the objective function Loss:

[0112]

[0113] Where C is the number of categories; is the real label; is the predicted label;

[0114] Under the guidance of labeled data, the proposed multimodal transportation recommendation initial model is optimized through back propagation;

[0115] S6. Using the training data set obtained in step S1, the multimodal traffic recommendation initial model constructed in step S5 is trained to obtain a multimodal traffic recommendation model based on a heterogeneous graph neural network;

[0116] S7. Use the multimodal traffic recommendation model based on heterogeneous graph neural network obtained in step S6 to perform real-time multimodal traffic recommendation.

[0117] like Figure 2 The figure shows the performance comparison results of the proposed method and the existing recommendation methods in F1, Pre, Rec and NDCG. THAN outperforms the baseline algorithms such as NMTRec, Hydra, HMTRL, HAN and HeCo in the four commonly used evaluation indicators, which shows the advantages of the proposed model THAN. Specifically, compared with the most advanced scheme, THAN achieves improvements in F1 (+0.8%), Pre (+6.2%), Rec (+0.8%) and NDCG (+4.9%). This is because THAN can not only learn the relationship between homogeneous nodes, but also enhance the feature representation from the perspective of heterogeneity. At the same time, the residual module in THAN can stabilize the training of the model, ensuring that the network has good performance when it is relatively deep.

[0118] like Figure 3Shown is a schematic diagram of the grouping robustness verification effect of the method of the present invention; in order to verify the robustness of THAN, the present application randomly divides the data into four groups, and uses each group of data to test the performance of the THAN model. The results show that the results of the four indicators of F1, Pre, Rec and NDCG of THAN in different groups of the Beijing data set are very stable. Specifically, in all tests using these four indicators, the difference in the result values ​​does not exceed 2%. The results verify that the THAN framework is robust for different transportation mode recommendation scenarios. In addition, the verification results on the Guangzhou, Shanghai and Shenzhen data sets show that the values ​​of the four indicators F1, Pre, Rec and NDCG are similar to the results obtained on the Beijing data set and also maintain a difference within 2%, further illustrating that THAN has good robustness.

[0119] like Figure 4 FIG. 1 is a schematic diagram of the ablation experiment effect of the method of the present invention; in order to further explore the effect of multimodal fusion in the THAN model, the present application conducts experimental tests on THAN and its variants THAN (ResNet) (excluding the ResNet layer), THAN (MBigraphE) (excluding MBigraphE), THAN (HierAtt) (excluding HierAtt) and THAN (ResNet & MBigraphE) (excluding ResNet and MBigraphE) on the Beijing dataset, and compares the values ​​of the four indicators generated. The results are shown in FIG. Figure 4 As shown. The results show that THAN outperforms THAN(ResNet), THAN(MBigraphE), THAN(HierAtt), and THAN(ResNet&MBigraphE). In addition, THAN(MBigraphE) and THAN(HierAtt) perform better than THAN(ResNet), which shows that residual blocks play a more important role in multimodal traffic recommendation. This is because ResNet helps solve the problem of over-smoothing and makes training better, at least ensuring that the current result is not worse than the last time. MBigraphE can balance the importance of features. This is because MBigraphE can not only learn neighbor embeddings, but also distinguish the importance of neighbors.

[0120] like Figure 5The figure shows the effect of the rationality verification of the multi-bipartite graph embedding of the method of the present invention; the present application hopes to verify the rationality of MBigraphE by evaluating the performance of three variants of THAN, which are: (1) THAN_o_d_od_pid does not consider MBigraphE in THAN, (2) THAN_o_d means that the interaction between o and d is not considered, and (3) THAN_od_pid means that the potential association between od and u is not considered. The comparison results of the four schemes on the Beijing dataset are shown in the following table. Figure 5 As shown in the figure, the results show that considering the potential connection between o and d or od and u is better than not considering the potential connection between o and d or od and u. Considering the interaction between o and d and od and u at the same time is better than considering the interaction between o and d or od and u alone. The experimental results further prove the effectiveness of MBigraphE.

Claims

1. A multimodal transportation recommendation method based on heterogeneous graph neural network, comprising the following steps: S1. Obtain historical travel data and process it to obtain a training data set; specifically, the steps include: The historical travel data includes user attribute data, starting point data, destination data, situational awareness data and transportation mode data, and is saved in a time series format; According to historical travel data, the corresponding starting points and destinations are combined into starting point-destination pairs (OD pairs); traffic data sets are collected and converted into graph structure data; the graph structure data includes users, transportation modes, OD pairs, starting points, destination points, edges with relationships and driving semantics; node data is formally represented using the following formula: Where X u is user attribute data, and N is the total data volume of the navigation data set, q is the dimension of the user attribute data; X o is the starting point data, and p is the dimension of the origin and destination; X d is the destination data, and X con is context-aware data, and k is the dimension of context data; X od is the OD pair data, and n is the dimension of OD pairs, and n=p+p; S2. Construct a multi-bipartite graph embedding scheme for the origin-destination region; specifically, the following steps are included: Extract the features of the origin o, destination d, OD pair od, and user u; Based on the OD pairs, a bipartite graph of origin and destination is constructed, and information from neighboring nodes is aggregated in the GCN layer to represent the points of the bipartite graph. Based on the data information of the user's travel on the OD pairs, a bipartite graph of user-OD pairs is constructed, and the node information is updated through the GAT layer. The GCN layer specifically updates the features of the origin and destination respectively through the following GCN model after obtaining the origin-destination bipartite graph: In the formula As the starting point i The embedding of the l+1th layer; N(o i ) is the starting point o i degree; N(d j ) is the destination d j degree; Destination j The embedding of the lth layer; Destination i The embedding of the l+1th layer; N(d i ) is the destination d i degree; N(o j ) is the starting point o j degree; As the starting point j The embedding of the lth layer; when the OD pair exists, e od =e o ||e d ,|| is a merge operation; The GAT layer is specifically that after obtaining the user-OD bipartite graph, the initialization representation matrix E of the user-OD bipartite graph is Where M is the number of OD pairs; Q is the number of users; is the initialization embedding of the jth OD pair, j = 1, 2, ..., M; is the initial embedding of the i-th user, i=1,2,...,Q; In order to obtain the user's characteristic information, the representation of user u is updated by aggregating the information propagated by user u's neighbors; Define the following aggregate function In the formula is user u at layer l+1 i Embedding of; N(u i ) is u i Neighbor aggregation; is the weight matrix obtained through training; For user u i andod j The attention value between is user u in layer l i andod j The attention value between Softmax() is the normalized exponential function softmax, is the importance of the i-th user to the j-th OD pair in the l-th layer, and ρ() is the activation function LeakyReLU, a is the attention weight vector, and is the weight parameter obtained through training, for u i In the representation at level l, for j Representation at level l; Through the GAT layer based on the user-OD bipartite graph, the user's characteristics are e u and e od ; The context information is normalized by standard to e m ; S3. Construct a heterogeneous traffic map according to the constructed origin-destination multi-bipartite graph embedding scheme; specifically, the steps include: According to the semantic structure (u, m, od) of navigation data, there is a user u using transportation mode m to travel on OD pair od to build a heterogeneous traffic graph. The initialization embedding vector of each node on the heterogeneous traffic graph is represented as e init =e u ||e od ||e m ,|| is a merge operation; e u is the initialization embedding vector of the user node, e od is the initialization embedding vector of the OD pair, e m Initialize embedding vectors for transportation modes; The final embedding of the transportation mode is represented as Where E u is the user node embedding, and E od is the OD pair node embedding, and E m is the transportation mode node embedding, and f1 is the size of the user node embedding dimension; f2 is the size of the OD node embedding dimension; f3 is the size of the transportation mode node embedding dimension; e' m ∈ε m , e' m Multimodal embedding vectors for each mode of transportation; S4. Design a hierarchical attention mechanism with residual networks to address the over-smoothing problem; S5. Aggregate and process the results of step S4 to obtain the final multimodal transportation recommendation results and construct an initial model for multimodal transportation recommendation; S6. Using the training data set obtained in step S1, the multimodal traffic recommendation initial model constructed in step S5 is trained to obtain a multimodal traffic recommendation model based on a heterogeneous graph neural network; S7. Use the multimodal traffic recommendation model based on heterogeneous graph neural network obtained in step S6 to perform real-time multimodal traffic recommendation.

2. According to claim 1, a multimodal traffic recommendation method based on heterogeneous graph neural network is characterized in that Step S4 describes designing a hierarchical attention mechanism with a residual network to solve the problem of over-smoothing, which specifically includes the following steps: The hierarchical attention mechanism with residual network includes residual network layer, node attention layer and weight attention layer; The residual network layer is used to maintain the stability of network training; the value of the residual network layer Directly affected by the l+1th layer hx perspective traffic mode node embedding and the layer l hx perspective traffic mode node embedding Impact: Where hx is the node type, including ho and he, where ho represents a homogeneous node and he represents a heterogeneous node; ψ s is a given set of meta-paths; σ() is the ReLU activation function; is the trainable weight matrix of the hx perspective of the l+2th layer; b is the embedding of the hx perspective transportation mode node at the l+1th layer; b hx(l+2) is the bias vector of the hx perspective of the l+2th layer; Node attention layer: The following formula is used to calculate the i-th transportation mode node embedding from the perspective of the l+1th layer hx: Where m j is the i-th transportation mode node m i Neighbor nodes; N(m i ) is the i-th transportation mode node m i The set of neighbor nodes of is m i and m j The attention value between is the trainable weight matrix of the lth layer; Weighted attention layer: The final embedding of each transportation mode m is calculated by the following formula: Where softmax() is the normalized exponential function softmax.

3. According to claim 2, a multimodal traffic recommendation method based on heterogeneous graph neural network is characterized in that The step S5 described in step S4 is aggregating and processing the results to obtain the final multimodal transportation recommendation results and construct the initial model of multimodal transportation recommendation, which specifically includes the following steps: Design a fusion neural layer to aggregate the results of step S4 and input the results into the fully connected layer to achieve multimodal transportation recommendation; Multimodal embedding e' with several fully connected layers m Used to predict the most suitable transportation mode m for user u to travel on od: In the formula Embedding of transportation mode nodes for homogeneous perspective; is the node embedding of transportation modes from heterogeneous perspectives; || is the merging operation; softmax() is the softmax function; θ() is the activation function ReLU; is the weight matrix obtained through training; is the final representation of the transportation mode m at the lth level; b (l) is the bias vector; y' m Score the final prediction; Sort the final scores of several modes of transportation used by the user to make multi-modal transportation recommendations for the user; During the model training process, supervised training is performed, and the following cross entropy loss is used as the objective function Loss: Where C is the number of categories; is the real label; is the predicted label; Under the guidance of labeled data, the proposed initial model of multimodal transportation recommendation is optimized through back propagation.

Images