Urban multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federated graph learning

Abstract

The present application relates to the technical field of city image prediction, in particular to a city multi-dimensional image prediction method fusing multi-source spatio-temporal data and federated graph learning. The present application trains a graph neural network by using local sample city data on each data holder, that is, the trained graph neural network is shared to each data holder, and each data holder shares the obtained local parameters to a central server, and the central server fuses each local parameter to obtain a final graph federated learning model. Since the present application shares the trained graph neural network instead of the local sample city data, data leakage can be prevented. By using the local sample city data on each data holder, the data dimension can be expanded, thereby improving the model training precision and further improving the model prediction precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of urban portrait prediction, in particular to a city multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federated graph learning. BACKGROUND

[0002] The multi-source spatio-temporal data of a city includes geographic information, traffic flow, productivity indicators, social activities, etc. of the city, and the city multi-dimensional portrait includes population density, traffic congestion index, gross domestic product, public safety events of the city. The multi-source spatio-temporal data is used as a training data set, and a model is trained using the training data set. The trained model can be used for portrait prediction. Since the multi-source spatio-temporal data is saved on different data sources, each data source prevents data leakage, so model training is only performed on the local server with the data source. The local server only has single-dimensional spatio-temporal data, and the single dimension of the data reduces the accuracy of model training, thereby affecting the prediction accuracy of the model.

[0003] To sum up, the prior art reduces the prediction accuracy of multi-dimensional portrait.

[0004] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0005] To solve the above technical problems, the present application provides a city multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federated graph learning, which solves the problem of reducing the prediction accuracy of multi-dimensional portrait in the prior art.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] The present application provides a city multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federated graph learning, comprising:

[0008] Obtain local parameters sent by each data holder, the local parameters being model parameters obtained by the data holder training a graph neural network based on local sample city data, and the local sample city data of each data holder constituting multi-source spatio-temporal data;

[0009] Obtain a graph federated learning model based on each local parameter;

[0010] Send the final parameters of the graph federated learning model to each data holder, obtain local prediction embeddings sent by each data holder, the local prediction embeddings being embeddings of the graph federated learning model generated by the data holder based on local city data and the final parameters, and apply the graph federated learning model to each local prediction embedding to obtain a prediction result of a city portrait.

[0011] In an implementation manner, the local parameters sent by each data holder are acquired, and the previous implementation manner further comprises:

[0012] The initialization parameters of the graph neural network are sent to each data holder.

[0013] The local initial embedding sent by each data holder is acquired, and the local initial embedding is the embedding of the graph neural network generated by the data holder based on the initialization parameters.

[0014] The local initial embedding is fused to obtain an initial fusion embedding, and the initial fusion embedding is sent to each data holder.

[0015] The local gradient sent by each data holder is acquired, and the local gradient is the gradient generated by the data holder based on the predicted value and the true value corresponding to the local sample city data, and the predicted value is the value output by the graph neural network with the initial fusion embedding based on the local sample city data.

[0016] The local gradient is fused to obtain a fusion gradient, and the fusion gradient is sent to each data holder, so that each data holder updates the initialization parameters based on the fusion gradient until the graph neural network on each data holder converges.

[0017] In an implementation manner, the local gradient sent by each data holder is an encrypted gradient.

[0018] In an implementation manner, the fusion gradient is sent to each data holder, so that each data holder updates the initialization parameters based on the fusion gradient until the graph neural network on each data holder converges, which comprises:

[0019] The initialization parameters updated based on the fusion gradient sent by each data holder are acquired.

[0020] Each updated initialization parameter is fused to obtain a fusion parameter, and the fusion parameter is sent to each data holder, so that each data holder iteratively updates the parameters of the graph neural network until the graph neural network converges.

[0021] In an implementation manner, each data holder interacts with the number of units to realize the alignment of the local sample city data, and the unit is a component of the city.

[0022] In an implementation manner, the number of units interacted is a number of units after hash encryption.

[0023] In an implementation manner, each data holder is verified by an interactive private key, and after verification, each data holder interacts with the number of the unit.

[0024] In an implementation manner, the local city data of each data holder includes satellite images, street view pictures, mobile phone signaling, and vehicle trajectories; and the city portrait includes a population portrait, a traffic portrait, and an environment portrait.

[0025] In an implementation manner, each node of the graph structure of the graph neural network is each unit of a city, and the edge between two nodes corresponding to two units is defined by the feature similarity of the two units.

[0026] In an implementation manner, the edge between two nodes corresponding to two units is defined by the feature similarity of the two units, including: when the feature similarity of the two units is greater than a threshold, an edge is established between the two nodes.

[0027] Beneficial effects: the present application trains a graph neural network by using local sample city data on each data holder, that is, the trained graph neural network is shared to each data holder, and each data holder shares the obtained local parameters to a central server, and the central server fuses each local parameter to obtain a final graph federated learning model. Since the present application shares the trained graph neural network instead of local sample city data, data leakage can be prevented. By using local sample city data on each data holder, the data dimension can be expanded, thereby improving the model training precision and further improving the model prediction precision. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The overall flowchart of the present application is shown in the figure.

[0029] Figure 2 The interaction between the central server and the data holder in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0030] The technical solutions in the present application are described in detail below in combination with the embodiments and the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0031] It is found through research that the multi-source spatio-temporal data of a city includes geographic information, traffic flow, productivity indicators, social activities, etc. of the city, and the multi-dimensional portrait of the city includes population density, traffic congestion index, gross domestic product, public safety events of the city. A set of final parameters is trained for all types of city portraits. When predicting, the city portrait is predicted based on the local city data of all data holders and the set of final parameters, wherein the local city data is satellite imagery, street view pictures, mobile signaling, vehicle trajectory, etc. The city portrait is a population portrait, a traffic portrait, an environment portrait, etc. That is, the set of final parameters trained can simultaneously obtain the prediction results of multiple portraits such as population portrait, traffic portrait, and environment portrait.

[0032] To solve the above technical problems, the present application provides a city multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federated graph learning, which solves the problem of reducing the prediction accuracy of the multi-dimensional portrait in the prior art.

[0033] The city multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federated graph learning of the embodiment can be applied to a terminal device, which can be a terminal product with data processing function, such as a central server, etc. In the embodiment, as shown in Figure 1 The city multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federated graph learning specifically includes the following steps:

[0034] S100, obtaining local parameters sent by each data holder, the local parameters being model parameters obtained by the data holder based on local sample city data training a graph neural network, and the local sample city data of each data holder constituting multi-source spatio-temporal data.

[0035] S200, obtaining a graph federated learning model based on each local parameter.

[0036] S300, sending final parameters of the graph federated learning model to each data holder, obtaining local prediction embeddings sent by each data holder, the local prediction embeddings being embeddings of the graph federated learning model generated by the data holder based on local city data and the final parameters, and applying the graph federated learning model to each local prediction embedding to obtain a prediction result of a city portrait.

[0037] A set of final parameters is trained for all types of city portraits. When predicting, the city portrait is predicted based on the local city data of all data holders and the set of final parameters, wherein the local city data is satellite imagery, street view pictures, mobile signaling, vehicle trajectory, etc. The city portrait is a population portrait, a traffic portrait, an environment portrait, etc. That is, the set of final parameters trained can simultaneously obtain the prediction results of multiple portraits such as population portrait, traffic portrait, and environment portrait.

[0038] As Figure 2 shown, the center server distributes the initialization parameters to various data holders (the data holders include clients used by the government platform, clients used by enterprises, clients used by the open platform, etc.), each data holder iteratively trains the graph neural network using its own local sample city data, and the local embedding generated by each iteration is uploaded to the center server to achieve the purpose of joint training through the center server until the graph neural network of each data holder converges. Figure 2

[0039] Each data holder uploads the local parameters finally trained to the center server, the center server aggregates the local parameters, and loads the aggregated parameters onto the graph neural network to form a graph federated learning model. When it is necessary to predict a population portrait, each data holder generates a local prediction embedding according to the local city data actually collected and the final model parameters, and then uploads the local prediction embedding to the center server, and the graph federated learning model on the center server fuses various local city data to predict the population portrait. The same method can be used to predict traffic portraits, economic portraits, health portraits, environmental portraits, and emotional portraits.

[0040] In embodiment one, the local sample city data includes satellite images, street view pictures, mobile signaling, and vehicle trajectories, and each local sample city data is data after preprocessing and alignment, and the preprocessing and alignment are performed on the data holder. The preprocessing and alignment include the following specific steps:

[0041] The spatial scale of the unit (dividing the city into a plurality of grids, and a grid is a unit) is unified among the various data holders, and a unique ID number is assigned to each unit.

[0042] The semantic features of the street view pictures are calculated in units, the number of various types of interest points on the street view pictures is identified, the area, number, and height of the buildings on the street view pictures are identified, and the total length and total area of the road network on the street view pictures are identified, and the semantic features, the number of various types of interest points, the area of the buildings, the number of the buildings, the height of the buildings, the total length of the road network, and the total area are taken as the feature information of the street view pictures.

[0043] For the mobile signaling data, the data is aggregated on the spatial unit according to the fixed time period of year, quarter, hour, day, week, etc., the visiting and traveling person flows of each unit in a certain time period are counted, and the person flow distribution of different age groups and cost levels is further subdivided to obtain the feature information of the mobile signaling.

[0044] ​In the second embodiment, the graph neural network is trained by using feature information of mobile phone signaling and feature information of street view pictures and other local sample city data of a sample city.

[0045] After the preprocessing is completed, the sample alignment is realized by the ID number of the interactive unit, that is, the graph neural network is trained by using the local sample city data on the same unit by each data holder. For example, there are two data holders A and B, A has local sample city data of units 1, 2 and 3, and 1, 2 and 3 are ID numbers; B has local sample city data of units 1, 2, 4 and 5, and the alignment means that A and B only use the local sample city data of units 1 and 2 to train the graph neural network.

[0046] In order to prevent leakage, each data holder interacts with the number of units by encryption, that is, each data holder generates a private key based on the elliptic curve Diffie-Hellman encryption algorithm. Then, each party performs hash calculation on the ID, maps the hash calculation result to the elliptic curve to generate an encrypted point, and then exchanges the encrypted points, calculates the data encryption result of the other party by using the received encrypted point and the private key, and decrypts the encrypted ID. Finally, the common ID used for joint training of the model is found by the intersection of the decrypted ID and the ID calculated by both parties, and the local sample city data on the unit corresponding to the common ID is used to train the model.

[0047] In this embodiment, data sources from multiple fields are combined, such as satellite images, street view images, points of interest, traffic flow, building information, social media data, etc. The above data has high correlation and complementarity in space and time. By combining the complex relationship modeling capability of the graph neural network and the distributed training mechanism of the federated learning, each data source performs feature extraction and model training locally. The aligned data not only retains its independence, but also realizes the collaborative training of multi-modal features through the aggregation mechanism of the federated learning framework.

[0048] In the third embodiment, the graph structure of the graph neural network is constructed on each data holder (the data holder is a participant), wherein the nodes of the graph structure are units of a city, each node The features of the corresponding local sample city data are , wherein is a feature set, is a feature dimension, is a node index number.

[0049] If the similarity between two nodes based on the features is greater than a threshold , then an edge is established between two nodes and the initial weight is assigned to the edge :

[0050] ;

[0051] wherein, is the feature of node , is the index number of node , is the cosine similarity between and , represents the two-norm calculation. The embodiment can also use the Euclidean distance and the Pearson correlation coefficient to calculate the similarity between two nodes.

[0052] The embodiment describes the relationship between nodes by each element in the adjacency matrix :

[0053] ;

[0054] Embodiment four, based on embodiment one or embodiment two or embodiment three, in the embodiment, before each data holder sends the local parameter obtained by training the local graph neural network to the center server, the following specific steps S01 to S05 are included:

[0055] S01, send the initialization parameter of the graph neural network to each data holder.

[0056] The initialization parameter includes the initialized weight matrix and the bias term, the weight matrix includes the weight matrix of each layer of graph convolution , and the bias term includes the bias term of each layer of graph convolution , represents the layer number.

[0057] The center server sends the initialization parameter to each data holder to realize the sharing of the initialization parameter.

[0058] S02, obtain the local initial embedding sent by each data holder, wherein the local initial embedding is the embedding of the graph neural network generated by the data holder based on the initialization parameter.

[0059] ;

[0060] ;

[0061] wherein, is the feature of node In the embedding of the last layer, the node is a neighbor node of the node , the node is the embedding before inputting into the graph convolution, that is , that is the embedding of the node before entering the graph convolution, is an activation function, is an average aggregation function, is a neighbor node set of the node , the neighbor node is a connection relationship between two nodes. Among them, , is a weight matrix before inputting into the graph convolution.

[0062] S03, fusing each of the local initial embeddings to obtain an initial fusion embedding, and sending the initial fusion embedding to each of the data holders.

[0063] In this embodiment, the center server fuses the local initial embeddings generated by each data holder by using the splicing fusion method. In each subsequent iteration training process, the center server also fuses the local embeddings generated by each data holder by using the splicing fusion method. Assuming that the data holders include data holders A, B and C, in an iteration process, the local embedding generated by the data holder A , the local embedding generated by the data holder B , and the local embedding generated by the data holder C , then the embedding after splicing fusion is , wherein represents the data holder A, represents the data holder B, represents the data holder C, represents splicing, represents global.

[0064] S04, obtaining the local gradient sent by each of the data holders, the local gradient being a gradient generated by the data holder based on a prediction value and a true value corresponding to the local sample city data, the prediction value being a value output by the graph neural network having the initial fusion embedding based on the local sample city data.

[0065] Each of the data holders receives the initial fusion embedding sent by the center server, and then the graph neural network of each data holder outputs prediction values for all nodes of the local sample city data based on the initial fusion embedding , each data holder generates a local gradient according to the prediction value and the true value corresponding to all nodes ​ :

[0066] ;

[0067] wherein is a loss function of the graph neural network, N is the total number of nodes.

[0068] In this embodiment, the local sample city data is always on the local client of each data holder, and is not uploaded to the central server. With the training strategy of "data does not move, model moves", the collaborative modeling of multi-source spatio-temporal data and the prediction of city multi-dimensional portrait are completed without sharing the original data. This embodiment adopts a federated learning framework, protects the privacy of each data holder through local training of the model and sharing of the parameters, and avoids the risks brought by data transmission.

[0069] S05, fusing each of the local gradients to obtain a fused gradient, and sending the fused gradient to each of the data holders, so that each of the data holders updates the initialization parameters based on the fused gradient until the graph neural network on each of the data holders converges.

[0070] Each data holder sends the local gradient generated by the data holder to the central server in an encrypted form. The central server calculates the gradient information of the fully connected layer based on the decrypted local gradient to obtain a fused gradient. The central server sends the fused gradient to each data holder.

[0071] wherein the central server calculates the gradient of the weight matrix based on :

[0072] ;

[0073] The central server calculates the gradient of the bias term based on :

[0074] ;

[0075] wherein represents the weight matrix of the fully connected layer, represents the bias term of the fully connected layer. represents the gradient of the weight of the output layer after the activation function is subjected to a nonlinear transformation, represents the gradient of the bias of the output layer after the activation function is subjected to a nonlinear transformation.

[0076] Each data holder calculates the gradient of the weight matrix ​​and the bias term The two parameters are updated:

[0077] ;

[0078] ;

[0079] wherein, represents the first the convolutional layer , represents the first the convolutional layer , represents the learning rate.

[0080] Each data holder updates the node embedding according to the weight matrix after the update and the bias term , and encrypts and uploads the updated local embedding for the center server to fuse the global embedding. Multiple iterations of local training and global aggregation are performed until the model converges, and finally a global model (the global model is a graph federated learning model) is generated.

[0081] In embodiment five, based on any one of embodiments one to four, the step S300 includes the following specific steps: all data holders obtain local prediction embeddings according to local city data, and then upload the local prediction embeddings to the center server; the center server generates portrait prediction results in any dimension of multiple dimensions such as population, traffic, and economy, for example, population density, traffic congestion index, gross domestic product, and public safety events, by using the global model.

[0082] In this embodiment, the center server can also share the global model to each data holder, and each data holder generates local embedding representation by using the final global model parameters in combination with local data (the local data is local city data), and encrypts and uploads to the center server. The center server generates portrait prediction results in multiple dimensions such as population, traffic, and economy by using the fused global embedding representation.

[0083] This embodiment jointly trains a graph neural network model by each data holder, comprehensively captures the interaction between the physical structure of the city, the distribution of facilities, and the human activity mode, and realizes feature sharing and knowledge integration among different tasks. The finally generated global model can accurately capture and represent the essential features and functional attributes of the urban area, support multiple downstream city tasks, and provide technical support for generating more accurate and comprehensive city portraits.

[0084] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program controlling relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0085] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for urban multi-dimensional portrait prediction by fusing multi-source spatio-temporal data and federated graph learning, characterized in that, The method comprises the following steps: Obtain the local parameters sent by each data holder, which are model parameters obtained by the data holder based on training the graph neural network based on local sample city data. The local sample city data of each data holder constitutes multi-source spatiotemporal data. The local sample city data of each data holder includes satellite images, street view pictures, mobile phone signals, and vehicle trajectories. Each data holder interacts with the unit number to achieve alignment of the local sample city data. The unit is a component of the city. The nodes of the graph structure of the graph neural network are the units of the city. When the feature similarity of two units is greater than a threshold, an edge is established between the two nodes, and the initial weight of the edge is : , For nodes Features, for Node index numbers other than for and The cosine similarity of Represents the two-norm calculation, where each data holder uses local sample city data on the same unit to train the graph neural network; Based on each of the local parameters, obtain a graph federated learning model; Send the final parameters of the graph federated learning model to each of the data holders, obtain the local prediction embedding sent by each of the data holders, the local prediction embedding being an embedding of the graph federated learning model generated by the data holder based on local city data and the final parameters, and apply the graph federated learning model to each of the local prediction embeddings to obtain a prediction result of a city portrait, the city portrait including a population portrait, a traffic portrait and an environment portrait; Obtain the local parameters sent by each data holder, which further comprises the following steps: Send the initialization parameters of the graph neural network to each of the data holders; obtaining local initial embeddings sent by each of the data holders , the local initial embeddings being embeddings of the graph neural network generated by the data holders based on the initialization parameters; ； ； In the formula, is the node In the embedding of the upper layer, the node is the node The neighbor node of the node The embedding before inputting to the graph convolution is That is is the node The embedding before entering the graph convolution, is an activation function, is an average aggregation function, is the node The neighbor node set of the node, the neighbor node is a connection relationship between two nodes, wherein , is a weight matrix before inputting to the graph convolution, is the weight matrix of each layer of graph convolution , is the bias term of each layer of graph convolution; Fuse each of the local initial embeddings to obtain an initial fusion embedding, and send the initial fusion embedding to each of the data holders; obtaining local gradients sent by each of the data holders : , the local gradient sent by each of the data holders is an encrypted gradient, the local gradient is a gradient based on a predicted value and a true value corresponding to the local sample city data generated by the data holder, the predicted value is a value output by the graph neural network with the initial fused embedding based on the local sample city data; Fusing the local gradients to obtain a fused gradient, and sending the fused gradient to each data holder so that each data holder updates the initialization parameters based on the fused gradient until the graph neural network on each data holder converges. The parameters include the weight matrix and bias : , , Representative Convolutional layer , Representative Convolutional layer , Represents the learning rate. 2.The method of claim 1, wherein, Send the fusion gradient to each of the data holders, so that each of the data holders updates the initialization parameters based on the fusion gradient until the graph neural network on each of the data holders converges, which comprises the following steps: Obtain the initialization parameters updated based on the fusion gradient and sent by each of the data holders; Fuse each of the updated initialization parameters to obtain a fusion parameter, and send the fusion parameter to each of the data holders, so that each of the data holders iteratively updates the parameters of the graph neural network until the graph neural network converges. 3.The method of claim 1, wherein The number interacted is the number after hash encryption. 4.The method of claim 1, wherein Each of the data holders verifies through the interaction of the private key, and after verification, each of the data holders interacts the number of the unit.

Images