Urban multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federal map learning

Abstract

The invention relates to the technical field of city portrait prediction, in particular to a city multi-dimensional portrait prediction method fusing multi-source spatio-temporal data and federal map learning. According to the method, the graph neural network is trained by using local sample city data on each data holder, that is, the trained graph neural network is shared to each data holder, each data holder shares local parameters obtained by training to the central server, the central server fuses the local parameters, and the local parameters of each data holder are integrated. And obtaining a final graph federal learning model. According to the method, the trained graph neural network is shared instead of local sample city data, so that data leakage can be prevented. And by adopting local sample city data of each data holder, the data dimension can be expanded, so that the model training precision is improved, and the model prediction precision is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of city image prediction, and in particular to a city multi-dimensional image prediction method integrating multi-source spatiotemporal data and federated graph learning. Background Art

[0002] The multi-source spatiotemporal data of a city includes the city's geographic information, traffic flow, productivity indicators, social activities, etc. The multi-dimensional portrait of the city includes the city's population density, traffic congestion index, gross domestic product, and public safety events. Use multi-source spatiotemporal data as a training data set, and use the training data set to train the model. The trained model can be used for portrait prediction. Since multi-source spatiotemporal data is stored on different data sources, each data source is only trained on the local server with the data source to prevent data leakage. The local server only has a single-dimensional spatiotemporal data. The single data will reduce the accuracy of model training, which in turn affects the prediction accuracy of the model.

[0003] In summary, the existing technology reduces the prediction accuracy of multi-dimensional portraits.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a method for predicting urban multidimensional portraits by integrating multi-source spatiotemporal data and federated graph learning, which solves the problem that the prior art reduces the prediction accuracy of multidimensional portraits.

[0006] To achieve the above object, the present invention adopts the following technical solutions: The present invention provides a method for predicting a city multi-dimensional portrait by integrating multi-source spatiotemporal data and federated graph learning, which includes: Acquire local parameters sent by each data holder, where the local parameters are model parameters obtained by the data holder based on training the graph neural network based on local sample city data, and the local sample city data of each data holder constitutes multi-source spatiotemporal data; Based on each of the local parameters, a graph federated learning model is obtained; The final parameters of the graph federated learning model are sent to each of the data holders, and the local prediction embedding sent by each of the data holders is obtained. The local prediction embedding is the embedding of the graph federated learning model generated by the data holder based on the local city data and the final parameters, and the graph federated learning model is applied to each of the local prediction embeddings to obtain a prediction result of the city portrait.

[0007] In one implementation, obtaining local parameters sent by each data holder also includes: Sending initialization parameters of the graph neural network to each of the data holders; Obtaining a local initial embedding sent by each of the data holders, where the local initial embedding is an embedding of the graph neural network generated by the data holder based on the initialization parameters; Fusing the local initial embeddings to obtain an initial fused embedding, and sending the initial fused embedding to the data holders; Obtaining a local gradient sent by each of the data holders, the local gradient being a gradient generated by the data holder based on a predicted value and a true value corresponding to the local sample city data, the predicted value being a value output by the graph neural network with the initial fusion embedding based on the local sample city data; The local gradients are fused to obtain fused gradients, and the fused gradients are sent to the data holders, so that the data holders update the initialization parameters based on the fused gradients until the graph neural network on the data holders converges.

[0008] In one implementation, the local gradient sent by each of the data holders is an encrypted gradient.

[0009] In one implementation, sending the fused gradient to each of the data holders so that each of the data holders updates the initialization parameter based on the fused gradient until the graph neural network on each of the data holders converges includes: Acquire the initialization parameters sent by each of the data holders after updating based on the fused gradient; The updated initialization parameters are fused to obtain fused parameters, and the fused parameters are sent to each data holder, so that each data holder iteratively updates the parameters of the graph neural network until the graph neural network converges.

[0010] In one implementation, the data holders interact with each other using unit numbers to achieve alignment of the local sample city data, where the unit is a component of a city.

[0011] In one implementation, the exchanged number is a number that has been hashed.

[0012] In one implementation, each of the data holders performs verification by exchanging private keys, and after passing the verification, each of the data holders exchanges the number of the unit.

[0013] In one implementation, the local city data of each data holder includes satellite images, street view pictures, mobile phone signals, and vehicle trajectories; the city portrait includes population portrait, traffic portrait, and environmental portrait.

[0014] In one implementation, each node of the graph structure of the graph neural network is a unit of the city, and the edge between two nodes corresponding to the two units is defined by the feature similarity of the two units.

[0015] In one implementation, feature similarity of the two units is used to define an edge between the two nodes corresponding to the two units, including: when the feature similarity of the two units is greater than a threshold, establishing an edge between the two nodes.

[0016] Beneficial effect: The present invention uses the local sample city data of each data holder to train the graph neural network, that is, the present invention shares the trained graph neural network with each data holder, and each data holder shares the local parameters obtained from the training to the central server, and the central server fuses the local parameters to obtain the final graph federated learning model. Since the present invention shares the trained graph neural network instead of the local sample city data, data leakage can be prevented. The use of local sample city data from each data holder can expand the data dimension, thereby improving the model training accuracy, and then improving the model prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall flow chart of the present invention; Figure 2 Schematic diagram of the interaction between the central server and the data holder in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solution of the present invention in combination with the embodiments and the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The study found that the city's multi-source spatiotemporal data includes the city's geographic information, traffic flow, productivity indicators, social activities, etc. The city's multi-dimensional portrait includes the city's population density, traffic congestion index, gross domestic product, and public safety events. Use multi-source spatiotemporal data as a training data set, and use the training data set to train the model. The trained model can be used for portrait prediction. Since multi-source spatiotemporal data is stored on different data sources, each data source is only trained on the local server with the data source to prevent data leakage. The local server only has a single-dimensional spatiotemporal data. The single data will reduce the accuracy of model training, which in turn affects the prediction accuracy of the model.

[0020] In order to solve the above technical problems, the present invention provides a method for predicting urban multidimensional portraits by integrating multi-source spatiotemporal data and federated graph learning, which solves the problem that the prior art reduces the prediction accuracy of multidimensional portraits.

[0021] The city multi-dimensional portrait prediction method integrating multi-source spatiotemporal data and federated graph learning in this embodiment can be applied to a terminal device, which can be a terminal product with data processing function, such as a central server. Figure 1 As shown in , the urban multi-dimensional portrait prediction method integrating multi-source spatiotemporal data and federated graph learning specifically includes the following steps: S100, obtaining local parameters sent by each data holder, wherein the local parameters are model parameters obtained by the data holder by training a graph neural network based on local sample city data, and the local sample city data of each data holder constitutes multi-source spatiotemporal data.

[0022] S200, obtaining a graph federated learning model based on each of the local parameters.

[0023] S300, sending the final parameters of the graph federated learning model to each of the data holders, obtaining the local prediction embedding sent by each of the data holders, the local prediction embedding being the embedding of the graph federated learning model generated by the data holder based on the local city data and the final parameters, and applying the graph federated learning model to each of the local prediction embeddings to obtain a prediction result of the city portrait.

[0024] A set of final parameters are trained for all types of city portraits. When predicting, the city portrait is predicted based on the local city data of all data holders and this set of final parameters, where the local city data includes satellite images, street view pictures, mobile phone signals, vehicle trajectories, etc.; the city portraits include population portraits, traffic portraits, environmental portraits, etc., that is, the prediction results of multiple portraits such as population portraits, traffic portraits, and environmental portraits can be obtained simultaneously using the trained set of final parameters.

[0025] like Figure 2 As shown, the central server distributes the initialization parameters to each data holder (data holders include Figure 2 Each data holder uses its own local sample city data to iteratively train the graph neural network. The local embedding generated by each iterative training is uploaded to the central server, and the purpose of joint training is achieved through the central server until the graph neural network of each data holder converges.

[0026] Each data holder uploads the local parameters obtained through the final training to the central server, which aggregates the local parameters and loads them onto the graph neural network to form a graph federated learning model. When it is necessary to predict the population portrait, each data holder generates a local prediction embedding based on the actual collected local city data and the final model parameters, and then uploads the local prediction embedding to the central server. The graph federated learning model on the central server fuses the local city data to predict the population portrait. The same method can be used to predict traffic portraits, economic portraits, health portraits, environmental portraits, and emotional portraits.

[0027] Example 1: In this example, local sample city data includes satellite images, street view pictures, mobile phone signals, and vehicle trajectories. Each local sample city data is preprocessed and aligned data, and preprocessing and alignment are performed on the data holder. The preprocessing and alignment include the following specific steps: The spatial scale of the unit is unified among all data holders (the city is divided into several grids, one grid is a unit), and each unit is assigned a unique ID number.

[0028] Taking units as units, the semantic features of street view images are calculated, the number of various points of interest on the street view images is identified, the building area, the number of buildings and the building height on the street view images are identified, the total length and the total area of ​​the road network on the street view images are identified, and the semantic features, the number of various points of interest, the building area, the number of buildings, the building height, the total length and the total area of ​​the road network are used as feature information of the street view images.

[0029] For mobile phone signaling data, data aggregation is performed on spatial units according to fixed time periods such as year, quarter, hour, day, and week. The visitor flow and travel flow of each unit within a certain time period are counted, and the flow distribution of people in different age groups and call fee levels is further subdivided to obtain characteristic information of mobile phone signaling.

[0030] Embodiment 2: This embodiment uses characteristic information of mobile phone signaling, characteristic information of street view pictures and other local sample city data to train the graph neural network, and uses the true value of the image corresponding to the local sample city data to calculate the loss function of the graph neural network.

[0031] After completing the preprocessing, each data holder realizes sample alignment through the ID number of the interactive unit. Alignment means that each data holder uses the local sample city data on the same unit to train the graph neural network. For example, there are two data holders A and B. A has the local sample city data of three units, unit 1, unit 2, and unit 3, where 1, 2, and 3 are ID numbers; B has the local sample city data of four units, unit 1, unit 2, unit 4, and unit 5. Alignment means that both A and B only use the local sample city data of the two common units, unit 1 and unit 2, to train the graph neural network.

[0032] In order to prevent leakage, each data holder exchanges the unit number in an encrypted manner, that is, each data holder generates its own private key based on the elliptic curve Diffie-Hellman encryption algorithm. After that, each party performs hash calculation on the ID, maps the hash calculation result to the elliptic curve to generate an encryption point, and then the two parties exchange encryption points, use the received encryption points and private keys to calculate the other party's data encryption results respectively, and decrypt the encrypted ID. Finally, through the intersection of the decrypted ID and the ID calculated by both parties, the common ID to be used for joint model training is found, and the model is trained with the local sample city data on the unit corresponding to the common ID.

[0033] This embodiment combines data sources from multiple fields, such as satellite images, street view images, points of interest, traffic flow, building information, social media data, etc. The above data are highly correlated and complementary in time and space. By combining the complex relationship modeling capabilities of graph neural networks and the distributed training mechanism of federated learning, each data source performs feature extraction and model training locally. The aligned data not only retains their independence, but also realizes the collaborative training of multimodal features through the aggregation mechanism of the federated learning framework.

[0034] Embodiment 3, based on Embodiment 1 or Embodiment 2, this embodiment constructs a graph structure of a graph neural network on each data holder (data holder is a participant), wherein the nodes of the graph structure are units of the city, and each node The corresponding local sample city data features are ,in is the feature set, is the feature dimension, The node index number.

[0035] If the feature-based similarity between two nodes is greater than a threshold , then an edge is established between the two nodes and an initial weight is given to the edge : ; In the formula, For Node Features, for Node index number other than for and The cosine similarity of This embodiment can also use the Euclidean distance and the Pearson correlation coefficient to calculate the similarity between two nodes.

[0036] This embodiment uses the adjacency matrix Each element in Describe the relationship between nodes: ; Embodiment 4, based on embodiment 1 or embodiment 2 or embodiment 3, in this embodiment, before each data holder sends the local parameters obtained by training the graph neural network locally to the central server, the following specific steps S01 to S05 are included: S01, sending the initialization parameters of the graph neural network to each data holder.

[0037] The initialization parameters include the initialized weight matrix and bias items. The weight matrix includes the weight matrix of each layer of graph convolution. , the bias term includes the bias term of each layer of graph convolution , Represents the layer number.

[0038] The central server sends the initialization parameters to each data holder to achieve sharing of the initialization parameters.

[0039] S02, obtaining the local initial embedding data sent by each data holder , the local initial embedding is the embedding of the graph neural network generated by the data holder based on the initialization parameters.

[0040] ; ; In the formula, Is a node In the previous layer of embedding, nodes Is a node Neighbor nodes of The embedding before being input into the graph convolution is ,Right now For Node Embedding before entering the graph convolution, is the activation function, is the average aggregation function, For Node A set of neighbor nodes, where a neighbor node is a connection relationship between two nodes. , is the weight matrix before being input into the graph convolution.

[0041] S03, fusing the local initial embeddings to obtain an initial fused embedding, and sending the initial fused embedding to the data holders.

[0042] In this embodiment, the central server uses the splicing and fusion method to fuse the local initial embeddings generated by each data holder. In each subsequent iterative training process, the central server also uses the splicing and fusion method to fuse the local embeddings generated by each data holder. If the data holders include data holders A, B, and C, in one iteration, the local embedding generated by data holder A , local embedding generated by data holder B , local embedding generated by data holder C , then the embedding after splicing and fusion is , where On behalf of data holder A, On behalf of data holder B, On behalf of data holder C, Represents splicing, Represents the global situation.

[0043] S04, obtaining the local gradient sent by each of the data holders, wherein the local gradient is a gradient generated by the data holder based on the predicted value and the true value corresponding to the local sample city data, and the predicted value is a value output by the graph neural network with the initial fusion embedding based on the local sample city data.

[0044] Each data holder receives the initial fusion embedding sent by the central server, and then the graph neural network of each data holder outputs the predicted values ​​for all nodes of the local sample city data based on the initial fusion embedding , each data holder based on the predicted value And the true value corresponding to all nodes Generate local gradients : ; in is the loss function of the graph neural network, , N is the total number of nodes.

[0045] In this embodiment, local sample city data is always on the local client of each data holder, but not uploaded to the central server. With the training strategy of "data does not move, model moves", the collaborative modeling of multi-source spatiotemporal data and the prediction of urban multi-dimensional portraits are completed without sharing the original data. This embodiment adopts a federated learning framework, and protects the privacy of each data holder by means of local training models and parameter sharing, avoiding the risks brought by data transmission.

[0046] S05, fusing each of the local gradients to obtain a fused gradient, and sending the fused gradient to each of the data holders, so that each of the data holders updates the initialization parameters based on the fused gradient until the graph neural network on each of the data holders converges.

[0047] Each data holder generates its own local gradient Sent to the central server in encrypted form, the central server uses the decrypted local gradient The gradient information of the fully connected layer is calculated to obtain the fused gradient, and the central server sends the fused gradient to each data holder respectively.

[0048] The central server is based on Calculate the gradient of the weight matrix : ; The central server is based on Calculate the gradient of the bias term : ; In the formula, represents the weight matrix of the fully connected layer, Represents the bias term of the fully connected layer. Represents the gradient of the output layer to the weight after the activation function has been nonlinearly transformed. Represents the gradient of the output layer to the bias after the activation function has been nonlinearly transformed.

[0049] Each data holder has a weight matrix and the bias term These two parameters are updated: ; ; In the formula, Representative Convolutional layer , Representative Convolutional layer , Represents the learning rate.

[0050] Each data holder is updated according to the weight matrix and the bias term Update the embedding of the node and encrypt and upload the updated local embedding for the central server to perform global embedding fusion. Repeat the local training and global aggregation multiple times until the model converges, and finally generate a global model (the global model is the graph federated learning model).

[0051] Embodiment 5, based on any one of embodiments 1 to 4, in this embodiment, step S300 includes the following specific steps: all data holders obtain local prediction embeddings based on their local city data, and then upload the local prediction embeddings to the central server, and the central server uses the global model to generate portrait prediction results in any dimension of multiple dimensions such as population, transportation, economy, etc., for example: population density, traffic congestion index, gross domestic product, public safety incidents, etc.

[0052] In this embodiment, the central server may share the global model with each data holder, and each data holder uses the final global model parameters in combination with local data (local data is local city data) to generate a local embedding representation, and encrypts and uploads it to the central server. The central server uses the fused global embedding representation to generate portrait prediction results in multiple dimensions such as population, transportation, and economy.

[0053] This embodiment uses the joint training of the graph neural network model by various data holders to fully capture the interactive relationship between the city's physical structure, facility distribution and human activity patterns, and realize feature sharing and knowledge integration between different tasks. The final generated global model can accurately capture and characterize the essential characteristics and functional attributes of urban areas, support multiple downstream urban tasks, and provide technical support for generating more accurate and comprehensive urban portraits.

[0054] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting urban multidimensional portraits by integrating multi-source spatiotemporal data and federated graph learning, characterized in that: include: Acquire local parameters sent by each data holder, where the local parameters are model parameters obtained by the data holder based on training the graph neural network based on local sample city data, and the local sample city data of each data holder constitutes multi-source spatiotemporal data; Based on each of the local parameters, a graph federated learning model is obtained; The final parameters of the graph federated learning model are sent to each of the data holders, and the local prediction embedding sent by each of the data holders is obtained. The local prediction embedding is the embedding of the graph federated learning model generated by the data holder based on the local city data and the final parameters, and the graph federated learning model is applied to each of the local prediction embeddings to obtain a prediction result of the city portrait.

2. The method for predicting a city multi-dimensional portrait by integrating multi-source spatiotemporal data and federated graph learning as claimed in claim 1, characterized in that: Get the local parameters sent by each data holder, which previously included: Sending initialization parameters of the graph neural network to each of the data holders; Obtaining a local initial embedding sent by each of the data holders, where the local initial embedding is an embedding of the graph neural network generated by the data holder based on the initialization parameters; Fusing the local initial embeddings to obtain an initial fused embedding, and sending the initial fused embedding to the data holders; Obtaining a local gradient sent by each of the data holders, the local gradient being a gradient generated by the data holder based on a predicted value and a true value corresponding to the local sample city data, the predicted value being a value output by the graph neural network with the initial fusion embedding based on the local sample city data; The local gradients are fused to obtain fused gradients, and the fused gradients are sent to the data holders, so that the data holders update the initialization parameters based on the fused gradients until the graph neural network on the data holders converges.

3. The method for predicting a city multi-dimensional portrait by integrating multi-source spatiotemporal data and federated graph learning as claimed in claim 2, characterized in that: The local gradient sent by each of the data holders is an encrypted gradient.

4. The method for predicting a city multi-dimensional portrait by integrating multi-source spatiotemporal data and federated graph learning as claimed in claim 2, characterized in that: Sending the fused gradient to each of the data holders so that each of the data holders updates the initialization parameters based on the fused gradient until the graph neural network on each of the data holders converges, including: Acquire the initialization parameters sent by each of the data holders after updating based on the fused gradient; The updated initialization parameters are fused to obtain fused parameters, and the fused parameters are sent to each data holder, so that each data holder iteratively updates the parameters of the graph neural network until the graph neural network converges.

5. The method for predicting a city multi-dimensional portrait by integrating multi-source spatiotemporal data and federated graph learning according to any one of claims 1 to 4, characterized in that: The data holders interact with each other by unit numbers to achieve alignment of the local sample city data, where the unit is a component of a city.

6. The method for predicting a city multi-dimensional image by integrating multi-source spatiotemporal data and federated graph learning as claimed in claim 5, characterized in that: The number exchanged is a number that has been hashed.

7. The method for predicting a city multi-dimensional portrait by integrating multi-source spatiotemporal data and federated graph learning as claimed in claim 5, characterized in that: Each of the data holders performs verification by exchanging private keys, and after passing the verification, each of the data holders exchanges the number of the unit.

8. The method for predicting a city multi-dimensional image by integrating multi-source spatiotemporal data and federated graph learning as claimed in claim 1, characterized in that: The local city data of each data holder includes satellite images, street view pictures, mobile phone signals, and vehicle trajectories; the city portrait includes population portrait, traffic portrait, and environmental portrait.

9. The method for predicting a city multi-dimensional portrait by integrating multi-source spatiotemporal data and federated graph learning as claimed in claim 1, characterized in that: Each node of the graph structure of the graph neural network is a unit of the city, and the edge between two nodes corresponding to the two units is defined by the feature similarity of the two units.

10. The method for predicting a city multi-dimensional portrait by integrating multi-source spatiotemporal data and federated graph learning as claimed in claim 9, characterized in that: The feature similarity of the two units is used to define an edge between the two nodes corresponding to the two units, including: when the feature similarity of the two units is greater than a threshold, establishing an edge between the two nodes.

Images