A Parallel Fusion Method, System, Terminal and Storage Medium for Multimodal Data of Urban Stock Space

Abstract

The present invention belongs to the technical field of data fusion, and discloses a parallel fusion method, system, terminal and storage medium for multi-modal data of urban stock space. The method includes: classifying, roughly extracting feature vectors and normalizing the multi-modal data of urban stock space in sequence to obtain a stock space data set of full vectors; the master node divides the stock space data set of full vectors and the neural network model based on the multi-modal spatio-temporal index structure, and correspondingly assigns them to each slave node; each slave node performs fine extraction of feature vectors through a deep learning algorithm, and performs preliminary fusion of multi-modal data through a multi-head attention mechanism; the master node uses the federated learning method to globally converge the local fusion feature vectors output by each slave node, and generates a target convergence model through iterative training. The present invention significantly improves the fusion processing efficiency of large-scale and diverse multi-source data of urban stock space.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and particularly to a parallel fusion method, system, terminal and computer-readable storage medium for multi-modal data of urban stock space. Background Art

[0002] With the continuous advancement of the urbanization process, the urban development mode has entered a new stage from "incremental expansion" to "stock improvement". In order to improve the utilization efficiency of urban stock space, it is necessary to converge and fuse stock space data, and conduct in-depth analysis to identify inefficient land use and implement reasonable transformation and upgrading. This process involves the integration and analysis of multi-source, multi-type, and multi-modal spatial data, including but not limited to vector data, remote sensing images, 3D models, and spatio-temporal flow data reflecting dynamic changes such as traffic flow and population movement. Currently, most urban data fusion methods have two main limitations: one is that they focus on a single data format, or although they can process multiple formats, they are all homogeneous data; the other is that most existing data fusion technologies are applicable to datasets of limited scale, and their computing efficiency and processing capabilities are insufficient when faced with multi-modal and massive urban spatial data. This limits the ability of urban planners and managers to comprehensively understand the current situation and development trend of the city, and thus affects the quality and efficiency of decision-making.

[0003] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0004] The main purpose of the present invention is to provide a parallel fusion method, system, terminal and computer-readable storage medium for multi-modal data of urban stock space, aiming to solve the problem of low processing efficiency of traditional data fusion methods when fusing and processing large-scale and diverse multi-source data of urban stock space.

[0005] To achieve the above-mentioned invention purpose, the present invention provides a parallel fusion method for multi-modal data of urban stock space, and the parallel fusion method for multi-modal data of urban stock space includes:

[0006] Obtain multi-modal data of urban stock space, and classify, roughly extract feature vectors and normalize the multi-modal data of urban stock space in sequence to obtain a stock space data set of full vectors;

[0007] Design a multi-modal spatio-temporal index structure including a master node and multiple slave nodes and construct a neural network model. The master node divides the stock space data set of full vectors and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly assigns the multiple sub-feature vectors and multiple sub-neural network models to each slave node;

[0008] Based on the sub-feature vectors and the sub-neural network models, each of the sub-nodes performs fine feature vector extraction through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors;

[0009] The main node obtains the local fusion feature vectors output by each of the sub-nodes, performs global aggregation based on the multiple local fusion feature vectors by using a federated learning method to obtain global fusion feature vectors, and generates a target aggregation model through iterative training based on the global fusion feature vectors;

[0010] Obtain multi-modal data of urban stock space to be fused, input the multi-modal data of urban stock space to be fused into the sub-neural network model to obtain target local fusion feature vectors, and input the target local fusion feature vectors into the target aggregation model to obtain a data fusion result.

[0011] Optionally, the obtaining of multi-modal data of urban stock space and the sequential classification, rough feature vector extraction, and normalization of the multi-modal data of urban stock space to obtain a full-vector stock space data set specifically include:

[0012] Obtain multi-modal data of urban stock space, and divide the multi-modal data of urban stock space into raster data, graph data, and time-series data;

[0013] Preprocess the raster data, the graph data, and the time-series data respectively to obtain preprocessed raster data, preprocessed graph data, and preprocessed time-series data, where the preprocessing includes denoising and missing value filling;

[0014] Unify the abstraction and modeling of the preprocessed raster data, the preprocessed graph data, and the preprocessed time-series data respectively to obtain a raster data model, a graph data model, and a time-series data model;

[0015] Use the raster data model, the graph data model, and the time-series data model to perform rough feature vector extraction on the preprocessed raster data, the preprocessed graph data, and the preprocessed time-series data respectively to obtain rough feature vectors corresponding to the raster data, rough feature vectors corresponding to the graph data, and rough feature vectors corresponding to the time-series data;

[0016] Normalize the rough feature vectors corresponding to the raster data, the rough feature vectors corresponding to the graph data, and the rough feature vectors corresponding to the time-series data respectively to obtain target rough feature vectors corresponding to the raster data, target rough feature vectors corresponding to the graph data, and target rough feature vectors corresponding to the time-series data;

[0017] Combine the target coarse feature vectors corresponding to the raster data, the target coarse feature vectors corresponding to the graph data, and the target coarse feature vectors corresponding to the time series data to obtain a stock space dataset of full vectors.

[0018] Optionally, the design includes a multi-modal spatio-temporal index structure of a master node and multiple slave nodes and constructing a neural network model. The master node divides the stock space dataset of full vectors and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly assigns the multiple sub-feature vectors and multiple sub-neural network models to each of the slave nodes, specifically including:

[0019] Design a multi-modal spatio-temporal index structure. Recursively divide in the spatial dimension using a quadtree to divide the two-dimensional space into multiple sub-regions. In the time dimension, use a linear index method to sort and retrieve data through timestamps. Among them, the quadtree includes a master node and multiple slave nodes. The master node represents the two-dimensional space, and each slave node represents each sub-region;

[0020] Construct a neural network model, which includes a raster data feature extraction module, a graph data feature extraction module, a time series data feature extraction module, and a multi-modal data preliminary fusion module;

[0021] Based on the multi-modal spatio-temporal index structure, the master node divides the stock space dataset of full vectors according to sub-regions and time ranges to obtain multiple sub-feature vectors, and the sub-feature vectors correspond one-to-one with the sub-regions;

[0022] Based on the multi-modal spatio-temporal index structure, the master node divides the neural network model according to sub-regions to obtain multiple sub-neural network models, and the sub-neural network models correspond one-to-one with the sub-regions;

[0023] Assign the sub-feature vectors and the sub-neural network models corresponding to each sub-region to the corresponding slave nodes;

[0024] Establish a task allocation strategy. Dynamically adjust the task allocation ratio according to the computing power and memory capacity of each slave node, and preferentially allocate tasks to slave nodes whose distance from the data storage location is less than a set value. Introduce a backup mechanism and allocate some tasks to multiple slave nodes simultaneously.

[0025] Optionally, in the step of allocating the sub-feature vectors and the sub-neural network models corresponding to the respective sub-regions to the corresponding respective sub-nodes, the sub-feature vectors corresponding to the respective sub-regions include sub-feature vectors corresponding to raster data, sub-feature vectors corresponding to graph data, and sub-feature vectors corresponding to time-series data; the sub-neural network models corresponding to the respective sub-regions include a sub-raster data feature extraction module, a sub-graph data feature extraction module, a sub-time-series data feature extraction module, and a sub-multi-modal data preliminary fusion module.

[0026] Optionally, the fine feature vectors include raster data feature vectors, graph data feature vectors, and time-series data feature vectors;

[0027] Based on the sub-feature vectors and the sub-neural network models, each sub-node performs fine extraction of feature vectors through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors, which specifically includes:

[0028] At each sub-node, the sub-feature vector corresponding to the raster data is sequentially passed through the 3D convolutional layer, GeLU (Gaussian Error Linear Unit) activation function layer, max pooling layer, residual block, global pooling layer, and first fully connected layer of the sub-raster data feature extraction module to perform fine extraction of the feature vector, obtaining a raster data feature vector ;

[0029] The sub-feature vector corresponding to the graph data is sequentially passed through the graph convolutional network, first ReLU (Rectified Linear Unit) activation function layer, multi-head attention network, and second fully connected layer of the sub-graph data feature extraction module to perform fine extraction of the feature vector, obtaining a graph data feature vector ;

[0030] The sub-feature vector corresponding to the time-series data is sequentially passed through the multi-layer long short-term memory network, third fully connected layer, second ReLU activation function layer, dropout layer, and fourth fully connected layer of the sub-time-series data feature extraction module to perform fine extraction of the feature vector, obtaining a time-series data feature vector ;

[0031] Through the multi-head attention mechanism, the raster data feature vector , the graph data feature vector , and the time-series data feature vector are used for preliminary fusion of multi-modal data to obtain local fusion feature vectors.

[0032] Optionally, through the multi-head attention mechanism, the raster data feature vector , the graph data feature vector and the time series data feature vector are initially fused for multi-modal data to obtain a local fusion feature vector, specifically including:

[0033] The raster data feature vector , the graph data feature vector and the time series data feature vector are respectively passed through different linear transformation layers to obtain the first query vector Q1, the first key vector K1, and the first value vector V1 corresponding to the raster data feature vector , the second query vector Q2, the second key vector K2, and the second value vector V2 corresponding to the graph data feature vector and the third query vector Q3, the third key vector K3, and the third value vector V3 corresponding to the time series data feature vector ;

[0034] The first similarity between the first query vector Q1 and the third key vector K3 is calculated in parallel through multiple attention heads , the second similarity between the second query vector Q2 and the second key vector K2 , and the third similarity between the third query vector Q3 and the first key vector K1 :

[0035] ;

[0036] ;

[0037] ;

[0038] where T represents transpose, represents the dimension of the first key vector, the second key vector, or the third key vector;

[0039] According to the first similarity , the second similarity and the third similarity , the first attention weight corresponding to the raster data is calculated using the Softmax function , the second attention weight corresponding to the graph data and the third attention weight corresponding to the time series data :

[0040] ;

[0041] ;

[0042] ;

[0043] According to the first attention weight , the second attention weight and the third attention weight , weighted summations are respectively performed on the third value vector V3, the second value vector V2, and the first value vector V1 to obtain a local fusion feature vector O:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] wherein, represents an attention mechanism.

[0049] Optionally, the master node obtains the local fusion feature vectors output by each slave node, and based on the local fusion feature vectors, uses a federated learning method for global aggregation to obtain a global fusion feature vector, and based on the global fusion feature vector, generates a target aggregation model through iterative training, specifically including:

[0050] The master node obtains the local fusion feature vectors output by each slave node, and based on the weights corresponding to each slave node, uses a federated learning algorithm to perform weighted fusion on the local fusion feature vectors output by each slave node to obtain a global fusion feature vector :

[0051] ;

[0052] wherein, i represents the i-th slave node, represents the local fusion feature vector output by the i-th slave node, represents the weight corresponding to the i-th slave node;

[0053] wherein, the weight corresponding to the i-th slave node is calculated according to the sample quantity

[0054] ;

[0055] wherein, represents the sample quantity of the first slave node, represents the sample quantity of the second slave node, Indicates the number of samples of the third child node, Indicates the number of samples of the fourth child node;

[0056] Use the global fusion feature vector generated by the master node to update the aggregation weight parameters of the master node, and feedback the updated aggregation weight parameters to each child node to obtain the target aggregation model;

[0057] Among them, the aggregation weight parameters of the master node include the weights corresponding to each of the child nodes.

[0058] To achieve the above invention purpose, the present invention also provides a parallel fusion system for multi-modal data of urban stock space. The parallel fusion system for multi-modal data of urban stock space includes:

[0059] Data preliminary processing module: used to obtain multi-modal data of urban stock space, and classify, roughly extract feature vectors and normalize the multi-modal data of urban stock space in sequence to obtain a full-vector stock space data set;

[0060] Distributed task allocation module: used to design a multi-modal spatio-temporal index structure including a master node and multiple child nodes and construct a neural network model. The master node divides the full-vector stock space data set and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly assigns the multiple sub-feature vectors and multiple sub-neural network models to each of the child nodes;

[0061] Preliminary fusion module: based on the sub-feature vectors and the sub-neural network models, each child node performs fine extraction of feature vectors through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors;

[0062] Global aggregation module: used for the master node to obtain the local fusion feature vectors output by each child node, perform global aggregation using the federated learning method based on the multiple local fusion feature vectors to obtain global fusion feature vectors, and generate a target aggregation model through iterative training based on the global fusion feature vectors;

[0063] Multi-modal data fusion module: used to obtain the multi-modal data of urban stock space to be fused, input the multi-modal data of urban stock space to be fused into the sub-neural network model to obtain target local fusion feature vectors, and input the target local fusion feature vectors into the target aggregation model to obtain a data fusion result.

[0064] To achieve the above-mentioned invention objectives, the present invention further provides a terminal, which includes: a memory, a processor, and a parallel fusion program of multi-modal data of urban stock space stored on the memory and operable on the processor. When the parallel fusion program of multi-modal data of urban stock space is executed by the processor, it implements the steps of the above-mentioned parallel fusion method of multi-modal data of urban stock space.

[0065] To achieve the above-mentioned invention objectives, the present invention further provides a computer-readable storage medium, which stores a parallel fusion program of multi-modal data of urban stock space. When the parallel fusion program of multi-modal data of urban stock space is executed by a processor, it implements the steps of the above-mentioned parallel fusion method of multi-modal data of urban stock space.

[0066] In the present invention, multi-modal data of urban stock space is acquired, and the multi-modal data of urban stock space is classified, coarsely extracted with feature vectors, and normalized in sequence to obtain a full-vector stock space data set; a multi-modal spatio-temporal index structure including a main node and multiple sub-nodes is designed and a neural network model is constructed. The main node divides the full-vector stock space data set and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly assigns the multiple sub-feature vectors and multiple sub-neural network models to each of the sub-nodes; based on the sub-feature vectors and the sub-neural network models, each sub-node performs fine extraction of feature vectors through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors; the main node acquires the local fusion feature vectors output by each sub-node, performs global aggregation based on the multiple local fusion feature vectors by using a federated learning method to obtain global fusion feature vectors, and generates a target aggregation model through iterative training based on the global fusion feature vectors; multi-modal data of urban stock space to be fused is acquired, the multi-modal data of urban stock space to be fused is input into the sub-neural network model to obtain target local fusion feature vectors, and the target local fusion feature vectors are input into the target aggregation model to obtain a data fusion result. The present invention significantly improves the fusion processing efficiency of large-scale and diverse multi-source data of urban stock space, supports complex and refined urban analysis and simulation, and provides strong technical support for urban planning and management. Description of the Drawings

[0067] Figure 1 is a flowchart of a preferred embodiment of the parallel fusion method of multi-modal data of urban stock space of the present invention;

[0068] Figure 2It is a schematic diagram of the multi-modal spatio-temporal index structure of the present invention;

[0069] Figure 3 It is a schematic diagram of the structure of the sub-neural network model of the present invention;

[0070] Figure 4 It is a schematic diagram of the structure of the sub-grid data feature vector extraction module of the present invention;

[0071] Figure 5 It is a schematic diagram of the structure of the sub-graph data feature vector extraction module of the present invention;

[0072] Figure 6 It is a schematic diagram of the structure of the sub-temporal data feature vector extraction module of the present invention;

[0073] Figure 7 It is a schematic diagram of the initial fusion of multi-modal data through the multi-head attention mechanism of the present invention;

[0074] Figure 8 It is a structural diagram of a preferred embodiment of the parallel fusion system for multi-modal data of the urban stock space of the present invention;

[0075] Figure 9 It is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed implementation manners

[0076] To make the objectives, technical solutions and advantages of the present invention clearer and more definite, the following further elaborates on the present invention by way of examples with reference to the accompanying drawings. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0077] With the continuous advancement of urbanization, the urban development model has entered a new stage from "incremental expansion" to "stock improvement". To improve the utilization efficiency of the urban stock space, it is necessary to converge and integrate the stock space data and conduct in-depth analysis to identify inefficient land use and implement reasonable transformation and upgrading. This process involves the integration and analysis of multi-source, multi-type, and multi-modal spatial data, including but not limited to vector data, remote sensing images, 3D models, and spatio-temporal flow data reflecting dynamic changes such as traffic flow and population movement. Due to the large volume, wide range, and complexity of these data, they each carry different geographical information and time dimension characteristics. Therefore, how to effectively converge and integrate these massive and diverse data has become a key technical issue that urgently needs to be solved in current urban planning and management. Currently, most urban data fusion methods have two main limitations: one is that they focus on a single data format, or although they can handle multiple formats, they are all homogeneous data; the other is that most existing data fusion technologies are applicable to datasets of limited scale, and their computational efficiency and processing capabilities are insufficient when faced with multi-modal and massive urban spatial data. This limits the ability of urban planners and managers to comprehensively understand the current situation and development trends of the city, thereby affecting the quality and efficiency of decision-making.

[0078] To solve the above technical problems, the present invention provides a parallel fusion method for multi-modal data of urban stock space, which acquires multi-modal data of urban stock space, and classifies, roughly extracts feature vectors and normalizes the multi-modal data of urban stock space in sequence to obtain a stock space data set with full vectors; designs a multi-modal spatio-temporal index structure including a master node and multiple slave nodes and constructs a neural network model. The master node divides the stock space data set with full vectors and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly assigns the multiple sub-feature vectors and the multiple sub-neural network models to each slave node; based on the sub-feature vectors and the sub-neural network models, each slave node performs fine extraction of feature vectors through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain locally fused feature vectors; the master node acquires the locally fused feature vectors output by each slave node, performs global aggregation based on the multiple locally fused feature vectors by using a federated learning method to obtain globally fused feature vectors, and generates a target aggregation model through iterative training based on the globally fused feature vectors; acquires multi-modal data of urban stock space to be fused, inputs the multi-modal data of urban stock space to be fused into the sub-neural network model to obtain target locally fused feature vectors, and inputs the target locally fused feature vectors into the target aggregation model to obtain a data fusion result. The present invention significantly improves the fusion processing efficiency of large-scale and diverse multi-source data of urban stock space, supports complex and refined urban analysis and simulation, and provides strong technical support for urban planning and management.

[0079] The following further describes the application content by describing embodiments in conjunction with the accompanying drawings.

[0080] A preferred embodiment of the parallel fusion method for multi-modal data of urban stock space according to the present invention is as Figure 1 shown and specifically includes:

[0081] S1. Acquire multi-modal data of urban stock space, and classify, roughly extract feature vectors and normalize the multi-modal data of urban stock space in sequence to obtain a stock space data set with full vectors.

[0082] In an implementation manner of this embodiment, the acquisition of multi-modal data of urban stock space, and the classification, rough extraction of feature vectors and normalization of the multi-modal data of urban stock space in sequence to obtain a stock space data set with full vectors specifically include:

[0083] Acquire multi-modal data of urban stock space, and divide the multi-modal data of urban stock space into raster data, graph data and time-series data;

[0084] Preprocessing the raster data, the graph data, and the time series data respectively to obtain preprocessed raster data, preprocessed graph data, and preprocessed time series data, wherein the preprocessing includes denoising and missing value filling;

[0085] Performing unified abstraction and modeling on the preprocessed raster data, the preprocessed graph data, and the preprocessed time series data to obtain a raster data model, a graph data model, and a time series data model;

[0086] Using the raster data model, the graph data model, and the time series data model, respectively, to perform coarse feature vector extraction on the preprocessed raster data, the preprocessed graph data, and the preprocessed time series data, to obtain coarse feature vectors corresponding to the raster data, coarse feature vectors corresponding to the graph data, and coarse feature vectors corresponding to the time series data;

[0087] Normalizing the coarse feature vectors corresponding to the raster data, the graph data, and the time series data, respectively, to obtain target coarse feature vectors corresponding to the raster data, the graph data, and the time series data;

[0088] The target coarse feature vector corresponding to the raster data, the target coarse feature vector corresponding to the graph data, and the target coarse feature vector corresponding to the time series data are combined to obtain a stock spatial data set of all vectors.

[0089] Specifically, first, the multi-modal data of urban stock space (multi-modal data refers to data of different modalities, such as images, texts, streaming data, etc.) is divided into raster data, graph data, time-series data, etc. Then, preprocessing is performed on these multi-source heterogeneous data, including denoising and missing value filling. Among them, denoising removes noise through filtering algorithms to ensure data quality. For remote sensing image data (i.e., raster data), adaptive filtering technology is used to reduce the influence of random noise; for time-series data, the sliding window smoothing method or Kalman filter is used to eliminate short-term fluctuations. For numerical data, the interpolation method is used to fill in missing values; for time-series data, a time-series prediction model is used to fill in missing values. Then, unified abstraction and modeling are performed on these multi-source heterogeneous data, and they are uniformly abstracted into raster data models, graph data models, and time-series data models for expression. The raster data model is used to describe continuous spatial data and supports efficient pixel-level analysis; the graph data model captures the connectivity and adjacency relationships between nodes by converting vector data into a graph structure, then constructs an entity-relationship graph, and calculates the adjacency matrix and degree matrix of the graph; the time-series data model is used to capture the changing patterns in the time dimension. Next, the feature vectors of the multi-modal data of urban stock space are roughly extracted using the raster data model, graph data model, and time-series data model. Coarse feature extraction of raster data: The principal component analysis is used to extract the main features of raster data, and the sliding window method is used to extract the texture features of local regions to enhance the modeling ability of spatial locality; Coarse feature extraction of graph data: The points, lines, and surfaces in vector data are converted into a graph structure, and topological analysis methods are used to extract the connectivity and adjacency relationships between nodes, generate the adjacency matrix and degree matrix of the graph, and then graph neural networks are used to extract the topological features of nodes and edges; Coarse feature extraction of time-series data: The long short-term memory network is used to extract the long-term dependencies in the time series, and the statistical features and frequency domain features of time-series data are extracted. Finally, the extracted coarse feature vectors are normalized, and the training set and test set are divided to form a unified and complete feature vector space (i.e., the stock space dataset of full vectors), ensuring that the data can be effectively input into a deep neural network (i.e., a neural network model). The present invention extracts global feature vectors (i.e., target coarse feature vectors) through raster data models, graph data models, and time-series data models, and finally forms a stock space dataset D of full vectors:

[0090] The i-th data in the stock space dataset D of full vectors is represented in the following format:

[0091] ;

[0092] where F1, F2, and F3 respectively represent the relevant features (i.e., target coarse feature vectors, it should be noted that the target coarse feature vectors are global feature vectors) extracted by the raster data model, graph data model, and time-series data model, Represents spatial coordinates, lat is latitude, lon is longitude, and t is timestamp.

[0093] S2. Design a multimodal spatiotemporal index structure including a main node and multiple sub-nodes and construct a neural network model. The main node divides the existing spatial data set of the full vector and the neural network model based on the multimodal spatiotemporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and assigns the multiple sub-feature vectors and the multiple sub-neural network models to each of the sub-nodes.

[0094] In one implementation of this embodiment, the design includes a multimodal spatiotemporal index structure of a master node and multiple sub-nodes and constructing a neural network model. The master node divides the existing spatial dataset of the full vector and the neural network model based on the multimodal spatiotemporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and assigns the multiple sub-feature vectors and the multiple sub-neural network models to each of the sub-nodes. Specifically, the design includes:

[0095] Design a multimodal spatiotemporal index structure, using a quadtree for recursive partitioning in the spatial dimension to divide the two-dimensional space into multiple sub-regions, and a linear indexing approach in the temporal dimension to sort and retrieve data by timestamp. The quadtree includes a main node and multiple child nodes, where the main node represents the two-dimensional space and each child node represents a sub-region.

[0096] Constructing a neural network model, the neural network model includes a raster data feature extraction module, a graph data feature extraction module, a time series data feature extraction module and a multimodal data preliminary fusion module;

[0097] Based on the multimodal spatiotemporal index structure, the master node divides the existing spatial dataset of the full vector into sub-regions and time ranges to obtain a plurality of sub-feature vectors, where the sub-feature vectors correspond one-to-one to the sub-regions;

[0098] Based on the multimodal spatiotemporal index structure, the master node divides the neural network model into sub-regions to obtain a plurality of sub-neural network models, and the sub-neural network models correspond one-to-one to the sub-regions;

[0099] Assigning the sub-feature vectors and the sub-neural network models corresponding to the respective sub-regions to the corresponding sub-nodes;

[0100] Establish a task allocation strategy, dynamically adjust the task allocation ratio according to the computing power and memory capacity of each child node, and give priority to assigning tasks to child nodes whose distance to the data storage location is less than the set value. In addition, introduce a backup mechanism to assign some tasks to multiple child nodes at the same time.

[0101] In one implementation of this embodiment, the sub-feature vectors and the sub-neural network models corresponding to each sub-region are assigned to corresponding sub-nodes, and the sub-feature vectors corresponding to each sub-region include sub-feature vectors corresponding to raster data, sub-feature vectors corresponding to graph data, and sub-feature vectors corresponding to time series data; the sub-neural network models corresponding to each sub-region include a sub-raster data feature extraction module, a sub-graph data feature extraction module, a sub-time series data feature extraction module, and a sub-multimodal data preliminary fusion module.

[0102] Specifically, first, a multimodal spatiotemporal index structure is designed. This multimodal spatiotemporal index structure is a unified spatial and temporal benchmark built based on the above-mentioned full-vector stock spatial dataset, which is used to improve the query efficiency of multimodal datasets in space, time, attributes and semantics, such as Figure 2 As shown. In the spatial dimension, quadtree is used for recursive partitioning. Quadtree is a hierarchical spatial index structure that can divide the two-dimensional space into multiple sub-areas. Each quadtree node (main node) represents a spatial range A(x y), recursively divide it into four sub-areas. A1, A2, A3, and A4 represent the four sub-areas corresponding to the main node respectively. The depth of the division is controlled by setting the condition for stopping the recursive division: the amount of data in the spatial range is less than the threshold N, and the spatial division depth reaches the maximum limit L. After recursive division, each sub-area corresponds to a unique spatial ID and contains all the data falling into the sub-area:

[0103] ;

[0104] in, , Respectively represent the horizontal and vertical coordinates of the spatial range A, Indicates sub-area and subregions The horizontal axis, Indicates sub-area and subregions The vertical coordinate, Indicates sub-area and subregions The horizontal axis, Indicates sub-area and subregions The vertical coordinate of

[0105] The time dimension is managed using a linear index, and data is sorted and retrieved by timestamp:

[0106] ;

[0107] Among them, T represents the entire time set, represents the nth timestamp;

[0108] By specifying a time range to retrieve data, the data that meets the time range is passed to the spatial index. After spatial partitioning, the output is obtained. Within the specified time range, the data set R that falls into a certain spatial sub-region:

[0109] ;

[0110] Among them, R represents the spatial set after spatio-temporal partitioning, represents the starting point of the time range, represents the end point of the time range. Figure 2 In represents the data with index 1 in the stock space data set D of the full vector (which can be understood as the first data in the stock space data set D of the full vector), represents the data with index 2 in the stock space data set D of the full vector (which can be understood as the second data in the stock space data set D of the full vector), represents the data with index 3 in the stock space data set D of the full vector (which can be understood as the third data in the stock space data set D of the full vector).

[0111] Then, based on the multi-modal spatio-temporal index structure, the full space of the feature vector is partitioned. The master node partitions the global feature vector (referring to the target coarse feature vector in the stock space data set of the full vector) and the neural network model into multiple sub-feature vectors and multiple sub-neural network models for subsequent distributed computing. First, the partitioning of the sub-feature vectors is carried out: According to the results of the multi-modal spatio-temporal index, the global feature vector is partitioned according to the spatial sub-region and the time range; each sub-feature vector corresponds to a specific spatial sub-region and time range, and contains the feature vectors of all data points within that region. Specifically, traverse the output result R of the multi-modal spatio-temporal index, extract the data set of each spatial sub-region; store the feature vectors in each data set independently to form sub-feature vectors. Then, the partitioning of the sub-neural network models is carried out: The neural network model (i.e., the global neural network model) is partitioned into multiple sub-neural network models according to the number of spatial sub-regions; each sub-neural network model is responsible for processing the calculation tasks of the corresponding sub-feature vectors, extracting local features, performing preliminary data fusion, and outputting intermediate results for subsequent global fusion.

[0112] Finally, perform distributed task allocation. First, select and initialize the computing nodes (i.e., child nodes). The master node distributes the divided sub-feature vectors and sub-neural network models to the processes of each computing node. Each computing node is responsible for processing its corresponding subtask, loading the sub-feature vectors and sub-neural network models. Then, establish a task allocation strategy. Dynamically adjust the task allocation ratio according to the computing power and memory capacity of each computing node to ensure workload balance among computing nodes; preferentially allocate tasks to computing nodes close to the data storage location to reduce data transmission overhead; to prevent task failure caused by computing node failures, introduce a backup mechanism and allocate some tasks to multiple computing nodes to improve system reliability.

[0113] S3. Based on the sub-feature vectors and the sub-neural network models, each of the child nodes performs fine extraction of the feature vectors through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors.

[0114] In an implementation manner of this embodiment, the fine feature vectors include raster data feature vectors, graph data feature vectors, and time series data feature vectors;

[0115] The step of, based on the sub-feature vectors and the sub-neural network models, each of the child nodes performing fine extraction of the feature vectors through a deep learning algorithm to obtain fine feature vectors, and performing preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors specifically includes:

[0116] At each child node, the sub-feature vectors corresponding to the raster data are sequentially passed through the 3D convolutional layer, GeLU activation function layer, max pooling layer, residual block, global pooling layer, and first fully connected layer of the sub-raster data feature extraction module to perform fine extraction of the feature vectors to obtain raster data feature vectors ;

[0117] The sub-feature vectors corresponding to the graph data are sequentially passed through the graph convolutional network, first ReLU activation function layer, multi-head attention network, and second fully connected layer of the sub-graph data feature extraction module to perform fine extraction of the feature vectors to obtain graph data feature vectors ;

[0118] The sub-feature vectors corresponding to the time series data are sequentially passed through the multi-layer long short-term memory network, third fully connected layer, second ReLU activation function layer, dropout layer, and fourth fully connected layer of the sub-time series data feature extraction module to perform fine extraction of the feature vectors to obtain time series data feature vectors ;

[0119] Through the multi-head attention mechanism, the raster data feature vectors and the graph data feature vector and the time series data feature vector are subjected to preliminary multi-modal data fusion to obtain a local fusion feature vector.

[0120] In an implementation manner of this embodiment, through the multi-head attention mechanism, the raster data feature vector and the graph data feature vector and the time series data feature vector are subjected to preliminary multi-modal data fusion to obtain a local fusion feature vector, which specifically includes:

[0121] The raster data feature vector and the graph data feature vector and the time series data feature vector respectively pass through different linear transformation layers to obtain the first query vector Q1, the first key vector K1, and the first value vector V1 corresponding to the raster data feature vector , the second query vector Q2, the second key vector K2, and the second value vector V2 corresponding to the graph data feature vector , and the third query vector Q3, the third key vector K3, and the third value vector V3 corresponding to the time series data feature vector ;

[0122] The first similarity between the first query vector Q1 and the third key vector K3 is calculated in parallel through multiple attention heads , the second similarity between the second query vector Q2 and the second key vector K2 , and the third similarity between the third query vector Q3 and the first key vector K1 :

[0123] ;

[0124] ;

[0125] ;

[0126] where T represents transpose, represents the dimension of the first key vector, the second key vector, or the third key vector;

[0127] According to the first similarity , the second similarity , and the third similarity , the first attention weight corresponding to the raster data is calculated using the Softmax function , the second attention weight corresponding to the graph data The third attention weight corresponding to the time series data :

[0128] ;

[0129] ;

[0130] ;

[0131] According to the first attention weight , the second attention weight and the third attention weight , weighted summations are respectively performed on the third value vector V3, the second value vector V2 and the first value vector V1 to obtain a local fusion feature vector O:

[0132] ;

[0133] ;

[0134] ;

[0135] ;

[0136] wherein represents the attention mechanism.

[0137] It should be noted that the structural compositions of the neural network model and the sub - neural network model of the present invention are the same, both as shown in Figure 3 . The structural compositions of the raster data feature extraction module and the sub - raster data feature extraction module are the same, both as shown in Figure 4 . The structures of the graph data feature extraction module and the sub - graph data feature extraction module are the same, both as shown in Figure 5 . The structures of the time series data feature extraction module and the sub - time series data feature extraction module are the same, both as shown in Figure 6 .

[0138] Specifically, as shown in Figure 3 , first, the sub - neural network model accesses according to the feature vectors of each source data, for example, the raster data uses convolutional layers and pooling layers for feature extraction; the graph data is processed using graph convolutional layers, graph attention networks and auto - encoders; the time series data is processed using recurrent neural networks. Then, the outputs of each sub - module are fused by a multi - head attention mechanism, thereby realizing the preliminary fusion (i.e., local fusion) of multi - source data of urban stock space. As shown in Figure 3 and Figure 4As shown, for raster data such as remote sensing images, spatial and spectral features of the images are first extracted through a 3D convolutional layer, with each convolutional kernel sized 7*7*3; and the GeLU activation function is used to enhance the non-linear expression ability of the model; then, through a max pooling layer (i.e., maximum pooling), the feature dimension is reduced while important information is retained; subsequently, the residual block contains two 3D convolutional layers with convolutional kernels of 3*3*3 and the GeLU activation function, and a multi-layer residual network is used to extract deeper features to alleviate the vanishing gradient problem; finally, a global pooling layer (i.e., global average pooling) is used to generate a fixed-length feature vector, and after adjusting the feature dimension through a fully connected layer, we get , for subsequent network processing. As Figure 3 and Figure 5 shown, for graph data, the structural information of nodes and their neighborhoods is extracted through a graph convolutional network, combined with the ReLU activation function to enhance the non-linear expression ability, and the relationship weights between nodes are calculated using a multi-head attention network to highlight important node relationships, and finally, a feature vector is further extracted through a fully connected layer . As Figure 3 and Figure 6 shown, for time series data, the dynamic dependencies in time are captured through a multi-layer LSTM (Long Short-Term Memory) network, then a fully connected layer is used for feature dimensionality reduction, and the generalization ability of the model is enhanced through the ReLU activation function and the dropout layer, and finally, a time series data feature vector is generated .

[0139] Among them, in each LSTM network, the calculation process at each time stamp t is implemented through four key components, the input gate, the forget gate, the cell state, and the output gate, to dynamically model time series data. The input gate determines how much of the input information at the current time stamp will be written into the cell state. It is calculated through a sigmoid activation function ( Figure 6 denoted by in ), combined with the current input and the hidden state at the previous time stamp to obtain a weight value between 0 and 1. The forget gate uses the sigmoid function to output a weight value based on the current input and the hidden state at the previous time stamp, which is used to determine the retention degree of each element in the cell state. The cell state Determine the hidden state of the current timestamp which information needs to be output. The hidden state not only contains the information of the current timestamp, but also retains historical information through the cell state, so as to capture long-term dependencies in the time series.

[0140] In multi-modal data fusion, such as Figure 7 shown, the feature vectors extracted from raster data, graph data, and time series data are effectively fused through the multi-head attention mechanism. First, the three obtained feature vectors are respectively passed through three different linear transformation layers to obtain the query (Query), key (Key), and value (Value) matrices, and the weights are adjusted to learn the correlation between different modalities. The raster data feature vector is linearly transformed to obtain K1, Q1, V1, the graph data feature vector is linearly transformed to obtain K2, Q2, V2, and the time series data feature vector is linearly transformed to obtain K3, Q3, V3. Cross-modal information fusion is performed between raster data and time series data. The similarities between Q1 and K3, Q3 and K1 are calculated in parallel through multiple attention heads. The dot product can reflect the correlation between different modalities, and the similarity Score is calculated. The SoftMax function is used to obtain the attention weights , , , so that the model can make weighted decisions between multi-modalities, thereby focusing on the most relevant information. According to the calculated attention weights, the values V1, V2, V3 are weighted. Through weighted summation, the model can dynamically select the most relevant part of the information. The multi-head attention mechanism captures different attention patterns through parallel computing, enabling the model to learn and fuse information from different modalities from multiple perspectives. Finally, a local fusion feature vector O that combines the features of all modalities is obtained, which can represent the input data more comprehensively.

[0141] S4. The master node obtains the local fusion feature vectors output by each slave node, and based on the multiple local fusion feature vectors, uses the federated learning method for global aggregation to obtain the global fusion feature vector, and based on the global fusion feature vector, generates the target aggregation model through iterative training.

[0142] In an implementation manner of this embodiment, the master node obtains the local fusion feature vectors output by each slave node, and based on the multiple local fusion feature vectors, uses the federated learning method for global aggregation to obtain the global fusion feature vector, and based on the global fusion feature vector, generates the target aggregation model through iterative training, specifically including:

[0143] The master node obtains the local fusion feature vectors output by each slave node, and based on the weights corresponding to each slave node, uses the federated learning algorithm to perform weighted fusion on the local fusion feature vectors output by each slave node to obtain the global fusion feature vector. :

[0144] ;

[0145] Where i represents the i-th slave node, represents the local fusion feature vector output by the i-th slave node, represents the weight corresponding to the i-th slave node;

[0146] Among them, the weight corresponding to the i-th slave node is calculated according to the number of samples of the i-th slave node as follows:

[0147] ;

[0148] Among them, represents the number of samples of the first slave node, represents the number of samples of the second slave node, represents the number of samples of the third slave node, represents the number of samples of the fourth slave node;

[0149] Use the global fusion feature vector generated by the master node to update the aggregation weight parameters of the master node, and feedback the updated aggregation weight parameters to each slave node to obtain the target aggregation model;

[0150] Among them, the aggregation weight parameters of the master node include the weights corresponding to each slave node.

[0151] Specifically, after each subprocess (slave node) finishes processing the data, it will return the calculation result vector and so on to the master node. After the master node receives the feature vectors of all subprocesses, it uses the federated averaging algorithm to perform weighted fusion on the parameters uploaded by each slave node to obtain the final aggregation result; then uses the final aggregation result (i.e., the global fusion feature vector) to update the aggregation weight parameters of the master node (i.e., the weights corresponding to each slave node); finally, feedback the updated aggregation weight parameters to each slave node to achieve one global collaborative training iteration, and finally generate a target aggregation model with stronger generalization ability. For the master node, this aggregation process is an aggregation model.

[0152] S5. Obtain the multi-modal data of the urban stock space to be fused, input the multi-modal data of the urban stock space to be fused into the sub-neural network model to obtain the target local fusion feature vector, and input the target local fusion feature vector into the target aggregation model to obtain the data fusion result.

[0153] Specifically, the present invention solves the problem of poor performance in multi-modal data fusion of urban stock space through a neural network for multi-modal data fusion of urban stock space; and solves the problem of poor performance in fusing massive urban stock space data through a spatial data distributed parallel fusion method. The present invention realizes distributed computing of each sub-node through the sub-processes of each sub-node, that is, distributed preliminary fusion of multi-modal data, and then realizes parallel fusion of multi-modal data of urban stock space, significantly improving the fusion processing efficiency of large-scale and diverse multi-source data of urban stock space, supporting complex and refined urban analysis and simulation, and providing strong technical support for urban planning and management.

[0154] In addition, based on the above parallel fusion method for multi-modal data of urban stock space, the present invention also provides a parallel fusion system for multi-modal data of urban stock space. Among them, a preferred embodiment of the parallel fusion system for multi-modal data of urban stock space is as Figure 8 shown, and specifically includes:

[0155] Data preliminary processing module 01: used to obtain the multi-modal data of the urban stock space, and classify, roughly extract feature vectors and normalize the multi-modal data of the urban stock space in sequence to obtain a full-vector stock space data set;

[0156] Distributed task allocation module 02: used to design a multi-modal spatio-temporal index structure including a master node and multiple slave nodes and construct a neural network model. The master node divides the full-vector stock space data set and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly assigns the multiple sub-feature vectors and multiple sub-neural network models to each slave node;

[0157] Preliminary fusion module 03: used to perform fine extraction of feature vectors by each slave node based on the sub-feature vectors and the sub-neural network models to obtain fine feature vectors, and perform preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors;

[0158] Global aggregation module 04: It is used for the master node to obtain the local fusion feature vectors output by each slave node, perform global aggregation on the basis of multiple local fusion feature vectors by using the federated learning method to obtain the global fusion feature vector, and generate the target aggregation model through iterative training based on the global fusion feature vector;

[0159] Multimodal data fusion module 05: It is used to obtain the multimodal data of the urban stock space to be fused, input the multimodal data of the urban stock space to be fused into the sub-neural network model to obtain the target local fusion feature vector, and input the target local fusion feature vector into the target aggregation model to obtain the data fusion result.

[0160] In addition, based on the above parallel fusion method and system for the multimodal data of the urban stock space, the present invention also correspondingly provides a terminal. Among them, a preferred embodiment of the terminal is as Figure 9 shown, specifically including a processor 10, a memory 20 and a display 30. Figure 9 Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0161] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as the hard disk or memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, and a Flash Card equipped on the terminal. Further, the memory 20 can also include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store the application software installed on the terminal and various types of data, such as storing the program code of the terminal. The memory 20 can also be used to temporarily store the data that has been output or will be output. In one embodiment, a parallel fusion program 40 for the multimodal data of the urban stock space is stored on the memory 20, and the parallel fusion program 40 for the multimodal data of the urban stock space can be executed by the processor 10, so as to implement the steps of the parallel fusion method for the multimodal data of the urban stock space in this application.

[0162] The processor 10 can be a Central Processing Unit (CPU), a microprocessor or other data processing chips in some embodiments, and is used to run the program code stored in the memory 20 or process data, such as executing the parallel fusion program 40 for the multimodal data of the urban stock space, etc.

[0163] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) toucher, etc. in some embodiments. The display 30 is used to display information on the terminal and to display a visual user interface.

[0164] In one embodiment, when the processor 10 executes the parallel fusion program 40 of the urban stock space multimodal data in the memory 20, the steps of the parallel fusion method of the urban stock space multimodal data as described above are implemented.

[0165] The present invention also correspondingly provides a computer-readable storage medium. Among them, the computer-readable storage medium stores a parallel fusion program of urban stock space multimodal data. When the parallel fusion program of urban stock space multimodal data is executed by a processor, the steps of the parallel fusion method of urban stock space multimodal data as described above are implemented.

[0166] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or terminal. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or terminal including that element.

[0167] Of course, those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0168] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description. All such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A parallel fusion method for multi-modal data of urban stock space, characterized in that, The parallel fusion method for multi-modal data of urban stock space includes: Obtain multi-modal data of urban stock space, and classify, roughly extract feature vectors, and normalize the multi-modal data of urban stock space in sequence to obtain a stock space data set of full vectors; Design a multi-modal spatio-temporal index structure including a master node and multiple slave nodes, and construct a neural network model. The master node divides the stock space data set of full vectors and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly assigns the multiple sub-feature vectors and multiple sub-neural network models to each slave node; Based on the sub-feature vectors and the sub-neural network models, each slave node performs fine extraction of feature vectors through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain locally fused feature vectors; The master node obtains the locally fused feature vectors output by each slave node, performs global aggregation based on the multiple locally fused feature vectors by using a federated learning method to obtain globally fused feature vectors, and generates a target aggregation model through iterative training based on the globally fused feature vectors; Obtain multi-modal data of urban stock space to be fused, input the multi-modal data of urban stock space to be fused into the sub-neural network model to obtain target locally fused feature vectors, and input the target locally fused feature vectors into the target aggregation model to obtain a data fusion result.

2. The parallel fusion method of multi-modal data of urban stock space according to claim 1, wherein The obtaining of multi-modal data of urban stock space, and the classification, rough extraction of feature vectors, and normalization of the multi-modal data of urban stock space in sequence to obtain a stock space data set of full vectors specifically includes: Obtain multi-modal data of urban stock space, and divide the multi-modal data of urban stock space into raster data, graph data, and time series data; Preprocess the raster data, the graph data, and the time series data respectively to obtain preprocessed raster data, preprocessed graph data, and preprocessed time series data, where the preprocessing includes denoising and missing value filling; Perform unified abstraction and modeling on the preprocessed raster data, the preprocessed graph data, and the preprocessed time series data respectively to obtain a raster data model, a graph data model, and a time series data model; Use the raster data model, the graph data model, and the time series data model to roughly extract feature vectors from the preprocessed raster data, the preprocessed graph data, and the preprocessed time series data respectively to obtain rough feature vectors corresponding to the raster data, rough feature vectors corresponding to the graph data, and rough feature vectors corresponding to the time series data; Normalize the rough feature vectors corresponding to the raster data, the rough feature vectors corresponding to the graph data, and the rough feature vectors corresponding to the time series data respectively to obtain target rough feature vectors corresponding to the raster data, target rough feature vectors corresponding to the graph data, and target rough feature vectors corresponding to the time series data; Combine the target coarse feature vectors corresponding to the raster data, the target coarse feature vectors corresponding to the graph data, and the target coarse feature vectors corresponding to the time series data to obtain a stock space dataset of full vectors.

3. The parallel fusion method for multi-modal data of urban stock space according to claim 2, characterized in that The design includes a multi-modal spatio-temporal index structure of a master node and multiple slave nodes and constructing a neural network model. The master node divides the stock space dataset of full vectors and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly assigns the multiple sub-feature vectors and multiple sub-neural network models to each of the slave nodes. Specifically, it includes: Design a multi-modal spatio-temporal index structure. In the spatial dimension, use a quadtree for recursive partitioning to divide the two-dimensional space into multiple sub-regions. In the time dimension, use a linear index method to sort and retrieve data through timestamps. Among them, the quadtree includes a master node and multiple slave nodes. The master node represents the two-dimensional space, and each slave node represents each sub-region; Construct a neural network model. The neural network model includes a raster data feature extraction module, a graph data feature extraction module, a time series data feature extraction module, and a multi-modal data preliminary fusion module; Based on the multi-modal spatio-temporal index structure, the master node divides the stock space dataset of full vectors according to sub-regions and time ranges to obtain multiple sub-feature vectors, and the sub-feature vectors correspond one-to-one with the sub-regions; Based on the multi-modal spatio-temporal index structure, the master node divides the neural network model according to sub-regions to obtain multiple sub-neural network models, and the sub-neural network models correspond one-to-one with the sub-regions; Assign the sub-feature vectors and the sub-neural network models corresponding to each sub-region to the corresponding slave nodes; Establish a task allocation strategy. Dynamically adjust the task allocation ratio according to the computing power and memory capacity of each slave node, and preferentially allocate tasks to slave nodes whose distance from the data storage location is less than a set value, and introduce a backup mechanism to allocate some tasks to multiple slave nodes at the same time.

4. The parallel fusion method for multi-modal data of urban stock space according to claim 3, characterized in that, In the step of assigning the sub-feature vectors and the sub-neural network models corresponding to each sub-region to the corresponding slave nodes, the sub-feature vectors corresponding to each sub-region include sub-feature vectors corresponding to raster data, sub-feature vectors corresponding to graph data, and sub-feature vectors corresponding to time series data; the sub-neural network models corresponding to each sub-region include a sub-raster data feature extraction module, a sub-graph data feature extraction module, a sub-time series data feature extraction module, and a sub-multi-modal data preliminary fusion module.

5. The parallel fusion method for multi-modal data of urban stock space according to claim 4, wherein, The fine feature vectors include raster data feature vectors, graph data feature vectors, and time series data feature vectors; Based on the sub-feature vectors and the sub-neural network models, each slave node performs fine extraction of feature vectors through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors. Specifically, it includes: At each child node, the sub-feature vectors corresponding to the grid data are sequentially subjected to fine feature extraction of the feature vectors through the 3D convolutional layer, GeLU activation function layer, max pooling layer, residual block, global pooling layer, and first fully connected layer of the sub-grid data feature extraction module to obtain grid data feature vectors ; The sub-feature vectors corresponding to the graph data are successively subjected to fine feature extraction of the feature vectors through the graph convolutional network, the first ReLU activation function layer, the multi-head attention network, and the second fully connected layer of the sub-graph data feature extraction module to obtain graph data feature vectors ; The sub-feature vectors corresponding to the timing data are sequentially passed through the multi-layer long short-term memory network, the third fully connected layer, the second ReLU activation function layer, the dropout layer, and the fourth fully connected layer of the sub-timing data feature extraction module to perform fine extraction of the feature vectors, obtaining the timing data feature vectors ; Through the multi-head attention mechanism, the raster data feature vector , the graph data feature vector and the time series data feature vector are initially fused for multi-modal data to obtain a local fusion feature vector.

6. The parallel fusion method for multi-modal data of urban stock space according to claim 5, characterized in that, Through the multi-head attention mechanism, the raster data feature vector , the graph data feature vector and the time series data feature vector are initially fused for multi-modal data to obtain a local fusion feature vector, specifically including: The raster data feature vector , the graph data feature vector and the time series data feature vector are respectively passed through different linear transformation layers to obtain the first query vector Q1, the first key vector K1, and the first value vector V1 corresponding to the raster data feature vector , the second query vector Q2, the second key vector K2, and the second value vector V2 corresponding to the graph data feature vector , and the third query vector Q3, the third key vector K3, and the third value vector V3 corresponding to the time series data feature vector ; Calculate the first similarity between the first query vector Q1 and the third key vector K3 in parallel through multiple attention heads , the second similarity between the second query vector Q2 and the second key vector K2 , the third similarity between the third query vector Q3 and the first key vector K1 : ； ； ； where T represents transpose, denotes the dimension of the first key vector, the second key vector, or the third key vector; According to the first similarity , the second similarity and the third similarity , use the Softmax function to calculate the first attention weight corresponding to the raster data , the second attention weight corresponding to the graph data and the third attention weight corresponding to the time series data : ； ； ； According to the first attention weight , the second attention weight and the third attention weight , weighted summations are respectively performed on the third value vector V3, the second value vector V2, and the first value vector V1 to obtain a local fusion feature vector O: ； ； ； ； Among them, represents the attention mechanism.

7. The parallel fusion method for multi-modal data of urban stock space according to claim 6, wherein The master node obtains the local fusion feature vectors output by each of the slave nodes, performs global aggregation on the basis of the multiple local fusion feature vectors by using a federated learning method to obtain a global fusion feature vector, and generates a target aggregation model through iterative training based on the global fusion feature vector, which specifically includes: The master node obtains the local fusion feature vectors output by each of the slave nodes, and based on the weights corresponding to each of the slave nodes, uses a federated learning algorithm to perform weighted fusion on the local fusion feature vectors output by each of the slave nodes to obtain a global fusion feature vector : ； Among them, i represents the i-th child node, represents the locally fused feature vector output by the i-th child node, represents the weight corresponding to the i-th child node; wherein, the weight corresponding to the i-th child node according to the number of samples of the i-th child node is calculated as follows: ； Among them, represents the number of samples of the first child node, represents the number of samples of the second child node, represents the number of samples of the third child node, represents the number of samples of the fourth child node; Updating the aggregation weight parameters of the master node by using the global fusion feature vector generated by the master node, and feeding back the updated aggregation weight parameters to each slave node to obtain a target aggregation model; Wherein, the aggregation weight parameters of the master node include the weights corresponding to each of the slave nodes.

8. A parallel fusion system for multi-modal data of urban stock space, characterized in that, The parallel fusion system for urban stock space multi-modal data includes: A data preliminary processing module: configured to obtain urban stock space multi-modal data, and classify, roughly extract feature vectors, and normalize the urban stock space multi-modal data in sequence to obtain a full-vector stock space data set; A distributed task allocation module: configured to design a multi-modal spatio-temporal index structure including a master node and multiple slave nodes and construct a neural network model, and the master node divides the full-vector stock space data set and the neural network model based on the multi-modal spatio-temporal index structure to obtain multiple sub-feature vectors and multiple sub-neural network models, and correspondingly allocate the multiple sub-feature vectors and multiple sub-neural network models to each of the slave nodes; A preliminary fusion module: configured to, based on the sub-feature vectors and the sub-neural network models, each slave node performs fine extraction of feature vectors through a deep learning algorithm to obtain fine feature vectors, and performs preliminary fusion of multi-modal data on the fine feature vectors through a multi-head attention mechanism to obtain local fusion feature vectors; A global aggregation module: configured to the master node obtains the local fusion feature vectors output by each of the slave nodes, performs global aggregation on the basis of the multiple local fusion feature vectors by using a federated learning method to obtain a global fusion feature vector, and generates a target aggregation model through iterative training based on the global fusion feature vector; A multi-modal data fusion module: configured to obtain urban stock space multi-modal data to be fused, input the urban stock space multi-modal data to be fused into the sub-neural network model to obtain a target local fusion feature vector, and input the target local fusion feature vector into the target aggregation model to obtain a data fusion result.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a parallel fusion program for urban stock space multi-modal data stored on the memory and executable on the processor, and when the parallel fusion program for urban stock space multi-modal data is executed by the processor, the steps of the parallel fusion method for urban stock space multi-modal data according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a parallel fusion program for urban stock space multi-modal data, and when the parallel fusion program for urban stock space multi-modal data is executed by a processor, the steps of the parallel fusion method for urban stock space multi-modal data according to any one of claims 1-7 are implemented.

Images