A Dynamic Load Balancing Method and System for Large-Scale Real-Time Spatial Computing

Abstract

The present invention discloses a dynamic load balancing method and system for large-scale real-time spatial computing. The method includes the following steps: obtaining a geographical spatial area to be calculated and performing rasterization preprocessing; performing downsampling processing on the original data within the rasterized geographical spatial area based on spatio-temporal step lengths to obtain downsampled geographical spatial data; constructing an LSTM+CNN prediction model, using the geographical spatial data of multiple time periods as inputs to predict future grid data within the rasterized geographical spatial area; performing dynamic spatial partitioning on the predicted future grid data to balance the data volume of each partition within a single node; and processing the data of each partition using a multi-threaded parallel computing method. The present invention relates to the technical field of spatial location data computing. By performing downsampling based on spatio-temporal step lengths and achieving load balancing for each process in a node through dynamic spatial partitioning, the maximum utilization of computing resources is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of spatial position data calculation, and more specifically, to a dynamic load balancing method and system for large-scale real-time spatial calculation. Background Art

[0002] Currently, with the rapid development of the Internet of Things technology and the widespread use of satellite positioning devices, the generation and processing of large-scale real-time spatial position data have become an important part of modern information systems. These data are widely used in fields such as transportation and logistics, and are of great significance for improving urban management efficiency and optimizing resource allocation.

[0003] However, in the face of massive and real-time updated spatial position data, traditional data processing methods face many challenges:

[0004] 1. Huge data volume: Real-time spatial position data is usually generated at a high frequency, with a huge data volume, posing extremely high requirements for storage and processing capabilities.

[0005] 2. Complex spatio-temporal characteristics: The distribution of spatial position data is uneven and dynamically changing, and fixed spatial partitioning will lead to problems of uneven computing load.

[0006] 3. High real-time requirements: Many application scenarios require data processing to be completed within an extremely short time (such as map matching) to ensure the timeliness and accuracy of decision-making.

[0007] In response to the above problems, existing solutions often adopt static resource allocation strategies and fixed processing flows, and cannot effectively cope with the dynamic changes and uneven loads of data.

[0008] Therefore, proposing a dynamic load balancing method for large-scale real-time spatial calculation is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0009] In view of this, the present invention provides a dynamic load balancing method and system for large-scale real-time spatial calculation, which realizes load balancing of each node in the computing cluster and load balancing of each process in the node in the large-scale real-time spatial position data processing scenario, so as to achieve the maximum utilization of computing resources.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] In the first aspect, the present invention provides a dynamic load balancing method for large-scale real-time spatial calculation, including the following steps:

[0012] S1. Obtain the geographical spatial area to be calculated and perform rasterization preprocessing;

[0013] S2. Downsample the original data within the rasterized geospatial region based on the spatio-temporal step size to obtain the downsampled geospatial data;

[0014] S3. Construct an LSTM+CNN prediction model, take the geospatial data of multiple time periods as input, and predict the future grid data within the rasterized geospatial region;

[0015] S4. Utilize the computing power resources of a single server node to perform dynamic spatial partitioning on the predicted future grid data to balance the data volume of each partition within the single node; Process the data of each partition using a multi-threaded parallel computing method.

[0016] Furthermore, it also includes:

[0017] S5. When using server cluster nodes for processing, according to the distribution of the predicted future grid data and the number of server cluster nodes, allocate the predicted future grid data to different cluster single nodes;

[0018] Utilize the corresponding computing power resources of the cluster single nodes to perform dynamic spatial partitioning on the allocated data and process the data of each partition.

[0019] Furthermore, in step S1, the rasterization preprocessing specifically includes:

[0020] Divide the outer envelope rectangle of the geospatial region to be calculated into grids;

[0021] Take the preset vertex as the origin, and perform grid increment encoding along the horizontal and vertical coordinate directions of the preset vertex to obtain grid coordinates.

[0022] Furthermore, step S2 specifically includes:

[0023] S21. In the time dimension, the time step size formula for the original data within the rasterized geospatial region changing within the time range t is:

[0024] DT = ceil(t / ST)

[0025] In the spatial dimension, the spatial step size formula for the original data within the rasterized geospatial region changing from grid (x1, y1) to grid (x2, y2) is:

[0026] DL = ceil((abs(x1 - x2) + abs(y1 - y2)) / SL)

[0027] where ST represents a preset time step size, SL represents a preset number of SL grids as a spatial step size, ceil represents the ceiling operation, and abs represents the absolute value operation;

[0028] S22. The spatio-temporal step D for the movement of the original data within the rasterized geospatial region is D = DT + DL. Sampling is performed every preset M spatio-temporal steps D to obtain downsampled geospatial data.

[0029] Further, in step S3, using the geospatial data of multiple time periods as input to predict the future grid data within the rasterized geospatial region includes:

[0030] Obtaining the geospatial data of V time periods at an interval of K time, and proportionally mapping them to the range of 0 - 255 to obtain V grayscale images;

[0031] Using the LSTM+CNN prediction model to predict the grayscale image of the grid at the next interval of K time, and performing inverse mapping processing to obtain the corresponding data as the future grid data.

[0032] Further, step S4 specifically includes:

[0033] S41. Performing partition processing on the computing power resources of a single server node;

[0034] S42. Obtaining a dynamic spatial partition task, which is to linearly allocate the predicted future grid data to each partition to achieve an equal number of data points in each partition;

[0035] S43. Constructing a partition list, the content of which is repeated partition numbers, and the length of the partition list is the number n of the future grid data;

[0036] S44. Randomly selecting a point within each partition range as the center point of the partition;

[0037] S45. Constructing a cost matrix with a dimension of n*n; n is the number of the future grid data, the row labels of the cost matrix are the numbers of the future grid data, and the column labels of the cost matrix are the labels of the partition list; the content of the cost matrix is the straight-line distance from the future grid data to the center point;

[0038] S46. Using the Jonker-Volgenant algorithm for iterative adjustment until the sum of the distances from the data points in the cost matrix to the corresponding partition center points is minimized to obtain the matching result between the data points and each partition;

[0039] S47. Updating the center point of each partition, reconstructing the cost matrix, and updating the matching result between the data points and each partition; stopping the update until the offset distance before and after the update of the center points of all partitions is less than a preset value to obtain the final matching result between the data points and each partition;

[0040] S48. Use the partition number with the largest proportion of data point partitions as the partition number of the grid where the data point is located to achieve dynamic load balancing for each partition.

[0041] Further, step S46 specifically includes:

[0042] Initialize the cost matrix and the potential values of the data points and partition centers;

[0043] Construct a matching array with a length of n to store the matching results of data points and partition centers;

[0044] Find an augmenting path, reassign data points, and increase the number of matches;

[0045] Until an augmenting path is found, reverse the matching status along this path to optimize the matching relationship;

[0046] Update the matching array according to the new matching relationship;

[0047] Continuously repeat finding the augmenting path and updating the matching array until all data points are optimally assigned;

[0048] Output the matching results of data points and each partition center, and calculate the total cost of all data points and partition centers.

[0049] Further, the finding of the augmenting path specifically includes:

[0050] Use breadth-first search to start from unassigned centers and find an augmenting path; find a better matching result by adjusting the slack value of data points;

[0051] Adjust the potential values of partition centers and data points to open more augmenting paths; according to the adjusted potential values, recalculate the slack value of each data point to reflect the latest potential value situation.

[0052] Further, in step S5, when using server cluster nodes for processing, according to the predicted distribution of future grid data and the number of server cluster nodes, allocate the predicted future grid data to different cluster single nodes; specifically including:

[0053] When using server cluster nodes for processing, according to the predicted distribution vector of future grid data, use cosine distance to calculate the similarity and distance between vectors to obtain a distance matrix;

[0054] Use the DBSCAN algorithm to cluster the distance matrix; obtain multiple data clusters with similar distribution characteristics;

[0055] According to the number of server cluster nodes, the data under each cluster is divided into corresponding parts, and each part is allocated to the corresponding single cluster node.

[0056] In a second aspect, the present invention provides a dynamic load balancing system for large-scale real-time spatial computing, including the following modules:

[0057] A preprocessing module: used to obtain the geospatial area to be calculated and perform rasterization preprocessing;

[0058] A downsampling processing module: used to perform downsampling processing on the original data in the rasterized geospatial area based on the spatio-temporal step length to obtain the downsampled geospatial data;

[0059] A prediction module: used to construct an LSTM+CNN prediction model, take the geospatial data of multiple time periods as input, and predict the future grid data in the rasterized geospatial area;

[0060] An equilibrium processing module: used to utilize the computing power resources of a single server node to perform dynamic spatial partitioning on the predicted future grid data to balance the data volume of each partition within the single node; and process the data of each partition by using a multi-threaded parallel computing method.

[0061] For the description of the second aspect in the present invention, reference can be made to the detailed description of the first aspect; and for the beneficial effects of the description of the second aspect, reference can be made to the analysis of the beneficial effects of the first aspect, which will not be elaborated here.

[0062] It can be seen from the above technical solutions that compared with the prior art, the present invention discloses a dynamic load balancing method for large-scale real-time spatial computing, which has the following beneficial effects:

[0063] By performing real-time geospatial data downsampling based on the spatio-temporal step length, the sampling frequency of data that is stationary for a long time or changes slowly for a long time is reduced, thereby reducing information redundancy, avoiding unnecessary computing overhead, and improving the subsequent data processing efficiency.

[0064] By predicting the raster data volume based on LSTM+CNN, the data volume that will appear in each grid in the future is estimated to cope with the dynamic changes in the spatial distribution of data, and further prepare for subsequent spatial equilibrium partitioning.

[0065] By achieving node process load balancing based on dynamic spatial partitioning and using the Jonker-Volgenant algorithm to solve the spatial partitioning problem, it is ensured that the data volume falling in each partition is basically equal, and each partition is a closed area to ensure that the topological relationship of geographical elements is not damaged.

[0066] Achieve load balancing of different nodes in the cluster through clustering based on spatial distribution feature vectors. Measure the similarity degree of the spatial distribution features of different objects through cosine similarity, and perform DBSCAN clustering based on this similarity. Randomly and evenly distribute the objects in each category to each cluster node to ensure that the data distribution of each node is consistent with the predicted result. Brief Description of the Drawings

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative efforts.

[0068] Figure 1 Schematic diagram of a dynamic load balancing method for large-scale real-time spatial computing provided by an embodiment of the present invention.

[0069] Figure 2 Schematic diagram of regional grid preprocessing provided by an embodiment of the present invention.

[0070] Figure 3 Schematic diagram of the input and output of the LSTM+CNN prediction model provided by an embodiment of the present invention.

[0071] Figure 4 Schematic diagram of the result of the partition number of the grid where the data point is located provided by an embodiment of the present invention.

[0072] Figure 5 Schematic diagram of the spatial partition results at different time periods provided by an embodiment of the present invention.

[0073] Figure 6 Schematic diagram of the DBSCAN clustering result provided by an embodiment of the present invention.

[0074] Figure 7 Schematic diagram of a dynamic load balancing system for large-scale real-time spatial computing provided by an embodiment of the present invention. Detailed Embodiments

[0075] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0076] Embodiment 1

[0077] An embodiment of the present invention discloses a dynamic load balancing method for large-scale real-time spatial computing. Refer to Figure 1 as shown, which includes steps S1 - S5.

[0078] The following details each step:

[0079] Step S1: Obtain the geospatial region to be calculated and perform rasterization preprocessing;

[0080] Facing large-scale and real-time updated spatial location data, obtain the geospatial region to be calculated, convert the continuous and infinite longitude and latitude point data in the region into discrete and finite spatial data to improve spatial processing efficiency. It is necessary to perform rasterization processing on the region and encode the rasters.

[0081] Refer to Figure 2 as shown, the specific rasterization encoding method is as follows:

[0082] (1) Determine the geospatial region to be calculated. For example, divide it according to the administrative region scope. Refer to Figure 2 the administrative region scope of Guangzhou City in

[0083] (2) Take the lower left corner point of the regional bounding rectangle as the origin, and generate a square grid with a side length of several meters centered on the origin to obtain the origin grid, which is encoded as (0, 0).

[0084] (3) Encoding method for the remaining grids: The first digit of the encoding increases from west to east, and the second digit of the encoding increases from south to north.

[0085] Step S2: Perform downsampling processing on the original data within the rasterized geospatial region based on the spatio-temporal step size to obtain the downsampled geospatial data;

[0086] Spatial data often has a large amount of redundant data with less information. For example, the longitude and latitude of a vehicle that remains stationary for a long time or moves slowly for a long time changes slightly over time. Another example is that due to equipment failures and other reasons, the upload time interval is too short in a short period. To address the above data redundancy problem, perform downsampling processing on the original spatial location data based on the spatio-temporal step size, thereby reducing information redundancy and improving the subsequent data processing efficiency.

[0087] The specific downsampling method is as follows:

[0088] (1) In the time dimension, take ST seconds as a time step size. That is, the time step size DT that an object moves within t seconds is DT = ceil(t / ST), where ceil represents the ceiling operation. For example: ceil(4 / 5) = 0, cel(6 / 5) = 1.

[0089] (2)In the spatial dimension, taking the span of SL grids as a spatial step, that is, the spatial step for an object to move from grid (x1, y1) to grid (x2, y2) is DL = ceil((abs(x1 - x2)+abs(y1 - y2)) / SL), where ceil represents the ceiling operation and abs represents the absolute value operation; x and y respectively represent the grid coordinate encodings where the original data is located before downsampling.

[0090] (3)Calculate the spatio-temporal step D = DT + DL of the object's movement, and sample once every M spatio-temporal steps, that is, the downsampled geospatial data is obtained.

[0091] In this embodiment, ST = 15 seconds, SL = 3, and M = 5 are set. The downsampling process of the geospatial time-series data is shown in Table 1:

[0092] Table 1

[0093]

[0094] Step S3, construct an LSTM+CNN prediction model, and predict the future grid data within the rasterized geospatial region according to the geospatial data of multiple time periods;

[0095] In order to estimate the data volume that will appear in each grid in the future and prepare for subsequent spatial equalization partitioning, this step performs raster data volume prediction.

[0096] The prediction steps refer to Figure 3 As shown, it is: process the grid data distribution of multiple recent time periods into multiple grayscale images, construct an LSTM+CNN prediction model to predict the future grid data volume, and output the prediction result. The specific steps are as follows:

[0097] (1)Process the grid data distribution of multiple recent time periods into multiple grayscale images

[0098] Taking K minutes as the statistical time interval, record the downsampled data volume of the grid from time t - K to time t as N(t, i, j), where i and j respectively represent the encoding values of the grid where the downsampled data is located, and obtain the data volumes of all grids in the recent V time periods, that is, N(t - (V - 1)*K, i, j), N(t - (V - 2)*K, i, j)... N(t, i, j).

[0099] Map the original grid data volume proportionally to the range of 0~255, record the maximum value among the data volumes of all time periods and all grids obtained as MAX_N, and record the mapped value as M(t, i, j). The calculation formula is M(t, i, j)=ceil(N(t, i, j) / MAX_N*255), where ceil represents the ceiling operation.

[0100] (2)Construct an LSTM+CNN prediction model to predict future grid data volume

[0101] The deep learning models LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network) are used to process time series data and spatial data respectively. LSTM+CNN is a hybrid architecture that combines these two models, aiming to utilize their respective advantages to solve more complex problems. First, CNN is used to extract the spatial features of the input data, and then these features are input into LSTM for time series modeling. This combination can capture both the spatial and temporal information of the data simultaneously.

[0102] The sample volume mapping value for each time period can be regarded as a grayscale image, obtaining V grayscale images. Use the LSTM+CNN model to predict the mapped data volume M(t+K,i,j) of the grid in the next K minutes, and perform inverse mapping processing N(t+K,i,j)=M(t+K,i,j)*MAX_N / 255.

[0103] Record the total predicted data volume from time t-K to time t as N_PREDICT, where .

[0104] Step S4: Utilize the computing power resources of a single server node to perform dynamic spatial partitioning on the predicted future grid data to balance the data volume of each partition within a single node; use a multi-threaded parallel computing method to process the data of each partition.

[0105] To make full use of the computing power resources of a single node, a multi-process parallel computing method is adopted for real-time map matching processing. Each process is responsible for the computing and processing of geospatial data within a partition. At the same time, to ensure that the computing load of each process is basically the same, it is necessary to perform reasonable spatial partitioning according to the predicted future data distribution situation in step S3 to achieve load balancing. The ideal spatial partitioning result is to make the data volume falling within each partition equal in the future time period.

[0106] Regard the spatial partitioning problem as a linear assignment problem (Linear Assignment Problem), and use the Jonker-Volgenant algorithm to solve the problem. The specific spatial partitioning method is as follows:

[0107] (1)If a single computing node has N_CORE CPU cores available for computing, then the entire area needs to be divided into N_CORE partitions, and the partition numbers are 1, 2,..., N_CORE respectively.

[0108] In this embodiment, a single computing node has 3 CPU cores available for computing. Therefore, the entire area needs to be divided into 3 partitions, and the partition numbers are 1, 2, and 3 respectively.

[0109] (2) Consider each data point as a "worker" in the linear task assignment problem, and consider the partition to which the data point is assigned as the "worker" being assigned to a certain "task" in the linear task assignment problem.

[0110] There are 10 data points in the area of this embodiment, and their longitude and latitude coordinates are shown in Table 2 as follows:

[0111] Table 2

[0112]

[0113] (3) Construct a "task" list. Construct a "task" list with a length of N_PREDICT. The content in the list is the partition numbers that repeat continuously, that is, 1, 2,..,, N_CORE, 1, 2,..., N_CORE, 1, 2,..., N_CORE,... until the length of the list reaches N_PREDICT and ends; the content of the list in this embodiment is 1, 2, 3, 1, 2, 3, 1, 2, 3, 1.

[0114] Through the above construction, the number of data points assigned to each partition is basically equal, thus achieving spatial balance.

[0115] (4) Randomly generate the center points of the partitions. Randomly select a point within the area range as the center point of each partition.

[0116] The center points of the partitions selected in this embodiment are shown in Table 3 as follows:

[0117] Table 3

[0118]

[0119] (5) Construct a "cost matrix". Construct a "cost matrix" with a dimension of N_PREDICT*N_PREDICT, that is, a 10*10 "cost matrix"; the row index in the matrix represents the index number of the data point ("worker"), denoted as point_index, and the column index represents the index label of the spatial partition ("task"), denoted as partition_index. The value at the position (point_index, partition_index) in the matrix represents the straight-line distance from the data point point_index to the center point of the partition partition_index. The cost matrix is shown in Table 4 as follows:

[0120] Table 4

[0121]

[0122] (6) Use the Jonker-Volgenant algorithm to solve the above linear task assignment problem, with the goal of minimizing the total cost, that is, minimizing the sum of the distances from the partition center points to the corresponding partition data points, so that the data points in each partition can be clustered together. The "task assignment result" is obtained by solving, that is, each data point is assigned to a specific spatial partition.

[0123] The Jonker-Volgenant algorithm is an efficient algorithm for solving the linear task assignment problem (also known as the minimum weight matching problem of a bipartite graph). In this embodiment, the linear task assignment problem is described as follows: Given a bipartite graph; one side is the set of workers, in this problem, it is the geographical space data points, and the other side is the set of tasks, in this problem, it is the spatial partitions or the spatial partition center points. There is a weight, that is, cost or benefit, between each task and worker. In this problem, the weight is the straight-line distance from the geographical space data point to the spatial partition center point. The goal is to find a matching scheme between workers and tasks to minimize the total weight; in this problem, the goal is to find a matching scheme between geographical space data points and spatial partitions to minimize the sum of the distances from all data points to the center points of their respective spatial partitions.

[0124] The core idea of the Jonker-Volgenant algorithm is to gradually reduce the total cost by iteratively adjusting the matching until the optimal matching is found. The specific steps are as follows:

[0125] (6.1) Initialize the cost matrix: Given an n*n cost matrix C, where C[i][j] represents the cost of assigning task i to worker j.

[0126] (6.2) Initialize the potential values: Initialize a potential value for each task and worker, which is initialized to zero in this embodiment. The potential value of task i is denoted as u[i], and the potential value of worker j is denoted as v[j].

[0127] (6.3) Construct the matching array: Initialize an array match of length n to store the matching results. match[j] represents the task assigned to worker j, and all elements are initially set to -1, indicating unmatched.

[0128] (6.4) Find an augmenting path: An augmenting path is an alternating path, and the edges on the path alternately belong to the matching and do not belong to the matching. The goal is to find an augmenting path from a task to a worker so that the number of matches can be increased by flipping the matching status on the path.

[0129] The ways to find an augmenting path include:

[0130] Breadth-First Search (BFS): Use BFS to start from all unassigned tasks and find an augmenting path. During the search, maintain a queue and an array 'parent' to record the path.

[0131] Relaxation operation: During the BFS process, for each task i and worker j, check if a better match can be found by adjusting the potential values. Specifically, check the following condition: C[i][j] - u[i] - v[j] < slack[j], where slack[j] is the slack value of worker j, representing the current minimum cost adjustment amount.

[0132] Adjust potential values: If no augmenting path is found, adjust the potential values of tasks and workers to allow more augmenting paths to be found. Specifically, for each unassigned task i, decrease its potential value u[i], and for each unassigned worker j, increase its potential value v[j].

[0133] Update slack values: Update the slack value slack[j] for each worker j according to the adjusted potential values.

[0134] (6.5) Update the matching: After finding an augmenting path, flip the matching status on the path. Specifically, for each edge on the path, if it currently belongs to the matching, remove it from the matching; if it does not belong to the matching, add it to the matching.

[0135] (6.6) Update the matching array: Update the'match' array to record the new matching relationship.

[0136] (6.7) Repeat the iteration: Repeat the process of finding augmenting paths, updating the matching, and updating the matching array until all tasks are assigned.

[0137] (6.8) Output the result: Finally, each element match[i][j] in the'match' array represents the task i assigned to worker j. Calculate the total cost, which is the sum of the costs of all matched tasks and workers:

[0138]

[0139] In this embodiment, after all tasks are assigned, it is obtained that:

[0140] Partition 1: Data points 2, 3, 8, 9; Partition 2: Data segments 6, 7, 10; Partition 3: Data points 1, 4, 5.

[0141] (7) Update the partition center points, that is, calculate the average of the longitudes and latitudes of all data points within each partition. The calculation results of the updated partition center points are shown in Table 5 below:

[0142] Table 5

[0143]

[0144] (8) Repeat step (5), and reconstruct the "cost matrix" using the updated partition center points, as shown in Table 6 below:

[0145] Table 6

[0146]

[0147] Repeat step (6) to obtain the "task assignment result", that is, each data point is assigned to a specific spatial partition. The task assignment result refers to the bold values in the cost matrix shown in Table 6. That is, Partition 1: Data points 2, 3, 8, 9; Partition 2: Data segments 4, 7, 10; Partition 3: Data points 1, 5, 6.

[0148] Until the offset distance of the center points before and after the update of all partitions is less than the preset value. In this embodiment, THRES_DIST = 0.0001, stop the iteration.

[0149] (9) Partition the grid. Refer to Figure 4 as shown, distinguish the positions of data points, the positions of spatial partition center points, and spatial grids with different markings; among them, different partitions have different colors, pink represents Partition 1, orange represents Partition 2, and gray represents Partition 3; use the partition number with the largest proportion of data points in the grid as the partition number to which the grid belongs. Refer to Figure 5 as shown, which is the spatial partition result at different time periods provided in this embodiment.

[0150] Step S5. When using server cluster nodes for processing, according to the predicted distribution of future grid data and the number of server cluster nodes, allocate the predicted future grid data to different cluster single nodes; utilize the computing power resources of the corresponding cluster single nodes to perform dynamic spatial partitioning on the allocated data and process the data in each partition.

[0151] In large-scale spatial location data calculation scenarios, in addition to achieving load balancing among different processes within a single node through dynamic spatial partitioning, it is often necessary to use a server cluster to meet the real-time processing requirements. This step takes vehicle analogy spatial location data as an example, and realizes the balanced distribution of computing loads among different nodes in the cluster by reasonably allocating vehicles to different cluster nodes.

[0152] In the embodiment, the position point distribution of each vehicle on the previous day is calculated at 0:00 a.m. every day, that is, the number of position points falling in each grid after vehicle downsampling, which is recorded as N_CAR(plate,i,j), where plate represents the license plate number, i and j represent grid codes, and they are first arranged in ascending order by i and then in ascending order by j to obtain the vehicle position spatial distribution feature vector V_CAR(plate)=(N_CAR(plate,0,0), N_CAR(plate,0,1), N_CAR(plate,0,2), ..., N_CAR(plate,i,j))...), which is called the vehicle data distribution vector.

[0153] The cosine distance is used to calculate the similarity between vectors. For example, the data distribution vector similarity between plate1 and plate2 is COS_SIM(plate1, plate2) = cos(V_CAR(plate1), V_CAR(plate2)); the distance between vectors is DIST(plate1, plate2)=1-COS_SIM(plate1, plate2). The vector distances between all pairs of vehicles are calculated to obtain the distance matrix.

[0154] The distance matrix is ​​input into the DBSCAN algorithm for clustering to obtain several vehicle clusters with similar travel distribution characteristics. The vehicles in each cluster are then randomly divided into Q parts, and each part is assigned to the corresponding cluster node.

[0155] Reference Figure 6 As shown in the figure, the DBSCAN clustering results are displayed. The horizontal axis is longitude and the vertical axis is latitude. The spatial data distribution maps of three vehicles are selected from the two clusters obtained by clustering for display. It can be seen that the spatial data distribution characteristics of vehicles in the same cluster are highly similar.

[0156] This embodiment uses the DBSCAN algorithm to obtain F clusters, with cluster labels C(1), C(2)...C(F), and the number of vehicles under each cluster is V(1), V(2)...V(F). Next, the vehicles of each category are randomly divided into Q parts, and the category 1 vehicles falling in node 1 are R(1,1), with a total of V(1) / Q vehicles, and the category F vehicles falling in node Q are R(Q,F), with a total of V(K) / Q vehicles, and the rest of the nodes are similar.

[0157] All vehicles R(1,1), R(1,2), ..., R(1,F) that fall on node 1 are merged to form the vehicle set of node 1, and all vehicles R(Q,1), R(Q,2), ..., R(Q,F) that fall on node Q are merged to form the vehicle set of node Q, and so on for the remaining nodes.

[0158] The present invention first obtains the geospatial region to be calculated and performs grid processing; secondly, a real-time geospatial data downsampling method based on spatio-temporal step lengths is adopted to filter redundant data within the grid, and then the LSTM+CNN model is used to predict the grid data volume in the future period. Multiprocess parallel processing is adopted inside a single node to process data in different spatial partitions. Through a dynamic spatial partitioning method based on linear assignment problem (LAP), according to the predicted grid data distribution, the Jonker-Volgenant algorithm is used to solve the spatial partitioning problem to ensure that the data volume falling in each partition is basically equal, and at the same time each partition is a closed region to ensure that the topological relationship of geographical elements is not damaged.

[0159] When using server cluster nodes for processing, the cosine similarity is also used to measure the similarity of the spatial distribution characteristics of different objects, and DBSCAN clustering is performed based on this similarity. The objects in each category are randomly and evenly assigned to each cluster node to ensure that the data distribution of each node is consistent with the predicted result, and finally, load balancing of multiple nodes within the cluster and load balancing of multiple processes within the node are achieved, making full use of computing resources.

[0160] Embodiment 2

[0161] An embodiment of the present invention discloses a dynamic load balancing system for large-scale real-time spatial computing. Referring to Figure 7 as shown, it includes the following modules:

[0162] A preprocessing module: used to obtain the geospatial region to be calculated and perform rasterization preprocessing;

[0163] A downsampling processing module: used to perform downsampling processing on the original data within the rasterized geospatial region based on spatio-temporal step lengths to obtain the downsampled geospatial data;

[0164] A prediction module: used to construct an LSTM+CNN prediction model and predict the future grid data within the rasterized geospatial region according to the geospatial data of multiple periods;

[0165] An equalization processing module: used to utilize the computing power resources of a single server node to perform dynamic spatial partitioning on the predicted future grid data to balance the data volume of each partition; and adopt a multi-thread parallel computing method to process the data of each partition.

[0166] In this embodiment, the preprocessing module first obtains the geospatial region to be calculated and performs rasterization preprocessing; then, through the downsampling processing module, the rasterized geospatial data is downsampled based on the spatio-temporal step, and the downsampled geospatial data is obtained after filtering redundant data; then, through the prediction module, the future grid data within the rasterized geospatial region is predicted to cope with the dynamic changes in the spatial distribution of the data, thereby preparing for subsequent spatial equalization zoning. Finally, through the equalization processing module, dynamic spatial zoning is performed to ensure that the amount of data falling in each zone is basically equal, and each zone is a closed area to ensure that the topological relationship of geographical elements is not damaged.

[0167] The equalization processing module of this embodiment is also used for when processing by server cluster nodes, according to the distribution of the predicted future grid data and the number of server cluster nodes, allocating the predicted future grid data to different cluster single nodes; and using the computing power resources of the corresponding cluster single nodes to perform dynamic spatial zoning on the allocated data and process the data in each zone.

[0168] This system solves the deficiencies in the prior art:

[0169] Deficiency one: The original spatial location data is not filtered according to the information gain, resulting in the processing of a large amount of redundant information, with unnecessary computational overhead and reduced real-time spatial location data processing efficiency.

[0170] Deficiency two: In the process of spatial zoning, it is not considered that each zone needs to form a closed area to maintain the topological relationship of geographical elements within the zone. For example, real-time map matching calculation needs to ensure the complete connectivity of road segments within each zone, and non-closed spatial zoning will lead to incorrect processing results.

[0171] Deficiency three: Spatial division is only based on the spatial distribution of historical data, without considering that the data distribution will change dynamically over time, and the future spatial distribution of data will be different from the historical spatial distribution to a certain extent, resulting in the current division scheme being unable to achieve a good load balancing effect in the future.

[0172] Deficiency four: In addition to considering the load balancing of different processes within a single node, the load balancing problem of different nodes is not considered, resulting in a large difference in the computing load between different nodes and affecting the real-time processing efficiency.

[0173] This system realizes the load balancing of each node in the computing cluster and the load balancing of each process in the node in the scenario of large-scale real-time spatial location data processing, so as to maximize the utilization of computing resources.

[0174] In the present specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts among the various embodiments can be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method section.

[0175] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dynamic load balancing method for large-scale real-time spatial computing, characterized in that: The following steps are involved: S1, obtain the geographic space area to be calculated and perform rasterization preprocessing; S2, downsampling the original data in the rasterized geographic spatial area based on the spatiotemporal step to obtain downsampled geographic spatial data; S3, constructing an LSTM+CNN prediction model, taking the geospatial data of multiple time periods as input, and predicting future grid data within the gridded geospatial area; S4, using the computing power resources of a single server node to dynamically partition the predicted future grid data, so that the amount of data in each partition within a single node is balanced; Use multi-threaded parallel computing to process the data of each partition; Wherein, step S4 specifically includes: S41, partitioning the computing resources of a single server node; S42, obtaining a dynamic space partitioning task, wherein the dynamic space partitioning task is to linearly distribute the predicted future grid data to each partition to achieve a balanced number of data points in each partition; S43, constructing a partition list, wherein the content of the partition list is repeated partition numbers, and the length of the partition list is the number n of the future grid data; S44, randomly selecting a point within each partition range as the center point of the partition; S45, constructing a cost matrix with a dimension of n*n; n is the number of the future grid data, the row label of the cost matrix is ​​the serial number of the future grid data, and the column label of the cost matrix is ​​the label of the partition list; the content of the cost matrix is ​​the straight-line distance from the future grid data to the center point; S46, using the Jonker-Volgenant algorithm to iteratively adjust until the sum of the distances from the data points to the corresponding partition center points in the cost matrix is ​​minimized, and the matching results between the data points and each partition are obtained; S47, updating the center point of each partition, reconstructing the cost matrix, and updating the matching results between the data point and each partition; until the offset distance before and after the update of the center point of all partitions is less than the preset value, then stop updating to obtain the final matching results between the data point and each partition; S48. The partition number of the data point partition with the largest proportion is used as the partition number of the grid where the data point is located, so as to achieve dynamic load balancing of each partition.

2. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: Also includes: S5. When server cluster node processing is adopted, the predicted future grid data is distributed to different cluster single nodes according to the distribution of the predicted future grid data and the number of server cluster nodes; Utilize the corresponding cluster single-node computing resources to dynamically partition the allocated data and process the data in each partition.

3. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: In step S1, the rasterization preprocessing specifically includes: Dividing the outer envelope rectangle of the geographic space area to be calculated into grids; The preset vertex is taken as the origin, and grid incremental coding is performed along the horizontal coordinate and the vertical coordinate direction of the preset vertex to obtain the grid coordinates.

4. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: Step S2 specifically includes: S21. In the time dimension, the time step formula of the original data in the rasterized geographic space area within the time range t is: DT=ceil(t / ST) In the spatial dimension, the spatial step formula for the original data in the rasterized geographic space area to change from the grid (x1, y1) to the grid (x2, y2) is: DL=ceil((abs(x1-x2)+abs(y1-y2)) / SL) Among them, ST represents a preset time step, SL represents a preset SL grids as a space step, ceil represents a rounding operation, and abs represents an absolute value operation; S22, the spatiotemporal step length D of the original data movement in the gridded geographic space area is DT+DL, and sampling is performed every M preset spatiotemporal steps D to obtain downsampled geographic space data.

5. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: In step S3, the geospatial data of multiple time periods are used as input to predict future grid data in the gridded geospatial area; including: Obtain the geographic spatial data of V time periods at intervals of K time, and map them proportionally to a range of 0 to 255 to obtain V grayscale images; The LSTM+CNN prediction model is used to predict the grayscale image of the grid at the next interval K time, and an inverse mapping process is performed to obtain the corresponding data as the future grid data.

6. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: Step S46 specifically includes: Initializing the cost matrix, initializing the potential values ​​of the data points and partition center points; Construct a matching array, the length of which is n, for storing the matching results of the data points and the partition center points; Find augmenting paths, redistribute data points, and increase the number of matches; Until an augmenting path is found, the matching state is reversed along the path to optimize the matching relationship; Update the matching array according to the new matching relationship; Repeat the process of finding augmenting paths and updating matching arrays until all data points are optimally allocated. Output the matching results of data points and each partition center point, and calculate the total cost of all data points and partition center points.

7. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 6, characterized in that: The finding of the augmenting path specifically includes: Use breadth-first search to find augmenting paths starting from the unassigned center point; find better matching results by adjusting the relaxation value of the data points; Adjust the potential values ​​of the partition center and data points to open more augmentation paths; based on the adjusted potential values, recalculate the relaxation value of each data point to reflect the latest potential value.

8. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 2, characterized in that: In step S5, when the server cluster node processing is adopted, the predicted future grid data is distributed to different cluster single nodes according to the distribution of the predicted future grid data and the number of server cluster nodes; specifically, it includes: When server cluster nodes are used for processing, the cosine distance is used to calculate the similarity and distance between vectors according to the distribution vectors of the predicted future grid data, and the distance matrix is ​​obtained; The distance matrix is ​​clustered using the DBSCAN algorithm to obtain multiple data clusters with similar distribution characteristics; According to the number of server cluster nodes, the data under each cluster is divided into corresponding shares, and each share is allocated to the corresponding cluster node.

9. A dynamic load balancing system for large-scale real-time spatial computing, characterized in that: Includes the following modules: Preprocessing module: used to obtain the geographic space area to be calculated and perform rasterization preprocessing; Downsampling processing module: used to downsample the original data in the rasterized geographic space area based on the spatiotemporal step to obtain downsampled geographic space data; Prediction module: used to construct an LSTM+CNN prediction model, taking the geospatial data of multiple time periods as input, and predicting future grid data in the gridded geospatial area; Balanced processing module: used to utilize the computing power resources of a single server node to dynamically partition the predicted future grid data, so that the amount of data in each partition within a single node is balanced; and the data of each partition is processed using multi-threaded parallel computing; The equalization processing module is specifically used for: Partition the computing resources of a single server node; Acquire a dynamic space partitioning task, wherein the dynamic space partitioning task is to linearly distribute the predicted future grid data to each partition so as to achieve a balanced number of data points in each partition; Constructing a partition list, wherein the content of the partition list is repeated partition numbers, and the length of the partition list is the number n of the future grid data; Randomly select a point within each partition as the center point of the partition; Construct a cost matrix with dimension n*n; n is the number of the future grid data, the row label of the cost matrix is ​​the number of the future grid data, and the column label of the cost matrix is ​​the label of the partition list; the content of the cost matrix is ​​the straight-line distance from the future grid data to the center point; The Jonker-Volgenant algorithm is used to iteratively adjust until the sum of the distances from the data points to the corresponding partition centers in the cost matrix is ​​minimized, and the matching results between the data points and each partition are obtained; Update the center point of each partition, reconstruct the cost matrix, and update the matching results between the data points and each partition; When the offset distance before and after the update of the center points of all partitions is less than the preset value, the update is stopped to obtain the final matching results of the data points and each partition; The partition number with the largest data point partition ratio is used as the partition number of the grid where the data point is located, so as to achieve dynamic load balancing for each partition.

Images