Dynamic load balancing method and system for large-scale real-time space calculation

Abstract

The invention discloses a dynamic load balancing method and system for large-scale real-time space calculation, and the method comprises the following steps: obtaining a to-be-calculated geographic space region, and carrying out the rasterization preprocessing; performing down-sampling processing on the original data in the rasterized geographic space region based on the space-time step length to obtain down-sampled geographic space data; constructing an LSTM + CNN prediction model, taking the geographic space data of a plurality of time periods as input, and predicting future grid data in the rasterized geographic space region; performing dynamic space partitioning on the predicted future grid data to enable the data volume of each partition in a single node to be balanced; and processing the data of each partition by adopting a multi-thread parallel computing mode. The invention relates to the technical field of spatial position data calculation, downsampling is carried out based on a space-time step length, and load balancing of each process in a node is realized through dynamic space partitioning, so that maximum utilization of calculation resources is achieved.

Description

Technical Field

[0001] The present invention relates to the field of spatial position data calculation technology, and more specifically to a dynamic load balancing method and system for large-scale real-time spatial calculation. Background Art

[0002] At present, with the rapid development of Internet of Things technology and the widespread use of satellite positioning equipment, the generation and processing of large-scale real-time spatial location data has become an important part of modern information systems. These data are widely used in transportation, logistics and other fields, and are of great significance for improving urban management efficiency and optimizing resource allocation.

[0003] However, faced with massive and real-time updated spatial location data, traditional data processing methods face many challenges:

[0004] 1. Huge amount of data: Real-time spatial location data is usually generated at a high frequency and in huge amounts, which places extremely high demands on storage and processing capabilities.

[0005] 2. Complex spatiotemporal characteristics: The distribution of spatial location data is uneven and dynamically changing, and fixed spatial partitions will lead to unbalanced computing load problems.

[0006] 3. High real-time requirements: Many application scenarios require that data processing must be completed in a very short time (such as map matching) to ensure the timeliness and accuracy of decision-making.

[0007] To address the above issues, existing solutions often adopt static resource allocation strategies and fixed processing procedures, which cannot effectively cope with dynamic changes in data and uneven loads.

[0008] Therefore, proposing a dynamic load balancing method for large-scale real-time spatial computing is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0009] In view of this, the present invention provides a dynamic load balancing method and system for large-scale real-time spatial computing, which realizes the load balancing of each node in the computing cluster and the load balancing of each process in the node in the large-scale real-time spatial location data processing scenario, thereby maximizing the utilization of computing resources.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides a dynamic load balancing method for large-scale real-time spatial computing, comprising the following steps:

[0012] S1. Obtain the geographic space area to be calculated and perform rasterization preprocessing;

[0013] S2, downsampling the original data in the rasterized geographic spatial area based on the spatiotemporal step to obtain downsampled geographic spatial data;

[0014] S3, constructing an LSTM+CNN prediction model, taking the geospatial data of multiple time periods as input, and predicting future grid data within the gridded geospatial area;

[0015] S4. Use the computing power resources of a single server node to dynamically partition the predicted future grid data in space, so that the amount of data in each partition within a single node is balanced; use multi-threaded parallel computing to process the data in each partition.

[0016] Furthermore, it also includes:

[0017] S5. When server cluster node processing is adopted, the predicted future grid data is distributed to different cluster single nodes according to the distribution of the predicted future grid data and the number of server cluster nodes;

[0018] Utilize the corresponding cluster single-node computing resources to dynamically partition the allocated data and process the data in each partition.

[0019] Furthermore, in step S1, the rasterization preprocessing specifically includes:

[0020] Dividing the outer envelope rectangle of the geographic space area to be calculated into grids;

[0021] The preset vertex is taken as the origin, and grid incremental coding is performed along the horizontal coordinate and the vertical coordinate direction of the preset vertex to obtain the grid coordinates.

[0022] Further, step S2 specifically includes:

[0023] S21. In the time dimension, the time step formula of the original data in the rasterized geographic space area within the time range t is:

[0024] DT=ceil(t / ST)

[0025] In the spatial dimension, the spatial step formula for the original data in the rasterized geographic space area to change from the grid (x1, y1) to the grid (x2, y2) is:

[0026] DL=ceil((abs(x1-x2)+abs(y1-y2)) / SL)

[0027] Among them, ST represents a preset time step, SL represents a preset SL grids as a space step, ceil represents a rounding operation, and abs represents an absolute value operation;

[0028] S22, the spatiotemporal step length D=DT+DL of the movement of the original data in the gridded geographic space area is sampled every M preset spatiotemporal steps D to obtain downsampled geographic space data.

[0029] Further, in step S3, the geospatial data of multiple time periods are used as input to predict future grid data in the gridded geospatial area; including:

[0030] Obtain the geographic spatial data of V time periods at intervals of K time, and map them proportionally to a range of 0 to 255 to obtain V grayscale images;

[0031] The LSTM+CNN prediction model is used to predict the grayscale image of the grid at the next interval K time, and an inverse mapping process is performed to obtain the corresponding data as the future grid data.

[0032] Further, step S4 specifically includes:

[0033] S41, partitioning the computing resources of a single server node;

[0034] S42, obtaining a dynamic space partitioning task, wherein the dynamic space partitioning task is to linearly distribute the predicted future grid data to each partition to achieve a balanced number of data points in each partition;

[0035] S43, constructing a partition list, wherein the content of the partition list is repeated partition numbers, and the length of the partition list is the number n of the future grid data;

[0036] S44, randomly selecting a point within each partition range as the center point of the partition;

[0037] S45, constructing a cost matrix with a dimension of n*n; n is the number of the future grid data, the row label of the cost matrix is ​​the serial number of the future grid data, and the column label of the cost matrix is ​​the label of the partition list; the content of the cost matrix is ​​the straight-line distance from the future grid data to the center point;

[0038] S46, using the Jonker-Volgenant algorithm to iteratively adjust until the sum of the distances from the data points to the corresponding partition center points in the cost matrix is ​​minimized, and the matching results between the data points and each partition are obtained;

[0039] S47, updating the center point of each partition, reconstructing the cost matrix, and updating the matching results between the data point and each partition; until the offset distance before and after the update of the center point of all partitions is less than the preset value, then stop updating to obtain the final matching results between the data point and each partition;

[0040] S48. The partition number of the data point partition with the largest proportion is used as the partition number of the grid where the data point is located, so as to achieve dynamic load balancing of each partition.

[0041] Further, step S46 specifically includes:

[0042] Initializing the cost matrix, initializing the potential values ​​of the data points and partition center points;

[0043] Construct a matching array, the length of which is n, for storing the matching results of the data points and the partition center points;

[0044] Find augmenting paths, redistribute data points, and increase the number of matches;

[0045] Until an augmenting path is found, the matching state is reversed along the path to optimize the matching relationship;

[0046] Update the matching array according to the new matching relationship;

[0047] Repeat the process of finding augmenting paths and updating matching arrays until all data points are optimally allocated.

[0048] Output the matching results of data points and each partition center point, and calculate the total cost of all data points and partition center points.

[0049] Furthermore, the finding of the augmented path specifically includes:

[0050] Use breadth-first search to find augmenting paths starting from the unassigned center point; find better matching results by adjusting the relaxation value of the data points;

[0051] Adjust the potential values ​​of the partition center and data points to open more augmentation paths; based on the adjusted potential values, recalculate the relaxation value of each data point to reflect the latest potential value.

[0052] Further, in step S5, when the server cluster node is used for processing, the predicted future grid data is distributed to different cluster single nodes according to the distribution of the predicted future grid data and the number of server cluster nodes; specifically, the following steps are included:

[0053] When server cluster nodes are used for processing, the cosine distance is used to calculate the similarity and distance between vectors according to the distribution vectors of the predicted future grid data, and the distance matrix is ​​obtained;

[0054] The distance matrix is ​​clustered using the DBSCAN algorithm to obtain multiple data clusters with similar distribution characteristics;

[0055] According to the number of server cluster nodes, the data under each cluster is divided into corresponding shares, and each share is allocated to the corresponding cluster node.

[0056] In a second aspect, the present invention provides a dynamic load balancing system for large-scale real-time spatial computing, comprising the following modules:

[0057] Preprocessing module: used to obtain the geographic space area to be calculated and perform rasterization preprocessing;

[0058] Downsampling processing module: used to downsample the original data in the rasterized geographic space area based on the spatiotemporal step to obtain downsampled geographic space data;

[0059] Prediction module: used to construct an LSTM+CNN prediction model, taking the geospatial data of multiple time periods as input, and predicting future grid data in the gridded geospatial area;

[0060] Balanced processing module: It is used to utilize the computing power resources of a single server node to dynamically partition the predicted future grid data in space, so that the amount of data in each partition within a single node is balanced; and the data of each partition is processed using multi-threaded parallel computing.

[0061] The description of the second aspect of the present invention can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second aspect can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0062] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a dynamic load balancing method for large-scale real-time spatial computing, which has the following beneficial effects:

[0063] By downsampling real-time geospatial data based on the spatiotemporal step, the sampling frequency of data that is static for a long time or changes slowly for a long time is reduced, thereby reducing information redundancy, avoiding unnecessary computing overhead, and improving the efficiency of subsequent data processing.

[0064] By predicting the amount of raster data based on LSTM+CNN, the amount of data that will appear in each grid in the future is estimated to cope with the dynamic changes in the spatial distribution of data, and thus prepare for the subsequent spatial balanced partitioning.

[0065] Through node process load balancing based on dynamic spatial partitioning, the Jonker-Volgenant algorithm is used to solve the spatial partitioning problem to ensure that the amount of data falling in each partition is basically equal. At the same time, each partition is a closed area to ensure that the topological relationship of geographical elements is not destroyed.

[0066] The load balancing of different nodes in the cluster is achieved by clustering based on spatial distribution feature vectors. The similarity of spatial distribution features of different objects is measured by cosine similarity, and DBSCAN clustering is performed based on this similarity. Objects of each category are randomly and evenly distributed to each cluster node to ensure that the data distribution of each node is consistent with the predicted results. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0068] Figure 1 A schematic diagram of a dynamic load balancing method for large-scale real-time spatial computing provided by an embodiment of the present invention.

[0069] Figure 2 A schematic diagram of regional rasterization preprocessing provided by an embodiment of the present invention.

[0070] Figure 3 Schematic diagram of the input and output of the LSTM+CNN prediction model provided in an embodiment of the present invention.

[0071] Figure 4 A schematic diagram of the partition numbering results of the grid where the data points are located provided in an embodiment of the present invention.

[0072] Figure 5 A schematic diagram of spatial partitioning results for different time periods provided by an embodiment of the present invention.

[0073] Figure 6 A schematic diagram of the DBSCAN clustering results provided by an embodiment of the present invention.

[0074] Figure 7 A block diagram of a dynamic load balancing system for large-scale real-time spatial computing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] Example 1

[0077] The embodiment of the present invention discloses a dynamic load balancing method for large-scale real-time spatial computing. Figure 1 As shown, steps S1-S5 are included.

[0078] The following are the steps in detail:

[0079] Step S1, obtaining the geographic space area to be calculated and performing rasterization preprocessing;

[0080] Faced with large-scale and real-time updated spatial location data, in order to obtain the geographic spatial area to be calculated, convert the continuous and infinite longitude and latitude point data in the area into discrete and finite spatial data, and improve the spatial processing efficiency, it is necessary to rasterize the area and encode the raster.

[0081] Reference Figure 2 As shown, the specific rasterization encoding method is:

[0082] (1) Determine the geographic spatial area to be calculated, such as dividing it by administrative area, referring to Figure 2 The administrative area of ​​Guangzhou City.

[0083] (2) Take the lower left corner of the outer envelope rectangle of the region as the origin, generate a square grid of several meters with the origin as the center, and obtain the origin grid, which is encoded as (0,0).

[0084] (3) Other grid coding methods: the first digit increases from west to east, and the second digit increases from south to north.

[0085] Step S2, downsampling the original data in the rasterized geographic spatial area based on the spatiotemporal step length to obtain downsampled geographic spatial data;

[0086] Spatial data often contains a large amount of redundant data with less information; for example, the longitude and latitude of a vehicle that has been stationary or moving slowly for a long time will change little over time, or the time interval for uploading data in a short time is too short due to equipment failure. To address the above data redundancy issues, the original spatial location data is downsampled based on the spatiotemporal step size to reduce information redundancy and improve the efficiency of subsequent data processing.

[0087] The specific downsampling method is as follows:

[0088] (1) In the time dimension, ST seconds is used as a time step, that is, the time step of an object moving within a time range of t seconds is DT = ceil (t / ST), where ceil represents the rounding down operation, for example: ceil (4 / 5) = 0, ceil (6 / 5) = 1.

[0089] (2) In the spatial dimension, the spatial step length is SL grids. That is, the spatial step length of an object moving from grid (x1, y1) to grid (x2, y2) is DL = ceil((abs(x1-x2)+abs(y1-y2)) / SL), where ceil represents the floor operation and abs represents the absolute value operation; x and y represent the grid coordinate encoding of the original data before downsampling.

[0090] (3) Calculate the spatiotemporal step length D = DT + DL of the object's movement, and perform sampling every M spatiotemporal steps to obtain the downsampled geospatial data.

[0091] In this embodiment, ST=15 seconds, SL=3, and M=5 are set, and the downsampling process of geospatial time series data is shown in Table 1:

[0092] Table 1

[0093]

[0094] Step S3, constructing an LSTM+CNN prediction model to predict future grid data in the rasterized geographic spatial area based on the geographic spatial data of multiple time periods;

[0095] In order to estimate the amount of data that will appear in each grid in the future and prepare for the subsequent spatial balanced partitioning, this step predicts the amount of grid data.

[0096] Prediction Steps Reference Figure 3 As shown in the figure, the grid data distribution of the recent multiple time periods is processed into multiple grayscale images, and the LSTM+CNN prediction model is constructed to predict the future grid data volume and output the prediction results. The specific steps are as follows:

[0097] (1) Process the grid data distribution of the recent multiple time periods into multiple grayscale images

[0098] Taking K minutes as the statistical time interval, the amount of downsampled data in the grid from time tK to time t is recorded as N(t,i,j), i and j respectively represent the encoding values ​​of the grid where the downsampled data is located, and the data volume of all grids in the most recent V time periods is obtained, that is, N(t-(V-1)*K,i,j), N(t-(V-2)*K,i,j)..., N(t,i,j).

[0099] The original grid data volume is proportionally mapped to the range of 0~255, the maximum value of the data volume of all time periods and all grids is recorded as MAX_N, and the mapped value is recorded as M(t,i,j). The calculation formula is M(t,i,j)=ceil(N(t,i,j) / MAX_N*255), where ceil represents the rounding down operation.

[0100] (2) Constructing LSTM+CNN prediction model to predict future grid data volume

[0101] The deep learning models LSTM (Long Short-Term Memory) and CNN (Convolutional Neural Network) process time series data and spatial data respectively. LSTM+CNN is a hybrid architecture that combines these two models, aiming to use their respective strengths to solve more complex problems. CNN is used to extract the spatial features of the input data, and then these features are input into LSTM for time series modeling. This combination can capture both spatial and temporal information of the data.

[0102] The sample volume mapping value of each time period can be regarded as a grayscale image, and V grayscale images are obtained. The LSTM+CNN model is used to predict the amount of data M(t+K,i,j) after mapping of the grid in the next K minutes, and the inverse mapping process N(t+K,i,j)=M(t+K,i,j)*MAX_N / 255 is performed.

[0103] The total amount of predicted data from time tK to time t is recorded as N_PREDICT, where .

[0104] Step S4, using the computing power resources of a single server node to dynamically partition the predicted future grid data, so that the amount of data in each partition within a single node is balanced; and multi-threaded parallel computing is used to process the data of each partition.

[0105] In order to make full use of the computing power resources of a single node, a multi-process parallel computing method is adopted for real-time map matching processing. Each process is responsible for the calculation and processing of geographic spatial data in a partition. At the same time, in order to ensure that the computing load of each process is basically the same, it is necessary to perform reasonable spatial partitioning according to the future data distribution predicted in step S3 to achieve load balancing. The ideal spatial partitioning result is to make the amount of data falling in each partition equal in the future period.

[0106] The spatial partition problem is regarded as a linear assignment problem and the Jonker-Volgenant algorithm is used to solve the problem. The specific spatial partition method is as follows:

[0107] (1) If a single computing node has N_CORE CPU cores available for computing, the entire area needs to be divided into N_CORE partitions, with the partition numbers 1, 2, ..., N_CORE.

[0108] In this embodiment, a single computing node has three CPU cores available for computing, so the entire area needs to be divided into three partitions, which are numbered 1, 2, and 3 respectively.

[0109] (2) Consider each data point as a “worker” in the linear task assignment problem, and consider the division of the data point into a partition as the assignment of a “worker” to a “task” in the linear task assignment problem.

[0110] In this embodiment, there are 10 data points in the area, and the coordinates of longitude and latitude are shown in Table 2:

[0111] Table 2

[0112]

[0113] (3) Construct a "task" list. Construct a "task" list of length N_PREDICT. The content of the list is a continuously repeated partition number, that is, 1,2,..,,N_CORE,1,2,...,N_CORE,1,2,...,N_CORE,..., until the list length reaches N_PREDICT; the list content of this embodiment is 1,2,3,1,2,3,1,2,3,1.

[0114] Through the above construction, the number of data points allocated to each partition is basically equal, thereby achieving spatial balance.

[0115] (4) Randomly generate partition center points and randomly select a point within the area as the center point of each partition.

[0116] The partition center points selected in this embodiment are shown in Table 3 below:

[0117] Table 3

[0118]

[0119] (5) Construct a "cost matrix". Construct a "cost matrix" of N_PREDICT*N_PREDICT dimensions, that is, a 10*10 "cost matrix". The row label in the matrix represents the index number of the data point ("worker"), denoted as point_index, and the column label represents the index number of the spatial partition ("task"), denoted as partition_index. The value of the position (point_index, partition_index) in the matrix represents the straight-line distance from the data point point_index to the center point of the partition partition_index. The cost matrix is ​​shown in Table 4 below:

[0120] Table 4

[0121]

[0122] (6) The Jonker-Volgenant algorithm is used to solve the above linear task assignment problem, with the optimization goal of minimizing the total cost, that is, minimizing the sum of the distances from the partition center point to the corresponding partition data point, so that the data points of each partition can be clustered together. The solution obtains the "task assignment result", that is, each data point is divided into a specific spatial partition.

[0123] The Jonker-Volgenant algorithm is an efficient algorithm for solving the linear task assignment problem (also known as the bipartite graph minimum weight matching problem). In this embodiment, the linear task assignment problem is described as follows: given a bipartite graph; one side is a set of workers, which in this problem is a geospatial data point, and the other side is a set of tasks, which in this problem is a spatial partition or a spatial partition center point. There is a weight between each task and worker, i.e., cost or benefit. In this problem, the weight is the straight-line distance from the geospatial data point to the spatial partition center point. The goal is to find a matching solution between workers and tasks that minimizes the total weight; in this problem, the goal is to find a matching solution between geospatial data points and spatial partitions that minimizes the sum of the distances of all data points to the center point of the spatial partition to which they belong.

[0124] The core idea of ​​the Jonker-Volgenant algorithm is to gradually reduce the total cost by iteratively adjusting the matching until the optimal matching is found. The specific steps are as follows:

[0125] (6.1) Initialize the cost matrix: Given an n*n cost matrix C, where C[i][j] represents the cost of assigning task i to worker j.

[0126] (6.2) Initialize potential value: Initialize a potential value for each task and worker. In this embodiment, it is initialized to zero. The potential value of task i is recorded as u[i], and the potential value of worker j is recorded as v[j].

[0127] (6.3) Construct a matching array: Initialize an array match of length n to store the matching results. match[j] represents the task assigned to worker j. Initially, all elements are set to -1, indicating no match.

[0128] (6.4) Finding an augmenting path: An augmenting path is an alternating path where the edges on the path alternately belong to a match and do not belong to a match. The goal is to find an augmenting path from a task to a worker such that the number of matches can be increased by flipping the matching state on the path.

[0129] Ways to find augmenting paths include:

[0130] Breadth First Search (BFS): Use BFS to start from all unmatched tasks and find augmenting paths. During the search, maintain a queue and an array parent to record the path.

[0131] Relaxation operation: During BFS, for each task i and worker j, check whether a better match can be found by adjusting the potential value. Specifically, check the following condition: C[i][j] - u[i] - v[j] < slack[j], where slack[j] is the slack value of worker j, indicating the current minimum cost adjustment amount.

[0132] Adjust the potential values: If no augmenting paths are found, adjust the potential values ​​of tasks and workers to allow more augmenting paths to be found. Specifically, for each unmatched task i, reduce its potential value u[i], and for each unmatched worker j, increase its potential value v[j].

[0133] Update slack: Update the slack[j] of each worker j according to the adjusted potential value.

[0134] (6.5) Update the matching: After finding the augmenting path, flip the matching state on the path. Specifically, for each edge on the path, if it currently belongs to the matching, remove it from the matching; if it does not belong to the matching, add it to the matching.

[0135] (6.6) Update the match array: Update the match array to record the new matching relationship.

[0136] (6.7) Repeat the iteration: Repeat the process of finding augmenting paths, updating matching, and updating the matching array until all tasks are matched.

[0137] (6.8) Output: Finally, each element match[i][j] in the match array represents the task i to which worker j is assigned. Calculate the total cost, which is the sum of the costs of all matching tasks and workers:

[0138]

[0139] In this embodiment, after all tasks are matched, the following is obtained:

[0140] Partition 1: data points 2, 3, 8, 9; Partition 2: data segments 6, 7, 10; Partition 3: data points 1, 4, 5.

[0141] (7) Update the partition center point, that is, calculate the average longitude and latitude of all data points in each partition. The calculation results of the updated partition center point are shown in Table 5 below:

[0142] Table 5

[0143]

[0144] (8) Repeat step (5) and use the updated partition center points to reconstruct the “cost matrix” as shown in Table 6 below:

[0145] Table 6

[0146]

[0147] Repeat step (6) to obtain the “task allocation result”, that is, each data point is divided into a specific spatial partition. The task allocation result refers to the bold value in the cost matrix shown in Table 6. That is, partition 1: data points 2, 3, 8, 9; partition 2: data segments 4, 7, 10; partition 3: data points 1, 5, 6.

[0148] The iteration is stopped until the offset distance of the center points of all partitions before and after the update is less than a preset value, THRES_DIST=0.0001 in this embodiment.

[0149] (9) Partition the grid, refer to Figure 4 As shown in the figure, the data point position, the spatial partition center point position and the spatial grid are distinguished by different labels; different partitions have different colors, pink represents partition 1, orange represents partition 2, and gray represents partition 3; the partition number with the largest proportion of data point partitions in the grid is taken as the partition number to which the grid belongs. Figure 5 As shown, the spatial partitioning results for different time periods provided by this embodiment.

[0150] Step S5, when server cluster node processing is adopted, the predicted future grid data is distributed to different cluster single nodes according to the distribution of the predicted future grid data and the number of server cluster nodes; the corresponding cluster single node computing resources are used to dynamically partition the allocated data in space, and the data of each partition is processed.

[0151] In large-scale spatial location data computing scenarios, in addition to load balancing different processes within a single node through dynamic spatial partitioning, server clusters are often required to meet real-time processing requirements. This step takes vehicle analogy spatial location data as an example, and achieves balanced distribution of computing loads on different nodes in the cluster by reasonably allocating vehicles to different cluster nodes.

[0152] In the embodiment, the position point distribution of each vehicle on the previous day is calculated at 0:00 a.m. every day, that is, the number of position points falling in each grid after vehicle downsampling, which is recorded as N_CAR(plate,i,j), where plate represents the license plate number, i and j represent grid codes, and they are first arranged in ascending order by i and then in ascending order by j to obtain the vehicle position spatial distribution feature vector V_CAR(plate)=(N_CAR(plate,0,0), N_CAR(plate,0,1), N_CAR(plate,0,2), ..., N_CAR(plate,i,j))...), which is called the vehicle data distribution vector.

[0153] The cosine distance is used to calculate the similarity between vectors. For example, the data distribution vector similarity between plate1 and plate2 is COS_SIM(plate1, plate2) = cos(V_CAR(plate1), V_CAR(plate2)); the distance between vectors is DIST(plate1, plate2)=1-COS_SIM(plate1, plate2). The vector distances between all pairs of vehicles are calculated to obtain the distance matrix.

[0154] The distance matrix is ​​input into the DBSCAN algorithm for clustering to obtain several vehicle clusters with similar travel distribution characteristics. The vehicles in each cluster are then randomly divided into Q parts, and each part is assigned to the corresponding cluster node.

[0155] Reference Figure 6 As shown in the figure, the DBSCAN clustering results are displayed. The horizontal axis is longitude and the vertical axis is latitude. The spatial data distribution maps of three vehicles are selected from the two clusters obtained by clustering for display. It can be seen that the spatial data distribution characteristics of vehicles in the same cluster are highly similar.

[0156] This embodiment uses the DBSCAN algorithm to obtain F clusters, with cluster labels C(1), C(2)...C(F), and the number of vehicles under each cluster is V(1), V(2)...V(F). Next, the vehicles of each category are randomly divided into Q parts, and the category 1 vehicles falling in node 1 are R(1,1), with a total of V(1) / Q vehicles, and the category F vehicles falling in node Q are R(Q,F), with a total of V(K) / Q vehicles, and the rest of the nodes are similar.

[0157] All vehicles R(1,1), R(1,2), ..., R(1,F) that fall on node 1 are merged to form the vehicle set of node 1, and all vehicles R(Q,1), R(Q,2), ..., R(Q,F) that fall on node Q are merged to form the vehicle set of node Q, and so on for the remaining nodes.

[0158] The present invention first obtains the geographic space area to be calculated and performs grid processing; secondly, a real-time geographic space data downsampling method based on space-time step is adopted to filter redundant data in the grid, and then the LSTM+CNN model is used to predict the amount of grid data in the future time period. A single node adopts multiple processes to parallelly process data of different spatial partitions. Through a dynamic spatial partitioning method based on linear task allocation LAP, according to the predicted distribution of grid data, the Jonker-Volgenant algorithm is used to solve the spatial partitioning problem to ensure that the amount of data falling in each partition is basically equal. At the same time, each partition is a closed area to ensure that the topological relationship of geographic elements is not destroyed.

[0159] When server cluster nodes are used for processing, the cosine similarity is used to measure the similarity of the spatial distribution characteristics of different objects, and DBSCAN clustering is performed based on the similarity. Objects of each category are randomly and evenly distributed to each cluster node to ensure that the data distribution of each node is consistent with the predicted results. Ultimately, load balancing of multiple nodes in the cluster and load balancing of multiple processes within the node are achieved, making full use of computing resources.

[0160] Example 2

[0161] The embodiment of the present invention discloses a dynamic load balancing system for large-scale real-time spatial computing. Figure 7 As shown, it includes the following modules:

[0162] Preprocessing module: used to obtain the geographic space area to be calculated and perform rasterization preprocessing;

[0163] Downsampling processing module: used to downsample the original data in the rasterized geographic space area based on the spatiotemporal step to obtain downsampled geographic space data;

[0164] Prediction module: used to build an LSTM+CNN prediction model to predict future grid data in the gridded geographic spatial area based on the geographic spatial data of multiple time periods;

[0165] Balanced processing module: It is used to utilize the computing power resources of a single server node to dynamically partition the predicted future grid data in space, so that the amount of data in each partition is balanced; and the data in each partition is processed using multi-threaded parallel computing.

[0166] This embodiment first obtains the geographic space area to be calculated through the preprocessing module and performs rasterization preprocessing; then, the downsampling processing module downsamples the rasterized geographic space data based on the spatiotemporal step, and obtains the downsampled geographic space data after filtering the redundant data; then, the prediction module predicts the future grid data in the rasterized geographic space area to cope with the dynamic changes in the data spatial distribution, and then prepares for the subsequent spatial balanced partitioning. Finally, the balanced processing module performs dynamic spatial partitioning to ensure that the amount of data falling in each partition is basically equal, and each partition is a closed area to ensure that the topological relationship of the geographic elements is not destroyed.

[0167] The balancing processing module of this embodiment is also used to distribute the predicted future grid data to different cluster single nodes according to the distribution of the predicted future grid data and the number of server cluster nodes when server cluster nodes are used for processing; and to dynamically partition the allocated data in space by using the corresponding cluster single node computing resources, and to process the data in each partition.

[0168] This system solves the deficiencies in the prior art:

[0169] Insufficient 1: The original spatial position data is not filtered according to the information gain, resulting in the processing of a large amount of redundant information, unnecessary computing overhead, and reduced efficiency in real-time spatial position data processing.

[0170] Insufficient 2: The spatial partitioning process does not take into account that each partition needs to form a closed area in order to maintain the topological relationship of the geographical elements within the area. For example, real-time map matching calculations need to ensure that the road sections within each partition have complete connectivity. Non-closed spatial partitions will lead to erroneous processing results.

[0171] Insufficient three: The spatial division is performed only based on the spatial distribution of historical data, without considering that the distribution of data will change dynamically over time. The spatial distribution of future data will be different from the historical spatial distribution, resulting in the current division scheme being unable to achieve a good load balancing effect in the future.

[0172] Insufficient 4: In addition to considering the load balancing of different processes within a single node, the load balancing problem of different nodes is not considered, resulting in large differences in computing loads between different nodes, affecting real-time processing efficiency.

[0173] This system realizes the load balancing of each node in the computing cluster and the load balancing of each process in the node in the large-scale real-time spatial location data processing scenario, thereby maximizing the utilization of computing resources.

[0174] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0175] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dynamic load balancing method for large-scale real-time spatial computing, characterized in that: The following steps are involved: S1, obtain the geographic space area to be calculated and perform rasterization preprocessing; S2, downsampling the original data in the rasterized geographic spatial area based on the spatiotemporal step to obtain downsampled geographic spatial data; S3, constructing an LSTM+CNN prediction model, taking the geospatial data of multiple time periods as input, and predicting future grid data within the gridded geospatial area; S4, using the computing power resources of a single server node to dynamically partition the predicted future grid data, so that the amount of data in each partition within a single node is balanced; Multi-threaded parallel computing is used to process the data of each partition.

2. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: Also includes: S5. When server cluster node processing is adopted, the predicted future grid data is distributed to different cluster single nodes according to the distribution of the predicted future grid data and the number of server cluster nodes; Utilize the corresponding cluster single-node computing resources to dynamically partition the allocated data and process the data in each partition.

3. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: In step S1, the rasterization preprocessing specifically includes: Dividing the outer envelope rectangle of the geographic space area to be calculated into grids; The preset vertex is taken as the origin, and grid incremental coding is performed along the horizontal coordinate and the vertical coordinate direction of the preset vertex to obtain the grid coordinates.

4. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: Step S2 specifically includes: S21. In the time dimension, the time step formula of the original data in the rasterized geographic space area within the time range t is: DT=ceil(t / ST) In the spatial dimension, the spatial step formula for the original data in the rasterized geographic space area to change from the grid (x1, y1) to the grid (x2, y2) is: DL=ceil((abs(x1-x2)+abs(y1-y2)) / SL) Among them, ST represents a preset time step, SL represents a preset SL grids as a space step, ceil represents a rounding operation, and abs represents an absolute value operation; S22, the spatiotemporal step length D=DT+DL of the movement of the original data in the gridded geographic space area is sampled every M preset spatiotemporal steps D to obtain downsampled geographic space data.

5. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: In step S3, the geospatial data of multiple time periods are used as input to predict future grid data in the gridded geospatial area; including: Obtain the geographic spatial data of V time periods at intervals of K time, and map them proportionally to a range of 0 to 255 to obtain V grayscale images; The LSTM+CNN prediction model is used to predict the grayscale image of the grid at the next interval K time, and an inverse mapping process is performed to obtain the corresponding data as the future grid data.

6. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 1, characterized in that: Step S4 specifically includes: S41, partitioning the computing resources of a single server node; S42, obtaining a dynamic space partitioning task, wherein the dynamic space partitioning task is to linearly distribute the predicted future grid data to each partition to achieve a balanced number of data points in each partition; S43, constructing a partition list, wherein the content of the partition list is repeated partition numbers, and the length of the partition list is the number n of the future grid data; S44, randomly selecting a point within each partition range as the center point of the partition; S45, constructing a cost matrix with a dimension of n*n; n is the number of the future grid data, the row label of the cost matrix is ​​the serial number of the future grid data, and the column label of the cost matrix is ​​the label of the partition list; the content of the cost matrix is ​​the straight-line distance from the future grid data to the center point; S46, using the Jonker-Volgenant algorithm to iteratively adjust until the sum of the distances from the data points to the corresponding partition center points in the cost matrix is ​​minimized, and the matching results between the data points and each partition are obtained; S47, updating the center point of each partition, reconstructing the cost matrix, and updating the matching results between the data point and each partition; until the offset distance before and after the update of the center point of all partitions is less than the preset value, then stop updating to obtain the final matching results between the data point and each partition; S48. The partition number of the data point partition with the largest proportion is used as the partition number of the grid where the data point is located, so as to achieve dynamic load balancing of each partition.

7. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 6, characterized in that: Step S46 specifically includes: Initializing the cost matrix, initializing the potential values ​​of the data points and partition center points; Construct a matching array, the length of which is n, for storing the matching results of the data points and the partition center points; Find augmenting paths, redistribute data points, and increase the number of matches; Until an augmenting path is found, the matching state is reversed along the path to optimize the matching relationship; Update the matching array according to the new matching relationship; Repeat the process of finding augmenting paths and updating matching arrays until all data points are optimally allocated. Output the matching results of data points and each partition center point, and calculate the total cost of all data points and partition center points.

8. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 7, characterized in that: The finding of the augmenting path specifically includes: Use breadth-first search to find augmenting paths starting from the unassigned center point; find better matching results by adjusting the relaxation value of the data points; Adjust the potential values ​​of the partition center and data points to open more augmentation paths; based on the adjusted potential values, recalculate the relaxation value of each data point to reflect the latest potential value.

9. A dynamic load balancing method for large-scale real-time spatial computing as claimed in claim 2, characterized in that: In step S5, when the server cluster node processing is adopted, the predicted future grid data is distributed to different cluster single nodes according to the distribution of the predicted future grid data and the number of server cluster nodes; specifically, it includes: When server cluster nodes are used for processing, the cosine distance is used to calculate the similarity and distance between vectors according to the distribution vectors of the predicted future grid data, and the distance matrix is ​​obtained; The distance matrix is ​​clustered using the DBSCAN algorithm to obtain multiple data clusters with similar distribution characteristics; According to the number of server cluster nodes, the data under each cluster is divided into corresponding shares, and each share is allocated to the corresponding cluster node.

10. A dynamic load balancing system for large-scale real-time spatial computing, characterized in that: Includes the following modules: Preprocessing module: used to obtain the geographic space area to be calculated and perform rasterization preprocessing; Downsampling processing module: used to downsample the original data in the rasterized geographic space area based on the spatiotemporal step to obtain downsampled geographic space data; Prediction module: used to construct an LSTM+CNN prediction model, taking the geospatial data of multiple time periods as input, and predicting future grid data in the gridded geospatial area; Balanced processing module: used to utilize the computing power resources of a single server node to dynamically partition the predicted future grid data, so that the amount of data in each partition within a single node is balanced; Multi-threaded parallel computing is used to process the data of each partition.

Images