A streaming computing job scheduling method based on time and space awareness

Abstract

The present invention relates to a streaming computing job scheduling method based on time-space perception, which belongs to the field of task scheduling of big data processing technology. First, two-stage clustering is performed on the moving objects at the current moment to achieve data uniformity while maintaining locality; secondly, on the basis of the two-stage clustering at the current moment, dynamic incremental clustering is performed on the moving objects at the latest moment to reduce the switching cost; then, data offset, data locality, switching cost and comprehensive indicators are used to evaluate the clustering quality of the dynamic incremental clustering algorithm; finally, each cluster after dynamic incremental clustering is mapped to different partitions of the streaming computing engine, and then each partition is scheduled to different computing tasks to perform various real-time calculations. The present invention solves the problem of real-time scheduling of tasks in large-scale traffic scenarios, and significantly saves computing resources by maintaining locality, reducing data offset, and reducing switching costs.

Description

Technical Field

[0001] The present invention relates to a streaming computing job scheduling method based on time-space perception, and belongs to the field of big data processing technology task scheduling. Background Art

[0002] In public transportation scenarios, many different real-time computing tasks (such as GPS trajectory map matching, route deviation detection, mobile object clustering, etc.) are attached to tens of thousands of mobile objects (such as buses, taxis or online ride-hailing cars). These tasks are the basis of a series of operations, public services, and emergency management. Before completing these computing tasks, the background must first perform task scheduling. Specifically, the GPS data of tens of thousands of vehicles is divided into multiple groups, and each group is scheduled to a different computing node. In this process, task scheduling is a key link. Inappropriate task scheduling methods will lead to a sharp decline in performance, and even fail to meet service quality requirements; especially in large-scale data scenarios, it will lead to a serious waste of a large amount of computing resources.

[0003] Some current classic scheduling algorithms, such as hash algorithms and grid partitioning, either fail to maintain data locality or produce serious data drift. Although some new partitioning methods can also alleviate the data drift problem while maintaining data locality, they still have the following three shortcomings:

[0004] (1) Unable to cope with the dynamic changes of moving objects. Some studies can propose a better partitioning method for the spatial distribution of moving objects in a specific time and space, which can basically achieve data uniformity while maintaining locality. However, as time goes by, the spatial position of a single vehicle will change, and the overall spatial distribution of moving objects will also change. If the task scheduling is still carried out according to the previous partitioning method, the performance of the system will drop sharply. Therefore, the existing methods cannot effectively cope with the changes in the temporal and spatial distribution of moving objects;

[0005] (2) The switching scheduling cost is too high. For mobile objects with spatiotemporal properties that are changing dynamically at all times, when the scheduling scheme is changed, the mobile objects need to be rescheduled according to the new scheme, and rescheduling will incur switching costs. If each object in the mobile object set needs to be switched, the scheduling period will have a serious negative impact on the performance of the system. For this scenario, there is an urgent need for a scheduling method that can perform incremental clustering and reduce switching costs.

[0006] (3) Lack of comprehensive quantitative indicators. Most scheduling methods partially solve data locality and data drift, and few research methods focus on switching costs. In real scheduling scenarios, the three factors have an important impact on the performance of the system. Existing methods have not established a unified comprehensive indicator to evaluate the pros and cons of scheduling methods.

[0007] The clustering method proposed in the patent "CN117828388A, an improved heuristic K-means clustering method for vehicle network nodes" makes full use of the dynamic changes in the spatiotemporal attributes of moving objects, improves the K-means clustering algorithm, and increases the clustering speed. However, it ignores the two key indicators of data locality and data offset, and the resulting cluster set has poor clustering quality.

[0008] Compared with the DBSCAN clustering algorithm, the clustering method proposed in the patent "CN117591904A, a truck clustering method based on density clustering" accelerates the search for points in the same cluster and enhances data locality by converting the truck GPS into OD points and calculating the similarity analysis of GeoHash values ​​and word vectors. However, it ignores the two factors of data offset and long code execution time.

[0009] Compared with the above patents, the present invention proposes a scheduling method based on (mobile object) spatiotemporal perception. Aiming at the characteristics that the spatial distribution of mobile objects changes dynamically over time, the scheduling method includes three core steps: clustering, mapping, and scheduling. First, the mobile objects at the current moment are clustered in two stages to maintain data locality while achieving data uniformity; secondly, based on the two-stage clustering at the current moment, the mobile objects at the latest moment are incrementally dynamically clustered to reduce the switching cost. Subsequently, three metrics (data offset, data locality, switching cost) and a comprehensive indicator are designed to evaluate the clustering quality of the dynamic clustering algorithm; finally, each cluster after dynamic clustering is mapped to different partitions of the streaming computing engine, and each partition is scheduled to different computing tasks to perform various real-time calculations. The present invention solves the problem of real-time scheduling of tasks in large-scale traffic scenarios, and significantly saves computing resources by maintaining data locality, reducing data offset, and reducing switching costs.

[0010] Compared with the above patents, the streaming computing job scheduling method based on time and space perception proposed in this invention mainly makes the following two contributions:

[0011] (1) While maintaining locality and uniformity, the switching cost of the scheduling algorithm is reduced. The two-stage clustering proposed in the present invention can obtain the cluster set with the best comprehensive index at a fixed time. When the spatiotemporal attributes of the mobile object change dynamically and the scheduling needs to be switched, the dynamic incremental clustering proposed in the present invention is optimized based on the cluster set obtained by the two-stage clustering at the previous time, minimizing the number of mobile objects that need to be switched. While maintaining data locality and data uniformity, the switching cost of the scheduling algorithm is significantly reduced.

[0012] (2) A systematic quantitative index is proposed to evaluate the pros and cons of the scheduling method. The comprehensive index proposed in this invention can comprehensively evaluate the clustering quality from three aspects: data locality, data offset, and switching cost, and verifies the effectiveness of the scheduling scheme proposed in this invention in real scheduling scenarios. Summary of the invention

[0013] The technical problem to be solved by the present invention is to provide a streaming computing job scheduling method based on time and space perception, which aims to solve the problems of high operating costs and low resource utilization in real-time scheduling of tasks in large-scale traffic scenarios.

[0014] The present invention analyzes the existing scheduling scheme and its shortcomings. Specifically, there are three shortcomings:

[0015] (1) Unable to cope with the dynamic changes of moving objects. Some studies can propose a better partitioning method for the spatial distribution of moving objects in a specific time and space, maintaining data locality while basically achieving data uniformity. However, as time goes by, the spatial position of a single vehicle will change, and the overall spatial distribution of moving objects will also change. If the task scheduling is still carried out according to the previous partitioning method, the performance of the system will drop sharply. Therefore, the existing methods cannot effectively cope with the changes in the temporal and spatial distribution of moving objects;

[0016] (2) The switching scheduling cost is too high. For mobile objects with spatiotemporal properties that are dynamically changing at all times, when the scheduling scheme is changed, the mobile objects need to be rescheduled according to the new scheme, which incurs switching costs. If each object in the mobile object set needs to be switched, the system performance will be seriously negatively affected during the scheduling period. For this scenario, there is an urgent need for a scheduling method that can perform incremental clustering and reduce switching costs.

[0017] (3) Lack of comprehensive quantitative indicators. Most scheduling methods partially solve data locality and data drift, but rarely focus on the study of switching costs. In real scheduling scenarios, the three factors have an important impact on the performance of the system. Existing methods have not established a unified comprehensive indicator to measure scheduling methods.

[0018] The technical solution of the present invention is: Based on the above analysis, a streaming computing job scheduling method based on time and space perception is proposed. In order to meet the requirements of streaming computing task scheduling for high resource utilization, fast switching scheduling, and low switching cost, the method proposed by the present invention is divided into three core steps when scheduling large-scale mobile objects: clustering, mapping, and scheduling. In the first step of clustering, the present invention considers the spatiotemporal attributes of the mobile object at the current moment to design a two-stage clustering, while maintaining data locality, reducing the data offset between clusters, and obtaining the optimal scheduling at the current moment; at the same time, considering the characteristics of the dynamic change of the spatiotemporal attributes of the mobile object, when switching scheduling, dynamic incremental clustering is proposed, based on the two-stage clustering results of the previous moment combined with the data of the current moment, only some mobile objects are re-clustered, while maintaining data locality and data uniformity, the switching cost is significantly reduced, and the problem of excessively high switching costs of existing solutions is avoided.

[0019] The specific method is as follows:

[0020] A streaming computing job scheduling method based on time and space perception includes the following steps:

[0021] Step 1: Get the spatial distribution of the mobile object set on the road network at the current moment;

[0022] Step 2: For a set of moving objects with a given spatial distribution, the latitude interval is divided into multiple sub-intervals of different latitude lengths according to the data density, ensuring that the number of moving objects in different sub-intervals is consistent. Each sub-interval is regarded as a large cluster after clustering in the first stage.

[0023] Step 3: In the large cluster obtained after the first stage of clustering, the longitude interval is divided into multiple longitude sub-intervals according to the data density to ensure that the number of moving objects in different sub-intervals is consistent. Each sub-interval is regarded as a small cluster after the second stage of clustering.

[0024] Step 4: Get the spatial distribution of the mobile object set on the road network at the latest moment;

[0025] Step 5: Set a set of parameter combinations, each of which includes three parameters: time interval, displacement threshold, and radius threshold;

[0026] Step 6: Based on the small clusters obtained after the second stage of clustering at the current moment, the current parameter combination is used in combination with the spatial distribution of the moving objects at the latest moment to perform dynamic incremental clustering, and the displacement of each moving object between two different moments is calculated. If the displacement exceeds the displacement threshold, the moving object is re-clustered; if the displacement does not exceed the displacement threshold, the moving object retains the cluster mark at the current moment as the cluster mark at the latest moment, and the small cluster set at the latest moment is obtained;

[0027] Step 7: Generate a cluster center for each small cluster at the latest moment. Add the longitude of each moving object in the same cluster and divide it by the number of moving objects in the cluster. Add the latitude and divide it by the number of moving objects in the cluster to obtain the longitude and latitude of the cluster center.

[0028] Step 8: For the moving objects that need to be re-clustered, a circular search area is generated with the moving object as the center and the radius threshold as the radius. The cluster center that appears in the search area is used as the candidate cluster for the moving object in the re-clustering;

[0029] Step 9: Among the candidate clusters, select the cluster with the least number of moving objects as the new cluster after re-clustering the moving objects;

[0030] Step 10: Iterate Step 8 and Step 9 until all moving objects that need to be re-clustered are processed and dynamic incremental clustering is completed;

[0031] Step 11: Use three metrics, data offset, data locality, and switching cost, to evaluate the clustering quality after dynamic incremental clustering using the current parameter combination to obtain a comprehensive indicator;

[0032] Step 12: Iterate Step 7 to Step 11 until all parameter combinations are traversed to obtain all comprehensive indicators under different parameter combinations;

[0033] Step 13: Select the parameter combination with the smallest comprehensive index as the optimal parameter combination of the dynamic incremental clustering algorithm, and use the optimal parameter combination each time the scheduling is switched;

[0034] Step 14: The time interval in the optimal parameter combination is used as the time interval for switching scheduling. Each time the scheduling is switched, each cluster after dynamic incremental clustering is mapped to a different partition of the streaming computing engine, and then each partition is scheduled to a different computing task to perform various real-time calculations.

[0035] Specifically, the clustering construction process of the first stage in Step 2 is as follows:

[0036] For a set of moving objects with a given spatiotemporal distribution, the latitude interval is divided into multiple sub-intervals of different latitude lengths according to the data density to ensure that the number of moving objects in different latitude sub-intervals is consistent. Each latitude sub-interval is regarded as a large cluster after the first stage of clustering.

[0037] The detailed process is as follows:

[0038] Step 2.1: Data preprocessing before the first clustering and definition and calculation of related variables, as described in formula (1):

[0039] (1)

[0040] In formula (1), It represents a list of the original data set sorted from small to large according to longitude. represents the number of large clusters, 1 represents the ideal amount of data for a large cluster in the first clustering;

[0041] Step 2.2: Get the data of each large cluster, as described in formula (2):

[0042] (2)

[0043] In formula (2), Indicates the number of the large cluster, Indicates A large cluster, express exist The corresponding subset in ;

[0044] Step 2.3: Add the corresponding cluster markers ;

[0045] Step 2.4: Iterate Step 2.2 and Step 2.3 until all moving objects complete the first stage of clustering.

[0046] Specifically, the second stage clustering construction process in Step 3 is as follows:

[0047] In each large cluster generated in Step 2, the longitude interval is divided into multiple longitude sub-intervals according to the data density to ensure that the number of moving objects in different longitude intervals is consistent. Each longitude interval is regarded as a small cluster after the second stage of clustering.

[0048] The detailed process is as follows:

[0049] Step 3.1: Data preprocessing before the second clustering and definition and calculation of related variables, as described in formula (3):

[0050] (3)

[0051] In formula (3), represents the number of small clusters in each large cluster, express The amount of data, ideal Indicates the ideal amount of data in each small cluster;

[0052] Step 3.2: Order Indicates Small clusters, looking for The cluster centers are sorted from large to small according to latitude. The data in the sorting , select the point with the largest latitude as the cluster center ;

[0053] Step 3.3: Search The same cluster points of are as described by formula (4) and formula (5):

[0054] (4)

[0055] In formula (4), the distance() function input and , output the perpendicular Euclidean distance between the two, Represents a distance list;

[0056] (5)

[0057] In formula (5), express The same cluster points of

[0058] Step 3.4: Order Represents the data list storing the jth small cluster. and join in , and add cluster markers;

[0059] Step 3.5: Command Indicates the data list storing all small clusters. join in ;

[0060] Step3.6: Iterate Step3.2 to Step3.5 until all data are divided into clusters.

[0061] Specifically, the comprehensive indicators in Step 11 are:

[0062] The comprehensive index is the weighted average of three indicators: data offset, data locality, and switching cost;

[0063] Data offset refers to the imbalance of data volume between different clusters after clustering of moving objects. The calculation method of data offset is described in formula (6):

[0064] (6)

[0065] In formula (6), Indicates data offset, Indicates the cluster number, represents the number of clusters, Indicates The number of mobile objects in a cluster, represents the ideal number of moving objects in each cluster after clustering, Represents the total number of moving objects. The smaller the data offset, the more balanced the data volume between different clusters. According to the scheduling scheme, these clusters are mapped to partitions, and the partitions are scheduled to tasks. After the calculation is performed, the time delay between tasks is lower;

[0066] Data locality is to judge the degree of distinct boundaries between different clusters from the perspective of spatial distribution. The calculation method of data locality is described by formulas (7) to (9):

[0067] (7)

[0068] (8)

[0069] (9)

[0070] In formulas (7) to (9), Indicates moving objects, express The set of circumscribed polygons of other clusters outside the cluster. Function Input and Output corresponding data locality , represents the total number of moving objects, It represents the average data locality of all moving objects. The larger the locality, the clearer the boundaries between different clusters.

[0071] The switching cost is the proportion of moving objects that are re-clustered when switching scheduling. The switching cost is calculated as described in formulas (10) and (11):

[0072] (10)

[0073] (11)

[0074] In formulas (10) and (11), express The switching cost, , Respectively At the upper and lower moments, represents the displacement threshold, Represents the switching cost of all moving objects. The smaller the switching cost, the lower the proportion of moving objects to be re-clustered, and the more computing resources are saved;

[0075] The calculation method of the comprehensive index corresponding to clustering quality is described in formulas (12) and (13):

[0076] (12)

[0077] (13)

[0078] In formulas (12) and (13), Represents a comprehensive indicator, , Respectively represent data offset , Data Locality , switching costs The smaller the comprehensive index is, the higher the data locality, the lower the data offset and the lower the switching cost.

[0079] Specifically, the optimal parameter combination of Step 13 is:

[0080] The three parameters are time interval , displacement threshold , Radius Threshold ,in Control the time interval of switching scheduling, Control switching costs, Controlling data locality and data offset, the process of finding the optimal scheduling solution can be regarded as the process of finding the optimal parameter combination. The way to find it is to use different parameter combinations to perform dynamic incremental aggregation on historical data, calculate the comprehensive index, and select the parameter combination with the optimal comprehensive index as the parameter used in each subsequent scheduling switch. The above process is shown in formula (14):

[0081] (14)

[0082] In formula (14), Indicates the time interval, represents the displacement threshold, represents the radius threshold, Represents a combination of parameters, Represents a collection of parameter combinations, Represents the optimal parameter combination. The parameters used in subsequent processes are derived from .

[0083] Specifically, the Step 14 is as follows:

[0084] Step:14.1: Obtain a set of heterogeneous clustering results through spatiotemporal-aware dynamic clustering. The clustering result set is ;

[0085] Step:14.2: Map the clustering result set to different partitions of the streaming computing engine. The partition result set is ;

[0086] Step:14.3: Start a task on each partition and start performing various calculations for moving objects;

[0087] Specifically, the dynamic incremental clustering construction process is as follows:

[0088] First, based on the two-stage clustering at the previous moment and combined with the new data at the current moment, search for moving objects that need to be re-clustered. Specifically, calculate the displacement of each moving object at the previous and next moments, and take the displacement greater than The moving objects are added to the list of classes that need to be re-clustered , the cluster labels of the remaining moving objects remain unchanged and are added to the list .

[0089] Dynamic incremental clustering is not for all moving objects, but only re-clustering a small part of them, which greatly reduces the switching cost. And because dynamic incremental clustering is based on the two-stage clustering at the previous moment, it greatly retains data locality without increasing data offset.

[0090] Specifically, the construction process requires the following three functions:

[0091] (1) : A function for finding cluster centers. Input a cluster list, calculate the mean longitude and latitude of all moving objects in the current cluster, create a cluster center, and output the cluster center.

[0092] (2) : A function that calculates the distance between two points. Input two points with longitude and latitude, and output the distance between the two points;

[0093] (3) search_center(): A function that finds a suitable new cluster for moving objects that need to be re-clustered.

[0094] Input a moving object that needs to be re-clustered, the radius threshold, the cluster center list, and the cluster data volume list. With the current moving object as the center and the radius threshold as the radius, a circular search area is formed. The cluster center list is queried to obtain the positions of all cluster centers. The cluster centers that appear in the search area are considered as candidate clusters to ensure that data locality is not reduced. The existing data volume of each candidate cluster in the cluster data volume list is queried, and the one with the smallest data volume is selected as the new cluster of the current moving object to avoid increasing data offset. Finally, the cluster center of the new cluster is output.

[0095] Based on the above three functions, the specific process of dynamic incremental clustering is as follows:

[0096] First, generate the cluster center and call the function for Each cluster in generates a cluster center ;

[0097] Secondly, traverse the change ,make Indicates the moving objects that currently need to be re-clustered;

[0098] Then, search for candidate clusters and call the function search_center() to search the current Alternative clusters of ;

[0099] Next, determine the new cluster and select the candidate cluster with the least moving objects as New clusters of

[0100] Finally, complete Re-cluster all moving objects in the cluster.

[0101] The beneficial effects of the present invention are:

[0102] (1) Reducing switching costs while coping with dynamic changes of mobile objects. The two-stage clustering proposed in the present invention can obtain the cluster set with the best comprehensive index at a fixed time. When the spatiotemporal attributes of mobile objects change dynamically and switching scheduling is required, the dynamic incremental clustering proposed in the present invention combines the cluster set obtained by the two-stage clustering at the previous time with the data at the current time, while maintaining data locality and data offset. The switching cost is reduced to the minimum.

[0103] (2) Comprehensive quantitative indicators are proposed. The comprehensive indicators proposed in this invention can comprehensively evaluate the clustering quality from three aspects: data locality, data offset, and switching cost, and verify the effectiveness of the scheduling scheme proposed in this invention in real scheduling scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 It is the scheduling flow chart of the present invention;

[0105] Figure 2 It is a schematic diagram of equal grid partitioning;

[0106] Figure 3 It is a schematic diagram of density clustering;

[0107] Figure 4 Schematic diagram of two-stage clustering;

[0108] Figure 5 Schematic diagram of dynamic incremental clustering. DETAILED DESCRIPTION

[0109] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods based on the description.

[0110] Example 1: Figure 1 As shown, a streaming computing job scheduling method based on time and space perception includes the following steps:

[0111] Step 1: Obtain the spatial distribution of 3,000 buses on the road network in Shenzhen at 9:00 a.m. on a certain day, and filter out abnormal longitude and latitude data;

[0112] Step 2: Clustering in the first stage. Set the number of large clusters after the first stage of clustering to 4. Then, for a given set of buses distributed in space, divide the latitude interval into 4 sub-intervals with different latitude lengths according to the data density, and ensure that the number of moving objects in different latitude sub-intervals is consistent, that is, approximately equal to 3000÷4=750 buses. Each latitude sub-interval is regarded as a large cluster after the first stage of clustering.

[0113] Step 3: The second stage of clustering. Each large cluster is divided into 8 small clusters. Specifically, the longitude interval is divided into 8 longitude sub-intervals according to the data density, ensuring that the number of buses in different longitude intervals is consistent, that is, they are approximately equal to buses, each longitude interval is regarded as a small cluster after the second stage of clustering, and there are 4*7=28 such small clusters in total;

[0114] Step 4: Get the spatial distribution of the bus collection on the road network at 9:05 am on the same day;

[0115] Step 5: Set a set of parameter combinations, and traverse each parameter combination for dynamic incremental clustering. Each parameter combination includes three parameters, namely time interval, displacement threshold, and radius threshold. In the set of parameter combinations, the range of the time interval is [5,30], and the traversal step is 5; the range of the displacement threshold is [500,4000], and the traversal step is 100; the range of the radius threshold is [500,3000], and the traversal step is 100, so the time interval has (30-5) / 5+1=6 values, the displacement threshold has (4000-500) / 100+1=36 values, and the radius threshold has (3000-500) / 100+1=26 values, so there are 6*36*26=5616 parameter combinations in total;

[0116] Step 6: Based on the small cluster set obtained by two-stage clustering of the 9:00 bus data, dynamic incremental clustering is performed using the current parameter combination and the spatial distribution of the 9:05 bus. Calculate the displacement of each bus in the past 5 minutes. If the displacement exceeds the displacement threshold, the bus is re-clustered, that is, it is divided into other clusters in an existing small cluster according to Step 8 and Step 9; if the displacement does not exceed the displacement threshold, the bus retains the cluster mark of 9:00 as the cluster mark of 9:05. Take two of the buses as an example: if the displacement threshold is 4000 meters, the cluster marks of the two buses 'BS08529D' and 'BS00012D' in the two-stage clustering at 9:00 are '1' and '16' respectively (representing the 1st small cluster and the 16th small cluster respectively), and the displacements from 9:00 to 9:05 are 4561 meters and 1356 meters respectively. The displacement of 'BS08529D' is greater than the displacement threshold and the cluster needs to be changed, that is, the cluster mark '1' of 9:00 is erased; the displacement of 'BS00012D' is less than the displacement threshold and the cluster mark remains unchanged, that is, whether it is 9:00 or 9:05, the cluster mark of this bus is '16';

[0117] Step 7: Generate cluster centers for the 28 small clusters at 9:05. Add the longitude and latitude of each bus in the same cluster and divide them by the number of existing buses in the small cluster to obtain the longitude and latitude of the cluster center.

[0118] Step 8: For buses that need to be re-clustered, such as bus 'BS08529D', a circular search area is constructed with the bus as the center and the current radius threshold of 3000 meters as the radius. It is found that 4 cluster centers appear in the area, and the clusters they belong to are clusters '24', '6', '13', and '2' respectively. These 4 clusters are candidate clusters for re-clustering bus 'BS08529D'.

[0119] Step 9: The number of buses in clusters '24', '6', '13', and '2' at 9:05 are 87, 25, 93, and 66 respectively. Cluster '6' has the least number of buses and is used as the new cluster after re-clustering bus 'BS08529D'.

[0120] Step 10: Iterate Step 8 and Step 9 until all buses that need to be re-clustered are processed;

[0121] Step 11: Use the three metrics of data offset, data locality and switching cost to evaluate the clustering quality after dynamic incremental clustering using the current parameter combination, and obtain a comprehensive index, which is the weighted sum of the three metrics, with weights of 0.3, 0.3 and 0.4 respectively;

[0122] Step 12: Iterate Step 7 to Step 11 until all parameter combinations are traversed to obtain all comprehensive indicators under different parameter combinations;

[0123] Step 13: After Step 12, it is found that the comprehensive index is the smallest when the parameter combination (5, 4000, 3000) is used for dynamic incremental clustering, so (5, 4000, 3000) is the optimal parameter combination. The optimal parameter combination (5, 4000, 3000) is selected each time the scheduling is switched;

[0124] Step 14: The time interval '5' in the optimal parameter combination (5, 4000, 3000) is used as the time interval for switching scheduling. Each time the scheduling is switched, the 28 clusters after dynamic incremental clustering are mapped to the 28 partitions of the streaming computing engine, and then the 28 partitions are scheduled to 28 computing tasks to perform various real-time calculations, such as map matching and trajectory anomaly detection.

[0125] Figure 1 The processing flow at the top corresponds to Step 1 to Step 13 of the above steps, where the two-stage clustering is only performed once during the first scheduling, which provides a basis for dynamic incremental clustering. Subsequent switching scheduling uses dynamic incremental clustering. Secondly, the exploration of the optimal parameter combination is only performed once. Specifically, historical data is used to repeatedly perform dynamic clustering on the basis of different parameter combinations, and then a comprehensive indicator of the feedback clustering quality is obtained. The parameter combination with the smallest comprehensive indicator is selected as the optimal parameter combination, and the obtained optimal parameter combination can be continuously used for switching scheduling. Figure 1 The processing flow at the bottom corresponds to Step 14 of the above steps, completing the mapping of clusters to partitions and the scheduling of partitions to tasks.

[0126] Figure 2 The schematic diagram of equal grid partitioning is shown in Figure 1. The scheduling switching includes three core steps: clustering, mapping, and scheduling. In the first step of clustering, the equal grid partitioning algorithm is used to divide the 3,000 buses at 9:00 and 9:05 into 28 equal grids, and then the 28 grids are mapped to 28 partitions, and the 28 partitions are scheduled to 28 tasks. Finally, three metrics (data locality, data offset, switching cost) and a comprehensive indicator (the weighted sum of the above three indicators) are used to quantify the division results. The experimental results are shown in Table 1. It can be observed that after using equal grid partitioning to cluster large-scale buses, although the data locality of different grids is perfect, there is a serious data offset between grids. After mapping and scheduling, this data offset will be manifested as a huge time delay between tasks; in addition, the experimental results in Table 1 show that the switching cost of equal grid partitioning as a clustering algorithm for switching scheduling is 100%, which seriously wastes computing resources.

[0127] Figure 3 Figure 1 is a schematic diagram of density clustering. Density clustering is selected as the clustering algorithm for switching scheduling. The 3,000 buses at 9:00 and 9:05 are divided into 28 clusters respectively, and the 28 clusters are mapped to 28 partitions. The 28 partitions are scheduled to 28 tasks. Finally, three metrics (data locality, data offset, and switching cost) and a comprehensive indicator (the weighted sum of the above three indicators) are used to quantify the division results. The experimental results are shown in Table 1. It can be observed that after using density clustering to divide large-scale buses, although there is no data offset between different clusters, the locality is insufficient, and the switching cost is 100%, which seriously wastes computing resources.

[0128] Figure 4 The schematic diagram of two-stage clustering is shown in Figure 1. Two-stage clustering is selected as the clustering algorithm for switching scheduling. The 3,000 buses at 9:00 and 9:05 are divided into 28 clusters respectively, and the 28 clusters are mapped to 28 partitions. The 28 partitions are scheduled to 28 tasks. Finally, three metrics and one comprehensive indicator are used to quantify the division results. The experimental results are shown in Table 1. It can be observed that after using two-stage clustering to divide large-scale buses, there is no data offset between different clusters and the data locality is perfect. However, it cannot cope with the continuous spatial changes of moving objects, which is reflected in the switching cost of 100%. The huge switching cost will bring serious waste of resources.

[0129] Figure 5 The following is a schematic diagram of dynamic incremental clustering. Two-stage clustering has divided the 3,000 buses at 9:00 into 28 clusters. However, if two-stage clustering is still used as the clustering algorithm for switching scheduling, the switching cost is too high. In order to reduce the switching cost, the 3,000 buses in these 28 clusters are incrementally clustered based on the 28 clusters obtained by two-stage clustering at 9:00, combined with the optimal parameter combination. The experimental results are shown in Table 1. The use of dynamic incremental clustering can maintain locality and data balance between clusters while only re-clustering 35 buses, significantly reducing the switching cost.

[0130] Table 1 Comparison of indicators of different clustering algorithms

[0131]

[0132] In summary, compared with the existing scheduling schemes, the scheduling scheme of the present invention can schedule large-scale moving objects with spatiotemporal attributes and dynamic changes in real time by designing two-stage clustering and dynamic clustering. At the same time, a comprehensive evaluation index is designed to prove that the scheduling scheme can significantly reduce the switching cost, improve resource utilization, and reduce the time delay between tasks while maintaining data locality and reducing data offset.

[0133] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A streaming computing job scheduling method based on time and space perception, characterized in that: Step 1: Get the spatial distribution of the mobile object set on the road network at the current moment; Step 2: For a set of moving objects with a given spatial distribution, the latitude interval is divided into multiple sub-intervals of different latitude lengths according to the data density, ensuring that the number of moving objects in different sub-intervals is consistent. Each sub-interval is regarded as a large cluster after clustering in the first stage. Step 3: In the large cluster obtained after the first stage of clustering, the longitude interval is divided into multiple longitude sub-intervals according to the data density to ensure that the number of moving objects in different sub-intervals is consistent. Each sub-interval is regarded as a small cluster after the second stage of clustering. Step 4: Get the spatial distribution of the mobile object set on the road network at the latest moment; Step 5: Set a set of parameter combinations, each of which includes three parameters: time interval, displacement threshold, and radius threshold; Step 6: Based on the small clusters obtained after the second stage of clustering at the current moment, dynamic incremental clustering is performed using the current parameter combination combined with the spatial distribution of the moving objects at the latest moment, and the displacement of each moving object between two different moments is calculated. If the displacement exceeds the displacement threshold, the moving object is re-clustered; If the displacement does not exceed the displacement threshold, the moving object retains the cluster mark at the current moment as the cluster mark at the latest moment, and obtains the small cluster set at the latest moment; Step 7: Generate a cluster center for each small cluster at the latest moment. Add the longitude of each moving object in the same cluster and divide it by the number of moving objects in the cluster. Add the latitude and divide it by the number of moving objects in the cluster to obtain the longitude and latitude of the cluster center. Step 8: For the moving objects that need to be re-clustered, a circular search area is generated with the moving object as the center and the radius threshold as the radius. The cluster center that appears in the search area is used as the candidate cluster for the moving object in the re-clustering. Step 9: Among the candidate clusters, select the cluster with the least number of moving objects as the new cluster after re-clustering the moving objects; Step 10: Iterate Step 8 and Step 9 until all moving objects that need to be re-clustered are processed and dynamic incremental clustering is completed; Step 11: Use three metrics, data offset, data locality, and switching cost, to evaluate the clustering quality after dynamic incremental clustering using the current parameter combination to obtain a comprehensive indicator; Step 12: Iterate Step 7 to Step 11 until all parameter combinations are traversed to obtain all comprehensive indicators under different parameter combinations; Step 13: Select the parameter combination with the smallest comprehensive index as the optimal parameter combination of the dynamic incremental clustering algorithm, and use the optimal parameter combination each time the scheduling is switched; Step 14: The time interval in the optimal parameter combination is used as the time interval for switching scheduling. Each time the scheduling is switched, each cluster after dynamic incremental clustering is mapped to a different partition of the streaming computing engine, and then each partition is scheduled to a different computing task to perform various real-time calculations; The specific construction process of the first stage clustering in Step 2 is as follows: Step 2.1: Data preprocessing before the first clustering and definition and calculation of related variables, as described in formula (1): （1）； In formula (1), It represents a list of the original data set sorted from small to large according to longitude. represents the number of large clusters, 1 represents the ideal amount of data for a large cluster in the first clustering; Step 2.2: Get the data of each large cluster, as described in formula (2): （2）； In formula (2), Indicates the number of the large cluster, Indicates A large cluster, express exist The corresponding subset in ; Step 2.3: Add corresponding cluster markers ; Step 2.4: Iterate Step 2.2 and Step 2.3 until all moving objects complete the first stage of clustering; The specific construction process of the second stage clustering in Step 3 is as follows: Step 3.1: Data preprocessing before the second clustering and definition and calculation of related variables, as described in formula (3): （3）； In formula (3), represents the number of small clusters in each large cluster, express The amount of data, ideal Indicates the ideal amount of data in each small cluster; Step 3.2: Order Indicates Small clusters, looking for The cluster centers are sorted from large to small according to latitude. The data in the sorting , select the point with the largest latitude as the cluster center ; Step 3.3: Search The same cluster points of are as described by formula (4) and formula (5): （4）； In formula (4), the distance() function input and , output the perpendicular Euclidean distance between the two, Represents a distance list; （5）； In formula (5), express The same cluster points of Step 3.4: Order Represents the data list storing the jth small cluster. and join in , and add cluster markers; Step 3.5: Command Indicates the data list storing all small clusters. join in ; Step 3.6: Iterate Step 3.2 to Step 3.5 until all data are divided into clusters; The comprehensive index in Step 11 is specifically: the comprehensive index is a weighted average of three indicators: data offset, data locality and switching cost; Data offset refers to the imbalance of data volume between different clusters after clustering of moving objects. The calculation method of data offset is described in formula (6): （6）； In formula (6), Indicates data offset, Indicates the cluster number, represents the number of clusters, Indicates The number of mobile objects in a cluster, represents the ideal number of moving objects in each cluster after clustering, Represents the total number of moving objects. The smaller the data offset, the more balanced the data volume between different clusters. According to the scheduling scheme, these clusters are mapped to partitions, and the partitions are scheduled to tasks. After the calculation is performed, the time delay between tasks is lower; Data locality is to judge the degree of distinct boundaries between different clusters from the perspective of spatial distribution. The calculation method of data locality is described by formulas (7) to (9): （7）； （8）； （9）； In formulas (7) to (9), Indicates moving objects, express The set of circumscribed polygons of other clusters outside the cluster. Function Input and Output corresponding data locality , represents the total number of moving objects, It represents the average data locality of all moving objects. The larger the locality, the clearer the boundaries between different clusters. The switching cost is the proportion of moving objects that are re-clustered when switching scheduling. The switching cost is calculated as described in formulas (10) and (11): （10）； （11）； In formulas (10) and (11), express The switching cost, , Respectively At the upper and lower moments, represents the displacement threshold, Represents the switching cost of all moving objects. The smaller the switching cost, the lower the proportion of moving objects to be re-clustered, and the more computing resources are saved; The calculation method of the comprehensive index corresponding to clustering quality is described in formulas (12) and (13): （12）； （13）； In formulas (12) and (13), Represents a comprehensive indicator, , Respectively represent data offset , Data Locality , switching costs The smaller the comprehensive index is, the higher the data locality, the lower the data offset and the lower the switching cost.

2. According to the method for scheduling streaming computing jobs based on time and space perception according to claim 1, it is characterized in that: The optimal parameter combination of Step 13 is: （14）； In formula (14), Indicates the time interval, represents the displacement threshold, represents the radius threshold, Represents a combination of parameters, Represents a collection of parameter combinations, represents the optimal parameter combination.

Images