Method and apparatus for generating training dataset for geographic computing cloud task scheduling

Abstract

The application discloses a training data set generation method and device for geographic computing cloud task scheduling, and belongs to the technical field of large-scale data processing. The method comprises the following steps: acquiring the code of a geographic computing cloud task; converting the code of the geographic computing cloud task into a geographic directed acyclic graph, wherein the geographic directed acyclic graph comprises a plurality of task nodes; dividing the geographic directed acyclic graph into a plurality of parallel subgraphs based on the plurality of task nodes; executing a sub geographic computing cloud task corresponding to each subgraph in parallel, collecting the running parameters of each task node in each sub geographic computing cloud task; acquiring a running feature vector corresponding to each subgraph based on the running parameters; performing unsupervised clustering on the plurality of subgraphs based on the running parameters and the running feature vectors corresponding to each subgraph, and obtaining a plurality of clustering clusters. The method can construct a high-quality training data set for geographic computing cloud task scheduling.

Description

Technical Field

[0001] This invention relates to the field of large-scale data processing technology, and in particular to a method and apparatus for generating training datasets for geographic computing cloud task scheduling. Background Technology

[0002] With the rapid development of technologies such as aerospace, sensors, wireless communication, and computers, geocomputing has entered the "big data era" characterized by massive volume, diversity, and high dimensionality. Geocomputing is a computational and analytical process centered on geospatial data, widely applied in typical scenarios such as environmental modeling, intelligent navigation, intelligent interpretation of remote sensing imagery, global climate change simulation, real-time location service analysis, and large-scale raster processing. Geocomputing tasks typically exhibit both computationally intensive and data-intensive characteristics, accompanied by complex spatiotemporal dependencies. For example, remote sensing image interpretation requires processing massive amounts of high-resolution image data, with spatial overlap and temporal correlations between images; climate simulation involves cross-scale raster computation and long-term data iteration. Cloud computing, with its powerful elastic resource allocation, on-demand service, and near-infinite computing resource pooling capabilities, has become an ideal platform for large-scale geocomputing tasks. By migrating geocomputing tasks to the cloud environment, computational efficiency can theoretically be significantly improved and hardware costs reduced.

[0003] In the process of developing cloud-based geocomputing, task scheduling optimization has become a core issue in improving computing performance and resource utilization.

[0004] In related technologies, artificial intelligence is used to schedule tasks in the geographic computing cloud. After training the AI ​​(e.g., using reinforcement learning algorithms), the trained AI can learn the optimal scheduling strategy by interacting with the environment and adapt to the dynamic changes in the cloud environment.

[0005] However, when using artificial intelligence (AI) for geocomputing cloud task scheduling, the performance of the AI ​​is highly dependent on the support of high-quality training datasets. Geocomputing tasks are diverse and varied, and their workflows often manifest as complex directed acyclic graphs (DAGs), making them difficult to use directly as training samples. Traditional monitoring data only records basic resource metrics (such as CPU and memory) and lacks the ability to perform deep feature extraction and correlation analysis from the perspectives of task code structure and data semantics. This results in feature vectors used for model training being of a single dimension and lacking sufficient information. If only geographic data is used to directly train AI, the performance of the trained AI will be insufficient to meet the requirements of geocomputing cloud task scheduling. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and storage medium for generating training datasets for geospatial cloud task scheduling, capable of constructing high-quality training datasets for geospatial cloud task scheduling. The technical solution includes at least the following:

[0007] In a first aspect, a method for generating a training dataset for geocomputing cloud task scheduling is provided, comprising: obtaining the code of a geocomputing cloud task; converting the code of the geocomputing cloud task into a directed acyclic graph (DAG), the DAG including multiple task nodes; dividing the DAG into multiple parallelizable subgraphs based on the multiple task nodes, each subgraph corresponding to a sub-geocomputing cloud task; executing the sub-geocomputing cloud tasks corresponding to each subgraph in parallel, and collecting the running parameters of each task node in each sub-geocomputing cloud task during the execution of each sub-geocomputing cloud task; obtaining the running feature vector corresponding to each subgraph based on the running parameters; and performing unsupervised clustering on the multiple subgraphs based on the running parameters and running feature vectors corresponding to each subgraph to obtain multiple clusters, each cluster corresponding to at least one subgraph, and each cluster being a type of sample in the training dataset for geocomputing cloud task scheduling.

[0008] Optionally, the step of dividing the geographic directed acyclic graph into multiple parallelizable subgraphs based on the multiple task nodes includes: obtaining the resource requirement vector of each task node; dividing the geographic directed acyclic graph into multiple candidate graphs based on the resource requirement vectors of each task node, while satisfying task dependencies; optimizing the execution order of task nodes within each candidate graph and verifying the executability of each candidate graph; if a candidate graph does not meet the executability requirements, adjusting the candidate graph that does not meet the executability requirements until each candidate graph meets the executability requirements, thereby obtaining multiple parallelizable subgraphs.

[0009] Optionally, based on the resource requirement vectors of each task node, and under the premise of satisfying task dependencies, the geographical directed acyclic graph is divided into multiple candidate graphs, including: dividing the geographical directed acyclic graph into multiple initial candidate graphs to obtain an initial candidate graph set; merging the first initial candidate graph and the second initial candidate graph if the similarity of the resource requirement vectors of the first initial candidate graph and the second initial candidate graph meets a resource requirement similarity threshold, the consistency of the I / O modes of the first initial candidate graph and the second initial candidate graph meets an I / O mode consistency threshold, and the spatial / temporal coverage overlap of the first initial candidate graph and the second initial candidate graph meets an overlap threshold; wherein, the first initial candidate graph and the second initial candidate graph are initial candidate graphs in the initial candidate graph set, and the first initial candidate graph and the second initial candidate graph are adjacent on the geographical directed acyclic graph.

[0010] Optionally, the operating parameters include CPU utilization, memory usage, disk read / write speed, I / O throughput, network bandwidth usage, task execution time, and data transfer time.

[0011] Optionally, obtaining the operational feature vector of each sub-geographic computing task based on the operational parameters includes: aggregating the operational parameters of different task nodes within the same sub-map to obtain a set of operational parameters corresponding to each sub-map; and determining the operational feature vector corresponding to each sub-map based on the set of operational parameters corresponding to each sub-map.

[0012] Optionally, the method further includes: structurally encapsulating multiple clusters to obtain multiple data objects, with each cluster corresponding to one data object, the data object including the cluster label, topological information of task nodes and edges in each subgraph of the cluster, computing resource requirements, and historical running parameters; storing the multiple data objects in a first database and establishing a multi-dimensional index for the multiple data objects; and establishing a standardized scheduling interface in the geographic scheduling engine, the scheduling interface being used to access the multiple data objects in the first database.

[0013] Secondly, a training dataset generation device for geocomputing cloud task scheduling is also provided, comprising: an acquisition module for acquiring the code of geocomputing cloud tasks; a conversion module for converting the code of the geocomputing cloud tasks into a directed acyclic graph (DAG), wherein the DAG includes multiple task nodes; a subgraph partitioning module for partitioning the DAG into multiple parallelizable subgraphs based on the multiple task nodes, each subgraph corresponding to a sub-geocomputing cloud task; a parameter acquisition module for executing the sub-geocomputing cloud tasks corresponding to each subgraph in parallel, and collecting the running parameters of each task node in each sub-geocomputing cloud task during the execution of each sub-geocomputing cloud task; a feature vector acquisition module for acquiring the running feature vector corresponding to each subgraph based on the running parameters; and a clustering module for performing unsupervised clustering on the multiple subgraphs based on the running parameters and running feature vectors corresponding to each subgraph, obtaining multiple clusters, each cluster corresponding to at least one subgraph, and each cluster being a type of sample in the training dataset for geocomputing cloud task scheduling.

[0014] Optionally, the subgraph partitioning module is further configured to obtain the resource requirement vector of each task node; based on the resource requirement vector of each task node, and under the premise of satisfying task dependencies, divide the geographic directed acyclic graph into multiple candidate graphs; optimize the execution order of task nodes within each candidate graph, and verify the executability of each candidate graph; if there is a candidate graph that does not meet the executability requirements, adjust the candidate graph that does not meet the executability requirements until each candidate graph meets the executability requirements, thereby obtaining multiple parallelizable subgraphs.

[0015] Optionally, the subgraph partitioning module is further configured to partition the geographic directed acyclic graph into multiple initial candidate graphs to obtain an initial candidate graph set; and merge the first initial candidate graph and the second initial candidate graph if the similarity of the resource demand vectors of the first initial candidate graph and the second initial candidate graph meets the resource demand similarity threshold, the consistency of the I / O patterns of the first initial candidate graph and the second initial candidate graph meets the I / O pattern consistency threshold, and the spatial / temporal coverage overlap of the first initial candidate graph and the second initial candidate graph meets the overlap threshold; wherein the first initial candidate graph and the second initial candidate graph are initial candidate graphs in the initial candidate graph set, and the first initial candidate graph and the second initial candidate graph are adjacent on the geographic directed acyclic graph.

[0016] Optionally, in the parameter acquisition module, the operating parameters include CPU utilization, memory usage, disk read / write speed, I / O throughput, network bandwidth usage, task execution time, and data transmission time.

[0017] Optionally, the feature vector acquisition module is further configured to aggregate the running parameters of different task nodes within the same subgraph to obtain a set of running parameters corresponding to each subgraph; and to determine the running feature vector corresponding to each subgraph based on the set of running parameters corresponding to each subgraph.

[0018] Optionally, the device further includes: a post-processing module for structurally encapsulating multiple clusters to obtain multiple data objects, each cluster corresponding to one data object, the data object including the cluster label, topological information of task nodes and edges in each subgraph of the cluster, computing resource requirements, and historical running parameters; storing the multiple data objects in a first database and establishing a multi-dimensional index of the multiple data objects; and establishing a standardized scheduling interface in the geographic scheduling engine, the scheduling interface being used to access the multiple data objects in the first database.

[0019] Thirdly, a computer device is also provided, comprising: a memory and a processor, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to perform the training dataset generation method for geospatial cloud task scheduling described in the above embodiments.

[0020] Fourthly, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform the training dataset generation method for geospatial cloud task scheduling described in the above embodiments.

[0021] Fifthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0022] The beneficial effects of the technical solution provided by this invention include at least the following:

[0023] This invention achieves high-quality training dataset construction for geocomputing cloud task scheduling by: acquiring the code of geocomputing cloud tasks; converting the code into a directed acyclic graph (DAG), which includes multiple task nodes; dividing the DAG into multiple parallelizable subgraphs, each corresponding to a sub-geocomputing cloud task; executing the sub-geocomputing cloud tasks corresponding to each subgraph in parallel, and collecting the running parameters of each task node in each sub-geocomputing cloud task during execution; obtaining the running feature vector corresponding to each subgraph based on the running parameters and running feature vectors; and performing unsupervised clustering on the multiple subgraphs based on the running parameters and running feature vectors corresponding to each subgraph, resulting in multiple clusters. Each cluster corresponds to at least one subgraph, and each cluster represents a class of samples in the training dataset for geocomputing cloud task scheduling. This method can replace traditional dataset generation methods that rely on manual construction or inefficient sampling. Under complex and ever-changing task load characteristics and resource constraints, this invention has good data generation efficiency, representativeness and adaptability, and can effectively support the training and optimization of scheduling algorithms. It is especially suitable for large-scale, dynamically changing geographic cloud computing environments. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1A flowchart illustrating a method for generating training datasets for geocomputing cloud task scheduling according to an exemplary embodiment of the present invention is shown.

[0026] Figure 2 A flowchart of a method for generating training datasets for geocomputing cloud task scheduling, provided by another exemplary embodiment of the present invention, is shown.

[0027] Figure 3 This is a schematic diagram of converting the code of a geocomputing cloud task into a geographic directed acyclic graph;

[0028] Figure 4 A schematic diagram of dividing a geographical directed acyclic graph into multiple parallel subgraphs;

[0029] Figure 5 This is a schematic diagram of the processing flow of each sub-graph;

[0030] Figure 6 This illustration shows a schematic diagram of the structure of a training dataset generation apparatus for geocomputing cloud task scheduling provided in an exemplary embodiment of the present invention.

[0031] Figure 7 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0032] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, but do not exclude other elements or objects.

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0034] Figure 1 A flowchart illustrating a method for generating training datasets for geospatial cloud task scheduling according to an exemplary embodiment of the present invention is shown. This method can be executed by a computer device. See also Figure 1 The method includes:

[0035] In step 101, the code for the geocomputing cloud task is obtained.

[0036] In step 102, the geographic dataset is converted into a geographic directed acyclic graph.

[0037] The geographically directed acyclic graph includes multiple task nodes.

[0038] In step 103, the geographical directed acyclic graph is divided into multiple parallel subgraphs based on multiple task nodes.

[0039] Each sub-map corresponds to a sub-geocomputing cloud task.

[0040] In step 104, the sub-geocomputing cloud tasks corresponding to each sub-map are executed in parallel. During the execution of each sub-geocomputing cloud task, the running parameters of each task node in each sub-geocomputing cloud task are collected.

[0041] In step 105, the running feature vector corresponding to each subgraph is obtained based on the running parameters.

[0042] In step 106, based on the running parameters and running feature vectors corresponding to each subgraph, unsupervised clustering is performed on multiple subgraphs to obtain multiple clusters.

[0043] Each cluster corresponds to at least one subgraph, and a cluster is a class of samples in the training dataset for geocomputing cloud task scheduling.

[0044] This invention can replace traditional methods of generating datasets that rely on manual construction or inefficient sampling, enabling the construction of high-quality training datasets for geospatial cloud task scheduling. Under complex and ever-changing task load characteristics and resource constraints, this invention exhibits good data generation efficiency, representativeness, and adaptability, effectively supporting the training and optimization of scheduling algorithms, and is particularly suitable for large-scale, dynamically changing geospatial cloud computing environments.

[0045] Figure 2 A flowchart illustrating a method for generating training datasets for geospatial cloud task scheduling, provided by another exemplary embodiment of the present invention, is shown. This method can be executed by a computer device. See also Figure 2 The method includes:

[0046] In step 201, the code for the geocomputing cloud task is obtained.

[0047] In this embodiment of the disclosure, the geocomputing cloud task exists in the form of code. The geocomputing cloud task can be a geocomputing cloud task that has been executed or a geocomputing cloud task that has not been executed.

[0048] In step 202, the code for the geocomputing cloud task is converted into a geographic directed acyclic graph.

[0049] A Geographic Directed Acyclic Graph (Geo-DAG) includes multiple task nodes.

[0050] Figure 3 This is a schematic diagram illustrating the conversion of code for geocomputing cloud tasks into a directed acyclic graph (DAG). The following section combines... Figure 3 Step 202 will be explained.

[0051] Optionally, step 202 includes steps 2021 to 2023.

[0052] Step 2021: Perform syntax parsing on the code of the geocomputing cloud task to obtain the first abstract syntax tree.

[0053] In addition, it is necessary to label the type of each geocomputation operator, the input and output data of each geocomputation operator, and the external calling information of each geocomputation operator in the abstract syntax tree.

[0054] Among them, the types of geographic computing operators include raster algebra operations, vector topology operations, spatial overlay analysis, buffer generation, interpolation and resampling, etc.

[0055] The input and output data of geospatial operators include the attributes of the input data (such as raster and vector), data path, and data dimension information (such as the number of bands, the number of features, and the spatial range).

[0056] External call information for each geospatial calculation operator includes external library functions and third-party service interfaces.

[0057] This results in a first abstract syntax tree with geographic computation semantic tags.

[0058] Step 2022: Perform semantic analysis on the first abstract syntax tree to obtain at least one task node, and determine the data dependency and control dependency relationships between the task nodes.

[0059] When dividing task nodes, composite operations are broken down into the smallest granularity of computational task nodes. For example, a composite operation such as "raster clipping + band operation + resampling" is decomposed into three independently executable task nodes.

[0060] Data dependency between task nodes refers to the data flow relationship between different task nodes, meaning that subsequent task nodes need to use the results produced by previous task nodes as their operands. For example, a raster calculation task can only be executed after the upstream pruning task is completed.

[0061] Control dependencies between task nodes refer to the fact that the execution of a task node depends on the execution results of previous task nodes, thus affecting the control flow of the program. For example, meteorological data needs to be processed day by day in time sequence.

[0062] Step 2023: Build a geographical directed acyclic graph based on the data dependencies and control dependencies between each task node.

[0063] Here, after obtaining multiple task nodes and the data dependencies and control dependencies between task nodes, the task nodes can be connected according to the data dependencies and control dependencies to obtain a geographic directed acyclic graph.

[0064] To ensure the executability of this geographic directed acyclic graph, a cycle detection algorithm can be used to detect and eliminate potential circular dependencies in the graph. If necessary, circular dependencies can be split or introduced into intermediate cache nodes. The final result is a standardized geographic directed acyclic graph.

[0065] Metadata information can also be appended to each task node in the geographic directed acyclic graph to support subsequent scheduling optimization.

[0066] The metadata information includes task identifier, data input / output mode, and resource requirement estimate (operator type and data size, preliminary estimate of CPU cores, memory usage, disk I / O requirements and data transfer cost).

[0067] Optionally, after obtaining multiple task nodes, each task node can be labeled. When labeling task nodes, the computation type, data attributes, start and end times and time granularity, physical storage location of the data, and data transfer cost of each task node can be labeled.

[0068] The computing types include CPU-intensive, memory-intensive, I / O-intensive, or hybrid.

[0069] Data attributes include raster and vector. For raster data, it is necessary to record features such as data resolution, spatial extent (represented by area or diagonal length), and number of bands; for vector data, it is necessary to label structural characteristics such as the number of features and topological complexity.

[0070] Time granularity is determined based on the time dimension of the data, and it is also possible to convert data with different time dimensions into the same time dimension. Here, the time dimension includes year, month, day, or hour.

[0071] Data transmission cost is estimated based on parameters such as data volume, network bandwidth, and transmission latency.

[0072] In this way, each task node has a complete structured description, providing data support for the subsequent step 203.

[0073] In step 203, the geographical directed acyclic graph is divided into multiple parallel subgraphs based on multiple task nodes.

[0074] Each sub-map corresponds to a sub-geocomputing cloud task.

[0075] Figure 4 This is a schematic diagram illustrating the division of a directed acyclic graph (DAG) into multiple parallel subgraphs. The following section combines... Figure 4 Step 203 will be explained.

[0076] Optionally, step 203 includes steps 2031 to 2034.

[0077] Step 2031: Obtain the resource requirement vector for each task node.

[0078] In implementing step 2031, the task nodes are structured and parsed based on their metadata and historical running characteristics to obtain a multi-dimensional resource requirement vector.

[0079] The resource demand vector includes five dimensions: computing resource demand, task operation characteristics, data scale and spatiotemporal attributes, data physical location and transmission time, and data dependency characteristics.

[0080] Computing resource requirements include CPU core count, memory usage, disk I / O throughput, and GPU requirements.

[0081] Task runtime characteristics, such as average runtime and peak resource utilization;

[0082] In terms of data scale and spatiotemporal attributes, raster data includes resolution, spatial coverage and number of bands, vector data includes the number of features and topological complexity, and time attributes include start and end time and time granularity.

[0083] Regarding the physical location of data and transmission time, the physical location of data refers to the location of the data storage node (which is known). The transmission time can be estimated based on the location of the data storage node, the amount of data, and the network bandwidth and latency.

[0084] Data dependency characteristics include constraints such as the amount of input and output data, generation order, and real-time or batch processing window.

[0085] By parsing the metadata of task nodes into structured resource requirement vectors, the resource requirements of task nodes can be structured and quantified, providing a foundation for subsequent steps 2032 to 2034.

[0086] Step 2032: Based on the resource requirement vector of each task node, the geographical directed acyclic graph is divided into multiple candidate graphs while satisfying task dependencies.

[0087] Optionally, step 2032 includes steps a to b as follows.

[0088] Step a: Divide the geographical directed acyclic graph into multiple initial candidate graphs to obtain an initial candidate graph set.

[0089] In a geographic directed acyclic graph, chains of similar continuous operators (such as map→filter→map with unbroken narrow dependencies in Spark RDDs) under the same coordinate reference system, spatial range, and resolution are split to avoid splitting at "high-cost spatial ranges" such as resampling, repartitioning, and reprojection, thus obtaining multiple first candidate graphs.

[0090] Then, without disrupting spatial adjacency, task nodes in multiple first candidate graphs that have no data dependency or only weakly correlated relationships that can be parallelized are decoupled and made into independent graphs to improve parallelism.

[0091] The first candidate graph after decoupling and the first candidate graph before decoupling together constitute the initial candidate graph set.

[0092] Step b: If the similarity of the resource demand vectors of the first initial candidate graph and the second initial candidate graph meets the resource demand similarity threshold, the consistency of the I / O mode of the first initial candidate graph and the second initial candidate graph meets the I / O mode consistency threshold, and the spatial / temporal coverage overlap of the first initial candidate graph and the second initial candidate graph meets the overlap threshold, then merge the first initial candidate graph and the second initial candidate graph.

[0093] The first initial candidate map and the second initial candidate map are initial candidate maps in the initial candidate map set, and the first initial candidate map and the second initial candidate map are adjacent on the geographical directed acyclic map.

[0094] Here, the resource demand similarity threshold, I / O mode consistency threshold, and overlap threshold are all preset thresholds, and their values ​​can be set according to requirements. This embodiment does not limit them.

[0095] By performing step b on each pair of first and second initial candidate graphs in the initial candidate graph set, the initial candidate graph set can be simplified, thereby obtaining multiple candidate graphs.

[0096] In steps a and b above, "soft verification" is continuously performed during the merging and segmentation of the geographical directed acyclic graph. This means that the peak CPU / MEM / IO / GPU concurrency of the candidate graph does not exceed the upper limit of the quota, the spatial localization rate is not lower than the set threshold, and idempotency, consistency, and correct spatial topology are maintained. For real-time / window-type tasks, time constraints are not violated, and parallel conflicts are avoided for tasks that require exclusive resources (such as GPU, write lock, and regionalization).

[0097] If any step results in resource out-of-bounds, insufficient locality, inconsistent CRS / temporal resolution, or improper cutting of the critical path, then the process is to backtrack and iterate in the following order: "backtrack at the geographic natural boundary nodes, merge with the subgraphs that are closest in space / time, adjust the partition or tile level and parallelism, and further subdivide at the boundaries of administrative regions / basins, etc." until a candidate graph partition that achieves a balance between transmission cost, critical path, and resource feasibility is obtained.

[0098] Step 2033: Optimize the execution order of task nodes within each candidate graph and verify the executability of each candidate graph.

[0099] When optimizing the execution order of task nodes within each candidate graph, nodes are topologically sorted based on the dependencies between tasks. Tasks without dependencies are grouped into parallel groups for execution, while tasks with dependencies or requiring shared resources are executed sequentially. During parallel group execution, tasks with similar spatial locations, consistent data partitions (such as the same tile, coordinate system, and resolution), and the same time window are prioritized for grouping together, keeping computation as close as possible to the nodes containing the data to reduce data transfer across racks or regions. For nodes involving high-cost operations such as shuffle, reprojection, resampling, and strong consistency writes, synchronization barriers are set at the boundaries to ensure sequential execution of preceding and following stages and avoid concurrency conflicts.

[0100] Verifying the executability of candidate graphs includes: whether dependencies are completely closed, whether peak resource consumption is within cluster limits, whether the data localization rate meets the threshold, whether the time window is continuous and complete, and whether write or aggregation operations are idempotent and recoverable.

[0101] Step 2034: If there are candidate graphs that do not meet the executability requirements, then the candidate graphs that do not meet the executability requirements are adjusted until each candidate graph meets the executability requirements, resulting in multiple parallelizable subgraphs.

[0102] When adjusting candidate graphs, the order or parallelism of task nodes is adjusted according to the principles of spatial proximity or temporal alignment until multiple subgraphs are formed that are resource-feasible, have reasonable dependencies, and are efficient in-situ.

[0103] Figure 5 This is a schematic diagram of the processing flow for each sub-graph. The following section combines... Figure 5 Steps 204 to 209 will be explained.

[0104] In step 204, the sub-geocomputing cloud tasks corresponding to each sub-map are executed in parallel. During the execution of each sub-geocomputing cloud task, the running parameters of each task node in each sub-geocomputing cloud task are collected.

[0105] Here, dividing a geographic directed acyclic graph into multiple parallelizable subgraphs essentially divides the original geographic computing cloud task into multiple sub-geocomputing cloud tasks, and these sub-geocomputing cloud tasks can be executed in parallel.

[0106] The parallel execution of the sub-geocomputing cloud tasks corresponding to each sub-graph in step 204 can be likened to a simulation of the sub-geocomputing cloud tasks, which can reflect the characteristics of each sub-geocomputing cloud task during runtime.

[0107] When collecting the running parameters of each task node in each sub-geocomputing cloud task, a lightweight monitoring unit can be injected into each task node. The lightweight monitoring unit can collect the running parameters of each task node in real time, ensuring that the resource consumption status of each task node during execution can be comprehensively and continuously reflected.

[0108] There are many ways to implement lightweight monitoring units, so we will omit the details here.

[0109] In this embodiment, the operating parameters include: CPU utilization, memory usage, disk read / write speed, I / O throughput, network bandwidth usage, task execution time, and data transmission time.

[0110] In step 205, the running feature vector corresponding to each subgraph is obtained based on the running parameters.

[0111] Optionally, step 205 includes steps 2051 to 2052.

[0112] Step 2051: Aggregate the running parameters of different task nodes within the same subgraph to obtain the running parameter set corresponding to each subgraph.

[0113] In implementation, the running parameters of different task nodes within the same time window and subgraph are aggregated, outliers caused by acquisition jitter or system interference are removed, duplicate records are eliminated, and the running parameters of each task node are aligned according to a unified time series and task identifier. The processed data is stored as a standardized resource consumption record (i.e., a set of running parameters) to ensure consistency for subsequent analysis.

[0114] Step 2052: Based on the set of running parameters corresponding to each subgraph, determine the running feature vector corresponding to each subgraph.

[0115] Here, the running feature vector can be a multi-dimensional vector, where one dimension of the multi-dimensional vector is a statistical feature of the set of running parameters. For example, the statistical features of the set of running parameters include mean, variance, peak value, volatility, resource utilization curve, etc.

[0116] Furthermore, the statistical feature values ​​of different dimensions need to be normalized to eliminate differences in units. This ultimately results in a standardized operational feature vector, which can efficiently characterize the resource usage patterns and performance characteristics of sub-geocomputing cloud tasks corresponding to each sub-map.

[0117] Optionally, after obtaining the running feature vector, anomaly detection can be performed based on the running feature vector. For example, threshold discrimination, cluster deviation detection, or machine learning prediction models can be used to compare and analyze the running feature vector. When a feature value in the running feature vector is detected to deviate significantly from the historical distribution or resource utilization anomalies, corresponding fault tolerance label information is generated, and a predefined fault tolerance mechanism is triggered to ensure the reliability and stability of the sub-geocomputing cloud task corresponding to the sub-map under abnormal conditions.

[0118] The operational feature vectors and parameters corresponding to each sub-map can be stored in the first database according to a standardized data model, and an interaction interface with the cloud-based geocomputing engine can be established in the first database. This interface supports subsequent feature data queries, statistical analysis, and scheduling optimization calls, enabling the scheduling engine (such as artificial intelligence) to dynamically adjust resource allocation and execution strategies based on historical operational feature vectors, thereby improving the overall computing efficiency and resource utilization of geocomputing.

[0119] In step 206, based on the running parameters and running feature vectors corresponding to each subgraph, unsupervised clustering is performed on multiple subgraphs to obtain multiple clusters.

[0120] Each cluster corresponds to at least one subgraph, and a cluster is a class of samples in the training dataset for geocomputing cloud task scheduling.

[0121] Before performing unsupervised clustering, preprocessing operations are performed on the running parameters and feature vectors to improve the clustering results. Preprocessing includes normalization or feature selection algorithms. Normalization eliminates the dimensional differences between different features, reducing dimensionality to decrease redundant information and computational complexity; feature selection algorithms extract the most discriminative subset of features, thereby generating a feature space suitable for cluster analysis.

[0122] An unsupervised clustering method is used to classify the preprocessed running parameters and running feature vectors, resulting in multiple clusters. For example, the unsupervised clustering method can be K-means, hierarchical clustering, density clustering (such as the DBSCAN clustering algorithm), or other machine learning clustering algorithms.

[0123] Cluster analysis is used to group subgraphs with similar resource requirements and operational characteristics into the same cluster, so that the scheduling engine can perform unified modeling and scheduling optimization for geospatial cloud tasks.

[0124] In addition, the effectiveness of these multiple clusters can be evaluated, for example, by using indicators such as silhouette coefficient, intra-cluster compactness and inter-cluster separation for quantitative evaluation.

[0125] For clusters whose validity evaluation results are invalid or whose boundaries are ambiguous, optimization can be achieved by iteratively adjusting parameters or changing the clustering algorithm, and finally, multiple clusters required in step 206 can be obtained.

[0126] These multiple clusters are labeled, and each cluster is given a descriptive label (such as "computation-intensive", "large-scale raster processing", "real-time location analysis"). The labeling results and their feature vectors are stored together in the first database to support subsequent fast retrieval and scheduling decisions.

[0127] Here, a cluster is equivalent to a class of samples, and the descriptive label corresponding to the cluster is the label of this class of samples, thus realizing the construction of a training dataset for geospatial cloud task scheduling.

[0128] After obtaining the training dataset for geocomputing cloud task scheduling, steps 207 to 209 can be used to link the training dataset with the scheduling engine.

[0129] In step 207, multiple clusters are structurally encapsulated to obtain multiple data objects.

[0130] Each cluster corresponds to a data object, which includes the cluster label, topological information of task nodes and edges in each subgraph of the cluster, computing resource requirements, and historical running parameters.

[0131] Computational resource requirements, such as CPU, memory, and I / O, can be obtained through a resource requirement vector. Historical runtime parameters include execution time, failure rate, and resource usage.

[0132] During the encapsulation process, it is necessary to define standard data models and field specifications to ensure consistency and integrity in subsequent storage and retrieval, and to perform necessary data cleaning and verification on data objects to remove outliers and redundant information, thereby ensuring encapsulation quality.

[0133] In step 208, multiple data objects are stored in the first database, and a multi-dimensional index of the multiple data objects is created.

[0134] For example, the first database is a storage system (such as a distributed database or object storage) that supports high-concurrency read and write operations and massive data management.

[0135] Multi-dimensional indexes should be created for these data objects in the scheduling engine, such as multi-dimensional indexes based on cluster labels, resource characteristics, and runtime statistics, to facilitate rapid retrieval by the scheduling engine. The index design should balance query efficiency and storage cost, supporting searches based on multiple criteria such as similarity, resource requirements, and historical performance to meet the real-time calling needs of the scheduling engine.

[0136] In step 209, a standardized scheduling interface is established in the geographic scheduling engine.

[0137] The scheduling interface is used to access multiple data objects in the first database.

[0138] This standardized scheduling interface enables the upper-layer scheduler to access data from each cluster. The interface needs to provide query, retrieval, filtering, and prediction capabilities, such as quickly matching similar tasks by cluster tags, predicting resource requirements based on historical runtime statistics, and generating optimal scheduling schemes. The interface should also support asynchronous calls, batch processing, and access control to ensure stability and security under high-concurrency task scheduling, while providing logging and monitoring functions for subsequent optimization of scheduling strategies.

[0139] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the above method embodiments.

[0140] Figure 6 A schematic diagram of a training dataset generation apparatus for geospatial cloud task scheduling provided in an exemplary embodiment of the present invention is shown. See also Figure 6 The training dataset device 600 for geospatial cloud task scheduling includes an acquisition module 601, a transformation module 602, a subgraph partitioning module 603, a parameter acquisition module 604, a feature vector acquisition module 605, and a clustering module 606.

[0141] Module 601 is used to obtain the code for geocomputing cloud tasks;

[0142] The conversion module 602 is used to convert the code of the geocomputing cloud task into a geographic directed acyclic graph, which includes multiple task nodes.

[0143] The subgraph partitioning module 603 is used to partition a geographic directed acyclic graph into multiple parallel subgraphs based on multiple task nodes, with each subgraph corresponding to a sub-geographic computing cloud task.

[0144] The parameter acquisition module 604 is used to execute the sub-geocomputing cloud tasks corresponding to each sub-map in parallel. During the execution of each sub-geocomputing cloud task, the module collects the running parameters of each task node in each sub-geocomputing cloud task.

[0145] The feature vector acquisition module 605 is used to acquire the running feature vector corresponding to each subgraph based on the running parameters;

[0146] The clustering module 606 is used to perform unsupervised clustering on multiple subgraphs based on the running parameters and running feature vectors corresponding to each subgraph, resulting in multiple clusters. Each cluster corresponds to at least one subgraph, and each cluster is a sample from the training dataset of the geospatial cloud task scheduling.

[0147] Optionally, the subgraph partitioning module 603 is also used to obtain the resource requirement vector of each task node; based on the resource requirement vector of each task node, the geographical directed acyclic graph is partitioned into multiple candidate graphs under the premise of satisfying task dependencies; the execution order of task nodes within each candidate graph is optimized, and the executability of each candidate graph is verified; if there is a candidate graph that does not meet the executability requirements, the candidate graph that does not meet the executability requirements is adjusted until each candidate graph meets the executability requirements, resulting in multiple parallelizable subgraphs.

[0148] Optionally, the subgraph partitioning module 603 is further configured to partition the geographic directed acyclic graph into multiple initial candidate graphs to obtain an initial candidate graph set; and merge the first initial candidate graph and the second initial candidate graph if the similarity of the resource demand vectors of the first initial candidate graph and the second initial candidate graph meets the resource demand similarity threshold, the consistency of the I / O modes of the first initial candidate graph and the second initial candidate graph meets the I / O mode consistency threshold, and the spatial / temporal coverage overlap of the first initial candidate graph and the second initial candidate graph meets the overlap threshold; wherein the first initial candidate graph and the second initial candidate graph are initial candidate graphs in the initial candidate graph set, and the first initial candidate graph and the second initial candidate graph are adjacent on the geographic directed acyclic graph.

[0149] Optionally, in the parameter acquisition module 604, the operating parameters include CPU utilization, memory usage, disk read / write speed, I / O throughput, network bandwidth usage, task execution time, and data transmission time.

[0150] Optionally, the feature vector acquisition module 605 is also used to aggregate the running parameters of different task nodes within the same subgraph to obtain a set of running parameters corresponding to each subgraph; and to determine the running feature vector corresponding to each subgraph based on the set of running parameters corresponding to each subgraph.

[0151] Optionally, the device further includes: a post-processing module 607, which is used to perform structured encapsulation on multiple clusters to obtain multiple data objects, with each cluster corresponding to one data object. The data objects include the cluster label, topological information of task nodes and edges in each subgraph of the cluster, computing resource requirements, and historical running parameters; store the multiple data objects in a first database and establish a multi-dimensional index for the multiple data objects; and establish a standardized scheduling interface in the geographic scheduling engine, which is used to access the multiple data objects in the first database.

[0152] It should be noted that the above embodiments of the training dataset generation device for geospatial cloud task scheduling are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the training dataset generation device for geospatial cloud task scheduling provided in the above embodiments and the training dataset generation method embodiments for geospatial cloud task scheduling belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0153] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment of the invention can be integrated into a single processor, exist as separate physical entities, or consist of two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0154] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or communication device, etc.) or processor to execute all or part of the steps of the method of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] Figure 7This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. Figure 7 As shown, the computer device 700 includes a processor 701 and a memory 702.

[0156] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0157] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 is used to store at least one instruction, which is executed by the processor 701 to implement the training dataset generation method for geospatial cloud task scheduling provided in this embodiment of the invention.

[0158] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0159] This invention also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of a computer device, the computer device is able to execute the training dataset generation method for geospatial cloud task scheduling provided in this invention.

[0160] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the training dataset generation method for geospatial cloud task scheduling provided in this invention.

[0161] The above description is merely an optional embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating a training dataset for geospatial cloud task scheduling, characterized in that, The method includes: Obtain the code for the geocomputing cloud task; The code of the geocomputing cloud task is converted into a geographic directed acyclic graph, which includes multiple task nodes. Based on the multiple task nodes, the geographic directed acyclic graph is divided into multiple parallel subgraphs, each subgraph corresponding to a sub-geographic computing cloud task. The sub-geocomputing cloud tasks corresponding to each sub-map are executed in parallel. During the execution of each sub-geocomputing cloud task, the running parameters of each task node in each sub-geocomputing cloud task are collected. Based on the aforementioned operating parameters, obtain the operating feature vector corresponding to each subgraph; Based on the running parameters and running feature vectors corresponding to each subgraph, unsupervised clustering is performed on multiple subgraphs to obtain multiple clusters. Each cluster corresponds to at least one subgraph, and each cluster is a sample in the training dataset of the geocomputing cloud task scheduling. Based on the multiple task nodes, the geographical directed acyclic graph is divided into multiple parallelizable subgraphs, including: Obtain the resource requirement vector for each task node; Based on the resource requirement vector of each task node, the geographical directed acyclic graph is divided into multiple candidate graphs while satisfying task dependencies. Optimize the execution order of task nodes within each candidate graph and verify the executability of each candidate graph; If there are candidate graphs that do not meet the executability requirements, then the candidate graphs that do not meet the executability requirements are adjusted until each candidate graph meets the executability requirements, resulting in multiple parallelizable subgraphs; Based on the resource requirement vectors of each task node, and under the premise of satisfying task dependencies, the geographical directed acyclic graph is divided into multiple candidate graphs, including: The geographical directed acyclic graph is divided into multiple initial candidate graphs to obtain an initial candidate graph set; If the similarity of the resource demand vectors of the first initial candidate graph and the second initial candidate graph meets the resource demand similarity threshold, the consistency of the I / O mode of the first initial candidate graph and the second initial candidate graph meets the I / O mode consistency threshold, and the spatial / temporal coverage overlap of the first initial candidate graph and the second initial candidate graph meets the overlap threshold, then the first initial candidate graph and the second initial candidate graph are merged. Wherein, the first initial candidate map and the second initial candidate map are initial candidate maps in the initial candidate map set, and the first initial candidate map and the second initial candidate map are adjacent on the geographical directed acyclic map.

2. The method for generating a training dataset for geospatial cloud task scheduling according to claim 1, characterized in that, The operating parameters include CPU utilization, memory usage, disk read / write speed, I / O throughput, network bandwidth usage, task execution time, and data transfer time.

3. The method for generating a training dataset for geospatial cloud task scheduling according to claim 1 or 2, characterized in that, The step of obtaining the operational feature vector for each sub-geographic computing task based on the operational parameters includes: Aggregate the running parameters of different task nodes within the same subgraph to obtain the set of running parameters corresponding to each subgraph; Based on the set of running parameters corresponding to each subgraph, the running feature vector corresponding to each subgraph is determined.

4. The method for generating a training dataset for geospatial cloud task scheduling according to claim 1 or 2, characterized in that, The method further includes: Multiple clusters are structurally encapsulated to obtain multiple data objects. Each cluster corresponds to one data object. The data object includes the cluster label, the topological information of task nodes and edges in each subgraph of the cluster, the computing resource requirements, and historical running parameters. The multiple data objects are stored in the first database, and a multi-dimensional index of the multiple data objects is established; A standardized scheduling interface is established in the geographic scheduling engine, which is used to access multiple data objects in the first database.

5. A training dataset generation device for geospatial cloud task scheduling, characterized in that, The device includes: The acquisition module is used to acquire code for geocomputing cloud tasks; A conversion module is used to convert the code of the geocomputing cloud task into a geographic directed acyclic graph, which includes multiple task nodes. The subgraph partitioning module is used to partition the geographic directed acyclic graph into multiple parallel subgraphs based on the multiple task nodes, with each subgraph corresponding to a sub-geographic computing cloud task. The parameter acquisition module is used to execute the sub-geocomputing cloud tasks corresponding to each sub-map in parallel. During the execution of each sub-geocomputing cloud task, the module collects the running parameters of each task node in each sub-geocomputing cloud task. The feature vector acquisition module is used to acquire the running feature vector corresponding to each subgraph based on the running parameters; The clustering module is used to perform unsupervised clustering on multiple subgraphs based on the running parameters and running feature vectors corresponding to each subgraph, resulting in multiple clusters. Each cluster corresponds to at least one subgraph, and each cluster is a sample from the training dataset of the geocomputing cloud task scheduling. The subgraph partitioning module is also used to obtain the resource requirement vector of each task node; Based on the resource requirement vector of each task node, the geographical directed acyclic graph is divided into multiple candidate graphs while satisfying task dependencies. Optimize the execution order of task nodes within each candidate graph and verify the executability of each candidate graph; If there are candidate graphs that do not meet the executability requirements, then the candidate graphs that do not meet the executability requirements are adjusted until each candidate graph meets the executability requirements, resulting in multiple parallelizable subgraphs; Based on the resource requirement vectors of each task node, and under the premise of satisfying task dependencies, the geographical directed acyclic graph is divided into multiple candidate graphs, including: The geographical directed acyclic graph is divided into multiple initial candidate graphs to obtain an initial candidate graph set; If the similarity of the resource demand vectors of the first initial candidate graph and the second initial candidate graph meets the resource demand similarity threshold, the consistency of the I / O mode of the first initial candidate graph and the second initial candidate graph meets the I / O mode consistency threshold, and the spatial / temporal coverage overlap of the first initial candidate graph and the second initial candidate graph meets the overlap threshold, then the first initial candidate graph and the second initial candidate graph are merged. Wherein, the first initial candidate map and the second initial candidate map are initial candidate maps in the initial candidate map set, and the first initial candidate map and the second initial candidate map are adjacent on the geographical directed acyclic map.

6. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the training dataset generation method for geospatial cloud task scheduling as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the training dataset generation method for geospatial cloud task scheduling as described in any one of claims 1 to 4.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the training dataset generation method for geospatial cloud task scheduling as described in any one of claims 1 to 4.

Images