Data-centered computing resource collaborative scheduling system and method for astronomical scientific research cloud

Abstract

The invention discloses an astronomical scientific research cloud-oriented data-centered computing resource collaborative scheduling system, method and device, and a storage medium, and belongs to the technical field of distributed computing. The system comprises a global task scheduler, a local task agent, a task wrapper, a task executor, a metadatabase and a result database. According to the method, a hierarchical framework of'global scheduling + local scheduling 'is adopted, global scheduling is based on data distribution information in a metadatabase and real-time loads of cloud nodes, and calculation tasks are intelligently scheduled to target nodes by taking data proximity and load balance as principles; a local task agent initiates and manages asynchronous execution of specific tasks through a task wrapper that follows a unified interface. According to the method, the problems of low global resource utilization rate and difficulty in application integration in geographically dispersed astronomical scientific research clouds are effectively solved, efficient collaborative scheduling taking data as a center is realized, and the expansibility and practicability of the system are improved.

Description

Technical Field

[0001] This invention belongs to the field of distributed computing technology, specifically relating to a data-centric collaborative scheduling system, method, device, and storage medium for computing resources in astronomical research clouds. Background Technology

[0002] An astronomical research cloud is a geographically distributed online research environment, consisting of multiple data center or computing center nodes distributed across different astronomical observatories. Each node has its own independent hardware and software environment, internal scheduling system, and operation and maintenance procedures, and continuously generates a large amount of observational data. Because transmitting large-scale data over a wide area network is costly and impractical, data-intensive computing can usually only be performed on the local node where the data resides.

[0003] The existing technology has the following main drawbacks: Incomplete global data layout information makes it difficult to efficiently perform data location-related computation scheduling on a global scale.

[0004] Lack of a global scheduling mechanism: It is impossible to monitor and optimize the allocation of computing resources across the entire astronomical research cloud in real time, resulting in low resource utilization.

[0005] Application integration difficulties: The lack of unified application model analysis and integration standards makes it impossible to flexibly support various types of scientific research computing tasks.

[0006] Therefore, there is an urgent need in this field for a solution that can achieve efficient and coordinated scheduling of global resources without significantly affecting the existing operating systems of each node. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a data-centric approach for astronomical research clouds. A computing resource collaborative scheduling system, method, equipment, and storage medium are developed to address the challenges of collaborative scheduling of data and computing resources in geographically dispersed environments.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a data-centric collaborative scheduling system for computing resources in astronomical research clouds, comprising: A global task scheduler, at least one local task agent, at least one task wrapper, at least one task executor, a metadata database, and a results database; The global task scheduler is deployed on the master node of the astronomical research cloud. It is used to receive computing tasks submitted by users and schedule the computing tasks to the target cloud node based on the dataset distribution information recorded in the metadata database and the real-time load status of each cloud node. The local task agent is deployed on each sub-node of the astronomical research cloud. It is used to receive tasks from the global task scheduler and, according to the preset concurrency control strategy, to start or manage the task execution process by calling the task wrapper. The task wrapper, serving as the interface between the local task agent and the task executor, is used to start the corresponding task execution program on the remote computing node, pass execution parameters, and provide feedback on the task execution status and results. The task executor is used to specifically execute the computational task; The metadata database is used to store and maintain the distribution metadata of all datasets in the astronomical research cloud; The results database is used to store the results data generated after the task is executed, or the storage path information of the results files.

[0009] The cloud node includes a master node and sub-nodes, and computing nodes are deployed within the cloud node.

[0010] Preferably, the process of the global task scheduler performing task scheduling includes: Based on the dataset name specified in the received computing task, query the metadata database to determine the set of candidate cloud nodes that store the dataset; Obtain the real-time load index of each node in the candidate cloud node set, select the candidate node with the lowest load as the target cloud node, and distribute the computing task to the task queue corresponding to the target cloud node. The node load is defined as the sum of the number of tasks currently running on the node, the number of scheduled tasks that are about to start, and the number of tasks waiting to be executed.

[0011] Preferably, the local task agent dynamically controls the number of task instances running at the same time according to a preset concurrency control strategy, and automatically retryes tasks that fail to execute according to a preset retry strategy.

[0012] Preferably, the task wrapper follows a unified calling interface specification, which is defined as: receiving a list of parameters and a single parameter, and returning a result in string form.

[0013] Preferably, the system maintains the status information of the task, and the task status includes one or more of the following: pending assignment, received, started, canceled, executed successfully, executed unsuccessfully, and retrying.

[0014] Secondly, this invention provides a data-centric collaborative scheduling method for computing resources in astronomical research clouds, comprising: Global task scheduling steps: The global task scheduler deployed on the master node of the astronomical research cloud receives the computing tasks submitted by users and schedules the tasks to the target cloud node based on the dataset distribution information recorded in the metadata database and the real-time load status of each cloud node. Local task agent scheduling steps: The local task agent deployed on the target cloud node receives tasks from the global task scheduler and calls the task wrapper to start or manage the task execution process according to the preset concurrency control strategy. Task packaging steps: Through the task packager, the corresponding task executor is started on the remote computing node, the execution parameters are passed, and the task execution status and results are fed back. Local task execution steps: The task execution steps are as follows: The computation task is specifically executed through the task executor; Metadata management steps: The metadata of all datasets in the astronomical research cloud is stored and maintained through the metadata database; Result storage steps: The result database stores the result data generated after the task is executed, or the storage path information of the result files, for users to query and download.

[0015] Preferably, in the global task scheduling step, if there are more than one candidate node storing the required dataset, the node with the lowest real-time load is further selected as the target cloud node.

[0016] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0017] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0018] The beneficial effects of this invention include: Data-centric, efficient scheduling: By understanding the global data distribution through a metadata database and combining it with node load conditions, Intelligent scheduling of computing tasks to the node where the data is located or to a node with a lighter load significantly improves resource utilization efficiency and task execution efficiency.

[0019] Powerful application integration capabilities: By designing a task wrapper with a unified interface, the scheduling system is integrated with specific computing tasks. The decoupling of application implementation details enables scientific research applications developed using different programming languages ​​and operating environments to be quickly integrated into the astronomical research cloud, forming a computing service co-construction model, which greatly improves the system's scalability and inclusiveness.

[0020] Hierarchical scheduling adapted to distributed environments: A hierarchical architecture of "global scheduling + local scheduling" is adopted. The global scheduler is responsible for macro-level task distribution, while the local scheduler is responsible for micro-level task execution and resource control. This architecture not only achieves a unified view and optimization of global resources, but also respects the autonomy of each node, perfectly adapting to the geographically dispersed and heterogeneous characteristics of astronomical research clouds.

[0021] This invention employs a hierarchical framework of "global scheduling + local scheduling." Global task scheduling is based on data distribution information in the metadata database and the real-time load of each cloud node, intelligently scheduling computing tasks to target nodes based on the principles of data proximity and load balancing. Local task agents initiate and manage the asynchronous execution of specific tasks through task wrappers that follow a unified interface. This invention effectively solves the problems of low global resource utilization and application integration difficulties in geographically dispersed astronomical research clouds, achieving efficient collaborative scheduling centered on data and improving the system's scalability and practicality. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the structure of a data-centric collaborative scheduling system for computing resources in astronomical research cloud environments, as described in this invention. Detailed Implementation

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] like Figure 1 As shown, the system of this invention is a loosely coupled, distributed system that links multiple resources. Its core components include: a global task scheduler, at least one local task agent, at least one task wrapper, at least one task executor, a metadata database, and a result database; The global task scheduler is deployed on the master node of the astronomical research cloud. It is used to receive computing tasks submitted by users and schedule the computing tasks to the target cloud node based on the dataset distribution information recorded in the metadata database and the real-time load status of each cloud node. The local task agent is deployed on each sub-node of the astronomical research cloud. It is used to receive tasks from the global task scheduler and, according to the preset concurrency control strategy, to start or manage the task execution process by calling the task wrapper. The task wrapper, serving as the interface between the local task agent and the task executor, is used to start the corresponding task execution program on the remote computing node, pass execution parameters, and provide feedback on the task execution status and results. The task executor is used to specifically execute the computational task; The metadata database is used to store and maintain the distribution metadata of all datasets in the astronomical research cloud; The results database is used to store the results data generated after the task is executed, or the storage path information of the results files.

[0025] In one embodiment, the process of the global task scheduler performing task scheduling includes: Based on the dataset name specified in the received computing task, query the metadata database to determine the set of candidate cloud nodes that store the dataset; Obtain the real-time load index of each node in the candidate cloud node set, select the candidate node with the lowest load as the target cloud node, and distribute the computing task to the task queue corresponding to the target cloud node. The node load is defined as the sum of the number of tasks currently running on the node, the number of scheduled tasks that are about to start, and the number of tasks waiting to be executed.

[0026] In one embodiment, the global task scheduler (NadcSched) acts as the system's "central hub," running on the master node. It maintains the task queues of each cloud node and makes decisions based on scheduling algorithms. For example, when it receives a task requiring the dataset "DS_A," it queries the metadata database and finds that "DS_A" exists on nodes Node_1 and Node_2. Subsequently, it compares the current load (number of running tasks + number of waiting tasks) of Node_1 and Node_2 and distributes the task to the less loaded node's queue.

[0027] In one embodiment, the local task agent dynamically controls the number of task instances running at the same time according to a preset concurrency control strategy, and automatically retryes tasks that fail to execute according to a preset retry strategy.

[0028] In one embodiment, the local task agent (NadcWorkerd) acts as a "branch" of the system, running on each cloud node. It listens to its own task queue, and when it receives a new task, it checks whether the current number of concurrent tasks has reached the limit (e.g., the concurrency is set to 5). If the limit has not been reached, the task is immediately started through the task wrapper; if the limit has been reached, the task is placed in the waiting queue.

[0029] In one embodiment, the task wrapper follows a unified calling interface specification, which is defined as: receiving a list of parameters and a single parameter, and returning a result in string form.

[0030] In one embodiment, a Python program for "galaxy spectral analysis" can be wrapped by a task wrapper. This wrapper receives parameters (such as spectral data ID and analysis mode), logs into the compute node via SSH, invokes the actual Python analysis script, and returns the path to the script's output file as a string to the local task agent.

[0031] In one embodiment, the unified interface for the task wrapper is designed as follows: def foo_task(self, *args, **kwargs) ->str Here, `self` contains information about the task itself (task UUID, etc.), `args` is a list of task-defined parameters, and `kwargs` is a dictionary of task-defined parameters. `kwargs` must include a key-value pair named `user`, whose content is the user ID who submitted the task. The return value is a string describing the task's execution result. The result string can be an error message, a download address, or a string representation of a complex data structure. The contents of `self` and `user` are automatically set by the scheduling system, and the string return value is automatically stored in the result database by the scheduling system. The return value and other parameters need to be configured according to the specific application.

[0032] In one embodiment, the system maintains the status information of the task, and the task status includes one or more of the following: pending assignment, received, started, canceled, executed successfully, executed failed, and retrying.

[0033] In one embodiment, the metadata database and the results database can be implemented using a relational database (such as MySQL) or a NoSQL database, respectively recording the "dataset-location" mapping and the "task UUID-result" mapping.

[0034] In one embodiment, users submit computation tasks via a web interface or API. In principle, task submission must specify the dataset name, task name, and other necessary parameters. The global task scheduler receives the submitted computation tasks and, based on the dataset distribution information recorded in the metadata database and the real-time load status of each cloud node, schedules the computation tasks to the target cloud node. A local task agent deployed on the target cloud node receives the task from the global task scheduler and, according to a preset concurrency control strategy, calls the task wrapper to start or manage the task execution process. The task wrapper starts the corresponding task executor on the remote computing node, passes execution parameters, and provides feedback on the task execution status and results. The task executor then executes the computation task. The metadata database stores and maintains the distribution metadata of all datasets in the astronomical research cloud. The results database stores the result data generated after task execution, or the storage path information of result files, for users to query and download.

[0035] In one embodiment, after receiving a computing task submitted by a user, the global task scheduler iterates through the cloud node status and the list of tasks registered within each cloud node in real time, and matches them with the current computing task to obtain a set of candidate cloud nodes that can support the computing task. If there is only one candidate cloud node, that candidate cloud node is the target cloud node, and the computing task is directly sent to that candidate cloud node; if there are more than one candidate cloud node, the current load of the candidate cloud nodes is calculated and compared, and the cloud node with the lowest load is the target cloud node. If there are no candidate cloud nodes (see label), the task is not assigned to any candidate cloud node. ), returns an error message.

[0036] In one embodiment, after the target cloud node receives a computing task, the local task agent determines, based on a preset concurrency control strategy, whether the running conditions are met. If so, it starts or manages the task execution process by calling the task wrapper. If the running conditions are not met temporarily, the task is retained until the conditions are met. The local task agent controls the number of concurrently running task processes based on the concurrency level. Considering that tasks are executed asynchronously and may encounter execution anomalies due to uncertainties, the system provides a retry mechanism. When a task fails, if the maximum number of retries has not been reached, the system will automatically restart the task until it completes normally, or record the failure status after reaching the maximum number of retries.

[0037] In one embodiment, the task executor specifically performs computational tasks.

[0038] In one embodiment, the metadata database stores and maintains the distribution metadata of all datasets in the astronomical research cloud.

[0039] In one embodiment, the results database stores the result data generated after the task is executed, or the storage path information of the result files, for users to query and download.

[0040] This invention employs a hierarchical framework of "global scheduling + local scheduling." Global task scheduling is based on data distribution information in the metadata database and the real-time load of each cloud node, intelligently scheduling computing tasks to target nodes based on the principles of data proximity and load balancing. Local task agents initiate and manage the asynchronous execution of specific tasks through task wrappers that follow a unified interface. This invention effectively solves the problems of low global resource utilization and application integration difficulties in geographically dispersed astronomical research clouds, achieving efficient collaborative scheduling centered on data and improving the system's scalability and practicality.

[0041] Through the above system and method, this invention effectively solves the problem of data and computing resource coordination in astronomical research cloud, and provides an efficient, flexible and scalable scheduling solution for large-scale, cross-institutional scientific computing.

Claims

1. A data-centric collaborative scheduling system for computing resources in astronomical research clouds, characterized in that: include A global task scheduler, at least one local task agent, at least one task wrapper, at least one task executor, a metadata database, and a results database; The global task scheduler is deployed on the master node of the astronomical research cloud. It is used to receive computing tasks submitted by users and schedule the computing tasks to the target cloud node based on the dataset distribution information recorded in the metadata database and the real-time load status of each cloud node. The local task agent is deployed on each node of the astronomical research cloud. It is used to receive tasks from the global task scheduler and start or manage the task execution process by calling the task wrapper according to the preset concurrency control strategy. The task wrapper, serving as the interface between the local task agent and the task executor, is used to start the corresponding task execution program on the remote computing node, pass execution parameters, and provide feedback on the task execution status and results. The task executor is used to specifically execute the computational task; The metadata database is used to store and maintain the distribution metadata of all datasets in the astronomical research cloud; The results database is used to store the results data generated after the task is executed, or the storage path information of the results files; The cloud node includes a master node and sub-nodes, and computing nodes are deployed within the cloud node.

2. The system according to claim 1, characterized in that, The process of task scheduling performed by the global task scheduler includes: Based on the dataset name specified in the received computing task, query the metadata database to determine the set of candidate cloud nodes that store the dataset; Obtain the real-time load index of each node in the candidate cloud node set, select the candidate node with the lowest load as the target cloud node, and distribute the computing task to the task queue corresponding to the target cloud node. The node load is defined as the sum of the number of tasks currently running on the node, the number of scheduled tasks that are about to start, and the number of tasks waiting to be executed.

3. The system according to claim 1 or 2, characterized in that, The local task agent dynamically controls the number of task instances running at the same time according to a preset concurrency control strategy, and automatically retryes tasks that fail to execute according to a preset retry strategy.

4. The system according to claim 1, characterized in that, The task wrapper follows a unified calling interface specification, which is defined as receiving a list of parameters and a single parameter, and returning a string result.

5. The system according to claim 1, characterized in that, The system maintains the status information of the task, and the task status includes one or more of the following: pending assignment, received, started, canceled, executed successfully, executed failed, and retrying.

6. A data-centric collaborative scheduling method for computing resources in an astronomical research cloud, applicable to the system described in any one of claims 1 to 5, characterized in that... include: Global task scheduling steps: The global task scheduler deployed on the master node of the astronomical research cloud receives the computing tasks submitted by users and schedules the tasks to the target cloud node based on the dataset distribution information recorded in the metadata database and the real-time load status of each cloud node. Local task agent scheduling steps: The local task agent deployed on the target cloud node receives tasks from the global task scheduler and calls the task wrapper to start or manage the task execution process according to the preset concurrency control strategy. Task packaging steps: Through the task packager, the corresponding task executor is started on the remote computing node, the execution parameters are passed, and the task execution status and results are fed back. Local task execution steps: The computational task is specifically executed through the task executor; Metadata management steps: The metadata of all datasets in the astronomical research cloud is stored and maintained through the metadata database; Result storage steps: The result database stores the result data generated after the task is executed, or the storage path information of the result files, for users to query and download.

7. The method according to claim 6, characterized in that, The process of task scheduling performed by the global task scheduler includes: Based on the dataset name specified in the received computing task, query the metadata database to determine the set of candidate cloud nodes that store the dataset; Obtain the real-time load index of each node in the candidate cloud node set, select the candidate node with the lowest load as the target cloud node, and distribute the computing task to the task queue corresponding to the target cloud node. The node load is defined as the sum of the number of tasks currently running on the node, the number of scheduled tasks that are about to start, and the number of tasks waiting to be executed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in claim 6 or 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in claim 6 or 7.

Images