Task scheduling method, system and related device

Abstract

The application discloses a task scheduling method and system and related equipment, and relates to the technical field of computing node clusters. A management node respectively establishes a mapping relationship between each node and a first number of guide information at the beginning, which is equivalent to allocating the first number of to-be-processed tasks to each working node. After receiving a task allocation request sent by the working node, a second number is determined according to a residual proportion between an unexecuted number and a total number of tasks, and a mapping relationship between the working node and the second number of guide information is established, which is equivalent to allocating the second number of to-be-processed tasks to the working node sending the task allocation request. The second number is less than or equal to the first number. In this way, the tasks are dynamically allocated according to the residual proportion between the unexecuted number of to-be-processed tasks and the total number of tasks, the computing efficiency is improved, and the working node obtains the to-be-processed tasks from a parallel file management system, so that the network load can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing node cluster, and particularly relates to a task scheduling method and system and related equipment. BACKGROUND

[0002] With the rapid growth of modern computing demands, large-scale computing node clusters have become the key infrastructure for processing massive amounts of data. The core task of these clusters is to efficiently manage and schedule thousands of computing tasks to ensure high efficiency and fast response in data processing. In particular, in the fields of cloud computing, big data analysis and artificial intelligence, high standards are placed on the scheduling system of computing node clusters. These application scenarios not only require the ability to process huge amounts of data, but also require computing tasks to be executed quickly and accurately.

[0003] In the related art, in the traditional method of processing large-scale data, a single large server (commonly known as a "fat node") with high-performance hardware is often used to complete all tasks. This method relies on the powerful computing power and large capacity of memory of a single node to try to complete as many computing tasks as possible within one node. However, this customized, single-task-oriented machine lacks flexibility and universality, making it difficult to adapt to changing computing demands. Compared with traditional supercomputer clusters, they usually use a static task scheduling strategy, that is, computing resources are allocated in advance and computing tasks are customized. This method requires high predictability in resource allocation, and if the preliminary planning is not proper, it may lead to uneven distribution of resources, thereby affecting the overall computing efficiency. In contrast, the scheduling management method of the popular Kubernetes (K8S) based cloud service cluster in the market builds a K8S based cluster, connects a large number of cloud servers, and uses Docker container technology to implement algorithm deployment and distribution. This mode uses the service mechanism of K8S to achieve load balancing and schedules according to the load at the start of the task. For large-scale computing tasks, this scheduling method has limitations. Since these computing tasks usually run for a long time and have uneven demands for CPU and memory during execution, simply scheduling according to the load at the start of the task may lead to uneven distribution of computing tasks in the cluster, thereby reducing computing efficiency. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application proposes a task scheduling method, system and related equipment, which can dynamically allocate tasks according to the remaining proportion between the number of unexecuted tasks and the total number of tasks, improve computing efficiency, and reduce network load by having the worker nodes obtain the tasks to be processed from the parallel file management system.

[0005] To achieve the above objectives, a first aspect of this application provides a task scheduling method applied to a management node in a computing node cluster, the method comprising:

[0006] Obtain the tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file for each task to be processed, and store the configuration file in the parallel file management system.

[0007] A configuration management file is generated based on each of the configuration files. The configuration management file includes multiple pieces of boot information. Each piece of boot information corresponds one-to-one with a configuration file. The boot information includes the storage address of the configuration file corresponding to the boot information.

[0008] Obtain the node identifier for each worker node;

[0009] A first mapping relationship is established between each node identifier and a first number of the guidance information, so that each working node obtains the corresponding target configuration file from the parallel file management system according to the storage address in the guidance information mapped to its respective node identifier, and executes the task to be processed corresponding to the target configuration file;

[0010] In response to a task assignment request sent by any of the work nodes, the number of unexecuted tasks to be processed is obtained, and the remaining proportion of the unexecuted number relative to the total number of tasks is determined;

[0011] A second number is determined based on the remaining ratio, and a second mapping relationship is established between the node identifier of the worker node that sends the task allocation request and the second number of guidance information, so that the worker node that sends the task allocation request executes the pending task corresponding to the second number of guidance information based on the second mapping relationship; the second number is less than or equal to the first number.

[0012] According to some embodiments of this application, before establishing a first mapping relationship between each node identifier and a first number of the guidance information, the following steps are included:

[0013] Each node file is created based on the node identifier of each working node, and the node files are stored in the parallel file management system; each node file corresponds one-to-one with the node identifier.

[0014] The step of establishing a first mapping relationship between each node identifier and a first number of guidance information includes:

[0015] The first number of guidance information are stored in each of the node files respectively, and the guidance information in each of the node files is different;

[0016] The step of establishing a second mapping relationship between the node identifier of the worker node that sends the task allocation request and the second number of guidance information includes:

[0017] The second number of boot information is stored in the node file corresponding to the working node.

[0018] According to some embodiments of this application, after obtaining the task to be processed, the method further includes:

[0019] A task list is created based on the pending tasks, and the task list includes a first task list; the first task list is used to record the pending tasks that have not been executed.

[0020] The step of obtaining the number of unexecuted tasks and determining the remaining proportion of the unexecuted number relative to the total number of tasks includes:

[0021] The number of pending tasks in the first task list is taken as the number of unexecuted tasks;

[0022] The remaining percentage is calculated based on the number of unexecuted tasks and the total number of tasks.

[0023] According to some embodiments of this application, the guidance information includes status information, which is used to indicate the status of the pending task corresponding to the guidance information; the task list further includes a second task list and a third task list, the second task list is used to record the pending tasks that are being executed; the third task list is used to record the pending tasks that have been completed.

[0024] After creating the task list based on the tasks to be processed, the following is also included:

[0025] In response to the status information of the guidance information in the node file changing to "being executed", the pending task corresponding to the guidance information is moved from the first task list to the second task list;

[0026] In response to the status information of the guidance information in the node file being changed to execution completed, the pending task corresponding to the guidance information is moved from the second task list to the third task list;

[0027] In response to the status information of the guidance information in the node file changing to "not executed", the pending task corresponding to the guidance information is moved from the second task list to the first task list.

[0028] To achieve the above objectives, a second aspect of this application provides a task scheduling method applied to worker nodes in a computing node cluster, the method comprising:

[0029] Based on the first mapping relationship, determine the first number of guiding information for its own node identifier mapping;

[0030] The target configuration file corresponding to the boot information is obtained from the parallel file management system according to the storage address of the boot information;

[0031] Execute the corresponding pending task according to the target configuration file;

[0032] In response to the worker node being in an idle state, a task allocation request is sent to the management node so that the management node obtains the number of unexecuted tasks to be processed, determines the remaining proportion of the unexecuted number relative to the total number of tasks, determines a second number based on the remaining proportion, and establishes a second mapping relationship between the node identifier of the worker node and the guidance information of the second number; the second number is less than or equal to the first number.

[0033] The pending task corresponding to the second number of the guidance information is executed based on the second mapping relationship.

[0034] According to some embodiments of this application, determining the first number of guiding information for its own node identifier mapping includes:

[0035] Based on its own node identifier, the system determines the target node file corresponding to the node identifier from the parallel file management system;

[0036] The boot information is obtained from the target node file.

[0037] According to some embodiments of this application, the parallel file management system stores a task list, which includes a first task list and a second task list; the first task list is used to record unexecuted pending tasks; the second task list is used to record currently executed pending tasks.

[0038] After executing the corresponding pending task according to the target configuration file, the process further includes:

[0039] Determine the execution time of the task to be processed;

[0040] If the execution duration exceeds a preset time threshold, the pending task is terminated, and the status information of the guidance information in the node file corresponding to the pending task is changed to "not executed," so that the management node moves the pending task from the second task list to the first task list, and changes the status information of the guidance information in the configuration management file to "not executed." The status information is used to indicate the status of the pending task corresponding to the guidance information.

[0041] According to some embodiments of this application, the step of executing the corresponding task to be processed according to the target configuration file includes:

[0042] At each first preset time interval, query the task list and the node file corresponding to itself;

[0043] When a query finds that the node file corresponding to itself has an unexecuted pending task, and the pending task is located in the first task list, the status information of the pending task is changed to "being executed", and the pending task is started within the second preset time period; wherein, the second preset time period is shorter than the first preset time period.

[0044] To achieve the above objectives, a third aspect of this application provides a task scheduling apparatus applied to a management node in a computing node cluster, the apparatus comprising:

[0045] The first acquisition module is used to acquire tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file based on each task to be processed, and store the configuration file in the parallel file management system.

[0046] A generation module is used to generate a configuration management file based on each of the configuration files. The configuration management file includes multiple pieces of boot information. Each piece of boot information corresponds one-to-one with a configuration file, and the boot information includes the storage address of the configuration file corresponding to the boot information.

[0047] The second acquisition module is used to acquire the node identifier of each working node;

[0048] The first mapping module is used to establish a first mapping relationship between each node identifier and a first number of the guidance information, so that each working node can obtain the corresponding target configuration file from the parallel file management system according to the storage address in the guidance information mapped to its respective node identifier, and execute the task to be processed corresponding to the target configuration file;

[0049] The proportion calculation module is used to respond to a task allocation request sent by any of the work nodes, obtain the number of unexecuted tasks to be processed, and determine the remaining proportion of the unexecuted number relative to the total number of tasks;

[0050] The second mapping module is used to determine a second number based on the remaining ratio, and to establish a second mapping relationship between the node identifier of the worker node that sends the task allocation request and the second number of guidance information, so that the worker node that sends the task allocation request executes the pending task corresponding to the second number of guidance information based on the second mapping relationship; the second number is less than or equal to the first number.

[0051] To achieve the above objectives, a fourth aspect of this application provides a task scheduling apparatus applied to worker nodes in a computing node cluster, the apparatus comprising:

[0052] The determination module is used to determine the first number of guiding information that are mapped to its own node identifier based on the first mapping relationship;

[0053] The third acquisition module is used to acquire the target configuration file corresponding to the boot information from the parallel file management system according to the storage address of the boot information;

[0054] The first execution module is used to execute the corresponding pending task according to the target configuration file;

[0055] The sending module is configured to send a task allocation request to the management node in response to the worker node being in an idle state, so that the management node obtains the number of unexecuted tasks to be processed, determines the remaining proportion of the unexecuted number relative to the total number of tasks, determines a second number based on the remaining proportion, and establishes a second mapping relationship between the node identifier of the worker node and the guidance information of the second number; the second number is less than or equal to the first number.

[0056] The second execution module is used to execute the task to be processed corresponding to the second number of the guidance information based on the second mapping relationship.

[0057] To achieve the above objectives, a fifth aspect of this application provides a task scheduling system applied to a computing node cluster, the computing node cluster including management nodes and worker nodes;

[0058] The management node is used for:

[0059] Obtain the tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file for each task to be processed, and store the configuration file in the parallel file management system.

[0060] A configuration management file is generated based on each of the configuration files. The configuration management file includes multiple pieces of boot information. Each piece of boot information corresponds one-to-one with a configuration file. The boot information includes the storage address of the configuration file corresponding to the boot information.

[0061] Obtain the node identifier for each worker node;

[0062] A first mapping relationship is established between each node identifier and a first number of the guidance information.

[0063] The working node is used for:

[0064] Based on the first mapping gate, the guidance information mapped to the node identifier of itself is determined;

[0065] The target configuration file corresponding to the boot information is obtained from the parallel file management system according to the storage address of the boot information;

[0066] Execute the corresponding pending task according to the target configuration file;

[0067] In response to the worker node being in an idle state, a task assignment request is sent to the management node;

[0068] The management node is also used for:

[0069] In response to a task assignment request sent by any of the work nodes, the number of unexecuted tasks to be processed is obtained, and the remaining proportion of the unexecuted number relative to the total number of tasks is determined;

[0070] A second number is determined based on the remaining ratio, and a second mapping relationship is established between the node identifier of the working node that sends the task allocation request and the guidance information of the second number; the second number is less than or equal to the first number;

[0071] The working node is also used for:

[0072] The pending task corresponding to the second number of the guidance information is executed based on the second mapping relationship.

[0073] To achieve the above objectives, a sixth aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the task scheduling method described in either the first aspect embodiment or the second aspect embodiment.

[0074] To achieve the above objectives, a seventh aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the task scheduling method described in either the first or second aspect of the embodiment.

[0075] The task scheduling method, system, and related devices of this application are applied to a computing node cluster. Initially, the management node of the computing node cluster establishes a first mapping relationship between each node identifier and a first number of guidance information, equivalent to assigning a first number of pending tasks to each worker node. Upon receiving a task allocation request from a worker node, it determines a second number based on the remaining ratio between the number of unexecuted tasks and the total number of tasks, and establishes a second mapping relationship between the worker node and the second number of guidance information, equivalent to assigning a second number of pending tasks to the worker node that issued the task allocation request. The second number is less than or equal to the first number. This achieves dynamic task allocation based on the remaining ratio between the number of unexecuted tasks and the total number of tasks, improving computational efficiency. Furthermore, each worker node determines the guidance information mapped to its own node identifier, and then retrieves the target configuration file corresponding to the guidance information from the parallel file management system according to the storage address of the guidance information. It then executes the corresponding pending tasks according to the target configuration file. Thus, instead of the management node distributing the configuration files of the pending tasks to the worker nodes, each worker node retrieves the configuration files from the parallel file management system, reducing the network load in the computing node cluster and improving the efficiency and stability of the computing node cluster network.

[0076] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0077] The present application will be further described below with reference to the accompanying drawings and embodiments, wherein:

[0078] Figure 1 This is a partial structural diagram of a computing node cluster according to an embodiment of this application;

[0079] Figure 2 This is a schematic diagram illustrating the execution of a task scheduling method by a computing node cluster according to an embodiment of this application.

[0080] Figure 3 This is a flowchart illustrating a task scheduling method applied to a management node in a computing node cluster according to an embodiment of this application.

[0081] Figure 4 This is a schematic diagram of the configuration file in an embodiment of this application;

[0082] Figure 5 This is a schematic diagram of a configuration management file according to one embodiment of this application;

[0083] Figure 6 for Figure 3 A detailed flowchart of step 350 is shown below;

[0084] Figure 7 This is a schematic diagram of a sub-process of a task scheduling method applied to the management node of a computing node cluster according to an embodiment of this application;

[0085] Figure 8 This is a flowchart illustrating a task scheduling method applied to worker nodes in a computing node cluster according to an embodiment of this application.

[0086] Figure 9 for Figure 8 A detailed flowchart of step S810 is shown below;

[0087] Figure 10 This is a schematic diagram of a sub-process of the task scheduling method applied to the worker nodes of a computing node cluster according to an embodiment of this application;

[0088] Figure 11 for Figure 8 A detailed flowchart of step S830 is shown below;

[0089] Figure 12 This is a schematic diagram of the task scheduling device applied to the management node of a computing node cluster according to an embodiment of this application;

[0090] Figure 13 This is a schematic diagram of the task scheduling device applied to the worker nodes of a computing node cluster according to an embodiment of this application;

[0091] Figure 14 This is a schematic diagram of the hardware structure of an electronic device according to another embodiment of this application. Detailed Implementation

[0092] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0093] In the description of this application, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0094] In the description of this application, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0095] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0096] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0097] First, let's analyze some of the terms used in this application:

[0098] A compute node cluster is a loosely integrated group of computers (hardware or software) that work together in a highly collaborative manner to complete various complex computational tasks. A cluster can be viewed as a unified computer system, where each computer is called a node, and they are interconnected through a specific method (such as a local area network). A compute node cluster includes worker nodes and a management node. In the basic components of a cluster, worker nodes (or compute nodes) are the computing units within the cluster, typically composed of ordinary computers connected and communicating via a network. They use the same operating system and applications, and each compute node has its own CPU, memory, and hard drive resources, enabling them to collaborate and complete computational tasks. The management node (or scheduler) is the control center of the cluster, responsible for managing and coordinating task execution and resource allocation on the compute nodes. The scheduler allocates tasks to idle compute nodes based on task priority and resource availability, and monitors the progress and status of task execution. Therefore, worker nodes and management nodes each play crucial roles in a compute node cluster, working together to ensure the efficient and stable operation of the cluster.

[0099] A Kubernetes cluster (K8s) is an open-source container orchestration system used to automate the deployment, scaling, and management of containerized applications. It provides a rich set of features, including application deployment, application maintenance and scaling, load balancing and network services, and automatic storage mounting. A K8s cluster consists of multiple nodes, which can be physical machines or virtual machines. Each node runs Kubernetes components, such as kubelet (node ​​proxy) and kube-proxy (network proxy). Nodes in the cluster are divided into two categories: management nodes and worker nodes. Management nodes are responsible for managing the entire cluster, including scheduling Pods to run on worker nodes; while worker nodes are responsible for running containerized applications. In a K8s cluster, applications are typically deployed and managed as Pods.

[0100] Pod: A Pod is a core component in Kubernetes and the basic unit of container orchestration. Each Pod can contain one or more containers that share the same network and storage resources, forming a tightly packed unit of work. The design of the Pod makes it the foundation for deploying, managing, and scaling applications in a Kubernetes cluster.

[0101] An image is a lightweight, executable, standalone software package that contains everything needed to run a software or application, such as code, runtime environment, system tools, libraries, and settings. In this way, images easily package applications, encapsulating all of the application's dependencies and configurations, thus simplifying application distribution and deployment. By using images, developers can ensure consistent application performance across any environment. This is because the image contains all the dependencies and environment configurations required for the application to run; regardless of the node or server deployed on, as long as that node supports container technology, the application will run in the same way. Images make application deployment extremely fast and simple. Once the image is built and stored in an image repository, the application can be quickly deployed to any node in the cluster using simple commands or Kubernetes configuration files.

[0102] Python middleware refers to software components that act as a connector between two applications or services. It can be a standalone process or a code library embedded within an application. The main function of middleware is to provide common services to applications, such as message queues, load balancing, and data caching, to simplify application development and deployment.

[0103] The OBS file system refers to the Object Storage Service (OBS) cloud storage service. It provides an optimized, high-performance file system that allows users to mount their created parallel file systems to cloud-based Linux servers. Within this system, users can process files and directories within the parallel file system online, just like operating a local file system, performing operations such as creating, deleting, renaming, and modifying files / directories. The advantage of the OBS file system lies in its parallel processing capabilities; users can process files stored within it anytime, anywhere, and quickly retrieve the processed files. Furthermore, in addition to supporting POSIX file semantics, it also supports the original object interface, meaning users can still use object-oriented processing methods to manipulate files, achieving interoperability between objects and files. The OBS file system also features high reliability, high scalability, and high security, meeting users' ever-growing data storage needs while ensuring data security. It is suitable for scenarios such as large-scale data storage, backup and archiving, multimedia content storage, cloud collaboration, and cloud-native applications.

[0104] In related technologies, load balancing is achieved using Kubernetes (K8S) service mechanisms, scheduling computing tasks based on their initial load. However, this scheduling method has limitations for large-scale computing tasks. Since these tasks typically have long execution times and uneven CPU and memory requirements during execution, simply scheduling based on initial load can lead to uneven distribution of computing tasks across the cluster, thus reducing computational efficiency.

[0105] Based on this, this application proposes a task scheduling method, system, and related equipment, which can dynamically allocate tasks according to the remaining ratio between the number of unexecuted tasks and the total number of tasks, and the worker nodes obtain the tasks to be processed from the parallel file management system, which can reduce network load.

[0106] The task scheduling method of this application embodiment can be applied to management nodes or worker nodes in a computing node cluster. The management node can be a terminal or a server, and the worker node can be a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, etc. The server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers. It can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the task scheduling method, etc., but is not limited to the above forms.

[0107] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0108] In one embodiment, the compute node cluster uses a Kubernetes (K8S) cluster. When executing the method of this embodiment, the compute node cluster needs to be deployed. (Refer to...) Figure 1 and Figure 2 , Figure 1 This is a partial structural diagram of a computing node cluster according to an embodiment of this application. Figure 2 This diagram illustrates the execution of a task scheduling method on a compute node cluster according to an embodiment of this application. First, a certain number of cloud service nodes are acquired, and a container service is installed on each cloud service node. These cloud service nodes serve as nodes in the compute node cluster, and worker nodes and management nodes are configured. Network plugins are configured for both the management node and each worker node to enable communication between them. On the management node, a Kubernetes etcd configuration file is created and the etcd service is started. etcd is a highly available key-value store system primarily used for shared configuration and service discovery. Next, a Kubernetes apiserver configuration file is created and the apiserver service is started. The apiserver is the front end of the Kubernetes cluster, responsible for handling requests from clients. Then, Kubernetes controller manager and scheduler configuration files are created and started. The controller manager manages and maintains various cluster resources, while the scheduler is responsible for scheduling Pods to appropriate nodes. In one embodiment, a deployment is created on the management node to start all pod compute nodes, and worker nodes use the same deployment to start the same image.

[0109] On each worker node, Kubernetes kubelet and kube-proxy configuration files are created, and these two services are started respectively. Kubelet is the most important "node proxy" on the node, managing the lifecycle of Pods and containers, and also responsible for volume (storage volume) and network management. Kube-proxy is responsible for implementing the communication and load balancing mechanisms of Kubernetes Services. In one embodiment, a Python middleware is deployed on the worker nodes to implement container management, process allocation, hot code replacement, process management, task management, log management, and data interaction. Then, the worker nodes are added to the Kubernetes cluster. Images are then configured for the management node and the containers on each worker node; for example, Miniconda 4.1.2 can be used as the base image, containing the necessary scientific computing libraries and tools to ensure consistency and dependency management of the computing environment. Packages such as GDAL spams and scikit-learn are added to the base image for reading, writing, and processing high-resolution image data. Finally, a parallel file management system is configured, using the OBS file system as the parallel file management system, enabling communication between the parallel file management system and the management node and each worker node.

[0110] The first aspect of this application provides a task scheduling method for a management node in a computing node cluster. (Refer to...) Figure 3 , Figure 3 This is a flowchart illustrating a task scheduling method applied to the management node of a computing node cluster, as described in an embodiment of this application. Figure 3 The method shown includes, but is not limited to, steps S310 to S360.

[0111] Step S310: Obtain the tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file based on each task to be processed, and store the configuration file in the parallel file management system.

[0112] It's important to note that "tasks to be processed" refers to computational tasks to be performed, which can involve calculations on remote sensing image data. Specifically, the management node obtains detailed information about the tasks to be processed, such as the computational objective and the storage path of the computational data. Then, it generates a configuration file based on this detailed information using a custom script. The configuration file records the detailed information of the tasks to be processed. After obtaining the configuration file, the worker nodes can execute the corresponding tasks based on its contents.

[0113] In one embodiment, reference is made to Figure 4 , Figure 4 This is a schematic diagram of the configuration file for an embodiment of this application.Figure 4 In this configuration file, the configuration file is a text file. In this configuration file, `logFileName` specifies the location of the log file; the filename is `trainsmp001.log`, and the path is ` / smpblocksAAAA / logs / `. `log_level` specifies the log level; common log levels are DEBUG (0), INFO (1), WARNING (2), ERROR (3), and CRITICAL (4). In this configuration file, 1 represents the INFO level. `sq_name` is the name of the SQLite database; the path is ` / smpblocksAAAA / `, and the filename is `blockinfo_train_2023geo.sqlite`. `sq_blockTable` is the table name in the SQLite database; in this configuration file, the table name is `blockinfo`. `sq_serieIDTable` is the table name in the SQLite database; in this configuration file, the table name is `series_id`. `serieFirst` and `serieLast` are parameters generally used to represent the start (`serieFirst`) and end (`serieLast`) sequence numbers of the processed data sequence. Figure 3 The illustrated configuration file starts from 1 and processes data up to 180. `outputFolder`: The directory for output files, with the path ` / smpblocksAAAA / train3pery`. `beginYearmmdd` and `endYearmmdd`: These two parameters represent the time range of the processed data, from January 1, 2013 to December 31, 2015. `dayFrequency`: Represents the frequency of data processing. The above configuration file illustration is merely an example and should not be construed as limiting this application.

[0114] Step S320: Generate a configuration management file based on each configuration file. The configuration management file includes multiple boot information items. Each boot information item corresponds to a configuration file, and the boot information item includes the storage address of the configuration file corresponding to the boot information item.

[0115] It is worth noting that multiple boot information items are first generated based on the configuration file. The boot information includes storage address and status information. The storage address is used to indicate the storage location of the configuration file corresponding to the boot information in the parallel file system. The status information is used to indicate the status of the task to be processed corresponding to the configuration file. The status can be one of not executed, executing, or completed.

[0116] In one embodiment, reference is made to Figure 5 , Figure 5 This is a schematic diagram of a configuration management file according to one embodiment of this application. Figure 5In this configuration file, each line represents a bootstrap message. "5769" is the sequence number of the task to be processed corresponding to this bootstrap message; "testb / testcc / 2022 / cls5769.yaml,imageclass_top3" is the storage address of the configuration file corresponding to this bootstrap message; and the terminating data "2" indicates status information. For example, if the terminating data is 0, it indicates that the task to be processed is not executed; if the terminating data is 1, it indicates that the task to be processed is being executed; and if the terminating data is 2, it indicates that the task to be processed is completed. Multiple bootstrap messages are then combined to build a configuration management file.

[0117] Step S330: Obtain the node identifier of each worker node. The node identifier is used to uniquely indicate the worker node in the compute node cluster.

[0118] In one embodiment, the hostname of the worker node is used as its node identifier. The worker node sends its own hostname to the management node.

[0119] In one embodiment, the IP address of the worker node is used as the node identifier of the worker node.

[0120] In one embodiment, the UUID of a worker node is used as its node identifier. UUID (Universally Unique Identifier): A UUID is a 128-bit string used to provide a unique ID for information. In a compute node cluster, a UUID can be generated for each worker node to ensure its uniqueness. The advantage of UUIDs is that they can remain unique across different systems and environments and do not require reliance on external factors (such as hostnames or IP addresses) to maintain their uniqueness.

[0121] Step S340: Establish a first mapping relationship between each node identifier and a first number of boot information, so that each working node can obtain the corresponding target configuration file from the parallel file management system according to the storage address in the boot information mapped to its respective node identifier, and execute the task to be processed corresponding to the target configuration file.

[0122] In one embodiment, the worker node determines the boot information corresponding to its own hostname from the parallel file management system, then obtains the target configuration file corresponding to the boot information from the parallel file management system based on the storage address of each boot information, and executes the corresponding pending task according to the target configuration file.

[0123] It should be noted that those skilled in the art can set the first number according to the actual situation.

[0124] Step S350: In response to a task assignment request sent by any worker node, obtain the number of unexecuted tasks to be processed, and determine the remaining proportion of the unexecuted number relative to the total number of tasks.

[0125] It's worth noting that a task assignment request refers to the message sent by a worker node to request a task from the management node. The number of unexecuted tasks refers to the number of pending tasks that have not yet been executed. For example, the number of unexecuted tasks could refer to the number of boot messages with a status of "unexecuted" in the configuration management file.

[0126] In one embodiment, the worker node periodically polls and checks the pending tasks running on its own. If it detects that the status information in the boot information mapped to its own hostname is all completed, it indicates that it has no pending tasks running. Then, it sends a task allocation request to the management node. After receiving the task allocation request, the management node determines the ratio between the number of unexecuted pending tasks and the total number of tasks as the remaining ratio.

[0127] Step S360: Determine the second number based on the remaining ratio, and establish a second mapping relationship between the node identifier of the work node that sends the task allocation request and the guidance information of the second number, so that the work node that sends the task allocation request executes the pending task corresponding to the guidance information of the second number based on the second mapping relationship; the second number is less than or equal to the first number.

[0128] In one embodiment, a ratio threshold is set. If the remaining ratio is greater than the ratio threshold, the second number is made equal to the first number. If the remaining ratio is less than the ratio threshold, the second number is made less than the first number.

[0129] In one embodiment, if the remaining percentage is greater than 0.5, the second number is equal to the first number. If the remaining percentage is less than 0.5 but greater than 0.2, the second number is the product of the first number and 0.5. If the remaining percentage is less than 0.2, the second number is the product of the first number and 0.2. If the product is not an integer, it can be rounded to the nearest integer.

[0130] In one embodiment, the second number is the product of the remaining proportion and the first number, and if the product is not an integer, it can be rounded to the nearest integer.

[0131] In this embodiment, through steps S310 to S360, the management node initially establishes a first mapping relationship between the node identifier of each worker node and a first number of guidance information, which is equivalent to assigning a first number of pending tasks to each worker node. Upon receiving a task allocation request from a worker node, a second number is determined based on the remaining ratio between the number of unexecuted tasks and the total number of tasks. A second mapping relationship is then established between the worker node that issued the task allocation request and the second number of guidance information, which is equivalent to assigning a second number of pending tasks to the worker node that issued the task allocation request. The second number is less than or equal to the first number. This achieves dynamic task allocation based on the remaining ratio between the number of unexecuted tasks and the total number of tasks, improving computational efficiency. Furthermore, each worker node determines the guidance information mapped to its own node identifier, and then retrieves the configuration file corresponding to the guidance information from the parallel file management system according to the storage address of the guidance information. The corresponding pending tasks are then executed according to the configuration file. Thus, there is no need for the management node to send the configuration files of the pending tasks to the worker nodes; instead, each worker node retrieves the configuration files from the parallel file management system, reducing the network load in the computing node cluster and improving the efficiency and stability of the computing node cluster network.

[0132] In one embodiment, before step S340, the management node creates a node file based on the node identifier of each worker node and stores the node files in the parallel file management system; the node files correspond one-to-one with the node identifiers. Specifically, when creating a node file, the name of the node file includes the node identifier of the worker node, thus ensuring a one-to-one correspondence between the node files and node identifiers, i.e., a one-to-one correspondence between the node files and the worker nodes. Then, in step S340, the management node stores a first number of guidance information entries in each of the node files, and the guidance information in each node file is different; thereby establishing a first mapping relationship between the first number of guidance information entries and each node identifier, where each node identifier maps to the first number of guidance information entries, and the guidance information mapped to each node identifier is different. This allows the worker node to determine its corresponding node file from the parallel file management system based on its own node identifier, then obtain guidance information from that file, and then obtain the target configuration file corresponding to the guidance information from the parallel file management system according to the storage address of the guidance information, and execute the corresponding task to be processed according to the target configuration file. Correspondingly, in step S360, the second number of boot information entries are stored in the node file corresponding to the worker node, thereby establishing a second mapping relationship between the worker node and the second number of boot information entries. Thus, compared to the related technologies where the management node directly sends configuration files to the worker nodes, this application facilitates the management node in assigning tasks to the worker nodes and also facilitates the worker nodes in obtaining configuration files from the parallel file management system, thereby reducing the network load on the computing node cluster.

[0133] In one embodiment, after step S310, the management node creates a task list based on the pending tasks. The task list includes a first task list; the first task list records unexecuted pending tasks. The task list is stored in a parallel file management system. (Refer to...) Figure 6 , Figure 6 for Figure 3 A detailed flowchart of step 350. Figure 6 The illustrated method flow includes, but is not limited to, steps S610 to S620.

[0134] Step S610: The number of pending tasks in the first task list is taken as the number of unexecuted tasks;

[0135] Step S620: Calculate the remaining percentage based on the number of unexecuted tasks and the total number of tasks.

[0136] Specifically, after receiving a task assignment request from a worker node, the management node queries the first task list from the parallel file management system. The number of unprocessed tasks in the first task list is taken as the unexecuted quantity, and the ratio of the unexecuted quantity to the total number of tasks is taken as the remaining percentage. Using the first task list, the unexecuted quantity can be quickly determined, thus enabling rapid calculation of the remaining percentage. This method is convenient and efficient, improving the efficiency of the computing node cluster.

[0137] In one embodiment, status information is used to indicate the status of the pending task corresponding to the guidance information; the task list further includes a second task list and a third task list, the second task list being used to record pending tasks currently being executed; the third task list being used to record pending tasks that have been completed. (Refer to...) Figure 7 , Figure 7 This is a schematic diagram of a sub-process of a task scheduling method applied to the management node of a computing node cluster according to an embodiment of this application. Figure 7 The illustrated method flow includes, but is not limited to, steps S710 to S730.

[0138] Step S710: In response to the status information of the boot information in the node file changing to "in execution", the pending task corresponding to the boot information is moved from the first task list to the second task list.

[0139] In one embodiment, after a worker node begins executing a task to be processed, it sends a message to the parallel file management system to change the status information of the corresponding boot information in the node file of the task to be processed to "being executed". This allows the management node to move the task to be processed corresponding to the boot information from the first task list to the second task list.

[0140] Step S720: In response to the status information of the boot information in the node file changing to execution completed, the pending task corresponding to the boot information is moved from the second task list to the third task list.

[0141] In one embodiment, after a worker node completes a task, it sends a message to the parallel file management system to change the status information of the corresponding boot information in the node file of the task to be completed. As a result, the management node moves the task corresponding to the boot information from the second task list to the third task list.

[0142] In step S730, in response to the change of the status information of the boot information in the node file to not executed, the pending task corresponding to the boot information is moved from the second task list to the first task list.

[0143] In one embodiment, when a worker node detects that a pending task has timed out, it stops the calculation of the pending task and changes the status information of the corresponding boot information in the node file corresponding to the pending task to "not executed". As a result, the pending task of the management node is moved from the second task list to the first task list, and the status information of the corresponding boot information in the configuration management file is changed to "not executed".

[0144] This application embodiment manages the task list through steps S710 to S730, so that when the management node receives a task allocation request sent by the worker node, it can determine a second number of tasks to be processed from the first task list and store the corresponding guidance information in the node file corresponding to the worker node.

[0145] It is worth noting that when the status information of the boot information in the node file is changed, the corresponding boot information in the configuration management file also needs to be changed in the same way.

[0146] In one embodiment, the task to be processed is to compute remote sensing image data. Based on year and latitude / longitude, the data is divided into 23 groups from 2000 to 2023, with 70,000 tiles (computing nodes) in each group, equivalent to 70,000 tasks to be processed. The processing time and computational load of each task are uneven. The computing node cluster has 600 machines available for computation. Depending on the cluster type, it can be expanded, with each group of computing machines (worker nodes) consisting of 200 machines. The management node generates 70,000 configuration files and 70,000 boot messages using scripts. These 70,000 boot messages are used to construct a configuration management file. Both the configuration files and the configuration management file are uploaded to a parallel file management system. Then, steps S310 to S360 are executed in the computing node cluster, where the first number is 4; if the remaining ratio is greater than 0.5, the second number is 4; if the remaining ratio is less than 0.5 but greater than 0.2, the second number is 2; if the remaining ratio is less than 0.2, the second number is 1.

[0147] A second aspect of this application provides a task scheduling method for worker nodes in a computing node cluster. (Refer to...) Figure 8 , Figure 8 This is a flowchart illustrating a task scheduling method applied to worker nodes in a computing node cluster, according to an embodiment of this application. Figure 8 The illustrated method flow includes, but is not limited to, steps S810 to S850.

[0148] Step S810: Determine the first number of guiding information for its own node identifier mapping based on the first mapping relationship;

[0149] In one embodiment, the management node establishes a first mapping relationship between the node identifier and a first number of guidance information, so that each node identifier maps to the first number of guidance information, and the guidance information mapped to each node identifier is different. Therefore, the worker node can determine the first number of guidance information mapped to its own node identifier.

[0150] Step S820: Obtain the target configuration file corresponding to the boot information from the parallel file management system according to the storage address of the boot information;

[0151] In one embodiment, the guidance information includes the storage address of the target configuration file of the task to be processed corresponding to the guidance information. Therefore, the worker node can obtain the target configuration file corresponding to the guidance information from the parallel file management system according to the storage address of the guidance information.

[0152] Step S830: Execute the corresponding pending task according to the target configuration file;

[0153] It is worth noting that, referring to Figure 2 A Python middleware is configured in the worker node. First, the main process is started through the Python middleware. The main process executes steps S810 to S820. Then, when the corresponding task to be processed is executed in step S830, the subprocess is started through the Python middleware. The subprocess performs the calculations on the task to be processed. Each task to be processed corresponds to one subprocess.

[0154] Step S840: In response to the worker node being in an idle state, a task allocation request is sent to the management node so that the management node can obtain the number of unexecuted tasks to be processed, determine the remaining proportion of the unexecuted number relative to the total number of tasks, determine a second number based on the remaining proportion, and establish a second mapping relationship between the node identifier of the worker node and the guidance information of the second number; the second number is less than or equal to the first number.

[0155] In one embodiment, after step S830, the worker node periodically polls itself using Python. If it finds no pending tasks being computed, it determines that it is idle and sends a task allocation request to the management node. This allows the management node to determine the remaining ratio between the number of unexecuted pending tasks and the total number of tasks. Based on this remaining ratio, a second number is determined, and the worker node's node identifier is mapped to the guidance information of this second number. If the worker node finds that it has pending tasks being computed, it does not need to send a task allocation request to the management node.

[0156] Step S850: Execute the task to be processed corresponding to the second number of guidance information based on the second mapping relationship.

[0157] In one embodiment, worker nodes periodically poll to check if there are any new guidance messages that map to their own node identifier. If a new guidance message is found, the corresponding pending task is executed. Therefore, after the management node maps worker nodes to a second number of guidance messages, the worker nodes can execute the pending tasks corresponding to that second number of guidance messages.

[0158] In this embodiment, through steps S810 to S850, the management node initially establishes a first mapping relationship between each node identifier and a first number of guidance information, which is equivalent to assigning a first number of pending tasks to each worker node. Upon receiving a task allocation request from a worker node, it determines a second number based on the remaining ratio between the number of unexecuted tasks and the total number of tasks, and establishes a second mapping relationship between the worker node that issued the task allocation request and the second number of guidance information, which is equivalent to assigning a second number of pending tasks to the worker node that issued the task allocation request. The second number is less than or equal to the first number. This achieves dynamic task allocation based on the remaining ratio between the number of unexecuted tasks and the total number of tasks, improving computational efficiency. Furthermore, each worker node determines the guidance information mapped to its own node identifier, and then retrieves the configuration file corresponding to the guidance information from the parallel file management system according to the storage address of the guidance information. It then executes the corresponding pending tasks according to the configuration file. Thus, there is no need for the management node to send the configuration files of the pending tasks to the worker nodes; instead, each worker node retrieves the configuration files from the parallel file management system, reducing the network load in the computing node cluster and improving the efficiency and stability of the computing node cluster network.

[0159] In one embodiment, reference is made to Figure 9 , Figure 9 for Figure 8 A schematic diagram of the specific process of step S810. Figure 9 The illustrated method flow includes, but is not limited to, steps S910 to S920.

[0160] Step S910: Determine the target node file corresponding to the node identifier from the parallel file management system based on its own node identifier;

[0161] Step S920: Obtain boot information from the target node file.

[0162] It is worth noting that because the management node includes the node identifier of the worker node in the node file name when creating the node file, there is a one-to-one correspondence between the node file and the node identifier, i.e., a one-to-one correspondence between the node file and the worker node. Furthermore, the management node stores a first set of boot information into each node file, and the boot information in each node file is different; thus, a mapping relationship is established between each node identifier and the first set of boot information, with each node identifier corresponding to a first set of boot information, and the boot information corresponding to each node identifier being different. Therefore, the worker node can determine the target node file corresponding to its own node identifier from the parallel file management system, and then retrieve the boot information from the target node file.

[0163] In one embodiment, reference is made to Figure 10 , Figure 10 This is a schematic diagram of a sub-process of the task scheduling method applied to the worker nodes of a computing node cluster according to an embodiment of this application. Figure 10 The illustrated process includes, but is not limited to, steps S1010 to S1020.

[0164] Step S1010: Determine the execution time of the task to be processed;

[0165] In one embodiment, the worker node starts a child process through a Python middleware. The child process is used to calculate the task to be processed, and the Python middleware records the cumulative time of the child process in calculating the task to be processed, which is used as the execution time of the task to be processed.

[0166] Step S1020: If the execution time exceeds a preset time threshold, the pending task is terminated, and the status information of the guidance information in the node file corresponding to the pending task is changed to "not executed". This causes the management node to move the pending task from the second task list to the first task list and change the status information of the corresponding guidance information in the configuration management file to "not executed". The status information is used to indicate the status of the pending task corresponding to the guidance information.

[0167] In one embodiment, if the execution time of a pending task exceeds a preset time threshold, indicating that the task has timed out, the Python middleware terminates the subprocess used to execute the task and changes the status information of the bootstrap information in the node file corresponding to the task to "not executed." This causes the management node to move the task from the second task list to the first task list and change the status information of the corresponding bootstrap information in the configuration management file to "not executed." This avoids the situation where a worker node continuously executes only one task due to its timeout, thereby reducing the computation time and idle time of a single worker node, improving computational efficiency, and optimizing the use of computing resources. If the execution time of a pending task is less than or equal to the preset time threshold, there is no need to terminate the subprocess.

[0168] In one embodiment, reference is made to Figure 11 , Figure 11 for Figure 8 A detailed flowchart of step S830 is shown. Figure 11 The illustrated process includes, but is not limited to, steps S1110 and S1120.

[0169] Step S1110: At each first preset time interval, query the task list and the node file corresponding to itself;

[0170] Step S1120: When it is found that there are unexecuted pending tasks in the corresponding node file and the pending tasks are in the first task list, the status information of the pending tasks is changed to "being executed" and the pending tasks are started within the second preset time period; wherein, the second preset time period is shorter than the first preset time period.

[0171] It's important to note that worker nodes periodically query the task list and their corresponding node files. When a node file contains guidance information indicating an "inactive" status, it means there are unprocessed pending tasks. If a node file contains an unprocessed pending task, and that task is in the first task list, its status is changed to "in execution." This involves updating the guidance information for that task to reflect the status. Changes need to be made in both the configuration management file and the node file. Due to potential delays in the interaction during this change, the time from the change to the task's official startup may be less than a second preset time period. If the second preset time period is longer than the first preset time period, a pending task might have its status changed to "in execution" but hasn't officially started yet. If this task is then queried again, the worker node won't start it because its status is still "in execution," preventing the task from ever being started. For example, if the first preset time period is 6.5 minutes, during the first query, if the node file corresponding to itself has an unexecuted pending task and the pending task is in the first task list, then the status information of the pending task will be changed to "being executed" and the start time of the pending task will be 7 minutes later. However, after 6.5 minutes, if a second query is performed, the worker node will be unable to start the pending task because the status information of the pending task has already been changed.

[0172] It is worth noting that in the specific process of step S850, steps S1110 to S1120 need to be executed.

[0173] The embodiments of this application limit the second preset time period to be shorter than the first preset time period, which can prevent the task to be processed from being queried again after the status information has changed but before it has been officially started, thereby preventing the task to be processed from being unable to start.

[0174] In one embodiment, a code storage area is set up in the parallel file management system to store the code files needed by each worker node. When a worker node needs to update or add code files, the management node uploads the updated or added code files to the code storage area, ensuring that the file name includes the worker node's node identifier. The corresponding code file is then retrieved from the code storage area using Python middleware based on its own node identifier. This enables hot code replacement, allowing for batch updates of running code without repackaging the image, reducing network load, and enabling dynamic code replacement testing at a finer granularity.

[0175] Reference Figure 12A third aspect of this application provides a task scheduling apparatus applied to a management node in a computing node cluster. The apparatus includes:

[0176] The first acquisition module 1210 is used to acquire tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file based on each task to be processed, and store the configuration file in the parallel file management system.

[0177] The generation module 1220 is used to generate a configuration management file based on various configuration files. The configuration management file includes multiple boot information items. The boot information corresponds one-to-one with the configuration file and includes the storage address of the configuration file corresponding to the boot information.

[0178] The second acquisition module 1230 is used to acquire the node identifier of each working node;

[0179] The first mapping module 1240 is used to establish a first mapping relationship between each node identifier and a first number of boot information, so that each working node can obtain the corresponding target configuration file from the parallel file management system according to the storage address in the boot information mapped to its respective node identifier, and execute the task to be processed corresponding to the target configuration file.

[0180] The ratio calculation module 1250 is used to respond to a task allocation request sent by any working node, obtain the number of unexecuted tasks to be processed, and determine the remaining ratio of the number of unexecuted tasks relative to the total number of tasks.

[0181] The second mapping module 1260 is used to determine a second number based on the remaining ratio, and to establish a second mapping relationship between the node identifier of the work node that sends the task allocation request and the guidance information of the second number, so that the work node that sends the task allocation request executes the pending task corresponding to the guidance information of the second number based on the second mapping relationship; the second number is less than or equal to the first number.

[0182] This task scheduling device can execute the task scheduling method of the first aspect of this application. The task scheduling device executes the task scheduling method to dynamically allocate tasks based on the remaining ratio between the number of unexecuted tasks and the total number of tasks, thereby improving computational efficiency. Furthermore, each worker node determines the guidance information mapped to its own node identifier based on its own node identifier. Then, according to the storage address of the guidance information, it retrieves the configuration file corresponding to the guidance information from the parallel file management system, and executes the corresponding task according to the configuration file. Thus, there is no need for the management node to send the configuration files of the tasks to the worker nodes; instead, each worker node retrieves the configuration files from the parallel file management system, which reduces the network load in the computing node cluster, thereby improving the efficiency and stability of the computing node cluster network.

[0183] The specific implementation of the task scheduling device is basically the same as that of the task scheduling method in the first aspect embodiment described above, and will not be repeated here. Under the premise of meeting the requirements of the embodiments of this application, the task scheduling device may also be provided with other functional modules to implement the task scheduling method in the first aspect embodiment described above.

[0184] Reference Figure 13 The fourth aspect of this application provides a task scheduling apparatus applied to worker nodes in a computing node cluster. The apparatus includes:

[0185] The determination module 1310 is used to determine the first number of guiding information of its own node identifier mapping based on the first mapping relationship;

[0186] The third acquisition module 1320 is used to obtain the target configuration file corresponding to the boot information from the parallel file management system according to the storage address of the boot information;

[0187] The first execution module 1330 is used to execute the corresponding pending task according to the target configuration file;

[0188] The sending module 1340 is used to send a task allocation request to the management node in response to the worker node being in an idle state, so that the management node can obtain the number of unexecuted tasks to be processed, determine the remaining proportion of the unexecuted number relative to the total number of tasks, determine a second number based on the remaining proportion, and establish a second mapping relationship between the node identifier of the worker node and the guidance information of the second number; the second number is less than or equal to the first number.

[0189] The second execution module 1350 is used to execute the pending task corresponding to the second number of guidance information based on the second mapping relationship.

[0190] This task scheduling device can execute the task scheduling method of the second aspect of this application. The task scheduling device executes the task scheduling method to dynamically allocate tasks based on the remaining ratio between the number of unexecuted tasks and the total number of tasks, thereby improving computational efficiency. Furthermore, each worker node determines the guidance information mapped to its own node identifier based on its own node identifier. Then, according to the storage address of the guidance information, it retrieves the configuration file corresponding to the guidance information from the parallel file management system, and executes the corresponding task according to the configuration file. Thus, instead of the management node distributing the configuration files of the tasks to the worker nodes, each worker node retrieves the configuration files from the parallel file management system, reducing the network load in the computing node cluster and improving the efficiency and stability of the computing node cluster network.

[0191] The specific implementation of the task scheduling device is basically the same as that of the task scheduling method in the second aspect embodiment described above, and will not be repeated here. Under the premise of meeting the requirements of the embodiments of this application, the task scheduling device may also be provided with other functional modules to implement the task scheduling method in the second aspect embodiment described above.

[0192] Reference Figure 1 The fifth aspect of this application provides a task scheduling system applied to a computing node cluster, the computing node cluster including management nodes and worker nodes;

[0193] The management node is used for:

[0194] Obtain the tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file for each task to be processed, and store the configuration file in the parallel file management system.

[0195] A configuration management file is generated based on each configuration file. The configuration management file includes multiple boot information items. Each boot information item corresponds one-to-one with a configuration file. The boot information item includes the storage address of the configuration file corresponding to the boot information item.

[0196] Obtain the node identifier for each worker node;

[0197] Establish a first mapping relationship between each node identifier and a first number of guidance information.

[0198] Work nodes are used for:

[0199] Guidance information for determining the node identifier mapping based on the first mapping relationship;

[0200] The target configuration file corresponding to the boot information is obtained from the parallel file management system based on the storage address of the boot information;

[0201] Execute the corresponding pending tasks according to the target configuration file;

[0202] In response to a worker node being idle, a task assignment request is sent to the management node;

[0203] The management node is also used for:

[0204] In response to a task assignment request sent by any worker node, obtain the number of unexecuted tasks and determine the remaining proportion of the unexecuted number relative to the total number of tasks;

[0205] The second number is determined based on the remaining proportion, and a second mapping relationship is established between the node identifier of the working node that sends the task allocation request and the guidance information of the second number; the second number is less than or equal to the first number;

[0206] Work nodes are also used for:

[0207] The pending tasks corresponding to the second number of guidance information are executed based on the second mapping relationship.

[0208] This task scheduling device can execute the task scheduling method of the first or second aspect of this application. The task scheduling device executes the task scheduling method to dynamically allocate tasks based on the remaining ratio between the number of unexecuted tasks and the total number of tasks, thereby improving computational efficiency. Furthermore, worker nodes determine the guidance information mapped to their own node identifier based on their own node identifier, and then retrieve the configuration file corresponding to the guidance information from the parallel file management system according to the storage address of the guidance information. They then execute the corresponding tasks according to the configuration file. Thus, instead of the management node distributing the configuration files of the tasks to the worker nodes, each worker node retrieves the configuration files from the parallel file management system, reducing the network load in the computing node cluster and improving the efficiency and stability of the computing node cluster network.

[0209] The specific implementation of the task scheduling device is basically the same as the specific implementation of the task scheduling method in the first or second aspect of the above-described embodiments, and will not be repeated here. Under the premise of meeting the requirements of the embodiments of this application, the task scheduling device may also be provided with other functional modules to implement the task scheduling method in the first or second aspect of the above-described embodiments.

[0210] To achieve the above objectives, a sixth aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the task scheduling method of either the first aspect embodiment or the second aspect embodiment. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0211] Reference Figure 14 , Figure 14 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0212] The processor 141 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0213] The memory 142 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 142 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 142 and is invoked by the processor 141 to execute the task scheduling method of the embodiments of this application.

[0214] Input / output interface 143 is used to implement information input and output;

[0215] The communication interface 144 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0216] Bus 145 transmits information between various components of the device (e.g., processor 141, memory 142, input / output interface 143, and communication interface 144);

[0217] The processor 141, memory 142, input / output interface 143 and communication interface 144 are connected to each other within the device via bus 145.

[0218] To achieve the above objectives, a seventh aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the task scheduling method of either the first aspect embodiment or the second aspect embodiment.

[0219] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0220] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0221] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0222] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0224] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0225] It should be understood that in this application, "at least one (item)" means one or more, and "more than one" means two or more. "And / or" is used to describe the mapping relationship between the mapped objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following mapped objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0226] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0227] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0229] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0230] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A task scheduling method, characterized in that, The method, applied to a management node in a compute node cluster, includes: Obtain the tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file for each task to be processed, and store the configuration file in the parallel file management system. A configuration management file is generated based on each of the configuration files. The configuration management file includes multiple pieces of boot information. Each piece of boot information corresponds one-to-one with a configuration file. The boot information includes the storage address of the configuration file corresponding to the boot information. Obtain the node identifier for each worker node; A first mapping relationship is established between each node identifier and a first number of the guidance information, so that each working node obtains the corresponding target configuration file from the parallel file management system according to the storage address in the guidance information mapped to its respective node identifier, and executes the task to be processed corresponding to the target configuration file; In response to a task assignment request sent by any of the work nodes, the number of unexecuted tasks to be processed is obtained, and the remaining proportion of the unexecuted number relative to the total number of tasks is determined; A second number is determined based on the remaining ratio, and a second mapping relationship is established between the node identifier of the worker node that sends the task allocation request and the second number of guidance information, so that the worker node that sends the task allocation request executes the pending task corresponding to the second number of guidance information based on the second mapping relationship; the second number is less than or equal to the first number.

2. The task scheduling method according to claim 1, characterized in that, Before establishing the first mapping relationship between each node identifier and the first number of guidance information, the process includes: Each node file is created based on the node identifier of each working node, and the node files are stored in the parallel file management system; each node file corresponds one-to-one with the node identifier. The step of establishing a first mapping relationship between each node identifier and a first number of guidance information includes: The first number of guidance information are stored in each of the node files respectively, and the guidance information in each of the node files is different; The step of establishing a second mapping relationship between the node identifier of the worker node that sends the task allocation request and the second number of guidance information includes: The second number of guidance messages are stored in the node file corresponding to the worker node that sent the task assignment request.

3. The task scheduling method according to claim 1, characterized in that, After obtaining the task to be processed, the following is also included: A task list is created based on the pending tasks, and the task list includes a first task list; the first task list is used to record the pending tasks that have not been executed. The step of obtaining the number of unexecuted tasks and determining the remaining proportion of the unexecuted number relative to the total number of tasks includes: The number of pending tasks in the first task list is taken as the number of unexecuted tasks; The remaining percentage is calculated based on the number of unexecuted tasks and the total number of tasks.

4. The task scheduling method according to claim 3, characterized in that, The guidance information includes status information, which is used to indicate the status of the task to be processed corresponding to the guidance information; the task list also includes a second task list and a third task list, wherein the second task list is used to record the task to be processed that is being executed. The third task list is used to record the pending tasks that have been completed. After creating the task list based on the tasks to be processed, the following is also included: In response to the status information of the guidance information in the node file changing to "being executed", the pending task corresponding to the guidance information is moved from the first task list to the second task list; In response to the status information of the guidance information in the node file being changed to execution completed, the pending task corresponding to the guidance information is moved from the second task list to the third task list; In response to the status information of the guidance information in the node file changing to "not executed", the pending task corresponding to the guidance information is moved from the second task list to the first task list.

5. A task scheduling method, characterized in that, The method, applied to worker nodes in a compute node cluster, includes: The first number of guiding information points are determined based on the first mapping relationship to determine the node identifier mapping of the node; the guiding information points correspond one-to-one with the configuration file, the configuration file is generated based on the task to be processed, and the configuration file is stored in the parallel file management system; the guiding information points include the storage address of the configuration file corresponding to the guiding information points. The target configuration file corresponding to the boot information is obtained from the parallel file management system according to the storage address of the boot information; Execute the corresponding pending task according to the target configuration file; In response to the worker node being in an idle state, a task allocation request is sent to the management node so that the management node obtains the number of unexecuted tasks to be processed, determines the remaining proportion of the unexecuted number relative to the total number of tasks, determines a second number based on the remaining proportion, and establishes a second mapping relationship between the node identifier of the worker node and the guidance information of the second number; the second number is less than or equal to the first number. The pending task corresponding to the second number of the guidance information is executed based on the second mapping relationship.

6. The task scheduling method according to claim 5, characterized in that, The first number of guiding information for determining its own node identifier mapping includes: Based on its own node identifier, the system determines the target node file corresponding to the node identifier from the parallel file management system; The boot information is obtained from the target node file.

7. The task scheduling method according to claim 6, characterized in that, The parallel file management system stores a task list, which includes a first task list and a second task list; the first task list is used to record the unexecuted tasks to be processed. The second task list is used to record the pending tasks that are currently being executed; After executing the corresponding pending task according to the target configuration file, the process further includes: Determine the execution time of the task to be processed; If the execution duration exceeds a preset time threshold, the pending task is terminated, and the status information of the guidance information in the node file corresponding to the pending task is changed to "not executed," so that the management node moves the pending task from the second task list to the first task list and changes the status information of the guidance information in the configuration management file to "not executed." The status information is used to indicate the status of the pending task corresponding to the guidance information.

8. The task scheduling method according to claim 7, characterized in that, The step of executing the corresponding pending task according to the target configuration file includes: At each first preset time interval, query the task list and the node file corresponding to itself; When a query finds that the node file corresponding to itself has an unexecuted pending task, and the pending task is located in the first task list, the status information of the pending task is changed to "being executed", and the pending task is started within a second preset time period; wherein, the second preset time period is shorter than the first preset time period.

9. A task scheduling device, characterized in that, The device, used as a management node in a computing node cluster, includes: The first acquisition module is used to acquire tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file based on each task to be processed, and store the configuration file in the parallel file management system. A generation module is used to generate a configuration management file based on each of the configuration files. The configuration management file includes multiple pieces of boot information. Each piece of boot information corresponds one-to-one with a configuration file, and the boot information includes the storage address of the configuration file corresponding to the boot information. The second acquisition module is used to acquire the node identifier of each working node; The first mapping module is used to establish a first mapping relationship between each node identifier and a first number of the guidance information, so that each working node can obtain the corresponding target configuration file from the parallel file management system according to the storage address in the guidance information mapped to its respective node identifier, and execute the task to be processed corresponding to the target configuration file; The proportion calculation module is used to respond to a task allocation request sent by any of the work nodes, obtain the number of unexecuted tasks to be processed, and determine the remaining proportion of the unexecuted number relative to the total number of tasks; The second mapping module is used to determine a second number based on the remaining ratio, and to establish a second mapping relationship between the node identifier of the worker node that sends the task allocation request and the second number of guidance information, so that the worker node that sends the task allocation request executes the pending task corresponding to the second number of guidance information based on the second mapping relationship; the second number is less than or equal to the first number.

10. A task scheduling device, characterized in that, The device is used on worker nodes in a computing node cluster and includes: The determination module is used to determine a first number of guiding information mappings of its own node identifier based on a first mapping relationship; the guiding information corresponds one-to-one with the configuration file, the configuration file is generated based on the task to be processed, and the configuration file is stored in the parallel file management system; the guiding information includes the storage address of the configuration file corresponding to the guiding information; The third acquisition module is used to acquire the target configuration file corresponding to the boot information from the parallel file management system according to the storage address of the boot information; The first execution module is used to execute the corresponding pending task according to the target configuration file; The sending module is configured to send a task allocation request to the management node in response to the worker node being in an idle state, so that the management node obtains the number of unexecuted tasks to be processed, determines the remaining proportion of the unexecuted number relative to the total number of tasks, determines a second number based on the remaining proportion, and establishes a second mapping relationship between the node identifier of the worker node and the guidance information of the second number; the second number is less than or equal to the first number. The second execution module is used to execute the task to be processed corresponding to the second number of the guidance information based on the second mapping relationship.

11. A task scheduling system, characterized in that, This is applied to a computing node cluster, which includes management nodes and worker nodes; The management node is used for: Obtain the tasks to be processed, determine the total number of tasks to be processed, generate a corresponding configuration file for each task to be processed, and store the configuration file in the parallel file management system. A configuration management file is generated based on each of the configuration files. The configuration management file includes multiple pieces of boot information. Each piece of boot information corresponds one-to-one with a configuration file. The boot information includes the storage address of the configuration file corresponding to the boot information. Obtain the node identifier for each worker node; A first mapping relationship is established between each node identifier and a first number of the guidance information; The working node is used for: The guidance information is determined based on the first mapping relationship, which is the node identifier mapping of itself. The target configuration file corresponding to the boot information is obtained from the parallel file management system according to the storage address of the boot information; Execute the corresponding pending task according to the target configuration file; In response to the worker node being in an idle state, a task assignment request is sent to the management node; The management node is also used for: In response to a task assignment request sent by any of the work nodes, the number of unexecuted tasks to be processed is obtained, and the remaining proportion of the unexecuted number relative to the total number of tasks is determined; A second number is determined based on the remaining ratio, and a second mapping relationship is established between the node identifier of the working node that sends the task allocation request and the guidance information of the second number; the second number is less than or equal to the first number; The working node is also used for: The pending task corresponding to the second number of the guidance information is executed based on the second mapping relationship.

12. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the task scheduling method according to any one of claims 1 to 8.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the task scheduling method according to any one of claims 1 to 8.

Images