Offline task execution method and device based on virtual node, and electronic device

Abstract

The present invention discloses a method and device for executing offline tasks based on virtual nodes, and an electronic device, relating to the field of distributed computing. The method comprises: receiving an offline task request, responding to the offline task request, selecting a target virtual node according to a preset scheduling algorithm, and having the target virtual node forward the offline task request to a physical container cluster corresponding to the target virtual node; updating resource status information transmitted by an offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; and receiving a task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster. The present invention solves the technical problem in related technologies of being unable to provide real-time resource status updates, resulting in delayed scheduling decisions and inefficient resource allocation.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology or other related fields, and in particular to a method and device for executing offline tasks based on virtual nodes, and an electronic device. Background Art

[0002] With the rapid development of big data and cloud computing technologies, the demand for offline task execution is increasing, especially in multi-cluster deployment environments. Traditional offline task execution methods rely on multi-cluster management platforms to allocate and schedule offline resources. However, this method has significant shortcomings. First, it is difficult for multi-cluster management platforms to fully and comprehensively grasp the offline resource status of all clusters in real time, resulting in limited timeliness and accuracy of resource allocation. Second, due to the need to maintain information on multiple clusters, this method has high maintenance costs and complexity, making it difficult to adapt to rapidly changing resource demands and large-scale cluster management. In other words, the technical solutions in related technologies cannot provide real-time resource status updates, resulting in delayed scheduling decisions and inefficient resource allocation. At the same time, existing resource scheduling methods may lead to inflexible resource allocation due to delayed updates of resource information.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present invention provide a method and device for offline task execution based on virtual nodes, and an electronic device, so as to at least solve the technical problem in related technologies that real-time resource status updates cannot be provided, resulting in delayed scheduling decisions and low resource allocation efficiency.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for executing offline tasks based on virtual nodes is provided, which is applied to a virtual offline resource group, wherein the virtual offline resource group includes multiple virtual nodes, and the virtual offline resource group corresponds to one or more physical container clusters and communicates with the physical container clusters through an offline resource management unit. The method for executing offline tasks based on virtual nodes includes: receiving an offline task request, wherein the offline task request includes at least: identification information of the offline task to be executed and the computing power resources required by the offline task; responding to the offline task request, selecting a target virtual node according to a preset scheduling algorithm, and having the target virtual node select the target virtual node; The offline task request is forwarded to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects nodes based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; receiving the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

[0006] Optionally, the step of selecting the target virtual node according to a preset scheduling algorithm includes: extracting the priority and resource type of the offline task from the identification information of the computing power resources required by the offline task; and based on the priority and resource type of the offline task, using the scheduling algorithm to select the target virtual node from the multiple virtual nodes.

[0007] Optionally, before updating the resource status information transmitted by the offline resource management unit, it also includes: monitoring the API service component of the physical container cluster through the offline resource management unit, collecting the resource status information in the physical container cluster, and updating the resource status information to the virtual offline resource group through an HTTP request.

[0008] Optionally, the step of forwarding the offline task request by the target virtual node to the physical container cluster corresponding to the target virtual node includes: performing a resource pre-allocation check based on the offline task request, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; if the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task, the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node.

[0009] Optionally, before receiving the task execution status transmitted by the offline resource management unit, the offline task execution method based on the virtual node also includes: the offline resource management unit regularly queries the API service component of the physical container cluster to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; the offline resource management unit packages the execution information of the offline task into a task status package, and transmits the task status package to the virtual offline resource group via an HTTP request.

[0010] Optionally, the offline task execution method based on virtual nodes also includes: the offline resource management unit performs regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check node status, network connectivity, and storage availability; when the monitoring results indicate that the physical container cluster has an abnormal condition, cluster health abnormality information is reported to the virtual offline resource group via an HTTP request.

[0011] Optionally, the step of receiving an offline task request includes: receiving an offline task request through a unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

[0012] According to another aspect of an embodiment of the present invention, there is also provided an offline task execution device based on a virtual node, which is applied to a virtual offline resource group, wherein the virtual offline resource group includes multiple virtual nodes, and the virtual offline resource group corresponds to one or more physical container clusters and communicates with the physical container cluster through an offline resource management unit. The offline task execution device based on the virtual node includes: a task request receiving unit for receiving an offline task request, wherein the offline task request includes at least: identification information of the offline task to be executed and the computing power resources required by the offline task; a virtual node selection unit for responding to the offline task request, selecting a target virtual node according to a preset scheduling algorithm, and the target virtual node sends the required computing power resources to the target virtual node. The offline task request is forwarded to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects nodes based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; a resource status updating unit is used to update the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; a task execution status receiving unit is used to receive the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

[0013] Optionally, the virtual node selection unit includes: an information extraction module for extracting the priority and resource type of the offline task from the identification information of the computing power resources required by the offline task; and a node selection module for selecting the target virtual node from the multiple virtual nodes based on the priority and resource type of the offline task using the scheduling algorithm.

[0014] Optionally, the offline task execution device based on the virtual node also includes: a resource status collection unit, which is used to monitor the API service component of the physical container cluster through the offline resource management unit before updating the resource status information transmitted by the offline resource management unit, collect the resource status information in the physical container cluster, and update the resource status information to the virtual offline resource group through an HTTP request.

[0015] Optionally, the virtual node selection unit also includes: a resource pre-allocation check module, which is used to perform a resource pre-allocation check based on the offline task request, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; and a request forwarding module, which is used to forward the offline task request to the physical container cluster corresponding to the target virtual node by the target virtual node when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task.

[0016] Optionally, the offline task execution device based on the virtual node also includes: a component query unit, which is used by the offline resource management unit to periodically query the API service component of the physical container cluster before receiving the task execution status transmitted by the offline resource management unit to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; a packaging unit, which is used by the offline resource management unit to package the execution information of the offline task into a task status package, and transmit the task status package to the virtual offline resource group through an HTTP request.

[0017] Optionally, the offline task execution device based on virtual nodes also includes: a cluster health monitoring unit, which is used by the offline resource management unit to perform regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check node status, network connectivity, and storage availability; a cluster health anomaly reporting unit, which is used to report cluster health anomaly information to the virtual offline resource group through an HTTP request when the monitoring results indicate that the physical container cluster has an abnormal condition.

[0018] Optionally, the task request receiving unit includes: a request receiving module, used to receive offline task requests through the unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned virtual node-based offline task execution methods.

[0020] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned virtual node-based offline task execution methods.

[0021] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements the steps of any one of the above-mentioned virtual node-based offline task execution methods.

[0022] In the present disclosure, after receiving an offline task request, the offline task request is responded to, a target virtual node is selected according to a preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects nodes based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation and updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node, and receiving the task execution status transmitted by the offline resource management unit, wherein the offline task request at least includes: the offline task to be executed and the identification information of the computing power resources required by the offline task, and the task execution status is the execution status of the offline task in the physical container cluster. In this embodiment, the distributed physical container cluster is abstracted into a unified virtual offline resource cluster through virtual node technology, and by introducing an offline resource management unit, real-time monitoring and dynamic update of the resource status of the physical container cluster are realized, thereby improving the timeliness and accuracy of resource scheduling, thereby solving the technical problem in related technologies that cannot provide real-time resource status updates, resulting in delayed scheduling decisions and inefficient resource allocation.

[0023] The present disclosure can use virtual node technology to enable the offline task scheduling platform to uniformly manage, allocate and schedule offline resources of multiple clusters as if operating a single cluster, which not only simplifies the complexity of task scheduling but also improves resource utilization and task execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 is a flowchart of an optional offline task execution method based on virtual nodes according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of a unified offline computing power supply system in a multi-cluster scenario according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of an optional offline task execution device based on virtual nodes according to an embodiment of the present invention;

[0028] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] To facilitate those skilled in the art to understand the present invention, some of the terms provided in the following embodiments of the present invention are explained below:

[0032] Hadoop technology, an open source distributed computing framework, is mainly used to store and process large-scale data sets. It can help enterprises handle massive amounts of data and realize data storage, processing, analysis and other functions.

[0033] Spark jobs, a big data processing framework, are used to process large datasets. In this framework, jobs can be written to perform various data processing tasks, such as data cleaning, transformation, and aggregation.

[0034] Flink jobs, a stream processing framework, can process real-time data streams and batch data. In this stream processing framework, you can use the DataStream API or Table API to write jobs.

[0035] In this embodiment, both Spark and Flink can execute offline tasks. The appropriate framework is selected according to the needs and scenarios, and its powerful functions are used to process large-scale data sets.

[0036] K8s cluster, also known as Kubernetes cluster, is a core component in cloud computing environments, used to automate the deployment, expansion, and management of container applications.

[0037] It should be noted that the offline task execution method based on virtual nodes and the device thereof in the present disclosure can be used in the field of distributed computing technology. When offline computing power supply based on a multi-cluster management platform is implemented based on distributed computing, it can also be used in any field other than the field of distributed computing technology. When offline computing power supply based on a multi-cluster management platform is implemented based on distributed computing, the application field of the offline task execution method based on virtual nodes and the device thereof is not limited in the present disclosure.

[0038] It should be noted that in this disclosure, when collecting and analyzing customer information, the corresponding operation entrance is provided for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0039] The following embodiments of the present invention can be applied to various systems / applications / devices for offline task execution based on virtual nodes. The offline task execution method based on virtual nodes provided by the present invention can be applied to large-scale, multi-cluster cloud computing environments, especially for batch computing tasks under the Hadoop technology system, such as the execution of Spark and Flink jobs. Compared with the offline computing power supply technical solution of multiple cluster management platforms, the present invention can overcome the shortcomings of the current offline computing power supply technical solution based on multiple cluster management platforms, such as poor timeliness and difficulty in maintenance, based on virtual node technology. At the same time, it can overcome the inability of cluster management platforms in related technologies to provide real-time resource status updates, resulting in delayed scheduling decisions and low resource allocation efficiency.

[0040] The present invention will be described in detail below with reference to various embodiments.

[0041] Example 1

[0042] According to an embodiment of the present invention, an embodiment of a method for executing offline tasks based on virtual nodes is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] It should be noted that the offline task execution method based on virtual nodes provided by the embodiment of the present invention is applicable to large-scale, multi-cluster cloud computing environments, for example, unified management of multi-cluster resources. In enterprise-level data centers or public cloud platforms, the resource pool is often composed of multiple physical container clusters (for example, physical K8S clusters), each of which may be distributed in different geographical locations or undertake different business tasks. The embodiment of the present invention uses virtual node technology to abstract these scattered physical K8S clusters into a unified virtual offline resource cluster, so that the offline task scheduling platform can uniformly manage, deploy and schedule the offline resources of multiple clusters like operating a single cluster, which not only simplifies the complexity of task scheduling, but also improves resource utilization and task execution efficiency. Alternatively, it is applicable to real-time resource status updates and dynamic scheduling. In a dynamically changing multi-cluster environment, the demand and supply status of offline resources are constantly changing. In an embodiment of the present invention, the offline resource management unit Agent can monitor the API service components of the physical K8S cluster in real time, collect and update the resource usage of each node in the cluster, and ensure that the virtual offline resource cluster can obtain the latest resource status information. Based on this real-time information, the scheduling algorithm can dynamically select the appropriate virtual node to execute offline tasks, avoiding the problem of resource waste and long task waiting time. Alternatively, it is applicable to offline task lifecycle management. This application not only focuses on the initial scheduling of offline tasks, but also covers the full lifecycle management of task execution. The virtual offline resource cluster and the offline resource management agent can continuously track the execution status of offline tasks, and can perform status updates and monitoring from the issuance of tasks to the completion or failure of tasks. This mechanism ensures the transparency of task execution, helps to quickly locate and solve problems, and also facilitates the secondary scheduling or resource recovery of offline tasks.

[0044] To ensure the smooth execution of offline tasks, this embodiment of the present invention also proposes a health check mechanism for the Kubernetes cluster. The offline resource management agent regularly checks multiple health indicators, including node status, network connectivity, and storage availability. If an anomaly is detected, it is immediately reported to the virtual offline resource cluster, triggering appropriate fault handling processes such as resource reallocation and task migration, thus ensuring the continuity and stability of offline task execution.

[0045] When an offline task is submitted, the virtual offline resource cluster estimates the feasibility of resource allocation based on the offline task's resource requirements and the current resource status of the virtual node. This mechanism ensures rapid offline task startup, avoiding resource contention and task delays. Furthermore, the scheduling algorithm comprehensively considers information such as resource type and priority to optimize resource scheduling, improving overall computing efficiency and resource utilization.

[0046] The technical solutions of the embodiments of the present invention are particularly suitable for offline tasks involving large-scale data sets, such as large-scale data analysis and machine learning model training. Through the unified management and dynamic scheduling of virtual nodes, efficient utilization of computing resources can be achieved, meeting the high demand for computing resources in large-scale data processing. Furthermore, through real-time resource status updates and task status tracking, uncertainties in the data processing process, such as sudden increases in data volume and adjustments to the priority of computing tasks, can be effectively addressed, ensuring the smooth progress of large-scale data processing.

[0047] The present application provides a virtual node-based offline task execution method, which is applied to a virtual offline resource group. The virtual offline resource group includes multiple virtual nodes. The virtual offline resource group corresponds to one or more physical container clusters and communicates with the physical container cluster through an offline resource management unit.

[0048] Under the above operating environment, this application provides Figure 1 The offline task execution method based on virtual nodes is shown. Figure 1 is a flowchart of an optional offline task execution method based on virtual nodes according to an embodiment of the present invention, such as Figure 1 As shown, the method includes the following steps:

[0049] Step S101: receiving an offline task request, wherein the offline task request at least includes: an offline task to be executed and identification information of computing resources required for the offline task.

[0050] It should be noted that the offline task request is submitted by the offline task scheduling platform or the user through the unified API interface of the virtual offline resource group. Optionally, the step of receiving the offline task request includes: receiving the offline task request through the unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, resource status update. The unified API interface mentioned in this embodiment can provide functions such as task submission, status query and resource status update. The task request not only contains the specific information of the offline task to be executed, but also contains the identification information of the computing resources required for the task, such as the type and quantity of CPU (Central Processing Unit), memory, GPU (Graphics Processing Unit) resources, as well as the priority and specific requirements of the task (such as storage requirements, network bandwidth, etc.).

[0051] Offline task requests are received through the virtual offline resource cluster's unified Application Programming Interface (API), an interface between software applications. This interface provides a standardized communication protocol, enabling offline task participants to interact with the virtual offline resource cluster in a unified manner. The interface design takes into account multiple functional requirements, including submitting offline tasks, querying task execution status, and real-time updates of virtual node resource status, thereby improving the flexibility and efficiency of task scheduling.

[0052] Step S102: In response to the offline task request, a target virtual node is selected according to a preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node. The scheduling algorithm selects nodes based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation.

[0053] Upon receiving a task request, the virtual offline resource group selects the most suitable virtual node based on a pre-set scheduling algorithm. This scheduling algorithm not only considers the current resource usage of each virtual node but also intelligently matches resources based on the offline task's resource requirements. Furthermore, the offline resource management unit in the target virtual node is responsible for forwarding the task request, ensuring that the task is accurately submitted to the corresponding physical container cluster.

[0054] The virtual node uses the offline resource management unit Agent to dynamically update the available offline resources of the cluster. The Agent collects available offline resources by communicating with the actual k8s cluster represented by the virtual node through the network.

[0055] Optionally, the step of selecting the target virtual node according to a preset scheduling algorithm includes: extracting the priority and resource type of the offline task from the identification information of the computing power resources required by the offline task; and selecting the target virtual node from multiple virtual nodes using a scheduling algorithm based on the priority and resource type of the offline task.

[0056] It's important to note that the scheduling algorithm uses the priority and resource type from the computing resource identifiers of offline tasks as a basis for selecting target virtual nodes. This ensures that high-priority tasks are assigned to nodes with sufficient resources, while also taking into account different resource requirements to achieve fine-grained resource scheduling.

[0057] Before forwarding the task request to the physical container cluster, the offline resource management unit on the target virtual node will perform a resource pre-allocation check to verify whether there are sufficient resources on the node to execute the offline task in the request. Only after the check is passed will the task request be forwarded to the physical container cluster to avoid cluster performance degradation caused by excessive resource allocation. Optionally, the step of forwarding the offline task request by the target virtual node to the physical container cluster corresponding to the target virtual node includes: performing a resource pre-allocation check based on the offline task request, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task, the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node.

[0058] After the offline task request is assigned to the target virtual node, the offline resource management unit first performs a resource pre-allocation check. The purpose of this check is to confirm whether the target virtual node has sufficient allocatable resources to meet the computing power requirements of the offline task. This check includes, but is not limited to, evaluating resources such as CPU, memory, storage space, and network bandwidth to ensure that the available quantities of these resources comply with the resource identification information specified in the request. By performing a resource pre-allocation check, the technical solution provided in this embodiment can avoid task execution failures or delays caused by insufficient resources, thereby improving resource utilization and task execution efficiency.

[0059] The resource pre-allocation check logic is usually based on the current resource occupancy of the physical container cluster and the real-time status of the allocatable resources on the target virtual node. The offline resource management unit monitors the API service components of the physical container cluster, such as the kube-api server, in real time to obtain the latest resource usage information and compares it with the resource requirements in the offline task request. If the resource check passes, that is, the allocatable resources on the target virtual node meet the computing power requirements of the offline task, then the offline resource management unit will continue with the subsequent task request forwarding steps; conversely, if the resource check fails, the offline resource management unit will prevent the execution of the task and may feedback information to the superior scheduling system or resource pool, requesting reallocation or scheduling of the task to another virtual node with sufficient resources.

[0060] Once the resource pre-allocation check confirms that the resources on the target virtual node meet the requirements, the offline resource management unit will forward the execution task request to the physical container cluster corresponding to the target virtual node. This process involves converting the specific parameters and operating environment configuration information of the offline task into instructions or task requests that the physical container cluster can understand and execute. The offline resource management unit uses HTTP (Hypertext Transfer Protocol request) requests or specific API calls through the communication interface with the physical container cluster (such as kube let or kube-apiserver) to send the offline task request to the physical cluster, thereby starting the execution of the offline task. Through this mechanism, the offline resource management unit acts as a bridge between the virtual node and the physical container cluster, realizing the seamless transmission of task requests between different levels.

[0061] During the process of forwarding task requests, the offline resource management unit will adopt an optimized forwarding mechanism to ensure that the task requests can be quickly and accurately received and executed by the physical container cluster. This may include converting the format of the task request so that it meets the input requirements of the physical container cluster; and adjusting the priority and strategy of task execution to adapt to the resource status of the physical cluster and the task execution environment. In addition, the forwarding mechanism also includes security and compliance checks on task requests to ensure that the execution of offline tasks does not pose a security risk to the physical container cluster or violate operating rules. Through this series of forwarding optimization measures, the technical solution provided in this embodiment can ensure the efficient execution of offline tasks on the physical container cluster, while reducing potential problems and risks in task execution.

[0062] Through the steps of resource pre-allocation checking and forwarding task requests to the physical container cluster, not only the accuracy and timeliness of task execution are improved, but also a flexible and controllable resource allocation and task execution management solution is provided for large-scale offline computing tasks. It is suitable for multi-cluster environments that require real-time resource status updates and dynamic scheduling.

[0063] Once the offline resource management unit completes the collection and preliminary processing of resource status information, the next key step is to update this information to the virtual offline resource group. Optionally, before updating the resource status information transmitted by the offline resource management unit, it also includes: monitoring the API service component of the physical container cluster through the offline resource management unit, collecting resource status information in the physical container cluster, and updating the resource status information to the virtual offline resource group through HTTP requests.

[0064] The offline resource management unit constructs an HTTP request containing resource status information. This request includes node information, resource usage, and possible resource warning information for the physical container cluster. The offline resource management unit sends this resource status information to the central management node of the virtual offline resource cluster in real time via HTTP requests. The central management node is responsible for receiving resource status updates from each virtual node and integrating this information into a global resource status view for subsequent scheduling algorithms. After receiving the HTTP request, the virtual offline resource cluster verifies the information to ensure the accuracy and completeness of the resource status. Once verified, the resource status information is stored and processed for subsequent task scheduling and resource allocation.

[0065] The update process may also include a feedback loop mechanism, that is, after receiving resource status information, the virtual offline resource group can send a confirmation or adjustment request to the corresponding offline resource management unit to optimize the frequency and efficiency of resource acquisition, or trigger the corresponding processing flow when the resource status is abnormal.

[0066] The technical solution provided by this embodiment ensures that the virtual offline resource group maintains real-time visibility into the resource status of the physical container cluster, thereby improving the accuracy and flexibility of task scheduling and reducing the complexity of resource management. Real-time resource status updates are key to efficient offline resource management in a multi-cluster environment. They enable the virtual offline resource group to rapidly respond to resource status changes, optimize resource allocation strategies, and provide a stable and reliable resource environment for offline task execution.

[0067] Step S103: updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node.

[0068] The offline resource management unit is deployed on each virtual node, and one of its core responsibilities is to monitor the resource status of the physical container cluster corresponding to the virtual node in real time. This monitoring is achieved by establishing a continuous communication connection with the API service component of the physical container cluster (such as kube-api server). The monitoring mechanism ensures that the offline resource management unit can promptly capture any changes in resource usage in the physical cluster, including but not limited to key indicators such as CPU usage, memory consumption, disk space usage, and network bandwidth usage. The offline resource management unit not only monitors the resource status, but is also responsible for collecting this status information. The collection process usually includes obtaining the latest resource usage data from the API service component of the physical container cluster, and performing preliminary processing on this data, such as data cleaning and format conversion, to ensure that it can be effectively understood and used by the virtual offline resource group.

[0069] The update process involves constructing an HTTP request containing resource status information and sending it to the virtual offline resource cluster's management server. Upon receiving the request, the management server verifies the integrity and accuracy of the information and then integrates it into the virtual offline resource cluster's global resource status database. Resource status information is updated continuously, ensuring that the virtual offline resource cluster has access to the most up-to-date resource status data at all times. This updated resource status information is used to make resource scheduling decisions for the virtual offline resource cluster. The scheduling algorithm dynamically adjusts task allocation strategies based on real-time resource status information to ensure that offline tasks are assigned to virtual nodes with sufficient resources that meet task requirements, thereby improving resource utilization and task execution efficiency.

[0070] Furthermore, resource status information is used to optimize virtual node resource management and task monitoring. For example, through resource status information, the management unit can monitor task resource consumption and promptly identify resource bottlenecks, thereby taking appropriate measures such as reserving resources and adjusting task priorities to ensure smooth task execution and rational resource allocation.

[0071] Alternatively, the virtual offline resource group provided in this embodiment may also establish an information feedback mechanism. This mechanism allows the group to send confirmation information or adjustment instructions to the offline resource management unit after receiving resource status information. For example, if the virtual offline resource group detects an abnormal resource status for a virtual node, it can instruct the resource management unit of that node to perform a more detailed resource check or adjust its monitoring frequency and collection strategy to optimize the efficiency and accuracy of resource management.

[0072] Optionally, before receiving the task execution status transmitted by the offline resource management unit, the offline task execution method based on the virtual node also includes: the offline resource management unit periodically queries the API service component of the physical container cluster to obtain the execution information of the offline task, wherein the execution information includes at least the task running status, resource consumption, and error information; the offline resource management unit packages the execution information of the offline task into a task status package, and transmits the task status package to the virtual offline resource group via an HTTP request.

[0073] Before resource status information is updated, the offline resource management unit will continuously monitor the API service component of the physical container cluster to ensure real-time monitoring of resource status. After the resource status information is collected and initially processed, the offline resource management unit will update this information to the virtual offline resource group in real time through HTTP requests.

[0074] The offline resource management unit is designed to periodically query the physical container cluster's API service components, such as the kube-apiserver, to obtain offline task execution information. This query process is periodic, and the query frequency can be adjusted based on the task's importance and real-time requirements, ensuring timely access to execution status without adding unnecessary system burden due to overly frequent queries.

[0075] Among them, the execution information of offline tasks mainly includes the task running status, resource consumption and possible error information. The task running status includes but is not limited to whether the task has started, the current progress, whether it has been completed or encountered an error, etc.; resource consumption information includes the usage of CPU, memory, storage and network resources during the task execution; error information covers any abnormalities or errors encountered during the task execution, such as insufficient resources, program errors, etc. The offline resource management unit packages the collected execution information into a task status package. This task status package is a structured data package that contains the execution information mentioned above. The packaging process usually also includes formatting and standardizing the execution information to ensure that the virtual offline resource group can quickly identify and process this information. After packaging is completed, the offline resource management unit transmits the task status package to the virtual offline resource group through an HTTP request. As a mature Web communication protocol, HTTP provides a reliable data transmission mechanism, and also supports asynchronous communication between the offline resource management unit and the virtual offline resource group, ensuring the real-time and accuracy of the task status information.

[0076] After receiving the task status package, the virtual offline resource group processes and integrates the information. Specifically, it verifies the integrity and accuracy of the task status package and then stores it in a central database, which provides a globally accessible view of resource and task status. This execution information is used in multiple ways. First, the virtual offline resource group uses task execution status and resource consumption to perform real-time task scheduling and resource optimization, ensuring that tasks are executed with optimal resource allocation. Second, the virtual offline resource group can automatically or manually trigger troubleshooting and recovery processes based on error information, such as resource reallocation and task restart, thereby improving system stability and reliability. The virtual offline resource group may also send feedback or adjustment instructions to the offline resource management unit based on the task execution status. For example, if an offline task is found to be consuming abnormal resources on a specific physical container cluster, the virtual offline resource group can instruct the corresponding resource management unit to increase the query frequency to obtain task status more frequently and take timely action. Furthermore, the virtual offline resource group can dynamically adjust task priorities or resource allocation strategies based on the real-time status of task execution to optimize overall computing efficiency and resource utilization.

[0077] Through the above implementation steps, the technical solution provided by the embodiment of the present invention realizes real-time monitoring, collection and updating of resource status information, ensures the accuracy and timeliness of resource scheduling, and also provides a stable and efficient resource environment for the execution of offline tasks.

[0078] Step S104: receiving a task execution status transmitted by the offline resource management unit, wherein the task execution status is an execution status of the offline task in the physical container cluster.

[0079] The task execution status refers to the running status of the offline task in the physical container cluster, including but not limited to the task start time, running status (such as running, paused, completed, failed), resource consumption (such as CPU, memory, disk IO, network bandwidth, etc.) and any related error messages or warnings. The offline resource management unit regularly obtains this status information from the API service component (such as kube-api server) of the physical container cluster associated with it to ensure that the collected data can fully reflect the real-time execution status of the task. The offline resource management unit formats the collected execution status information and packages it into a task status package. This task status package not only contains the current running status and resource usage of the task, but may also include other metadata, such as task ID, virtual node identifier, task priority, etc., so that the virtual offline resource group can quickly identify and process this information. After packaging is completed, the offline resource management unit securely and efficiently transmits the task status package to the central management node of the virtual offline resource group through HTTP requests.

[0080] Using HTTP requests to transmit task status packages has several advantages: First, HTTP is a widely supported protocol that ensures compatibility across different systems and platforms. Second, HTTP supports asynchronous communication, which means that offline resource management units can send task status packages without affecting their main functions (such as resource management and task scheduling). Finally, HTTP provides security. By using secure protocols such as HTTPS, task status information can be prevented from being intercepted or tampered with during transmission.

[0081] After receiving the task status packet, the virtual offline resource group verifies and processes the information. First, it verifies the data's integrity and consistency, then integrates it into the global task execution status database to facilitate subsequent resource scheduling and task management decisions. Based on this status information, the virtual offline resource group can perform the following operations: If the task execution status packet indicates that a task's resource consumption on the current physical container cluster is abnormal, or if execution fails, the virtual offline resource group can immediately adjust the task schedule, reassigning the task to a virtual node with more abundant resources or a better fit for the task, ensuring successful task completion. Based on the task execution status, the virtual offline resource group can dynamically adjust resource allocation strategies. For example, after a resource-intensive task completes, it can release the resources it occupies for use by other tasks, or it can pre-reserve sufficient resources on virtual nodes for resource-intensive tasks to improve task execution efficiency. Error messages or warnings in the task status packet can help the virtual offline resource group quickly identify problems in task execution, such as insufficient resources or program errors. Based on this information, the system can automatically execute fault recovery procedures, such as resource reallocation and task restart, to minimize interruptions and delays in task execution.

[0082] The virtual offline resource group can generate real-time task monitoring reports based on the collected task execution status for task schedulers, system administrators or users to view, which improves the transparency of task execution and facilitates the monitoring and analysis of task execution.

[0083] In addition to receiving task execution status, the virtual offline resource group may also need to send feedback information to the offline resource management unit. For example, when the virtual offline resource group makes scheduling adjustments based on task execution status, it sends update instructions to the resource management unit of the relevant virtual node to ensure consistency between resource status and task status. This feedback loop mechanism ensures that the system can quickly respond to changes in task execution, improving overall resource utilization efficiency and task execution success rate.

[0084] Through the above steps, after receiving the offline task request, the offline task request can be responded to, the target virtual node is selected according to the preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects the node based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation and updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node, and receiving the task execution status transmitted by the offline resource management unit, wherein the offline task request includes at least: the offline task to be executed and the identification information of the computing power resources required by the offline task, and the task execution status is the execution status of the offline task in the physical container cluster. In this embodiment, by introducing the offline resource management unit, real-time monitoring and dynamic update of the resource status of the physical container cluster are realized, thereby improving the timeliness and accuracy of resource scheduling, thereby solving the technical problem that the related art cannot provide real-time resource status updates, resulting in delayed scheduling decisions and low resource allocation efficiency.

[0085] In a multi-cluster environment, the technical feature of periodic health status monitoring helps to improve the stability and availability of the entire offline computing system. Optionally, the offline task execution method based on virtual nodes also includes: the offline resource management unit performs periodic health status monitoring on the physical container cluster, wherein the periodic health status monitoring is used to regularly check the node status, network connectivity, and storage availability; when the monitoring results indicate that there is an abnormal condition in the physical container cluster, the cluster health abnormality information is reported to the virtual offline resource group through HTTP request.

[0086] In this embodiment, the offline resource management unit is not only responsible for scheduling offline tasks and monitoring the execution status, but also undertakes the task of regular health status monitoring of the physical container cluster. Through regular health monitoring, the stability and reliability of the system are enhanced, ensuring that offline tasks can be executed in a healthy and stable physical environment.

[0087] The offline resource management unit regularly monitors the health of the physical container cluster to comprehensively assess the cluster's operational status and prevent potential system failures or performance degradation. Specific monitoring scope includes, but is not limited to: a. Node status: Checks whether each node in the physical container cluster is operating normally, including monitoring the node's CPU usage, memory usage, disk space usage, and load. Monitoring node status helps promptly detect node failures, such as hardware failures and operating system issues, to ensure stable cluster operation. b. Network connectivity: Monitors the network communication status between nodes, including network latency, packet loss rate, and connection status. Monitoring network connectivity is crucial for ensuring that offline tasks can successfully transmit data and perform computations within the cluster, especially in multi-node distributed computing scenarios. c. Storage availability: Monitors the physical container cluster's storage system to ensure the reliability of data storage and access. This includes checking storage read and write speeds, remaining space, and data redundancy. Monitoring storage availability helps avoid data loss or task interruptions due to storage failures.

[0088] The offline resource management unit establishes a communication connection with the API service component of the physical container cluster (such as the kube-api server) and sends monitoring requests periodically (e.g., every minute, every five minutes, etc.) to obtain the various indicator data within the aforementioned health monitoring range. This data is collected and analyzed by the offline resource management unit to determine the health status of the cluster. In addition, the offline resource management unit may also deploy additional monitoring tools or agents to monitor the operating status of the physical cluster in more detail, improving the accuracy and comprehensiveness of monitoring.

[0089] When analyzing monitoring data, the offline resource management unit determines whether the physical container cluster is experiencing an anomaly based on preset thresholds or anomaly detection rules. For example, if a node's CPU usage consistently exceeds a threshold, network latency exceeds normal limits, or remaining storage space falls below a safety threshold, the offline resource management unit will determine that the cluster health is abnormal. Once monitoring results indicate an anomaly in the physical container cluster, the offline resource management unit immediately reports the anomaly to the virtual offline resource cluster via HTTP. This report includes not only the type and severity of the anomaly, but may also include a preliminary analysis of the fault and recommended remediation measures. Reporting anomaly information via HTTP not only ensures real-time and secure information transmission but also supports asynchronous communication, allowing the virtual offline resource cluster to implement appropriate recovery and resource reallocation strategies based on the specific circumstances after receiving the anomaly report. Upon receiving the anomaly report, the virtual offline resource cluster verifies and analyzes the information and then takes appropriate remediation measures based on the severity and type of the anomaly. This may include rescheduling affected offline tasks and migrating them to a healthy physical container cluster, or notifying the system administrator to troubleshoot and remediate the problem to restore the cluster to a healthy state.

[0090] In the process of handling abnormal information, the virtual offline resource group may also send feedback information to the offline resource management unit, instructing it to adjust the monitoring strategy or frequency to more accurately monitor the abnormal situation, or notify the offline resource management unit to resume normal monitoring mode after the abnormal situation is resolved to avoid excessive resource consumption.

[0091] The following describes in detail another optional specific implementation.

[0092] This embodiment can be applied to offline batch computing tasks under the Hadoop technology system, specifically including spark tasks, flink tasks, etc. When K8S provides offline resources for these batch computing tasks, due to the large resource demand, multiple K8S clusters are often involved. In this case, the batch computing task scheduling platform needs to connect to multiple K8S clusters, which is more complex and less flexible. This technical solution simulates a single k8s cluster into a host node through virtual nodes, so that the batch computing task scheduling platform only needs to connect to one k8s cluster, simplifying the process.

[0093] Figure 2 is a schematic diagram of a unified offline computing power supply system in a multi-cluster scenario according to an embodiment of the present invention. Figure 2As shown, the system includes: an offline task party, a virtual offline resource cluster (including multiple virtual nodes), and multiple K8S clusters, wherein an offline resource management unit Agent is deployed in each K8S cluster, and each virtual node can correspond to a K8S cluster.

[0094] Among them, the first part: virtual offline resource cluster.

[0095] The virtual offline resource cluster serves as a unified offline computing power supply portal, simplifying the offline task scheduling process. The virtual offline resource cluster has the following key functions:

[0096] (1) Task reception and scheduling: Receive offline tasks submitted by offline task parties, and based on the internal scheduling algorithm, quickly select the virtual node that is most suitable for executing the task to achieve efficient task allocation.

[0097] (2) Task lifecycle management: Comprehensively manage the lifecycle of offline tasks, including task dispatching, status tracking, and final completion or failure status reporting, to ensure transparency and controllability of task execution.

[0098] (3) Virtual node management: Responsible for the life cycle management of virtual nodes, including the creation, deletion, and dynamic adjustment of virtual node specifications to adapt to the ever-changing offline task requirements and physical cluster resource conditions.

[0099] Part 2: Virtual Nodes.

[0100] A virtual node is an abstract representation of offline resources in a K8S cluster. It communicates with the physical K8S cluster via HTTP requests. By mapping a single K8S cluster to a virtual host node, the batch computing task scheduling platform can treat it as a single resource supply node, thereby simplifying the docking process with multiple physical K8S clusters. The roles of virtual nodes include:

[0101] (1) Task request forwarding: As an intermediary, it directs the received offline task request to the physical K8S cluster it represents to ensure that the task can be executed.

[0102] (2) Resource status update: Update the available offline resource status of the K8S cluster it represents in real time, providing accurate resource information for scheduling decisions of the virtual offline resource cluster.

[0103] The virtual node uses the offline resource management unit Agent to dynamically update the available offline resources of the cluster. The Agent collects available offline resources by communicating with the actual k8s cluster represented by the virtual node through the network.

[0104] When forwarding offline task requests to the representative K8S cluster, the virtual node adopts the forwarding logic. The batch task scheduling platform sends the batch task request. Then the virtual offline resource cluster runs the scheduling algorithm, selects a virtual node with sufficient available offline resources, and sends the batch task request. After that, the offline resource management unit Agent communicates with the representative's actual K8s cluster and sends the batch task request.

[0105] Part 3: Offline resource management agent.

[0106] To coordinate and monitor the offline resources and task status of the physical K8S cluster, the offline resource management agent is deployed in the physical K8S cluster associated with each virtual node. This agent has the following key responsibilities:

[0107] (1) Resource monitoring and updating: For example, continuously monitoring the resource usage of the API service components (such as kube-apiserver) of the physical K8S cluster to ensure the real-time and accuracy of resource status information and provide updated resource data for virtual nodes.

[0108] (2) Task status tracking and reporting: Track the execution status of offline tasks on the physical cluster, including task operation status, resource consumption, and error information, and report this status information to the virtual offline resource cluster through HTTP requests to provide a basis for task scheduling and resource optimization. For example, by querying the kube-api server of the representative k8s cluster, query the fields representing the running status of batch tasks, and update the query results to the custom resource-related fields representing the batch tasks in the virtual offline resource cluster.

[0109] (3) Cluster health check and status reporting: Regularly perform health checks to evaluate the overall health of the physical K8S cluster, including node status, network connectivity, storage availability, etc. Once an anomaly is detected, it is immediately reported to the virtual offline resource cluster, triggering the corresponding fault handling process.

[0110] Compared to traditional multi-cluster management platforms, this implementation optimizes offline computing power provisioning based on virtual node technology. By updating resource status and task execution status in real time, it ensures that resource allocation and task scheduling decisions are based on the most accurate information, significantly improving response speed and resource allocation efficiency. Furthermore, by introducing virtual nodes and deploying an offline resource management agent, the complexity of connecting the batch computing task scheduling platform with multiple physical Kubernetes clusters is reduced, simplifying system management and maintenance processes and improving overall system stability.

[0111] Furthermore, by integrating virtual offline resource clusters, virtual nodes, and offline resource management agents, this technical solution provides a unified, efficient, and stable resource management and scheduling platform for batch computing offline tasks, significantly improving the execution efficiency and resource utilization of offline tasks in a multi-cluster environment.

[0112] The following describes it in detail with reference to another embodiment.

[0113] Example 2

[0114] The virtual node-based offline task execution device provided in this embodiment includes multiple implementation units, each implementation unit corresponding to each implementation step in the above-mentioned embodiment 1.

[0115] An embodiment of the present invention provides an offline task execution device based on virtual nodes, which is applied to a virtual offline resource group. The virtual offline resource group includes multiple virtual nodes. The virtual offline resource group corresponds to one or more physical container clusters and communicates with the physical container cluster through an offline resource management unit.

[0116] Figure 3 is a schematic diagram of an optional offline task execution device based on virtual nodes according to an embodiment of the present invention, such as Figure 3 As shown, the offline task execution device based on virtual nodes may include: a task request receiving unit 31, a virtual node selecting unit 32, a resource status updating unit 33, and a task execution status receiving unit 34.

[0117] The task request receiving unit 31 is configured to receive an offline task request, wherein the offline task request includes at least: identification information of an offline task to be executed and computing resources required for the offline task;

[0118] A virtual node selection unit 32 is configured to respond to offline task requests, select a target virtual node based on a preset scheduling algorithm, and forward the offline task request to the physical container cluster corresponding to the target virtual node. The scheduling algorithm selects nodes based on the current resource usage of each virtual node and the resource requirements of the offline task. The offline resource management unit in the target virtual node is responsible for executing the forwarding operation.

[0119] A resource status updating unit 33 is configured to update the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node;

[0120] The task execution status receiving unit 34 is configured to receive the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

[0121] The above-mentioned offline task execution device based on virtual nodes can receive offline task requests through the task request receiving unit 31. The offline task request includes at least: the offline task to be executed and the identification information of the computing power resources required by the offline task. The offline task request is responded to by the virtual node selection unit 32, and the target virtual node is selected according to the preset scheduling algorithm. The target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects the node based on the current resource occupancy of each virtual node and the resource requirements of the offline task. The offline resource management unit in the target virtual node is responsible for performing the forwarding operation, and the resource status information transmitted by the offline resource management unit is updated through the resource status updating unit 33, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node, and the task execution status transmitted by the offline resource management unit is received by the task execution status receiving unit 34, wherein the task execution status is the execution status of the offline task in the physical container cluster. In this embodiment, by introducing an offline resource management unit, real-time monitoring and dynamic updating of the resource status of the physical container cluster are achieved, thereby improving the timeliness and accuracy of resource scheduling, thereby solving the technical problem in related technologies that cannot provide real-time resource status updates, resulting in delayed scheduling decisions and inefficient resource allocation.

[0122] Optionally, the virtual node selection unit includes: an information extraction module for extracting the priority and resource type of the offline task from the identification information of the computing power resources required by the offline task; a node selection module for selecting a target virtual node from multiple virtual nodes based on the priority and resource type of the offline task using a scheduling algorithm.

[0123] Optionally, the offline task execution device based on the virtual node also includes: a resource status collection unit, which is used to monitor the API service component of the physical container cluster through the offline resource management unit before updating the resource status information transmitted by the offline resource management unit, collect the resource status information in the physical container cluster, and update the resource status information to the virtual offline resource group through an HTTP request.

[0124] Optionally, the virtual node selection unit also includes: a resource pre-allocation check module, which is used to perform a resource pre-allocation check based on the offline task request, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; and a request forwarding module, which is used to forward the offline task request to the physical container cluster corresponding to the target virtual node by the target virtual node when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task.

[0125] Optionally, the offline task execution device based on the virtual node also includes: a component query unit, which is used by the offline resource management unit to periodically query the API service component of the physical container cluster before receiving the task execution status transmitted by the offline resource management unit to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; a packaging unit, which is used by the offline resource management unit to package the execution information of the offline task into a task status package, and transmit the task status package to the virtual offline resource group through an HTTP request.

[0126] Optionally, the offline task execution device based on the virtual node also includes: a cluster health monitoring unit, which is used by the offline resource management unit to perform regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check the node status, network connectivity, and storage availability; a cluster health anomaly reporting unit, which is used to report cluster health anomaly information to the virtual offline resource group through an HTTP request when the monitoring results indicate that the physical container cluster has an abnormal condition.

[0127] Optionally, the task request receiving unit includes: a request receiving module, used to receive offline task requests through a unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

[0128] The above-mentioned offline task execution device based on virtual nodes can also include a processor and a memory. The above-mentioned task request receiving unit 31, virtual node selection unit 32, resource status update unit 33, task execution status receiving unit 34, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0129] The processor includes a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and kernel parameters can be adjusted to enable offline task processing based on virtual nodes, providing offline computing power. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory. The memory includes at least one memory chip.

[0130] Example 3

[0131] An embodiment of the present application may provide an electronic device, Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4Only one is shown) processor 402, memory 404, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0132] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the offline task execution method and device based on virtual nodes in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned offline task execution method based on virtual nodes. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0133] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive an offline task request, wherein the offline task request at least includes: the offline task to be executed and the identification information of the computing power resources required by the offline task; respond to the offline task request, select the target virtual node according to the preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects the node based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; update the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; receive the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

[0134] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: extract the priority and resource type of the offline task from the identification information of the computing power resources required by the offline task; based on the priority and resource type of the offline task, use the scheduling algorithm to select the target virtual node from multiple virtual nodes.

[0135] The processor can call the information and applications stored in the memory through the transmission device to perform the following steps: before updating the resource status information transmitted by the offline resource management unit, monitor the API service component of the physical container cluster through the offline resource management unit, collect the resource status information in the physical container cluster, and update the resource status information to the virtual offline resource group through HTTP request.

[0136] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: based on the offline task request, perform a resource pre-allocation check, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task, the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node.

[0137] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: before receiving the task execution status transmitted by the offline resource management unit, the offline resource management unit regularly queries the API service component of the physical container cluster to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; the offline resource management unit packages the execution information of the offline task into a task status package, and transmits the task status package to the virtual offline resource group through an HTTP request.

[0138] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: the offline resource management unit performs regular health monitoring on the physical container cluster, wherein the regular health monitoring is used to regularly check the node status, network connectivity, and storage availability; when the monitoring results indicate that the physical container cluster has an abnormal condition, the cluster health abnormality information is reported to the virtual offline resource group through an HTTP request.

[0139] The processor can call the information and applications stored in the memory through the transmission device to perform the following steps: receive offline task requests through the unified API interface of the virtual offline resource group, where the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

[0140] It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a Mobile Internet Device (MID), or a PAD. Figure 4 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 4 Different configurations shown.

[0141] Those skilled in the art will appreciate that all or part of the steps in the various virtual node-based offline task execution methods of the above-mentioned embodiments can be completed by instructing the hardware related to the terminal device through a program. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0142] Example 4

[0143] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the offline task execution method based on virtual nodes provided in the first embodiment.

[0144] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the virtual node-based offline task execution methods in the above-mentioned embodiment 1.

[0145] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0146] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the offline task execution method based on virtual nodes described in various embodiments of the present application.

[0147] The present application also provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the offline task execution method based on virtual nodes described in each embodiment of the present application are implemented.

[0148] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0149] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0151] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0152] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0154] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for executing offline tasks based on virtual nodes, characterized in that: Applied to a virtual offline resource group, the virtual offline resource group includes multiple virtual nodes, the virtual offline resource group corresponds to one or more physical container clusters, and communicates with the physical container clusters through an offline resource management unit. The offline task execution method based on the virtual node includes: Receive an offline task request, wherein the offline task request includes at least: identification information of an offline task to be executed and computing resources required for the offline task; In response to the offline task request, a target virtual node is selected according to a preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects the node based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; Updating resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; Receive a task execution status transmitted by the offline resource management unit, wherein the task execution status is an execution status of the offline task in the physical container cluster.

2. The offline task execution method according to claim 1, characterized in that: The steps of selecting a target virtual node according to a preset scheduling algorithm include: Extracting the priority and resource type of the offline task from the identification information of the computing resources required by the offline task; Based on the priority and resource type of the offline task, the target virtual node is selected from the multiple virtual nodes using the scheduling algorithm.

3. The offline task execution method according to claim 1, characterized in that: Before updating the resource status information transmitted by the offline resource management unit, the method further includes: The offline resource management unit monitors the API service component of the physical container cluster, collects resource status information in the physical container cluster, and updates the resource status information to the virtual offline resource group through an HTTP request.

4. The offline task execution method according to claim 1, characterized in that: The step of forwarding the offline task request by the target virtual node to the physical container cluster corresponding to the target virtual node includes: Based on the offline task request, performing a resource pre-allocation check, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; If the check result indicates that the allocatable resources on the target virtual node meet the computing resources required by the offline task, the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node.

5. The offline task execution method according to claim 1, characterized in that: Before receiving the task execution status transmitted by the offline resource management unit, the method further includes: The offline resource management unit periodically queries the API service component of the physical container cluster to obtain execution information of the offline task, wherein the execution information at least includes task running status, resource consumption, and error information; The offline resource management unit packages the execution information of the offline task into a task status package, and transmits the task status package to the virtual offline resource group through an HTTP request.

6. The offline task execution method according to claim 1, characterized in that: Also includes: The offline resource management unit performs regular health monitoring on the physical container cluster, wherein the regular health monitoring is used to regularly check node status, network connectivity, and storage availability; When the monitoring result indicates that the physical container cluster has an abnormal condition, cluster health abnormality information is reported to the virtual offline resource group via an HTTP request.

7. The offline task execution method according to claim 1, characterized in that: The steps of receiving an offline task request include: An offline task request is received through the unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

8. An offline task execution device based on virtual nodes, characterized in that: Applied to a virtual offline resource group, the virtual offline resource group includes multiple virtual nodes, the virtual offline resource group corresponds to one or more physical container clusters, and communicates with the physical container clusters through an offline resource management unit. The offline task execution device based on the virtual node includes: A task request receiving unit, configured to receive an offline task request, wherein the offline task request includes at least: identification information of an offline task to be executed and computing resources required for the offline task; a virtual node selection unit, configured to respond to the offline task request, select a target virtual node according to a preset scheduling algorithm, and forward the offline task request to the physical container cluster corresponding to the target virtual node by the target virtual node, wherein the scheduling algorithm selects nodes based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; a resource status updating unit, configured to update the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; The task execution status receiving unit is configured to receive the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

9. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the virtual node-based offline task execution method described in any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the offline task execution method based on virtual nodes described in any one of claims 1 to 7 are implemented.

Images