Virtual node-based offline task execution method and apparatus, and electronic device

Abstract

The invention discloses an off-line task execution method and device based on virtual nodes and electronic equipment, and relates to the field of distributed computation.The method comprises the steps that an off-line task request is received, the off-line task request is responded, a target virtual node is selected according to a preset scheduling algorithm, and the target virtual node is executed according to the target virtual node; the target virtual node forwards the offline task request to a physical container cluster corresponding to the target virtual node, resource state information transmitted by an offline resource management unit is updated, and the resource state information is obtained by monitoring the physical container cluster corresponding to the target virtual node by the offline resource management unit; and receiving a task execution state transmitted by the offline resource management unit, wherein the task execution state is an execution state of the offline task in the physical container cluster. According to the method and the device, the technical problems of scheduling decision lag and low resource allocation efficiency caused by incapability of providing real-time resource state update in related technologies are solved.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology or other related fields, and in particular to an offline task execution method and device based on a virtual node, and an electronic device. Background Art

[0002] With the rapid development of big data and cloud computing technologies, the demand for offline task execution is increasing, especially in multi-cluster deployment environments. Traditional offline task execution methods rely on multi-cluster management platforms to allocate and schedule offline resources, but this method has significant disadvantages. First, it is difficult for multi-cluster management platforms to grasp the offline resource status of all clusters in real time and comprehensively, resulting in limited timeliness and accuracy of resource allocation. Secondly, due to the need to maintain information of multiple clusters, this method has high maintenance costs and complexity, and is difficult to adapt to rapidly changing resource requirements and large-scale cluster management. That is, the technical solutions in the relevant technologies cannot provide real-time resource status updates, resulting in delayed scheduling decisions and inefficient resource allocation. At the same time, the existing resource scheduling methods may cause inflexible resource allocation due to delayed updates of resource information.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for executing offline tasks based on virtual nodes, and an electronic device, so as to at least solve the technical problem in the related art that real-time resource status updates cannot be provided, resulting in delayed scheduling decisions and low resource allocation efficiency.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for executing offline tasks based on virtual nodes is provided, which is applied to a virtual offline resource group, wherein the virtual offline resource group includes multiple virtual nodes, and the virtual offline resource group corresponds to one or more physical container clusters and communicates with the physical container clusters through an offline resource management unit. The method for executing offline tasks based on virtual nodes includes: receiving an offline task request, wherein the offline task request at least includes: identification information of the offline task to be executed and the computing power resources required by the offline task; responding to the offline task request, selecting a target virtual node according to a preset scheduling algorithm, and having the target virtual node execute the task; The offline task request is forwarded to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects nodes based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; receiving the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

[0006] Optionally, the step of selecting the target virtual node according to a preset scheduling algorithm includes: extracting the priority and resource type of the offline task from the identification information of the computing power resources required for the offline task; based on the priority and resource type of the offline task, using the scheduling algorithm to select the target virtual node from the multiple virtual nodes.

[0007] Optionally, before updating the resource status information transmitted by the offline resource management unit, it also includes: monitoring the API service component of the physical container cluster through the offline resource management unit, collecting the resource status information in the physical container cluster, and updating the resource status information to the virtual offline resource group through an HTTP request.

[0008] Optionally, the step of forwarding the offline task request by the target virtual node to the physical container cluster corresponding to the target virtual node includes: performing a resource pre-allocation check based on the offline task request, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required for the offline task; when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required for the offline task, the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node.

[0009] Optionally, before receiving the task execution status transmitted by the offline resource management unit, the offline task execution method based on the virtual node also includes: the offline resource management unit periodically queries the API service component of the physical container cluster to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; the offline resource management unit packages the execution information of the offline task into a task status package, and transmits the task status package to the virtual offline resource group via an HTTP request.

[0010] Optionally, the offline task execution method based on virtual nodes also includes: the offline resource management unit performs regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check node status, network connectivity, and storage availability; when the monitoring results indicate that the physical container cluster has an abnormal condition, cluster health abnormality information is reported to the virtual offline resource group via an HTTP request.

[0011] Optionally, the step of receiving an offline task request includes: receiving an offline task request through a unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

[0012] According to another aspect of an embodiment of the present invention, there is also provided an offline task execution device based on a virtual node, which is applied to a virtual offline resource group, wherein the virtual offline resource group includes multiple virtual nodes, and the virtual offline resource group corresponds to one or more physical container clusters and communicates with the physical container clusters through an offline resource management unit. The offline task execution device based on the virtual node includes: a task request receiving unit, which is used to receive an offline task request, wherein the offline task request includes at least: identification information of the offline task to be executed and the computing power resources required for the offline task; a virtual node selection unit, which is used to respond to the offline task request, select a target virtual node according to a preset scheduling algorithm, and the target virtual node sends the required computing power resources to the target virtual node. The offline task request is forwarded to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm performs node selection based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; a resource status updating unit is used to update the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; a task execution status receiving unit is used to receive the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

[0013] Optionally, the virtual node selection unit includes: an information extraction module, used to extract the priority and resource type of the offline task from the identification information of the computing power resources required for the offline task; and a node selection module, used to select the target virtual node from the multiple virtual nodes based on the priority and resource type of the offline task and using the scheduling algorithm.

[0014] Optionally, the offline task execution device based on the virtual node also includes: a resource status collection unit, which is used to listen to the API service component of the physical container cluster through the offline resource management unit before updating the resource status information transmitted by the offline resource management unit, collect the resource status information in the physical container cluster, and update the resource status information to the virtual offline resource group through an HTTP request.

[0015] Optionally, the virtual node selection unit also includes: a resource pre-allocation check module, which is used to perform a resource pre-allocation check based on the offline task request, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; and a request forwarding module, which is used to forward the offline task request to the physical container cluster corresponding to the target virtual node by the target virtual node when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task.

[0016] Optionally, the offline task execution device based on the virtual node also includes: a component query unit, which is used for the offline resource management unit to periodically query the API service component of the physical container cluster before receiving the task execution status transmitted by the offline resource management unit to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; a packaging unit, which is used for the offline resource management unit to package the execution information of the offline task into a task status package, and transmit the task status package to the virtual offline resource group via an HTTP request.

[0017] Optionally, the offline task execution device based on virtual nodes also includes: a cluster health monitoring unit, which is used by the offline resource management unit to perform regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check node status, network connectivity, and storage availability; a cluster health abnormality reporting unit, which is used to report cluster health abnormality information to the virtual offline resource group via an HTTP request when the monitoring results indicate that the physical container cluster is in an abnormal condition.

[0018] Optionally, the task request receiving unit includes: a request receiving module, used to receive offline task requests through a unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned virtual node-based offline task execution methods.

[0020] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned offline task execution methods based on virtual nodes.

[0021] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned offline task execution methods based on virtual nodes are implemented.

[0022] In the present disclosure, after receiving an offline task request, the offline task request is responded to, a target virtual node is selected according to a preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm performs node selection based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation and updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node, and receiving the task execution status transmitted by the offline resource management unit, wherein the offline task request at least includes: the offline task to be executed and the identification information of the computing power resources required by the offline task, and the task execution status is the execution status of the offline task in the physical container cluster. In this embodiment, the decentralized physical container cluster is abstracted into a unified virtual offline resource cluster through virtual node technology, and by introducing an offline resource management unit, real-time monitoring and dynamic update of the resource status of the physical container cluster is achieved, thereby improving the timeliness and accuracy of resource scheduling, thereby solving the technical problem that the related technology cannot provide real-time resource status updates, resulting in delayed scheduling decisions and inefficient resource allocation.

[0023] The present disclosure can use virtual node technology to enable an offline task scheduling platform to uniformly manage, deploy and schedule offline resources of multiple clusters just like operating a single cluster, which not only simplifies the complexity of task scheduling, but also improves resource utilization and task execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 is a flowchart of an optional offline task execution method based on a virtual node according to an embodiment of the present invention;

[0026] Figure 2 is a schematic diagram of a unified offline computing power supply system in a multi-cluster scenario according to an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of an optional offline task execution device based on a virtual node according to an embodiment of the present invention;

[0028] Figure 4 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] To facilitate those skilled in the art to understand the present invention, some terms provided in the following embodiments of the present invention are explained below:

[0032] Hadoop technology, an open source distributed computing framework, is mainly used to store and process large-scale data sets. It can help enterprises process massive data and realize data storage, processing, analysis and other functions.

[0033] Spark jobs, a big data processing framework, are used to process large data sets. In this big data processing framework, jobs can be written to perform various data processing tasks, such as data cleaning, transformation, aggregation, etc.

[0034] Flink jobs, a stream processing framework, can process real-time data streams and batch data. In this stream processing framework, you can use the DataStream API or Table API to write jobs.

[0035] Whether it is Spark or Flink, both can execute offline task jobs in this embodiment, select the appropriate framework according to needs and scenarios, and use its powerful functions to process large-scale data sets.

[0036] K8s cluster, also known as Kubernetes cluster, is a core component in a cloud computing environment, used to automate the deployment, expansion, and management of container applications.

[0037] It should be noted that the offline task execution method based on virtual nodes and the device thereof in the present disclosure can be used in the field of distributed computing technology. When offline computing power supply based on a multi-cluster management platform is implemented based on distributed computing, it can also be used in any field outside the field of distributed computing technology. When offline computing power supply based on a multi-cluster management platform is implemented based on distributed computing, the application field of the offline task execution method based on virtual nodes and the device thereof is not limited in the present disclosure.

[0038] It should be noted that in the present disclosure, when collecting and analyzing customer information, corresponding operation entrances are provided for users to choose to agree or reject the automated decision results; if the user chooses to reject, the expert decision-making process is entered.

[0039] The following embodiments of the present invention can be applied to various systems / applications / devices for offline task execution based on virtual nodes. The offline task execution method based on virtual nodes provided by the present invention can be applicable to large-scale, multi-cluster cloud computing environments, especially batch computing tasks under the Hadoop technology system, such as the execution of Spark and Flink jobs. Compared with the offline computing power supply technical solution of the multi-cluster management platform, the present invention can overcome the shortcomings of the current offline computing power supply technical solution based on the multi-cluster management platform, such as poor timeliness and difficulty in maintenance, based on the virtual node technology, and can also overcome the inability of the cluster management platform in the related technology to provide real-time resource status updates, resulting in delayed scheduling decisions and low resource allocation efficiency.

[0040] The present invention is described in detail below in conjunction with various embodiments.

[0041] Embodiment 1

[0042] According to an embodiment of the present invention, an embodiment of an offline task execution method based on a virtual node is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] It should be noted that the offline task execution method based on virtual nodes provided by the embodiment of the present invention is applicable to large-scale, multi-cluster cloud computing environments, for example, unified management of multi-cluster resources. In enterprise-level data centers or public cloud platforms, resource pools are often composed of multiple physical container clusters (for example, physical K8S clusters), each of which may be distributed in different geographical locations or undertake different business tasks. The embodiment of the present invention abstracts these scattered physical K8S clusters into a unified virtual offline resource cluster through virtual node technology, so that the offline task scheduling platform can uniformly manage, allocate and schedule offline resources of multiple clusters like operating a single cluster, which not only simplifies the complexity of task scheduling, but also improves resource utilization and task execution efficiency. Alternatively, it is suitable for real-time resource status update and dynamic scheduling. In a dynamically changing multi-cluster environment, the demand and supply status of offline resources are constantly changing. In an embodiment of the present invention, the offline resource management unit Agent can monitor the API service component of the physical K8S cluster in real time, collect and update the resource usage of each node in the cluster, and ensure that the virtual offline resource cluster can obtain the latest resource status information. Based on these real-time information, the scheduling algorithm can dynamically select appropriate virtual nodes to execute offline tasks, avoiding the problem of resource waste and long task waiting time. Alternatively, it is applicable to offline task lifecycle management. This application not only focuses on the initial scheduling of offline tasks, but also covers the full lifecycle management of task execution. The virtual offline resource cluster and the offline resource management agent can continuously track the execution status of offline tasks, and can perform status updates and monitoring from task issuance to task completion or failure. This mechanism ensures the transparency of task execution, helps to quickly locate and solve problems, and also facilitates secondary scheduling or resource recovery of offline tasks.

[0044] In order to ensure the smooth execution of offline tasks, the embodiment of the present invention also proposes a health check mechanism for the K8S cluster. The offline resource management agent regularly checks multiple health indicators including node status, network connectivity, and storage availability. Once an abnormality is detected, it can be immediately reported to the virtual offline resource cluster to trigger the corresponding fault handling process, such as resource reallocation, task migration, etc., to ensure the continuity and stability of offline task execution.

[0045] When an offline task is submitted, the virtual offline resource cluster estimates the feasibility of resource allocation based on the resource requirements of the offline task and the current resource status of the virtual node. This mechanism ensures that offline tasks can be started quickly, avoiding resource competition and task delays. At the same time, the scheduling algorithm comprehensively considers information such as resource type and priority to achieve optimal resource scheduling, thereby improving overall computing efficiency and resource utilization.

[0046] The technical solution of the embodiment of the present invention is particularly suitable for offline tasks that process large-scale data sets, such as large-scale data analysis, machine learning model training, etc. Through the unified management and dynamic scheduling of virtual nodes, efficient use of computing resources can be achieved to meet the high demand for computing resources in large-scale data processing. At the same time, through real-time resource status updates and task status tracking, it is possible to effectively deal with uncertainties in the data processing process, such as a sudden increase in data volume, priority adjustment of computing tasks, etc., to ensure the smooth progress of large-scale data processing.

[0047] The present application provides a virtual node-based offline task execution method, which is applied to a virtual offline resource group. The virtual offline resource group includes multiple virtual nodes. The virtual offline resource group corresponds to one or more physical container clusters and communicates with the physical container cluster through an offline resource management unit.

[0048] Under the above operating environment, this application provides Figure 1 The offline task execution method based on virtual nodes is shown. Figure 1 is a flowchart of an optional offline task execution method based on a virtual node according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0049] Step S101: receiving an offline task request, wherein the offline task request at least includes: an offline task to be executed and identification information of computing resources required for the offline task.

[0050] It should be noted that the offline task request is submitted by the offline task scheduling platform or the user through the unified API interface of the virtual offline resource group. Optionally, the step of receiving the offline task request includes: receiving the offline task request through the unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, resource status update. The unified API interface mentioned in this embodiment can provide functions such as task submission, status query and resource status update. The task request not only contains the specific information of the offline task to be executed, but also contains the identification information of the computing power resources required for the task, such as the type and quantity of CPU (Central Processing Unit), memory, GPU (Graphics Processing Unit) resources, as well as the priority and specific requirements of the task (such as storage requirements, network bandwidth, etc.).

[0051] Offline task requests are received through the unified API interface (Application Programming Interface, which is an interactive interface between software applications) of the virtual offline resource group. The interface provides a standardized communication protocol, allowing offline task parties to interact with the virtual offline resource group in a unified manner. The interface design takes into account multiple functional requirements, including submission of offline tasks, query of task execution status, and real-time update of virtual node resource status, thereby improving the flexibility and efficiency of task scheduling.

[0052] Step S102, respond to the offline task request, select the target virtual node according to the preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm performs node selection based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation.

[0053] After receiving the task request, the virtual offline resource group will select the most suitable virtual node according to the preset scheduling algorithm. The scheduling algorithm not only considers the current resource occupancy of each virtual node, but also makes intelligent matches based on the resource requirements of the offline task. In addition, the offline resource management unit in the target virtual node is responsible for executing the forwarding operation of the task request to ensure that the task can be accurately submitted to the corresponding physical container cluster.

[0054] The virtual node uses the offline resource management unit Agent to dynamically update the available offline resources of the cluster. The Agent collects available offline resources by communicating with the actual k8s cluster represented by the virtual node through the network.

[0055] Optionally, the step of selecting a target virtual node according to a preset scheduling algorithm includes: extracting the priority and resource type of the offline task from the identification information of the computing power resources required for the offline task; and selecting the target virtual node from multiple virtual nodes using a scheduling algorithm based on the priority and resource type of the offline task.

[0056] It should be noted that when selecting the target virtual node, the scheduling algorithm will extract the priority and resource type from the computing resource identification information required by the offline task as the basis for selection. This ensures that high-priority tasks can be assigned to nodes with sufficient resources first, while also taking into account different types of resource requirements to achieve fine resource scheduling.

[0057] Before forwarding the task request to the physical container cluster, the offline resource management unit on the target virtual node will perform a resource pre-allocation check to verify whether there are sufficient resources on the node to execute the offline task in the request. Only after the check is passed will the task request be forwarded to the physical container cluster to avoid cluster performance degradation caused by excessive resource allocation. Optionally, the step of forwarding the offline task request by the target virtual node to the physical container cluster corresponding to the target virtual node includes: based on the offline task request, performing a resource pre-allocation check, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task, the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node.

[0058] After the offline task request is assigned to the target virtual node, the offline resource management unit first performs a resource pre-allocation check. The purpose of this check is to confirm whether there are enough allocatable resources on the target virtual node to meet the computing power requirements of the offline task. The check includes but is not limited to the evaluation of resources such as CPU, memory, storage space, and network bandwidth to ensure that the available amount of these resources meets the resource identification information specified in the request. By performing a resource pre-allocation check, the technical solution provided in this embodiment can avoid task execution failure or delay caused by insufficient resources, thereby improving resource utilization and task execution efficiency.

[0059] The resource pre-allocation check logic is usually based on the current resource occupancy of the physical container cluster and the real-time status of the allocatable resources on the target virtual node. The offline resource management unit monitors the API service components of the physical container cluster, such as the kube-api server, in real time to obtain the latest resource usage information and compare it with the resource requirements in the offline task request. If the resource check passes, that is, the allocatable resources on the target virtual node meet the computing power requirements of the offline task, then the offline resource management unit will continue with the subsequent task request forwarding steps; conversely, if the resource check fails, the offline resource management unit will prevent the execution of the task and may feedback information to the superior scheduling system or resource pool, requesting to reallocate or schedule the task to another virtual node with sufficient resources.

[0060] Once the resource pre-allocation check confirms that the resources on the target virtual node meet the requirements, the offline resource management unit will forward the execution task request to the physical container cluster corresponding to the target virtual node. This process involves converting the specific parameters and operating environment configuration information of the offline task into instructions or task requests that the physical container cluster can understand and execute. The offline resource management unit uses HTTP (Hypertext Transfer Protocol request) requests or specific API calls through the communication interface with the physical container cluster (such as kube let or kube-apiserver) to send the offline task request to the physical cluster, thereby starting the execution of the offline task. Through this mechanism, the offline resource management unit acts as a bridge between the virtual node and the physical container cluster, realizing the seamless transmission of task requests between different levels.

[0061] During the process of forwarding task requests, the offline resource management unit will adopt an optimized forwarding mechanism to ensure that the task requests can be quickly and accurately received and executed by the physical container cluster. This may include format conversion of task requests to meet the input requirements of the physical container cluster; and adjusting the priority and strategy of task execution to adapt to the resource status of the physical cluster and the task execution environment. In addition, the forwarding mechanism also includes security and compliance checks on task requests to ensure that the execution of offline tasks does not cause security risks to the physical container cluster or violate operating rules. Through this series of forwarding optimization measures, the technical solution provided in this embodiment can ensure the efficient execution of offline tasks on the physical container cluster, while reducing potential problems and risks in task execution.

[0062] Through the steps of resource pre-allocation check and task request forwarding to the physical container cluster, not only the accuracy and timeliness of task execution are improved, but also a flexible and controllable resource allocation and task execution management solution is provided for large-scale offline computing tasks, which is suitable for multi-cluster environments that require real-time resource status updates and dynamic scheduling.

[0063] Once the offline resource management unit completes the collection and preliminary processing of resource status information, the next key step is to update this information to the virtual offline resource group. Optionally, before updating the resource status information transmitted by the offline resource management unit, it also includes: monitoring the API service component of the physical container cluster through the offline resource management unit, collecting resource status information in the physical container cluster, and updating the resource status information to the virtual offline resource group through HTTP requests.

[0064] The offline resource management unit constructs an HTTP request containing resource status information. The request contains node information of the physical container cluster, resource usage, and possible resource warning information. The offline resource management unit sends the resource status information to the central management node of the virtual offline resource group in real time through HTTP requests. The central management node is responsible for receiving resource status updates from each virtual node and integrating this information into the global resource status view for subsequent scheduling algorithms. After the virtual offline resource group receives the HTTP request, it will verify the information to ensure the accuracy and completeness of the resource status. After verification, the resource status information will be stored and processed for subsequent task scheduling and resource allocation.

[0065] The update process may also include a feedback loop mechanism, that is, after receiving resource status information, the virtual offline resource group can send a confirmation or adjustment request to the corresponding offline resource management unit to optimize the frequency and efficiency of resource acquisition, or trigger the corresponding processing flow when the resource status is abnormal.

[0066] The technical solution provided in this embodiment can ensure that the virtual offline resource group has real-time knowledge of the resource status of the physical container cluster, thereby improving the accuracy and flexibility of task scheduling and reducing the complexity of resource management. Real-time resource status information update is the key to achieving efficient management of offline resources in a multi-cluster environment. It enables the virtual offline resource group to quickly respond to changes in resource status, optimize resource allocation strategies, and provide a stable and reliable resource environment for the execution of offline tasks.

[0067] Step S103: updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node.

[0068] The offline resource management unit is deployed on each virtual node, and one of its core responsibilities is to monitor the resource status of the physical container cluster corresponding to the virtual node in real time. This monitoring is achieved by establishing a continuous communication connection with the API service component of the physical container cluster (such as kube-api server). The monitoring mechanism ensures that the offline resource management unit can capture any changes in resource usage in the physical cluster in a timely manner, including but not limited to key indicators such as CPU usage, memory consumption, disk space usage, and network bandwidth usage. The offline resource management unit not only monitors the resource status, but is also responsible for collecting this status information. The collection process usually includes obtaining the latest resource usage data from the API service component of the physical container cluster, and performing preliminary processing on this data, such as data cleaning and format conversion, to ensure that it can be effectively understood and used by the virtual offline resource group.

[0069] The update process includes constructing an HTTP request containing resource status information and then sending it to the management server of the virtual offline resource group. After receiving the request, the management server verifies the integrity and accuracy of the information and then integrates the information into the global resource status database of the virtual offline resource group. The update of resource status information is continuous, ensuring that the virtual offline resource group can access the most timely resource status data at any time. The updated resource status information is used for resource scheduling decisions of the virtual offline resource group. The scheduling algorithm dynamically adjusts the task allocation strategy based on the real-time resource status information to ensure that offline tasks can be allocated to virtual nodes with sufficient resources and meet the task requirements, thereby improving resource utilization and task execution efficiency.

[0070] In addition, resource status information is also used to optimize the resource management and task monitoring of virtual nodes. For example, through resource status information, the management unit can monitor the resource consumption of tasks and promptly discover resource bottlenecks, so as to take corresponding measures, such as resource reservation and task priority adjustment, to ensure the smooth execution of tasks and the reasonable allocation of resources.

[0071] Alternatively, the virtual offline resource group provided in this embodiment may also establish an information feedback mechanism, that is, after receiving the resource status information, it can send confirmation information or adjustment instructions to the offline resource management unit. For example, if the virtual offline resource group detects that the resource status of a virtual node is abnormal, it can instruct the resource management unit of the node to perform a more detailed resource check, or adjust its monitoring frequency and collection strategy to optimize the efficiency and accuracy of resource management.

[0072] Optionally, before receiving the task execution status transmitted by the offline resource management unit, the offline task execution method based on the virtual node also includes: the offline resource management unit periodically queries the API service component of the physical container cluster to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; the offline resource management unit packages the execution information of the offline task into a task status package, and transmits the task status package to the virtual offline resource group via an HTTP request.

[0073] Before the resource status information is updated, the offline resource management unit will continue to monitor the API service component of the physical container cluster to ensure real-time monitoring of the resource status. After the resource status information is collected and initially processed, the offline resource management unit will update this information to the virtual offline resource group in real time through HTTP requests.

[0074] The offline resource management unit is designed to periodically query the API service components of the physical container cluster, such as kube-apiserver, to obtain the execution information of offline tasks. This query process is periodic, and the query frequency can be adjusted according to the importance and real-time requirements of the task, ensuring that the execution status can be obtained in a timely manner without adding unnecessary system burden due to too frequent queries.

[0075] Among them, the execution information of offline tasks mainly includes the task running status, resource consumption and possible error information. The task running status includes but is not limited to whether the task has started, the current progress, whether it has been completed or encountered an error, etc.; resource consumption information includes the use of CPU, memory, storage and network resources during task execution; error information covers any abnormal or error conditions encountered during task execution, such as insufficient resources, program errors, etc. The offline resource management unit packages the collected execution information into a task status package, which is a structured data package containing the above-mentioned execution information. The packaging process usually also includes formatting and standardizing the execution information to ensure that the virtual offline resource group can quickly identify and process this information. After packaging, the offline resource management unit transmits the task status package to the virtual offline resource group through an HTTP request. As a mature Web communication protocol, HTTP provides a reliable data transmission mechanism and also supports asynchronous communication between the offline resource management unit and the virtual offline resource group, ensuring the real-time and accuracy of the task status information.

[0076] After receiving the task status package, the virtual offline resource group will process and integrate the information. Specifically, the virtual offline resource group will verify the integrity and accuracy of the task status package, and then store it in the central database, corresponding to the globally accessible resource and task status view. These execution information are used in many aspects. First, the virtual offline resource group uses the task running status and resource consumption to perform real-time task scheduling and resource optimization to ensure that the task can be executed with the optimal resource configuration. Secondly, the virtual offline resource group can automatically or manually trigger the troubleshooting and recovery process based on the error information, such as resource reallocation, task restart, etc., to improve the stability and reliability of the system. The virtual offline resource group may also send feedback information or adjustment instructions to the offline resource management unit based on the task execution status. For example, if it is found that the resource consumption of an offline task on a specific physical container cluster is abnormal, the virtual offline resource group can instruct the corresponding resource management unit to increase the query frequency so as to obtain the task status more frequently and take timely measures. In addition, the virtual offline resource group may also dynamically adjust the task priority or resource allocation strategy according to the real-time situation of task execution to optimize the overall computing efficiency and resource utilization.

[0077] Through the above implementation steps, the technical solution provided by the embodiment of the present invention realizes real-time monitoring, collection and updating of resource status information, ensures the accuracy and timeliness of resource scheduling, and also provides a stable and efficient resource environment for the execution of offline tasks.

[0078] Step S104: receiving a task execution status transmitted by the offline resource management unit, wherein the task execution status is an execution status of the offline task in the physical container cluster.

[0079] The task execution status refers to the running status of the offline task in the physical container cluster, including but not limited to the task start time, running status (such as running, paused, completed, failed), resource consumption (such as CPU, memory, disk IO, network bandwidth, etc.) and any related error information or warnings. The offline resource management unit regularly obtains this status information from the API service component (such as kube-api server) of the physical container cluster associated with it to ensure that the collected data can fully reflect the real-time execution of the task. The offline resource management unit formats the collected execution status information and packages it into a task status package. This task status package not only contains the current running status and resource usage of the task, but may also include other metadata, such as task ID, virtual node identifier, task priority, etc., so that the virtual offline resource group can quickly identify and process this information. After packaging, the offline resource management unit transmits the task status package to the central management node of the virtual offline resource group securely and efficiently through HTTP requests.

[0080] There are several advantages to using HTTP requests to transmit task status packages. First, HTTP is a widely supported protocol that ensures compatibility across different systems and platforms. Second, HTTP supports asynchronous communication, which means that the offline resource management unit can send task status packages without affecting its main functions (such as resource management and task scheduling). Finally, HTTP provides security protection. By using secure protocols such as HTTPS, task status information can be prevented from being intercepted or tampered with during transmission.

[0081] After receiving the task status package, the virtual offline resource group will verify and process the information. First, the integrity and consistency of the data are verified, and then it is integrated into the global task execution status database to facilitate subsequent resource scheduling and task management decisions. The virtual offline resource group can perform the following operations based on this status information: If the task execution status package shows that the resource consumption of a task on the current physical container cluster is abnormal, or the execution fails, the virtual offline resource group can immediately adjust the task scheduling and reallocate the task to a virtual node with more abundant resources or more suitable for the task to run, to ensure that the task can be completed smoothly. Based on the task execution status, the virtual offline resource group can dynamically adjust the resource allocation strategy, such as releasing the occupied resources of a task with high resource consumption after it is completed for use by other tasks, or reserving sufficient resources on the virtual node in advance for resource-intensive tasks to improve task execution efficiency. Error information or warnings in the task status package can help the virtual offline resource group quickly locate problems in task execution, such as insufficient resources, program errors, etc. Based on this information, the system can automatically execute fault recovery processes, such as resource reallocation and task restart, to reduce interruptions and delays in task execution.

[0082] The virtual offline resource group can generate real-time task monitoring reports based on the collected task execution status for task schedulers, system administrators or users to view, which improves the transparency of task execution and facilitates the monitoring and analysis of task execution.

[0083] In addition to receiving the task execution status, the virtual offline resource group may also need to send feedback information to the offline resource management unit. For example, when the virtual offline resource group makes scheduling adjustments based on the task execution status, it will send update instructions to the resource management unit of the relevant virtual node to ensure the consistency of the resource status and task status. This feedback loop mechanism ensures that the system can quickly respond to changes in task execution, improving the overall resource utilization efficiency and task execution success rate.

[0084] Through the above steps, after receiving the offline task request, the offline task request can be responded to, the target virtual node is selected according to the preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm is based on the current resource occupancy of each virtual node and the resource demand of the offline task to select the node, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation, updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node, and receiving the task execution status transmitted by the offline resource management unit, wherein the offline task request at least includes: the offline task to be executed and the identification information of the computing power resources required by the offline task, and the task execution status is the execution status of the offline task in the physical container cluster. In this embodiment, by introducing the offline resource management unit, the real-time monitoring and dynamic update of the resource status of the physical container cluster are realized, the timeliness and accuracy of resource scheduling are improved, thereby solving the technical problem that the related technology cannot provide real-time resource status updates, resulting in delayed scheduling decisions and low resource allocation efficiency.

[0085] In a multi-cluster environment, the technical feature of regular health status monitoring helps to improve the stability and availability of the entire offline computing system. Optionally, the offline task execution method based on virtual nodes also includes: the offline resource management unit performs regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check the node status, network connectivity, and storage availability; when the monitoring results indicate that the physical container cluster has an abnormal condition, the cluster health abnormality information is reported to the virtual offline resource group through an HTTP request.

[0086] In this embodiment, the offline resource management unit is not only responsible for scheduling offline tasks and monitoring the execution status, but also undertakes the task of regular health status monitoring of the physical container cluster. Through regular health monitoring, the stability and reliability of the system are enhanced, ensuring that offline tasks can be executed in a healthy and stable physical environment.

[0087] The offline resource management unit performs regular health monitoring of the physical container cluster to comprehensively evaluate the operation status of the cluster to prevent potential system failures or performance degradation. The specific monitoring scope includes but is not limited to: a. Node status: Check whether each node in the physical container cluster is operating normally, including monitoring the node's CPU usage, memory usage, disk space usage, and node load. Monitoring the node status helps to timely detect node failures, such as hardware failures, operating system problems, etc., to ensure the stable operation of the cluster. b. Network connectivity: Monitor the network communication status between nodes, including network latency, packet loss rate, and connection status. Monitoring network connectivity is crucial to ensure that offline tasks can smoothly transmit data and perform calculations in the cluster, especially in multi-node distributed computing scenarios. c. Storage availability: Monitor the storage system of the physical container cluster to ensure the reliability of data storage and access. This includes checking the storage read and write speed, remaining space, and data redundancy. Monitoring storage availability helps avoid data loss or task interruption due to storage failure.

[0088] The offline resource management unit establishes a communication connection with the API service component of the physical container cluster (such as kube-api server), and sends monitoring requests regularly (such as every minute, every five minutes, etc.) to obtain the various indicator data within the above health status monitoring range. These data are collected and analyzed by the offline resource management unit to determine the health status of the cluster. In addition, the offline resource management unit may also deploy additional monitoring tools or agents to monitor the operating status of the physical cluster in more detail and improve the accuracy and comprehensiveness of monitoring.

[0089] When analyzing the monitoring data, the offline resource management unit determines whether the physical container cluster has an abnormal condition based on the preset threshold or abnormality judgment rules. For example, if the CPU usage of the node continues to exceed the threshold, the network delay exceeds the normal range, or the remaining storage space is lower than the safety threshold, the offline resource management unit will determine that the cluster health is abnormal. Once the monitoring results indicate that the physical container cluster has an abnormal condition, the offline resource management unit will immediately report the cluster health abnormality information to the virtual offline resource group through HTTP requests. The report includes not only the type and degree of the abnormality, but also a preliminary analysis of the fault and recommended countermeasures. Reporting abnormal information through HTTP requests not only ensures the real-time and security of information transmission, but also supports asynchronous communication mechanisms, allowing the virtual offline resource group to take appropriate fault recovery and resource reallocation strategies according to the specific situation after receiving the abnormality report. After receiving the abnormal information report, the virtual offline resource group will verify and analyze the information, and then take appropriate countermeasures according to the severity and type of the abnormal situation. This may include rescheduling the affected offline tasks and migrating them to a healthy physical container cluster; or notifying the system administrator to troubleshoot and repair the fault to restore the health of the cluster.

[0090] In the process of handling abnormal information, the virtual offline resource group may also send feedback information to the offline resource management unit, instructing it to adjust the monitoring strategy or frequency in order to more accurately monitor the abnormal situation, or notify the offline resource management unit to resume normal monitoring mode after the abnormal situation is resolved to avoid excessive resource consumption.

[0091] Another optional specific implementation is described in detail below.

[0092] This implementation method can be applied to offline batch computing tasks under the Hadoop technology system, including spark tasks, flink tasks, etc. When K8S provides offline resources for these batch computing tasks, multiple K8S clusters are often involved due to the large resource demand. In this case, the batch computing task scheduling platform needs to connect to multiple K8S clusters, which is more complex and less flexible; this technical solution simulates a single k8s cluster into a host node through virtual nodes, so that the batch computing task scheduling platform only needs to connect to one k8s cluster, simplifying the process.

[0093] Figure 2 is a schematic diagram of a unified offline computing power supply system in a multi-cluster scenario according to an embodiment of the present invention. Figure 2As shown, the system includes: an offline task party, a virtual offline resource cluster (including multiple virtual nodes), and multiple K8S clusters, wherein an offline resource management unit Agent is deployed in each K8S cluster, and each virtual node can correspond to a K8S cluster.

[0094] Among them, the first part: virtual offline resource cluster.

[0095] As a unified offline computing power supply portal, the virtual offline resource cluster simplifies the offline task scheduling process. The virtual offline resource cluster has the following key functions:

[0096] (1) Task reception and scheduling: Receive offline tasks submitted by offline task parties and quickly select the most suitable virtual node to execute the task based on the internal scheduling algorithm to achieve efficient task allocation.

[0097] (2) Task lifecycle management: Comprehensively manage the lifecycle of offline tasks, including task dispatch, status tracking, and final completion or failure status reporting, to ensure transparency and controllability of task execution.

[0098] (3) Virtual node management: Responsible for the life cycle management of virtual nodes, including the creation, deletion, and dynamic adjustment of virtual node specifications to adapt to the ever-changing offline task requirements and physical cluster resource conditions.

[0099] Part 2: Virtual Nodes.

[0100] A virtual node is an abstract representation of offline resources in a K8S cluster. It communicates with a physical K8s cluster through HTTP requests. By mapping a single K8S cluster to a virtual host node, the batch computing task scheduling platform can treat it as a single resource supply node, thereby simplifying the docking process with multiple physical K8S clusters. The roles of virtual nodes include:

[0101] (1) Task request forwarding: As an intermediary, it directs the received offline task request to the physical K8S cluster it represents to ensure that the task can be executed.

[0102] (2) Resource status update: Updates the available offline resource status of the K8S cluster it represents in real time, providing accurate resource information for scheduling decisions of the virtual offline resource cluster.

[0103] The virtual node uses the offline resource management unit Agent to dynamically update the available offline resources of the cluster. The Agent collects available offline resources by communicating with the actual k8s cluster represented by the virtual node through the network.

[0104] When forwarding offline task requests to the representative K8S cluster, the virtual node adopts the forwarding logic, the batch task scheduling platform sends the batch task request, and then the virtual offline resource cluster runs the scheduling algorithm, selects the virtual node with sufficient available offline resources, and sends the batch task request. After that, the offline resource management unit Agent communicates with the representative's actual k8s cluster and sends the batch task request.

[0105] Part 3: Offline resource management agent.

[0106] In order to coordinate and monitor the offline resources and task status of the physical K8S cluster, the offline resource management agent is deployed in the physical K8S cluster associated with each virtual node. The agent has the following key responsibilities:

[0107] (1) Resource monitoring and updating: For example, continuously monitoring the resource usage of the API service components (such as kube-apiserver) of the physical K8S cluster to ensure the timeliness and accuracy of resource status information and provide updated resource data for virtual nodes.

[0108] (2) Task status tracking and reporting: Track the execution status of offline tasks on the physical cluster, including the running status of tasks, resource consumption, and error information, and report this status information to the virtual offline resource cluster through HTTP requests to provide a basis for task scheduling and resource optimization. For example, by querying the kube-api server of the representative k8s cluster, query the field representing the running status of the batch task, and update the query results to the custom resource-related fields representing the batch task of the virtual offline resource cluster.

[0109] (3) Cluster health check and status reporting: Perform health checks regularly to evaluate the overall health of the physical K8S cluster, including node status, network connectivity, storage availability, etc. Once an abnormality is detected, it is immediately reported to the virtual offline resource cluster to trigger the corresponding fault handling process.

[0110] Compared with the traditional multi-cluster management platform, this implementation method optimizes the supply of offline computing power based on virtual node technology. By updating the resource status and task execution status in real time, it ensures that the decision of resource allocation and task scheduling can be based on the most accurate information, which significantly improves the response speed and resource allocation efficiency. At the same time, by introducing virtual nodes and deploying offline resource management agents, the complexity of connecting the batch computing task scheduling platform with multiple physical K8S clusters is reduced, the system management and maintenance process is simplified, and the overall stability of the system is improved.

[0111] Furthermore, by integrating virtual offline resource clusters, virtual nodes and offline resource management agents, this technical solution provides a unified, efficient and stable resource management and scheduling platform for batch computing offline tasks, significantly improving the execution efficiency and resource utilization level of offline tasks in a multi-cluster environment.

[0112] The following is a detailed description in conjunction with another embodiment.

[0113] Embodiment 2

[0114] The offline task execution device based on virtual nodes provided in this embodiment includes multiple implementation units, each of which corresponds to each implementation step in the above-mentioned embodiment 1.

[0115] An embodiment of the present invention provides an offline task execution device based on virtual nodes, which is applied to a virtual offline resource group. The virtual offline resource group includes multiple virtual nodes. The virtual offline resource group corresponds to one or more physical container clusters and communicates with the physical container cluster through an offline resource management unit.

[0116] Figure 3 is a schematic diagram of an optional offline task execution device based on a virtual node according to an embodiment of the present invention, such as Figure 3 As shown, the offline task execution device based on virtual nodes may include: a task request receiving unit 31, a virtual node selecting unit 32, a resource status updating unit 33, and a task execution status receiving unit 34.

[0117] The task request receiving unit 31 is used to receive an offline task request, wherein the offline task request at least includes: an offline task to be executed and identification information of computing resources required for the offline task;

[0118] A virtual node selection unit 32 is used to respond to an offline task request, select a target virtual node according to a preset scheduling algorithm, and forward the offline task request to a physical container cluster corresponding to the target virtual node by the target virtual node, wherein the scheduling algorithm performs node selection based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for executing the forwarding operation;

[0119] A resource status updating unit 33, used to update the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node;

[0120] The task execution status receiving unit 34 is used to receive the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

[0121] The above-mentioned offline task execution device based on virtual nodes can receive offline task requests through the task request receiving unit 31, and the offline task request at least includes: the offline task to be executed and the identification information of the computing power resources required by the offline task. The virtual node selection unit 32 responds to the offline task request, selects the target virtual node according to the preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm selects the node based on the current resource occupancy of each virtual node and the resource requirements of the offline task. The offline resource management unit in the target virtual node is responsible for performing the forwarding operation, and the resource status information transmitted by the offline resource management unit is updated through the resource status updating unit 33, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node, and the task execution status transmitted by the offline resource management unit is received through the task execution status receiving unit 34, wherein the task execution status is the execution status of the offline task in the physical container cluster. In this embodiment, by introducing an offline resource management unit, real-time monitoring and dynamic updating of the resource status of the physical container cluster are achieved, thereby improving the timeliness and accuracy of resource scheduling, thereby solving the technical problem in related technologies that real-time resource status updates cannot be provided, resulting in delayed scheduling decisions and inefficient resource allocation.

[0122] Optionally, the virtual node selection unit includes: an information extraction module, used to extract the priority and resource type of the offline task from the identification information of the computing power resources required for the offline task; a node selection module, used to select the target virtual node from multiple virtual nodes based on the priority and resource type of the offline task using a scheduling algorithm.

[0123] Optionally, the offline task execution device based on the virtual node also includes: a resource status collection unit, which is used to monitor the API service component of the physical container cluster through the offline resource management unit before updating the resource status information transmitted by the offline resource management unit, collect the resource status information in the physical container cluster, and update the resource status information to the virtual offline resource group through an HTTP request.

[0124] Optionally, the virtual node selection unit also includes: a resource pre-allocation check module, which is used to perform a resource pre-allocation check based on the offline task request, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; and a request forwarding module, which is used to forward the offline task request to the physical container cluster corresponding to the target virtual node by the target virtual node when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task.

[0125] Optionally, the offline task execution device based on the virtual node also includes: a component query unit, which is used by the offline resource management unit to periodically query the API service component of the physical container cluster before receiving the task execution status transmitted by the offline resource management unit to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; a packaging unit, which is used by the offline resource management unit to package the execution information of the offline task into a task status package, and transmit the task status package to the virtual offline resource group via an HTTP request.

[0126] Optionally, the offline task execution device based on virtual nodes also includes: a cluster health monitoring unit, which is used by the offline resource management unit to perform regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check the node status, network connectivity, and storage availability; a cluster health exception reporting unit, which is used to report cluster health exception information to the virtual offline resource group through an HTTP request when the monitoring results indicate that the physical container cluster has an abnormal condition.

[0127] Optionally, the task request receiving unit includes: a request receiving module, used to receive offline task requests through a unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

[0128] The above-mentioned offline task execution device based on virtual nodes may also include a processor and a memory. The above-mentioned task request receiving unit 31, virtual node selection unit 32, resource status update unit 33, task execution status receiving unit 34, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0129] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and offline task processing based on virtual nodes can be implemented by adjusting kernel parameters to provide offline computing power. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM), and the memory includes at least one storage chip.

[0130] Embodiment 3

[0131] An embodiment of the present application may provide an electronic device, Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4(only one is shown) processor 402, memory 404, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0132] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the offline task execution method and device based on virtual nodes in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned offline task execution method based on virtual nodes. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving an offline task request, wherein the offline task request at least includes: identification information of the offline task to be executed and the computing power resources required by the offline task; responding to the offline task request, selecting a target virtual node according to a preset scheduling algorithm, and the target virtual node forwarding the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm performs node selection based on the current resource occupancy of each virtual node and the resource requirements of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; updating the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; receiving the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

[0134] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: extract the priority and resource type of the offline task from the identification information of the computing power resources required by the offline task; based on the priority and resource type of the offline task, use a scheduling algorithm to select a target virtual node from multiple virtual nodes.

[0135] The processor can call the information and applications stored in the memory through the transmission device to perform the following steps: before updating the resource status information transmitted by the offline resource management unit, the offline resource management unit monitors the API service component of the physical container cluster, collects the resource status information in the physical container cluster, and updates the resource status information to the virtual offline resource group through an HTTP request.

[0136] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: based on the offline task request, perform a resource pre-allocation check, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; when the check result indicates that the allocatable resources on the target virtual node meet the computing power resources required by the offline task, the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node.

[0137] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: before receiving the task execution status transmitted by the offline resource management unit, the offline resource management unit periodically queries the API service component of the physical container cluster to obtain the execution information of the offline task, wherein the execution information at least includes the task running status, resource consumption, and error information; the offline resource management unit packages the execution information of the offline task into a task status package, and transmits the task status package to the virtual offline resource group through an HTTP request.

[0138] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: the offline resource management unit performs regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check the node status, network connectivity, and storage availability; when the monitoring results indicate that the physical container cluster has an abnormal condition, the cluster health abnormality information is reported to the virtual offline resource group through an HTTP request.

[0139] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive offline task requests through the unified API interface of the virtual offline resource group, where the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

[0140] It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, etc. Figure 4 The structure of the electronic device is not limited. Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 4 Different configurations shown.

[0141] A person of ordinary skill in the art can understand that all or part of the steps in the various virtual node-based offline task execution methods in the above-mentioned embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0142] Embodiment 4

[0143] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the offline task execution method based on virtual nodes provided in the first embodiment.

[0144] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the virtual node-based offline task execution methods in the above-mentioned embodiment 1.

[0145] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0146] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the offline task execution method based on virtual nodes described in each embodiment of the present application.

[0147] The present application also provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the offline task execution method based on virtual nodes described in each embodiment of the present application are implemented.

[0148] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0149] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0152] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0154] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for executing offline tasks based on virtual nodes, characterized in that: Applied to a virtual offline resource group, the virtual offline resource group includes multiple virtual nodes, the virtual offline resource group corresponds to one or more physical container clusters, and communicates with the physical container clusters through an offline resource management unit. The offline task execution method based on the virtual node includes: Receiving an offline task request, wherein the offline task request includes at least: identification information of an offline task to be executed and computing resources required for the offline task; In response to the offline task request, a target virtual node is selected according to a preset scheduling algorithm, and the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node, wherein the scheduling algorithm performs node selection based on the current resource occupancy of each virtual node and the resource demand of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; Updating resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; Receive a task execution status transmitted by the offline resource management unit, wherein the task execution status is an execution status of the offline task in the physical container cluster.

2. The offline task execution method according to claim 1, characterized in that: The step of selecting a target virtual node according to a preset scheduling algorithm includes: Extracting the priority and resource type of the offline task from the identification information of the computing resources required by the offline task; Based on the priority and resource type of the offline task, the target virtual node is selected from the multiple virtual nodes using the scheduling algorithm.

3. The offline task execution method according to claim 1, characterized in that: Before updating the resource status information transmitted by the offline resource management unit, the method further includes: The offline resource management unit monitors the API service component of the physical container cluster, collects resource status information in the physical container cluster, and updates the resource status information to the virtual offline resource group through an HTTP request.

4. The offline task execution method according to claim 1, characterized in that: The step of forwarding the offline task request by the target virtual node to the physical container cluster corresponding to the target virtual node includes: Based on the offline task request, perform a resource pre-allocation check, wherein the resource pre-allocation check is used to confirm whether the allocatable resources on the target virtual node meet the computing power resources required by the offline task; When the check result indicates that the allocatable resources on the target virtual node meet the computing resources required by the offline task, the target virtual node forwards the offline task request to the physical container cluster corresponding to the target virtual node.

5. The offline task execution method according to claim 1, characterized in that: Before receiving the task execution status transmitted by the offline resource management unit, the method further includes: The offline resource management unit periodically queries the API service component of the physical container cluster to obtain execution information of the offline task, wherein the execution information at least includes task running status, resource consumption, and error information; The offline resource management unit packages the execution information of the offline task into a task status package, and transmits the task status package to the virtual offline resource group through an HTTP request.

6. The offline task execution method according to claim 1, characterized in that: Also includes: The offline resource management unit performs regular health status monitoring on the physical container cluster, wherein the regular health status monitoring is used to regularly check node status, network connectivity, and storage availability; When the monitoring result indicates that the physical container cluster is in an abnormal state, cluster health abnormality information is reported to the virtual offline resource group through an HTTP request.

7. The offline task execution method according to claim 1, characterized in that: The steps of receiving an offline task request include: An offline task request is received through the unified API interface of the virtual offline resource group, wherein the unified API interface is used to complete the following functions: offline task submission, task status query, and resource status update.

8. An offline task execution device based on virtual nodes, characterized in that: Applied to a virtual offline resource group, the virtual offline resource group includes multiple virtual nodes, the virtual offline resource group corresponds to one or more physical container clusters, and communicates with the physical container clusters through an offline resource management unit. The offline task execution device based on the virtual node includes: A task request receiving unit, configured to receive an offline task request, wherein the offline task request at least includes: identification information of an offline task to be executed and computing resources required for the offline task; A virtual node selection unit, used to respond to the offline task request, select a target virtual node according to a preset scheduling algorithm, and forward the offline task request to the physical container cluster corresponding to the target virtual node by the target virtual node, wherein the scheduling algorithm performs node selection based on the current resource occupancy of each virtual node and the resource demand of the offline task, and the offline resource management unit in the target virtual node is responsible for performing the forwarding operation; A resource status updating unit, configured to update the resource status information transmitted by the offline resource management unit, wherein the resource status information is obtained by the offline resource management unit monitoring the physical container cluster corresponding to the target virtual node; The task execution status receiving unit is used to receive the task execution status transmitted by the offline resource management unit, wherein the task execution status is the execution status of the offline task in the physical container cluster.

9. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the virtual node-based offline task execution method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the offline task execution method based on virtual nodes described in any one of claims 1 to 7 are implemented.

Images