Resource allocation method based on Kubernetes cluster and related equipment

Abstract

The present application relates to a resource allocation method based on a Kubernetes cluster and related equipment. The method includes: obtaining the current running status of an application running on a Kubernetes cluster; determining a target task based on the current running status; adjusting the Spark code of the application based on the target task to generate an operator associated with the target task; using the operator to batch apply for at least one executor that matches the target task within a preset time range; allocating resources corresponding to the application based on at least one executor, wherein: if the number of at least one executor is less than the target number corresponding to the target task, terminating the operation of the application and regenerating at least one executor; otherwise, directly running the application. By dynamically allocating resources corresponding to the target task of the application, the present application can improve the migration efficiency of Spark in the Kubernetes cluster, reduce the probability of application failure, and enhance the stability of the Kubernetes cluster.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a resource allocation method based on a Kubernetes cluster and related equipment. Background Art

[0002] With the rise of big data, traditional database systems are increasingly unable to adapt to the needs of enterprise digital transformation. Instead, enterprises are frequently adopting Spark to build data systems. Spark is a unified analytics engine for large-scale data processing that can run on clusters. When Spark runs on a cluster, the driver can connect to a cluster manager to allocate resources across applications. Cluster managers include Spark's own cluster manager, Mesos, YARN, or Kubernetes, forming different scheduling systems.

[0003] In actual applications, we often encounter scenarios where we need to migrate large-scale Spark cluster tasks from other scheduling systems such as Mesos and YARN to Kubernetes (i.e., K8S). However, during the specific migration process, the following problems may arise:

[0004] 1. etcd is prone to overflow. etcd is a key component of the Kubernetes cluster, used for shared configuration and service discovery. However, when migrating from other scheduling systems to Kubernetes, the sheer volume of data generated by the numerous simultaneous events and the resulting new versions with each update can quickly overwhelm etcd.

[0005] 2. Resource deadlock: When using Kubernetes to schedule Spark tasks, if multiple tasks simultaneously request a certain number of executors, these tasks may not receive sufficient resources to run and may only receive partial resources, leading to resource deadlock.

[0006] 3. Resource allocation issues. Spark tasks require users to specify the number of executors required in the task configuration, but there's no way to measure whether this number of executors matches the task size. If the number of executors is too small, the corresponding task will be terminated due to insufficient resources; if the number of executors is too large, resources will be wasted.

[0007] 4. Resource recycling. In Spark versions 2.4 and above, dynamic collection is generally enabled to return completed executors to the scheduler to reduce pod idle time. However, pod recycling also causes the rearrangement output files stored in the pod to be recycled, forcing downstream tasks to recalculate the RDD, impacting their execution.

[0008] 5. Event log issues: Since the history server of a Spark on Kubernetes cluster is generally stored on the cluster's persistent volume, which has capacity limitations, and Spark's event logs are particularly large, they can easily fill up the disk capacity, leading to insufficient capacity.

[0009] Therefore, the above migration process not only fails to know the amount of resources required for task operation, which leads to resource waste and reduced migration efficiency, but also causes the occurrence of various program failures, increasing the risk of data system instability, and urgently needs further improvement. Summary of the Invention

[0010] In view of this, this application proposes a resource allocation method based on Kubernetes cluster and related equipment, which can dynamically allocate resources corresponding to the target tasks of the application, thereby improving the migration efficiency of Spark in the Kubernetes cluster, reducing the probability of application failure, and enhancing the stability of the Kubernetes cluster.

[0011] According to one aspect of the present application, a resource allocation method based on a Kubernetes cluster is provided, and the resource allocation method based on a Kubernetes cluster includes: obtaining the current running status of an application running on the Kubernetes cluster; determining a target task corresponding to the application based on the current running status of the application; adjusting the Spark code of the application based on the target task of the application, and generating an operator associated with the target task; using the operator to batch apply for at least one executor matching the target task within a preset time range; allocating resources corresponding to the application based on the at least one executor, wherein: if the number of the at least one executor is less than the target number corresponding to the target task, terminating the running of the application and regenerating the at least one executor; if the number of the at least one executor is greater than or equal to the target number corresponding to the target task, directly running the application.

[0012] According to another aspect of the present application, a resource allocation device based on a Kubernetes cluster is provided, comprising: a status acquisition module for acquiring the current running status of an application running on a Kubernetes cluster; a task determination module for determining a target task corresponding to the application based on the current running status of the application; an operator generation module for adjusting the Spark code of the application based on the target task of the application, and generating an operator associated with the target task; an executor application module for using the operator to batch apply for at least one executor matching the target task within a preset time range; a resource allocation module for allocating resources corresponding to the application based on the at least one executor, wherein: if the number of the at least one executor is less than the target number corresponding to the target task, the operation of the application is terminated and the at least one executor is regenerated; if the number of the at least one executor is greater than or equal to the target number corresponding to the target task, the application is directly run.

[0013] In some embodiments of the present application, a computer device may also be provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-described method when executing the computer program.

[0014] In some embodiments of the present application, a computer-readable storage medium may be provided, on which a computer program is stored. When the computer program is run on a computer, the computer is caused to execute the steps of the method as described above.

[0015] By obtaining the current running status of the application on the Kubernetes cluster, combining the target task of the application, adjusting the Spark code of the application to generate an operator associated with the target task, and then using the operator to batch apply for at least one executor matching the target task within a preset time range, and finally allocating resources corresponding to the application based on the at least one executor, according to various aspects of the present application, resources corresponding to the target task of the application can be dynamically allocated, thereby improving the migration efficiency of Spark in the Kubernetes cluster, reducing the probability of application failure, and enhancing the stability of the Kubernetes cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of the application environment of the resource allocation method based on the Kubernetes cluster according to an embodiment of the present application is shown.

[0018] Figure 2 A schematic diagram illustrating a flow chart of a resource allocation method based on a Kubernetes cluster according to an embodiment of the present application is shown.

[0019] Figure 3 A schematic diagram showing the Kubernetes cluster architecture of an embodiment of the present application.

[0020] Figure 4 A schematic diagram of a Kubernetes cluster-based resource allocation device according to an embodiment of the present application is shown.

[0021] Figure 5 A schematic diagram showing the structure of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0023] The resource allocation method based on Kubernetes cluster provided in this application can be applied to Figure 1 In the application environment shown, terminal 101 can communicate with server 102 via a network. Specifically, server 102 can obtain the current running status of an application on the Kubernetes cluster, adjust the Spark code of the application based on the target task of the application, generate an operator associated with the target task, and then use the operator to batch apply for at least one executor matching the target task within a preset time range. Finally, resources corresponding to the application are allocated based on the at least one executor.

[0024] The terminal 101 may be a terminal device such as a personal computer, a laptop computer, a smart phone, a tablet computer, or a portable wearable device, a smart TV or other smart terminals.

[0025] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0026] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0027] Specifically, such as Figure 2 As shown, a schematic diagram of the resource allocation method based on Kubernetes cluster in an embodiment of the present application is shown, in which the method is applied to Figure 1 Taking the server in

[0065] as an example, the method may include:

[0028] Step S1: Get the current running status of the application running on the Kubernetes cluster;

[0029] Applications running on Kubernetes clusters can be implemented based on Spark, a unified analytics engine for large-scale data processing that provides high-level APIs in programming languages like Java and Python and supports optimization of general execution graphs.

[0030] Spark runs on Windows and Unix-like systems (e.g., Linux, macOS), and on any platform running a supported version of Java, such as the JVM. Spark can run standalone or on multiple existing cluster managers. In this application, Spark can be deployed on a Kubernetes cluster.

[0031] Specifically, Spark applications can run as independent processes on a cluster, coordinated by an object in the Spark main program (called a driver program). In practice, to run on a cluster, the driver program can connect to several types of cluster managers (such as Spark's own cluster manager, Mesos, YARN, or Kubernetes), which can be responsible for allocating resources between applications. Once the driver program connects to the corresponding cluster manager, Spark obtains executors on the nodes in the cluster, then sends the application code to the executors, and finally the SparkContext sends the tasks to the executors for execution.

[0032] Furthermore, the application's running status may include running, failed, retrying, submitting, successful, etc. Different running statuses correspond to different target tasks. It is understood that the application's running status may also have other types, and this application does not limit the running status of the application.

[0033] Step S2: determining a target task corresponding to the application according to the current running state of the application;

[0034] For example, application code can be defined by a JAR or Python program that is passed to the Spark context. Each application has its own executor process, which can remain running for the duration of the application and run tasks in multiple threads. This helps isolate applications from each other both on the scheduling side (each driver program schedules its own tasks) and on the execution side (tasks from different applications run in different platforms).

[0035] In this application, applications built on Spark can be run on a cluster using a driver and executor. Spark is independent of the underlying cluster manager. In one example, a Spark application can be submitted directly to a Kubernetes cluster using the spark-submit command.

[0036] For example, if the current running state of the application is "Running," it means that the application has not yet completed execution within a certain period of time. In this case, the configured task-related information corresponding to the application can be read to obtain the target task corresponding to the application. For another example, if the current running state of the application is "Submitting," it means that the application has completed execution within a certain period of time. In this case, the relevant log information corresponding to the application can be read to obtain the target task corresponding to the application.

[0037] Step S3: adjusting the Spark code of the application according to the target task of the application to generate an operator associated with the target task;

[0038] The operator may be a code snippet generated by adjusting the Spark code of the application program, and is used to batch apply for at least one executor that matches the target task.

[0039] In practice, the Operator pattern can describe the key objectives of a service or set of services. These operators can encapsulate pre-written task automation code, thereby automating repetitive tasks within the Kubernetes application runtime. Notably, in this application, the Operator can extend the capabilities of the Kubernetes cluster by defining one or more custom resource-association controllers associated with the target tasks, without modifying the Kubernetes code itself. In other words, the Operator can act as a controller for custom resources within the Kubernetes cluster.

[0040] When deploying the operator, custom resources and their associated controllers can be added to the cluster. For example, the operator can be implemented using any language or runtime that supports the Kubernetes API client. It is understood that this application does not limit the specific implementation of the operator.

[0041] Furthermore, adjusting the Spark code of the application according to the target task of the application to generate an operator associated with the target task may include:

[0042] Step S31: merging multiple source configuration files in the Kubernetes cluster into a target configuration file, wherein the source configuration file is used to store configuration information related to cluster users and authentication mechanisms;

[0043] The source configuration file may be a Kubeconfig file, of which there may be multiple, and is used to store information about cluster users, namespaces, and authentication mechanisms to select clusters and related information for cluster API server communication. Exemplarily, the source configuration file may include cluster parameters, user parameters, context parameters, and the current context, and the kubecm tool is used to merge multiple source configuration files in the Kubernetes cluster into a target configuration file.

[0044] Step S32: Based on the target configuration file and in combination with the target task of the application, the Spark code of the application is adjusted to generate an operator associated with the target task.

[0045] In this application, the merging of the target configuration files can occur before the operator is generated. Due to the adoption of a more aggressive merging strategy, the embodiment of this application can further improve the migration efficiency of Spark in the Kubernetes cluster.

[0046] To more clearly illustrate the resource allocation method based on the Kubernetes cluster, the following briefly introduces the architecture of the Kubernetes cluster.

[0047] Figure 3 A schematic diagram of the Kubernetes cluster architecture of an embodiment of the present application is shown.

[0048] like Figure 3 As shown, in this application, the architecture of a Kubernetes cluster may include a control plane component (ControlPanel), a cloud interface, and several nodes.

[0049] The control plane component makes global decisions for the entire cluster, including resource scheduling and detecting and responding to cluster events, such as starting new containers when the replicas field of a deployment is not met. The control plane component can run on any node in the cluster.

[0050] The API server can be a subcomponent of the Kubernetes control plane. The API server is responsible for exposing the Kubernetes API and handling requests. The API server acts as a front-end for the Kubernetes control plane and can be implemented using kube-apiserver.

[0051] etcd is a consistent and highly available key-value store that serves as the backend database for all Kubernetes cluster data.

[0052] The kube-scheduler is a subcomponent of the control plane component that monitors newly created containers and containers that do not have a specific node to run on, and selects nodes to run the containers in the Kubernetes cluster. During the specific scheduling process, factors that may be considered include but are not limited to the resource requirements of individual containers and container collections, software and hardware and policy constraints, affinity and anti-affinity specifications, data location, interference between workloads, and deadlines.

[0053] As a subcomponent of the control plane component, kube-controller-manager can be used to run controller processes. Logically, each controller is a separate process, but to reduce complexity, controllers can all be compiled into the same executable file and run in the same process.

[0054] The Cloud Interface, also known as the Cloud Controller Manager, connects the cluster to the cloud provider's API and separates components that interact with the cloud platform from those that interact with the cluster.

[0055] The Node component runs on each node and is responsible for maintaining running containers and providing the Kubernetes runtime environment. Kubelet runs on every node in the cluster to ensure stable container operation. Kube-proxy, a network proxy running on every node in the cluster, implements part of the Kubernetes service concept. The container runtime environment is the software responsible for running containers.

[0056] Step S4: using the operator to batch apply for at least one executor matching the target task within a preset time range;

[0057] Specifically, using the operator to batch apply for at least one executor matching the target task within a preset time range may include:

[0058] Step S41: Obtain the current etcd capacity of the Kubernetes cluster;

[0059] etcd is a highly available key-value store primarily used for shared configuration and service discovery. It uses a consensus algorithm to replicate logs and ensure strong consistency. Generally, etcd can be thought of as a highly available and consistent service discovery repository.

[0060] Step S42: adjusting the current etcd capacity of the Kubernetes cluster to a target etcd capacity that matches the target task according to the target task;

[0061] Since the current etcd capacity of the Kubernetes cluster is usually small, which affects the normal operation of the application, the current etcd capacity of the Kubernetes cluster can be increased to the target etcd capacity that is compatible with the target task according to the target task, that is, etcd is expanded.

[0062] Step S43: Based on the target etcd capacity, use the operator to batch apply for at least one executor that matches the target task within a preset time range.

[0063] It should be noted that in some cases, if the current etcd capacity is too large and affects resource allocation, the embodiment of the present application can also appropriately reduce the current etcd capacity based on the target task. By dynamically adjusting the etcd capacity, the embodiment of the present application can further improve the migration efficiency of Spark in the Kubernetes cluster.

[0064] Step S5: Allocate resources corresponding to the application based on the at least one executor, wherein: if the number of the at least one executor is less than the target number corresponding to the target task, terminate the operation of the application and regenerate the at least one executor; if the number of the at least one executor is greater than or equal to the target number corresponding to the target task, directly run the application.

[0065] Specifically, allocating resources corresponding to the application based on the at least one executor may include:

[0066] Step S51: registering a listener corresponding to the application, wherein the listener is used to collect information related to current storage and reorder data volume;

[0067] The listener can be implemented based on Spark. The current storage information can include information such as current memory and disk usage, and the rearranged data volume can include shuffle data volume. It is understood that in actual applications, the listener can also include other task-related storage information and data information, which is not limited in this application.

[0068] Furthermore, registering a listener corresponding to the application includes:

[0069] Step S511: Using external reordering storage to store objects in the Kubernetes cluster;

[0070] The reordered storage is also called shuffle storage, which can support various commonly used object storages. In one example, the object storage can be, for example, s3, gcs, etc., as shown below:

[0071] spark.shuffle.externalStorage.enabled

[0072] spark.shuffle.externalStorage.backend

[0073] spark.shuffle.externalStorage.bucket

[0074] Enabled, backend, and bucket can correspond to three types of external storage objects. It is understood that this application does not limit the types of various stored objects.

[0075] Step S512: exposing the stored object as a persistent volume claim;

[0076] A Persistent Volume Claim (PVC) expresses a user's request for storage. A Persistent Volume Claim consumes Persistent Volume (PV) resources. A Persistent Volume is a piece of storage in a cluster that can be pre-configured or dynamically configured using a StorageClass. A Persistent Volume Claim can request a specific size and access mode, for example, requiring a PV to be mounted in ReadWriteOnce, ReadOnlyMany, or ReadWriteMany modes.

[0077] Step S513: Mount the persistent volume claim to the corresponding container based on the storage request corresponding to the persistent volume claim.

[0078] It should be noted that persistent volume claims allow applications to consume abstract storage resources and use persistent volumes with different performance for different tasks. After receiving the storage request corresponding to the persistent volume claim, the persistent volume claim can be mounted to the corresponding container (Pod).

[0079] It is worth noting that the container (Pod) of this application can be the smallest deployable computing unit created and managed in a Kubernetes cluster. In the process of submitting a Spark application to a Kubernetes cluster, you can first create a Spark application running in a container, then create a driver container and an executor container and connect them, and then execute the application code. When the application code is executed, the executor container is terminated and cleaned up, while the driver container can retain logs until it is garbage collected or manually cleaned up.

[0080] A container can be a single instance or a group of instances. A group of containers can share storage, networking, and declarations about how to run them. The contents of a container can be collocated and scheduled together, running within a shared context. Generally, a container can be considered a "logical host" for an application, containing one or more application containers that are relatively tightly coupled together.

[0081] Furthermore, registering a listener corresponding to the application further includes:

[0082] Step S514: Acquire the allocated resources and at least one event that has occurred corresponding to the target task;

[0083] The allocated resources may be resources allocated to the application based on the at least one executor. The allocated resources may be dynamic and associated with a specific task. The event may be generated during the application's attempt to run.

[0084] Step S515: Separately store the allocated resources and the at least one event that has occurred, wherein the storage location of the at least one event is the same.

[0085] That is, the storage of the at least one event can be stored separately and centrally. By separating the storage of resources from the storage of events, the embodiment of the present application can solve the problem of etcd being easily overwhelmed due to excessive time.

[0086] Step S52: generating a statistical report based on the current storage information and the amount of rearranged data collected by the listener, wherein the statistical report includes the task amount of the target task;

[0087] Specifically, the statistical report displays information related to the target task's workload. Furthermore, the statistical report also includes current storage information and the amount of rearranged data collected by the listener. The statistical report corresponding to the target task is generated by combining the information related to the target task's workload with the current storage information and the amount of rearranged data collected by the listener.

[0088] Step S53: Automatically generate configuration information corresponding to the target task according to the statistical report.

[0089] In practical applications, the statistical report can perform statistical analysis on the relevant information collected by the listener, and then generate configuration information corresponding to the target task, so as to recommend the configuration corresponding to the target task to the user based on the configuration information. Compared with the prior art of manually configuring information related to the target task, the embodiment of the present application can improve the efficiency of configuring the target task while avoiding the adverse effects of manual configuration.

[0090] Furthermore, the resource allocation method based on the Kubernetes cluster also includes:

[0091] Step S61: intercepting log information that exceeds a preset time range;

[0092] The preset time range may be set as required, for example, the preset time range may be 24 hours or 48 hours, and this application does not limit the preset time range.

[0093] Step S62: storing the log information in the rearrangement storage.

[0094] Exemplarily, the reordered storage may be an object storage such as s3 in step S511. By storing the log information in the reordered storage, embodiments of the present application can address the limited storage space issue in a Kubernetes cluster, thereby improving the efficiency of Spark migration in a Kubernetes cluster, reducing the probability of application failures, and enhancing the stability of the Kubernetes cluster.

[0095] In summary, this application obtains the current running status of the application on the Kubernetes cluster, combines the target task of the application, adjusts the Spark code of the application to generate an operator associated with the target task, and then uses the operator to batch apply for at least one executor that matches the target task within a preset time range. Finally, based on the at least one executor, resources corresponding to the application are allocated. It can dynamically allocate resources corresponding to the target task of the application, know the resources required for the task to run when running the target task of the application, and apply for resources more reasonably, thereby improving the migration efficiency of Spark in the Kubernetes cluster, reducing the probability of application failure, and enhancing the stability of the Kubernetes cluster. At the same time, using external storage as Spark's rearrangement storage can reduce the probability of rearranged data needing to be recalculated due to node recycling, thereby solving the efficiency and storage problems in the process of large-scale cluster migration of Spark.

[0096] In order to better implement the above method, accordingly, an embodiment of the present application also provides a resource allocation device based on a Kubernetes cluster, and the resource allocation device based on a Kubernetes cluster is specifically integrated in a terminal or a server.

[0097] Figure 4 A schematic diagram of a resource allocation device based on a Kubernetes cluster according to an embodiment of the present application is shown. Figure 4 , the device comprises:

[0098] A status acquisition module 41 is used to obtain the current running status of the application running on the Kubernetes cluster;

[0099] A task determination module 42 is configured to determine a target task corresponding to the application according to the current running state of the application;

[0100] An operator generation module 43 is configured to adjust the Spark code of the application according to the target task of the application and generate an operator associated with the target task;

[0101] An executor application module 44 is configured to use the operator to batch apply for at least one executor that matches the target task within a preset time range;

[0102] The resource allocation module 45 is used to allocate resources corresponding to the application based on the at least one executor, wherein: if the number of the at least one executor is less than the target number corresponding to the target task, the operation of the application is terminated and the at least one executor is regenerated; if the number of the at least one executor is greater than or equal to the target number corresponding to the target task, the application is directly run.

[0103] In addition, the embodiment of the present application also provides a computer device, which can be a terminal or a server, such as Figure 5 , which shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0104] The computer device may include one or more processing core processors 401, one or more computer readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will understand that Figure 5 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0105] Processor 401 is the control center of the computer device. It connects the various components of the entire computer device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 402 and accessing data stored in memory 402, it performs various functions of the computer device and processes data, thereby providing overall monitoring of the computer device. Optionally, processor 401 may include one or more processing cores; preferably, processor 401 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0106] Memory 402 can be used to store software programs and modules. Processor 401 executes various functional applications and data processing by running the software programs and modules stored in memory 402. Memory 402 may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the like; the data storage area may store data created based on the use of the computer device. In addition, memory 402 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 402 may also include a memory controller to provide processor 401 with access to memory 402.

[0107] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0108] The computer device may further include an input unit 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0109] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the computer device will load the executable files corresponding to one or more application processes into the memory 402 according to the following instructions, and the processor 401 will run the application stored in the memory 402 to implement various functions as follows:

[0110] Step S1: Get the current running status of the application running on the Kubernetes cluster;

[0111] Step S2: determining a target task corresponding to the application according to the current running state of the application;

[0112] Step S3: adjusting the Spark code of the application according to the target task of the application to generate an operator associated with the target task;

[0113] Step S4: using the operator to batch apply for at least one executor matching the target task within a preset time range;

[0114] Step S5: Allocate resources corresponding to the application based on the at least one executor, wherein: if the number of the at least one executor is less than the target number corresponding to the target task, terminate the operation of the application and regenerate the at least one executor; if the number of the at least one executor is greater than or equal to the target number corresponding to the target task, directly run the application.

[0115] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0116] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0117] To this end, an embodiment of the present application also provides a computer-readable storage medium, which stores multiple instructions that can be loaded by a processor to execute the methods provided in various optional implementations of the embodiment of the present application.

[0118] According to one aspect of the present application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0119] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0120] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0121] Since the instructions stored in the storage medium can execute the steps in the method provided in the embodiment of the present application, the beneficial effects that can be achieved by the method provided in the embodiment of the present application can be achieved. Please refer to the previous embodiment for details and will not be repeated here.

[0122] The above is a detailed introduction to the resource allocation method, device, computer equipment and storage medium based on the Kubernetes cluster provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core ideas. At the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A resource allocation method based on Kubernetes cluster, characterized in that: The method comprises: Get the current running status of the application running on the Kubernetes cluster; Determining a target task corresponding to the application according to a current running state of the application; Adjusting the Spark code of the application according to the target task of the application to generate an operator associated with the target task; Using the operator to batch apply for at least one executor that matches the target task within a preset time range; Allocating resources corresponding to the application based on the at least one executor, wherein: if the number of the at least one executor is less than a target number corresponding to the target task, terminating the running of the application and regenerating the at least one executor; if the number of the at least one executor is greater than or equal to the target number corresponding to the target task, directly running the application; Adjusting the Spark code of the application according to the target task of the application to generate an operator associated with the target task includes: Merge multiple source configuration files in the Kubernetes cluster into a target configuration file, wherein the source configuration file is used to store configuration information corresponding to cluster users and authentication mechanisms; Based on the target configuration file and in combination with the target task of the application, the Spark code of the application is adjusted to generate an operator associated with the target task.

2. The resource allocation method based on Kubernetes cluster according to claim 1, characterized in that Using the operator to batch apply for at least one executor matching the target task within a preset time range includes: Get the current etcd capacity of the Kubernetes cluster; Adjust the current etcd capacity of the Kubernetes cluster to the target etcd capacity that matches the target task according to the target task; Based on the target etcd capacity, the operator is used to batch apply for at least one executor that matches the target task within a preset time range.

3. The resource allocation method based on Kubernetes cluster according to claim 1, characterized in that Allocating resources corresponding to the application based on the at least one executor includes: Registering a listener corresponding to the application, wherein the listener is used to collect information related to current storage and reorder data volume; generating a statistical report based on the current storage information and the amount of rearranged data collected by the listener, wherein the statistical report includes the task amount of the target task; Configuration information corresponding to the target task is automatically generated according to the statistical report.

4. The resource allocation method based on Kubernetes cluster according to claim 3, characterized in that Register the listener corresponding to the application, including: Using external reordering storage to store objects in the Kubernetes cluster; Exposing the stored object as a persistent volume claim; The persistent volume claim is mounted to the corresponding container based on the storage request corresponding to the persistent volume claim.

5. The resource allocation method based on Kubernetes cluster according to claim 4, characterized in that Registering a listener corresponding to the application also includes: Acquire allocated resources and at least one occurred event corresponding to the target task; The allocated resources and the at least one event that has occurred are stored separately, wherein the at least one event is stored in the same location.

6. The resource allocation method based on Kubernetes cluster according to claim 4, characterized in that The Kubernetes cluster-based resource allocation method further includes: Intercept log information that exceeds the preset time range; The log information is stored on the reorder storage.

7. A resource allocation device based on Kubernetes cluster, characterized in that: include: The status acquisition module is used to obtain the current running status of the application running on the Kubernetes cluster; A task determination module, configured to determine a target task corresponding to the application according to the current running state of the application; An operator generation module, configured to adjust the Spark code of the application according to the target task of the application and generate an operator associated with the target task; An executor application module, configured to use the operator to batch apply for at least one executor matching the target task within a preset time range; a resource allocation module, configured to allocate resources corresponding to the application based on the at least one executor, wherein: if the number of the at least one executor is less than the target number corresponding to the target task, terminate the running of the application and regenerate the at least one executor; if the number of the at least one executor is greater than or equal to the target number corresponding to the target task, directly run the application; The operator generation module is also used for: Merge multiple source configuration files in the Kubernetes cluster into a target configuration file, wherein the source configuration file is used to store configuration information corresponding to cluster users and authentication mechanisms; Based on the target configuration file and in combination with the target task of the application, the Spark code of the application is adjusted to generate an operator associated with the target task.

8. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is run on a computer, the computer is caused to perform the steps of the method according to any one of claims 1 to 6.

Images