Resource allocation method, system and equipment based on k8s and storage medium

Abstract

The invention provides a resource allocation method, system and device based on k8s and a storage medium, and relates to the field of distribution.The method comprises the steps that the method is used for a k8s master node, spark software is loaded in the k8s master node, and the method comprises the steps that a spark task is received, the spark task is split according to a first preset resource quantity and issued to a preset number of work nodes, and a suspended task is obtained; wherein the first preset resource quantity is the maximum resource quantity which can be accepted by the working node; and acquiring the node number N according to the resource quantity required by the suspended task, applying for N virtual nodes from the k8s base, and issuing the suspended task to the virtual nodes after successful application. The problem that when spark software is adopted to carry out data calculation, resource scheduling is difficult to carry out is solved.

Description

Technical Field

[0001] The present application relates to the field of distributed systems, and in particular to a resource allocation method, system, device and storage medium based on k8s. Background Art

[0002] Spark is a big data computing software, suitable for computing non-real-time batch data. However, the amount of data varies in different business scenarios.

[0003] When using big data spark software for data calculation, it is necessary to estimate in advance how much resources are needed. When the estimated resources exceed the existing computing resources, the operation and maintenance personnel need to manually expand the cluster. Due to the difference between the estimated resources and the actual resources, it is easy to waste resources or inefficient execution.

[0004] In view of the above defects, there is an urgent need for a k8s-based resource allocation method, system, device and storage medium that can solve the problem of difficulty in resource scheduling when using spark software for data calculation in the prior art. Summary of the invention

[0005] The present application provides a k8s-based resource allocation method, system, device and storage medium to solve the problem that it is difficult to perform resource scheduling when spark software performs data calculations.

[0006] In a first aspect, the present application provides a resource allocation method based on k8s, which is used for a k8s master node, in which spark software is loaded, and the method includes:

[0007] Receive a spark task, split the spark task according to a first preset resource amount and send it to a preset number of working nodes, and obtain a suspended task; wherein the first preset resource amount is a maximum resource amount that the working node can accept;

[0008] The number of nodes N is obtained according to the required resources of the suspended task, and N virtual nodes are applied to the k8s base. After the application is successful, the suspended task is sent to the virtual node.

[0009] In a possible design, splitting the spark task according to the first preset resource amount and sending it to a preset number of working nodes to obtain the suspended task includes:

[0010] Split the spark task into the smallest task that can be completed by a single node, and send the smallest task to the working node in a queue;

[0011] When the amount of resources required for the task received by the working node is greater than the first preset amount of resources, it is determined that the task of the working node has been sent, and the minimum task in the queue is sent to the next working node until all the working nodes have completed the sending;

[0012] The unissued tasks in the queue are suspended as suspended tasks.

[0013] In a possible design, obtaining the number of nodes N according to the required amount of resources of the suspended task and applying for N virtual nodes from the k8s base include:

[0014] The number of nodes N is obtained by rounding the ratio of the required resource amount of the suspended task to the second preset resource amount, and a first message is sent to the k8s base, wherein the first message is used to indicate the addition of N virtual nodes whose available resource amount is not less than the third preset resource amount, wherein the third preset resource amount is greater than the second preset resource amount.

[0015] In a possible design, sending the suspended task to the virtual node includes:

[0016] The smallest task among the suspended tasks is queued and sent to the virtual node. When the amount of resources required for the task received by the virtual node is greater than the third preset amount of resources, it is determined that the virtual node task has been sent, and the smallest task in the queue is sent to the next virtual node until the suspended task is sent.

[0017] In a possible design, after applying for N virtual nodes from the k8s base, the method further includes:

[0018] If a second message sent by the k8s base is received, the second message is used to indicate that the virtual node is successfully created, and the second message also includes the address information of the virtual node;

[0019] Configuration is performed according to the address information of the virtual node to establish a transmission channel.

[0020] In one possible design, the method further includes:

[0021] Monitor the available resource amount of the working node, and if the available resource amount of the working node is greater than the first preset resource amount, mark the working node as an idle working node, and the idle working node is used to receive and execute the sent tasks.

[0022] In a second aspect, the present application provides a resource allocation system based on k8s, including:

[0023] Spark software node, deployed in the k8s master node, is used to receive spark tasks;

[0024] The k8s master node is used to execute the above-mentioned k8s-based resource allocation method;

[0025] Worker node, used to receive and execute tasks issued by the k8s master node;

[0026] A k8s base, used to create a virtual node according to a first message sent by the k8s master node;

[0027] Also used to send a second message to the k8s master node, the second message is used to indicate that the new virtual node is successfully created, and the second message also includes address information of the virtual node;

[0028] The virtual node is used to receive and execute tasks issued by the k8s master node.

[0029] In a third aspect, the present application provides a resource allocation device based on k8s, the device comprising:

[0030] The first module is used to receive a spark task, split the spark task according to a first preset resource amount and send it to a preset number of working nodes, and obtain a suspended task; wherein the first preset resource amount is the maximum resource amount that the working node can accept;

[0031] The second module is used to obtain the number of nodes N according to the required resources of the suspended task, apply for N virtual nodes from the k8s base, and send the suspended task to the virtual node after the application is successful.

[0032] In a fourth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0033] The memory stores computer-executable instructions;

[0034] The processor executes the computer-executable instructions stored in the memory to implement the above-mentioned k8s-based resource allocation method.

[0035] In a fifth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above-mentioned k8s-based resource allocation method.

[0036] The k8s-based resource allocation method, system, device and storage medium provided in this application deploy spark software on k8s master. After k8s master allocates spark tasks to working nodes, suspended tasks are dynamically allocated to virtual nodes to complete the tasks. Users do not need to estimate resource usage in advance, nor do they need to apply for resources according to the maximum resource amount, thereby improving resource utilization and reducing the economic cost of establishing a cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 Schematic diagram of the process of resource allocation method based on k8s provided in the embodiment of the present application Figure 1 ;

[0039] Figure 2 Schematic diagram of the process of resource allocation method based on k8s provided in the embodiment of the present application Figure 2 ;

[0040] Figure 3 A schematic diagram of the structure of a resource allocation system based on k8s provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of the structure of a resource allocation device based on k8s provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims, rather than all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0045] It should be noted that the terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example.

[0046] It should be noted that the “at…” in the embodiments of the present application may be the instant when a certain situation occurs, or may be a period of time after the occurrence of a certain situation, and the embodiments of the present application do not specifically limit this.

[0047] First, the terms in this application are explained:

[0048] Kubernetes: k8s for short, a distributed architecture solution based on container technology, an open source container cluster management system, a one-stop complete distributed system development and support platform, without any intrusion to existing programming languages, programming frameworks, and middleware. It provides comprehensive management tools that cover all aspects including development, deployment testing, operation and maintenance monitoring. k8s master is the control node in Kubernetes.

[0049] This application deploys the spark software on the k8s master node, allocates spark tasks to the working nodes through the k8s master node, and then replaces the large working nodes with small virtual nodes for suspended tasks, and dynamically allocates them to the virtual nodes to complete the tasks. Users do not need to estimate resource usage in advance, nor do they need to apply for resources according to the maximum resource amount, thereby improving resource utilization and reducing the economic cost of establishing a cluster.

[0050] The k8s-based resource allocation method provided in the embodiment of the present application is introduced in detail below in conjunction with the accompanying drawings.

[0051] Figure 1 Schematic diagram of the process of resource allocation method based on k8s provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the spark software is deployed on the k8s master. The methods include:

[0052] S101, receiving a spark task, splitting the spark task according to a first preset resource amount and sending it to a preset number of working nodes, and obtaining a suspended task; wherein the first preset resource amount is a maximum resource amount that the working node can accept;

[0053] Specifically, before receiving the spark task, receiving the preset number sent by the user, creating the preset number of working nodes; obtaining the available resource amounts of the multiple working nodes as the first preset resource amount;

[0054] Specifically, the spark tasks are split and distributed to the respective working nodes according to the first preset resource amount. After the working nodes are allocated, the remaining tasks are suspended as suspended tasks.

[0055] S102. Obtain the number of nodes N according to the required resources of the suspended task, apply for N virtual nodes from the k8s base, and send the suspended task to the virtual node after the application is successful.

[0056] Specifically, based on the principle of distributing the suspended tasks to the virtual nodes as evenly as possible according to the required amount of resources, the number of nodes N is obtained and a virtual node application is initiated to the k8s base. After the application is completed, the suspended tasks are sent to the N virtual nodes.

[0057] The embodiment of the present application provides a method, by deploying spark software on k8s master, after k8s master assigns spark tasks to working nodes, for suspended tasks, dynamically assigns them to virtual nodes to complete the tasks, the user does not need to estimate resource usage in advance, nor does he need to apply for resources according to the maximum resource amount, thereby improving resource utilization and reducing the economic cost of establishing a cluster.

[0058] Figure 2 Schematic diagram of the process of resource allocation method based on k8s provided in the embodiment of the present application Figure 2 ;like Figure 2 As shown, for the above k8s master, the method includes:

[0059] S201, splitting the spark task into the minimum tasks that can be completed by a single node, sending the minimum tasks to a preset number of working nodes according to the queue, and treating one or more minimum tasks that have not been sent as pending tasks;

[0060] Specifically, a work node queue of a preset number of work nodes is established, and the minimum task is assigned to the first work node in the work node queue according to the order of the minimum task queue; and at the same time, the required resource amount of the accepted task returned by the work node is monitored;

[0061] Specifically, if when assigned to a minimum task, the amount of resources required for the working node to accept the task is greater than the first preset amount of resources, it is determined that the task of the working node has been issued; the minimum task is issued to the next working node in the queue; and so on until the tasks of all working nodes have been issued; thereafter, the tasks that have not been issued in the minimum task queue are suspended as suspended tasks.

[0062] Furthermore, the working nodes that can accept tasks are marked as idle working nodes;

[0063] Furthermore, the method of this embodiment also includes: monitoring the available resource amount of the working node, if the available resource amount of the working node is greater than the first preset resource amount, marking the working node as an idle working node to facilitate the completion of the next spark task.

[0064] S202. Obtain the number of nodes N according to the required amount of resources for the suspended task, and send a first message to the k8s base, where the first message is used to indicate the addition of N virtual nodes whose available resources are greater than a third preset amount of resources.

[0065] Specifically, the ratio of the required amount of resources for the suspended task to the second preset amount of resources is obtained, the ratio is rounded to obtain N, where N is the required number of virtual nodes, and a first message is sent to the k8s base; the first message is used to instruct the k8s base to create N virtual nodes whose available resources are not less than the third preset amount of resources;

[0066] Furthermore, the second preset resource amount is smaller than the third preset resource amount, that is, the required resource amount of the suspended task is evenly distributed according to the second preset resource amount to obtain the number of virtual nodes, and when the task is actually issued, the upper limit of the resource amount allocated to each virtual node is the third preset resource amount; this is because in the actual operation process, due to the different resource amounts required for the minimum task, it is impossible to truly achieve an even distribution, so the second preset resource amount is set to be smaller than the third preset resource amount to avoid the failure of the suspended task due to allocation errors and the total amount of available resources of the virtual node being less than the required amount of resources to complete the suspended task.

[0067] In some embodiments, the third preset resource amount is the available resource amount of the virtual node, and the third preset resource amount is 105% to 110% times the second preset resource amount.

[0068] In response, the k8s base creates N virtual nodes whose available resources are not less than the third preset resource amount and sends a second message to the k8s master, where the second message is used to indicate that the virtual node is successfully established and also includes the address / interface information of the virtual node.

[0069] Furthermore, the virtual node is implemented through virtual kubelet, which is used to use other ordinary servers as working nodes to complete suspended tasks; after the task is completed, the server on the virtual node is deleted to make it a normal server again.

[0070] Furthermore, if the available resources of the virtual node are less than those of the working node, the virtual node will run a smaller amount of tasks and a higher degree of concurrency;

[0071] In some embodiments, the third preset resource amount is no greater than 60% of the first preset resource amount.

[0072] S203. Receive the second message sent by the k8s base, and configure the virtual node to establish a transmission channel with the virtual node.

[0073] Specifically, the second message is used to indicate that the virtual node is successfully established. The second message also includes address / interface information of the virtual node. Configuration is performed according to the address information of the virtual node to establish a transmission channel.

[0074] S204. The smallest task in the suspended tasks is queued and sent to the virtual node. When the amount of resources required for the task received by the virtual node is greater than the third preset amount of resources, it is determined that the virtual node task has been sent, and the smallest task in the queue is sent to the next virtual node until the suspended task is sent.

[0075] Specifically, for suspended tasks, according to the queue information of the above-mentioned minimum task, continue to allocate the minimum tasks to the virtual nodes, and allocate the minimum tasks to one of the virtual nodes in the order of the minimum task queue; at the same time, monitor the required amount of resources for the accepted tasks returned by the virtual node. When a certain minimum task is allocated, if the required amount of resources for the virtual node to accept the task is greater than the third preset amount of resources, it is determined that the task of the virtual node has been issued; issue the minimum task to the next virtual node in the random queue; and so on until the suspended tasks are issued.

[0076] Furthermore, the k8s master is also used to monitor the virtual node. If the virtual node indicates that the task is completed, the virtual node is released;

[0077] Further, in response, the virtual node receives and executes the sent task.

[0078] For example, a Spark task that originally required 100 fixed working nodes with 64 cores and 256G memory is replaced with 50 fixed working nodes, and the remaining ones are replaced with small-sized virtual nodes, such as 8 cores, 64G, etc., and they are released immediately after use; using the resource allocation method of this application. The user's cost will be reduced by an average of 35% to 40%, and because the virtual node specifications are small, the tasks originally run by 10 fixed nodes may be split into 40 virtual nodes. This distributed running method improves concurrency, so the overall performance of user tasks is also improved by an average of 15% to 20%.

[0079] The present application provides a method that can achieve the following technical effects:

[0080] Use small-scale virtual nodes to replace large-scale fixed working nodes to reduce machine costs.

[0081] This embodiment can establish an appropriate number of virtual nodes to complete the suspended task according to the amount of resources required for the suspended task. The user does not need to estimate the resource usage in advance, nor does he need to apply for resources according to the maximum resource amount, eliminating the work of manual expansion and reduction of the operation and maintenance personnel.

[0082] After allocating spark tasks to working nodes through the k8s master node, the unassigned tasks are suspended. For suspended tasks, small-sized virtual nodes are used to replace large-sized fixed nodes, and they are dynamically allocated to virtual nodes to complete the tasks, thereby improving task completion efficiency and resource utilization.

[0083] Figure 3 A schematic diagram of the structure of a resource allocation system based on k8s provided in an embodiment of the present application; Figure 3 As shown, the system 30 includes:

[0084] Spark software 301 is deployed in the k8s master node to receive spark tasks;

[0085] The k8s master node 302 is used to execute the above-mentioned k8s-based resource allocation method;

[0086] Working node 303, used to receive and execute tasks issued by the k8s master node;

[0087] k8s base 304, used to create a virtual node according to the first message sent by the k8s master node;

[0088] It is also used to send a second message to the k8s master node, the second message is used to indicate that the new virtual node is successfully created, and the second message also includes the address information of the virtual node;

[0089] Virtual node 305 is used to receive and execute tasks issued by the k8s master node.

[0090] The k8s-based resource allocation system provided in this embodiment can execute the above-mentioned k8s-based resource allocation method, and its implementation principle and technical effects are similar, which will not be repeated in this embodiment.

[0091] The embodiment of the present invention can divide the electronic device or the main control device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0092] Figure 4 A schematic diagram of a k8s-based resource allocation device provided in an embodiment of the present application; Figure 4 , the device 40 comprises:

[0093] The first module 401 is used to receive a spark task, split the spark task according to a first preset resource amount and send it to a preset number of working nodes, and obtain a suspended task; wherein the first preset resource amount is a maximum resource amount that the working node can accept;

[0094] The second module 402 is used to obtain the number of nodes N according to the required amount of resources for the suspended task, apply for N virtual nodes from the k8s base, and send the suspended task to the virtual node after the application is successful.

[0095] Furthermore, the first module 401 is specifically used for:

[0096] Split the spark task into the smallest task that can be completed by a single node, and send the smallest task to the working node in a queue;

[0097] When the amount of resources required for the task received by the working node is greater than the first preset amount of resources, it is determined that the task of the working node is completed, and the minimum task in the queue is sent to the next working node until all working nodes have completed the sending;

[0098] Suspend the unsent tasks in the queue as suspended tasks.

[0099] Furthermore, the second module 402 is specifically used for:

[0100] The number of nodes N is obtained by rounding the ratio of the required resource amount of the suspended task to the second preset resource amount, and a first message is sent to the k8s base. The first message is used to indicate the addition of N virtual nodes whose available resource amount is not less than the third preset resource amount, wherein the third preset resource amount is greater than the second preset resource amount.

[0101] Furthermore, the second module 402 is further configured to:

[0102] The smallest task in the suspended tasks is queued and sent to the virtual node. When the amount of resources required for the task received by the virtual node is greater than the third preset amount of resources, it is determined that the virtual node task has been sent, and the smallest task in the queue is sent to the next virtual node until the suspended task is sent.

[0103] Furthermore, the second module 402 is further configured to:

[0104] If a second message sent by the k8s base is received, the second message is used to indicate that the new virtual node is successfully created, and the second message also includes the address information of the virtual node;

[0105] The transmission channel is established by configuring according to the address information of the virtual node.

[0106] Furthermore, the second module 402 is further configured to:

[0107] Monitor the available resources of the working node. If the available resources of the working node are greater than the first preset resource amount, mark the working node as an idle working node. The idle working node is used to receive and execute the sent tasks.

[0108] The k8s-based resource allocation device provided in this embodiment can execute the above-mentioned k8s-based resource allocation method, and its implementation principle and technical effects are similar, which will not be repeated in this embodiment.

[0109] In the aforementioned specific implementation of a k8s-based resource allocation device, each module can be implemented as a processor, and the processor can execute computer execution instructions stored in the memory, so that the processor executes the above-mentioned k8s-based resource allocation method.

[0110] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 50 includes: at least one processor 501 and a memory 502. The electronic device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0111] During the specific implementation process, at least one processor 501 executes the computer execution instructions stored in the memory 502, so that at least one processor 501 executes a k8s-based resource allocation method as executed on the electronic device side above.

[0112] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0113] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0114] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0115] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0116] The above-mentioned functions implemented by the electronic device and the main control device introduce the scheme provided by the embodiment of the present invention. It can be understood that in order to implement the above-mentioned functions, the electronic device or the main control device includes a hardware structure and / or software module corresponding to the execution of each function. In combination with the units and algorithm steps of each example described in the embodiment disclosed in the embodiment of the present invention, the embodiment of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present invention.

[0117] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned k8s-based resource allocation method is implemented.

[0118] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0119] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in an electronic device or a main control device.

[0120] The present application also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0121] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource allocation method based on k8s, characterized in that: Used for a k8s master node, in which the spark software is loaded, the method includes: Receive a spark task, split the spark task according to a first preset resource amount and send it to a preset number of working nodes, and obtain a suspended task; wherein the first preset resource amount is a maximum resource amount that the working node can accept; The number of nodes N is obtained according to the required resources of the suspended task, and N virtual nodes are applied to the k8s base. After the application is successful, the suspended task is sent to the virtual node.

2. The method according to claim 1, characterized in that The step of splitting the spark task according to the first preset resource amount and sending the task to a preset number of working nodes to obtain the suspended task includes: Split the spark task into the smallest task that can be completed by a single node, and send the smallest task to the working node in a queue; When the amount of resources required for the task received by the working node is greater than the first preset amount of resources, it is determined that the task of the working node has been sent, and the minimum task in the queue is sent to the next working node until all the working nodes have completed the sending; The unissued tasks in the queue are suspended as suspended tasks.

3. The method according to claim 1, characterized in that The step of obtaining the number of nodes N according to the required amount of resources for the suspended task and applying for N virtual nodes from the k8s base includes: The number of nodes N is obtained by rounding the ratio of the required resource amount of the suspended task to the second preset resource amount, and a first message is sent to the k8s base, wherein the first message is used to indicate the addition of N virtual nodes whose available resource amount is not less than the third preset resource amount, wherein the third preset resource amount is greater than the second preset resource amount.

4. The method according to claim 3, characterized in that The sending the suspended task to the virtual node includes: The smallest task among the suspended tasks is queued and sent to the virtual node. When the amount of resources required for the task received by the virtual node is greater than the third preset amount of resources, it is determined that the virtual node task has been sent, and the smallest task in the queue is sent to the next virtual node until the suspended task is sent.

5. The method according to claim 1, characterized in that After applying for N virtual nodes from the k8s base, the method further includes: If a second message sent by the k8s base is received, the second message is used to indicate that the virtual node is successfully created, and the second message also includes the address information of the virtual node; Configuration is performed according to the address information of the virtual node to establish a transmission channel.

6. The method according to claim 1, characterized in that The method further comprises: Monitor the available resource amount of the working node, and if the available resource amount of the working node is greater than the first preset resource amount, mark the working node as an idle working node, and the idle working node is used to receive and execute the sent tasks.

7. A resource allocation system based on k8s, characterized in that: include: Spark software is deployed in the k8s master node to receive spark tasks; The k8s master node is used to execute the method according to any one of claims 1 to 6; Worker node, used to receive and execute tasks issued by the k8s master node; A k8s base, used to create a virtual node according to a first message sent by the k8s master node; Also used to send a second message to the k8s master node, the second message is used to indicate that the new virtual node is successfully created, and the second message also includes address information of the virtual node; The virtual node is used to receive and execute tasks issued by the k8s master node.

8. A resource allocation device based on k8s, characterized in that: include: The first module is used to receive a spark task, split the spark task according to a first preset resource amount and send it to a preset number of working nodes, and obtain a suspended task; wherein the first preset resource amount is the maximum resource amount that the working node can accept; The second module is used to obtain the number of nodes N according to the required resources of the suspended task, apply for N virtual nodes from the k8s base, and send the suspended task to the virtual node after the application is successful.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

Images