Presto scheduling-based elastic scaling method and elastic scaling device

Abstract

The application provides a Presto scheduling-based elastic scaling method and device. The elastic scaling method comprises the following steps: in the case that a query request needs to be independently run and is a core service, obtaining a target performance index of an initial cluster and target query information of the query request, wherein the target performance index at least comprises memory usage and CPU occupancy of the initial cluster, and the target query information at least comprises metadata information and context information; and performing elastic scaling on the initial cluster according to at least the target performance index and the target query information. The application realizes dynamic arrangement and control of resources, alleviates the queuing problem caused by resource competition, avoids the problem that the number of PODs is uncontrollable caused by the elastic scaling taking only the CPU and memory usage (i.e. the target performance index corresponding to the initial cluster) as the monitoring index, and achieves the purpose of improving the stability and resource utilization of the service.

Description

Technical Field

[0001] This application relates to the field of container cloud technology, and more specifically, to a Presto-based elastic scaling method, elastic scaling device, and computer-readable storage medium. Background Technology

[0002] Current compute and storage separation deployment schemes are based on a combination of cloud computing and big data. However, separating Presto compute from HDFS (Hadoop Distributed File System) storage leads to the failure of Presto's local scheduling optimization strategies. Since the monitoring metrics for elastic scaling in Kubernetes container clusters are CPU and memory usage, newly added PODs may be overwhelmed by Presto's compute tasks, consuming all available CPU and memory. If the Kubernetes cluster continues to expand, this can result in an uncontrollable number of PODs.

[0003] Therefore, there is an urgent need for a method that can avoid the uncontrollable number of PODs caused by elastic scaling based on CPU and memory usage as monitoring indicators. Summary of the Invention

[0004] The main objective of this application is to provide a Presto-based elastic scaling method, elastic scaling device, and computer-readable storage medium to solve the problem of uncontrollable POD quantity caused by elastic scaling based on CPU and memory usage as monitoring indicators in the prior art.

[0005] According to one aspect of the present invention, a Presto-based elastic scaling method is provided, comprising: when a query request needs to run independently and is a core business, obtaining target performance metrics of an initial cluster and target query information of the query request, wherein the initial cluster includes multiple PODs, the target performance metrics include at least the memory utilization and CPU utilization of the initial cluster, and the target query information includes at least metadata information and context information; and elastically scaling the initial cluster based at least on the target performance metrics and the target query information.

[0006] Optionally, the initial cluster is elastically scaled based at least on the target performance metrics and the target query information, including: determining the target number of PODs based on the target performance metrics and the target query information, wherein the target number of PODs is the number of PODs to be expanded from the initial cluster; determining the target memory space required by the PODs to expand the target number of PODs based on the target number of PODs; and elastically scaling the initial cluster based on the HDFS address corresponding to the query request and the target memory space.

[0007] Optionally, the initial cluster is elastically scaled based on the HDFS address corresponding to the query request and the target memory space, including: splitting the HDFS address to obtain multiple HDFS sub-addresses; traversing the multiple HDFS sub-addresses to determine the remaining memory space corresponding to each HDFS sub-address; if there is a remaining memory space greater than or equal to the target memory space, popping the target number of PODs from the corresponding HDFS sub-address; if all remaining memory spaces are less than the target memory space, popping the target number of PODs from the target cluster, wherein the target cluster is the cluster where the initially reserved spare PODs are located.

[0008] Optionally, after elastically scaling the initial cluster based at least on the target performance metric and the target query information, the elastic scaling method includes: adding the target number of PODs to a candidate queue, and determining whether the number of tasks processed by each POD in the candidate queue reaches a target value, wherein the target value is an upper limit on the number of tasks processed by a target POD; if the number of tasks processed by a target POD does not reach the target value, submitting unassigned tasks in the task queue to the target POD, wherein the target POD is one of a plurality of PODs, and the unassigned tasks are unassigned tasks corresponding to the query request; if the number of tasks run by each POD reaches the target value, adding the unassigned tasks to the task queue.

[0009] Optionally, the process of determining whether the query request runs independently includes: determining the priority of the query request; determining that the query request runs independently if the priority is higher than the target level and / or the amount of data corresponding to the query request is greater than or equal to the target quantity; and determining that the query request does not run independently if the priority is lower than the target level and / or the amount of data corresponding to the query request is less than the target quantity.

[0010] Optionally, after determining that the query request will not run independently when the priority is lower than the target level and / or the amount of data corresponding to the query request is less than the target number, the elastic scaling method further includes: determining whether the target POD in the candidate queue has reached the target value, wherein the target POD is one of the multiple PODs, and the target value is the upper limit of the number of tasks processed by a target POD; if the number of tasks processed by the target POD has not reached the target value, submitting unassigned tasks in the task queue to the target POD, wherein the unassigned tasks are the unassigned tasks corresponding to the query request; and if the number of tasks run by each POD has reached the target value, adding the unassigned tasks to the arriving task queue.

[0011] Optionally, when the query request needs to run independently and is a core business, before obtaining the target performance indicators of the initial cluster and the target query information of the query request, the elastic scaling method further includes: determining whether to enable a forced scheduling policy, wherein the forced scheduling policy is used to characterize whether to perform elastic scaling based on the target performance indicators and the target query information; if the forced scheduling policy is not enabled, determining the HDFS address corresponding to the query request; determining the corresponding target rack location based on the HDFS address, and adding the POD corresponding to the target rack location to the candidate queue.

[0012] Optionally, the elastic scaling method further includes: sending the query result to the client after all tasks corresponding to the query request have been completed, wherein the query result is the query result corresponding to the query request.

[0013] According to another aspect of the present invention, an elastic scaling device based on Presto scheduling is also provided, comprising: an acquisition unit, configured to acquire target performance indicators of an initial cluster and target query information of the query request when the query request needs to run independently and is a core business, wherein the initial cluster includes multiple PODs, the target performance indicators include at least the memory utilization rate and CPU utilization rate of the initial cluster, and the target query information includes at least metadata information and context information; and an elastic scaling unit, configured to elastically scale the initial cluster based at least on the target performance indicators and the target query information.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes any of the described Presto-based elastic scaling methods.

[0015] In this embodiment of the invention, the Presto-based elastic scaling method, when a received query request needs to run independently and is a core business function, obtains the target performance indicators of the initial cluster corresponding to the query request and the target query information corresponding to the query request. Then, elastic scaling is performed on the initial cluster based at least on the target performance indicators and the target query information. Compared with the prior art method that elastically scales the initial cluster based only on the target performance indicators corresponding to the initial cluster, the elastic scaling method of this application elastically scales the initial cluster based at least on the target performance indicators corresponding to the initial cluster and the target query information corresponding to the query request. That is, it realizes elastic scaling of the initial cluster based on the target performance indicators corresponding to the initial cluster and the custom elastic scaling indicators (i.e., the context information and metadata information corresponding to the query request). This achieves dynamic orchestration and control of resources, alleviates the waiting and queuing problem caused by resource contention, and avoids the problem of uncontrollable POD number caused by elastic scaling based only on CPU and memory usage (i.e., the target performance indicators corresponding to the initial cluster) as monitoring indicators, thereby improving service stability and resource utilization. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A flowchart illustrating an embodiment of the elastic scaling method based on Presto scheduling of this application is shown;

[0018] Figure 2 An architecture diagram of a containerized deployment scheme for computing and storage according to an embodiment of this application is shown;

[0019] Figure 3 An architecture diagram of an elastic scaling technology solution based on Presto scheduling according to an embodiment of this application is shown;

[0020] Figure 4 A timing diagram of a Presto-based elastic scaling technology solution according to an embodiment of this application is shown;

[0021] Figure 5 A schematic diagram of the structure of a Presto-based elastic scaling device according to an embodiment of this application is shown;

[0022] Figure 6 A flowchart illustrating an embodiment of the elastic scaling scheme based on Presto scheduling of this application is shown. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0027] Presto: A distributed SQL query engine for big data developed in Java;

[0028] Coordinator: A component of Presto, responsible for parsing SQL statements, generating execution plans, and distributing execution tasks to Worker nodes for execution;

[0029] Worker: A component of Presto, responsible for actually executing query tasks and interacting with HDFS to read data;

[0030] Operator: A framework for wrapping, running, and managing Kubernetes applications, which can acquire or manage resources such as cluster nodes, tags, and services;

[0031] Client-go is a library in the Kubernetes cluster that provides services for extension and customization. It can also be used to manage resource objects in the cluster.

[0032] As mentioned in the background section, the number of PODs caused by elastic scaling based on CPU and memory usage as monitoring indicators in the prior art is uncontrollable. In order to solve the above problems, a typical embodiment of this application provides an elastic scaling method, elastic scaling device and computer-readable storage medium based on Presto scheduling.

[0033] According to an embodiment of this application, a Presto-based elastic scaling method is provided.

[0034] Figure 1 This is a flowchart of a Presto-based elastic scaling method according to an embodiment of this application. Figure 1 As shown, the elastic stretching method includes the following steps:

[0035] Step S101: When the query request needs to run independently and is a core business, obtain the target performance indicators of the initial cluster and the target query information of the query request. The initial cluster includes multiple PODs, the target performance indicators include at least the memory usage and CPU utilization of the initial cluster, and the target query information includes at least metadata information and context information.

[0036] Step S102: Based at least on the aforementioned target performance indicators and the aforementioned target query information, perform elastic scaling on the aforementioned initial cluster.

[0037] In the aforementioned Presto-based elastic scaling method, when a received query request needs to run independently and is a core business function, the target performance metrics of the initial cluster corresponding to the query request and the target query information corresponding to the query request are obtained. Then, elastic scaling of the initial cluster is performed based at least on the target performance metrics and target query information of the initial cluster. Compared to existing methods that only rely on the target performance metrics of the initial cluster for elastic scaling, the elastic scaling method of this application performs elastic scaling based at least on the target performance metrics of the initial cluster and the target query information corresponding to the query request. This achieves elastic scaling of the initial cluster based on the target performance metrics of the initial cluster and custom elastic scaling metrics (i.e., the context information and metadata information corresponding to the query request). This enables dynamic orchestration and control of resources, alleviates queuing problems caused by resource contention, and avoids the uncontrollable number of PODs caused by elastic scaling based solely on CPU and memory usage (i.e., the target performance metrics of the initial cluster) as monitoring metrics. This ultimately improves service stability and resource utilization.

[0038] In practical applications, the aforementioned metadata information can be data describing the object, including descriptive information about the object's attributes, operations, and relationships. The aforementioned context information can be the storage location of the query information corresponding to the query request.

[0039] Specifically, the initial cluster mentioned above is the cluster initialized to handle query requests when a query request is received from a client. This initial cluster includes multiple PODs, where a POD is the smallest resource management component in a Kubernetes cluster and also the smallest resource object for running containerized applications. One POD represents a process running in the cluster. A POD may also include Presto Workers.

[0040] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0041] In one embodiment of this application, the initial cluster is elastically scaled based at least on the aforementioned target performance metrics and target query information, including: determining the target number of PODs based on the target performance metrics and target query information, wherein the target number of PODs is the number of PODs used to expand the initial cluster; determining the target memory space required for the PODs to expand the target number of PODs based on the target number of PODs; and elastically scaling the initial cluster based on the HDFS address corresponding to the query request and the target memory space. In this embodiment, determining the target number of PODs based on the target performance metrics of the initial cluster and the target query information corresponding to the query request ensures that the determined target number of PODs is relatively accurate. Furthermore, based on the target memory space occupied by the PODs within the target number of PODs and the HDFS address corresponding to the query request, the creation location of the PODs can be determined relatively easily, achieving a more reasonable elastic scaling of the initial cluster.

[0042] To further improve the utilization of cluster resources, in another embodiment of this application, the initial cluster is elastically scaled according to the HDFS address corresponding to the query request and the target memory space, including: splitting the HDFS address to obtain multiple HDFS sub-addresses; traversing the multiple HDFS sub-addresses to determine the remaining memory space corresponding to each HDFS sub-address; if there is a remaining memory space greater than or equal to the target memory space, popping the target number of PODs from the corresponding HDFS sub-address; if all the remaining memory spaces are less than the target memory space, popping the target number of PODs from the target cluster, wherein the target cluster is the cluster where the initially reserved spare PODs are located.

[0043] In another embodiment of this application, after elastically scaling the initial cluster based at least on the target performance indicators and the target query information, the elastic scaling method includes: adding the target number of PODs to a candidate queue, and determining whether the number of tasks processed by each POD in the candidate queue reaches a target value, wherein the target value is an upper limit for the number of tasks processed by a target POD; if the number of tasks processed by a target POD does not reach the target value, submitting unassigned tasks in the task queue to the target POD, wherein the target POD is one of a plurality of PODs, and the unassigned tasks are unassigned tasks corresponding to the query request; if the number of tasks run by each POD reaches the target value, adding the unassigned tasks to the task queue. Specifically, if the number of tasks processed by the target POD in the candidate queue has not reached its upper limit, the unassigned tasks are submitted to the target POD so that they can be processed relatively quickly. If the number of tasks processed by each POD in the candidate queue has reached the target value, the unassigned tasks are added to the task queue to wait for processing. This can further alleviate the queuing problem of unassigned tasks caused by resource contention and further ensure the stability of service operation.

[0044] In one specific embodiment of this application, the assigned unassigned tasks can be processed based on the corresponding Workers on each POD.

[0045] To more easily determine whether a query request needs to be run independently, in another embodiment of this application, the process of determining whether the query request should be run independently includes: determining the priority of the query request; if the priority is higher than the target level and / or the amount of data corresponding to the query request is greater than or equal to the target quantity, determining that the query request should be run independently; if the priority is lower than the target level and / or the amount of data corresponding to the query request is less than the target quantity, determining that the query request should not be run independently.

[0046] In practical applications, the priority of a query request can also be interpreted as the business timeliness of the corresponding query request, i.e., the urgency of the corresponding query request.

[0047] To further improve resource utilization and alleviate queuing issues caused by resource competition, in one embodiment of this application, after determining that the query request will not run independently when the priority is lower than the target level and / or the amount of data corresponding to the query request is less than the target number, the elastic scaling method further includes: determining whether the target POD in the candidate queue has reached the target value, wherein the target POD is one of a plurality of PODs, and the target value is an upper limit on the number of tasks processed by one target POD; if the number of tasks processed by the target POD has not reached the target value, submitting unassigned tasks in the task queue to the target POD, wherein the unassigned tasks are the unassigned tasks corresponding to the query request; and if the number of tasks run by each POD has reached the target value, adding the unassigned tasks to the task queue.

[0048] In another embodiment of this application, when the query request needs to run independently and is a core business, before obtaining the target performance indicators of the initial cluster and the target query information of the query request, the elastic scaling method further includes: determining whether to enable a forced scheduling policy, wherein the forced scheduling policy is used to characterize whether to perform elastic scaling based on the target performance indicators and the target query information; if the forced scheduling policy is not enabled, determining the HDFS address corresponding to the query request; determining the corresponding target rack location based on the HDFS address, and adding the POD corresponding to the target rack location to the candidate queue. In this embodiment, when the forced scheduling policy is not enabled, determining the POD corresponding to the target rack location based on the HDFS address corresponding to the query request and adding the corresponding POD to the candidate queue achieves more flexible service deployment and ensures lower costs.

[0049] In another embodiment of this application, the above-mentioned elastic scaling method further includes: after all the tasks corresponding to the above-mentioned query requests have been completed, sending the query results to the client, wherein the query results are the query results corresponding to the above-mentioned query requests. This ensures that the client can receive the query results corresponding to the query requests in a timely manner, thus ensuring a better user experience.

[0050] In one specific embodiment of this application, such as Figure 2 As shown, this application also provides a containerized deployment scheme for computing and storage. Specifically, this application utilizes container technology to achieve online deployment of images and orchestration scripts. It mainly involves containerized deployment of a Presto-based computing layer, a Hadoop component-based read-write separation layer, and a containerized cloud and IaaS (Infrastructure as a Service)-based storage separation layer. Specifically, the Presto-based computing layer consists of a Presto hybrid real-time data analysis cluster, a Presto data quality analysis cluster, a Presto elastic business dedicated cluster, and a Presto elastic business computing dedicated cluster, with reserved space for elastic scaling for Pod expansion. The Hadoop component-based read-write separation layer extracts and encapsulates software functions from big data, uses containerized management software for cluster deployment design, and configures Hadoop read-write separation, delegating read-write consistency issues to big data and providing read-write services. Storage separation layer based on container cloud and IaaS: The storage separation design provides storage for upper-layer services through Persistent Volumes, shielding the differences between different storage solutions.

[0051] In another specific embodiment of this application, such as Figure 3 As shown, a Presto-based elastic scaling technology solution is provided. The specific implementation steps are as follows:

[0052] S1: The SQL query statement corresponding to the client's query request is parsed by the Coordinator in Presto, and the target query information corresponding to the query request, namely Query Context and Query Metadata, is sent to Client Go through the Restful API interface;

[0053] S2: Client Go uses the Kubernetes cluster's api-server to obtain the initial cluster's target performance metrics (including memory usage and CPU utilization) from ETCD. Based on these metrics and the target query information provided by the Coordinator, it manages the number of PODs in the initial cluster.

[0054] S3: Based on the K8S cluster's Client go, elastically scale the initial cluster and send the query results to the client.

[0055] In one specific embodiment of this application, such as Figure 4 As shown, Figure 3 The corresponding elastic scaling technology solution's sequence diagram is shown. Specifically, Client Go obtains the initial cluster's target performance metrics from ETCD, as well as context and metadata information provided by Presto (Coordinator), such as the number of query requests currently running and queued in the Presto cluster. The specific steps are as follows:

[0056] S1: Presto interacts with the K8S Client go, providing elastic scaling metric data customized through the Operator, i.e., the target query information corresponding to the query request, and sending POD scaling requests to the Client go;

[0057] S2: Client Go determines the target number of PODs to be expanded based on the target performance metrics of the initial cluster and the target query information corresponding to the query request. Then, it sends the expansion request with the target number of PODs to the api-server corresponding to the initial cluster so that the api-server can create PODs.

[0058] S3: After receiving a scaling request with a target number of PODs, the api-server revises the ETCD and creates the PODs;

[0059] S4: After the api-server has created the target number of PODs, it returns a message to the Client go indicating that the PODs have been successfully created.

[0060] S5: After receiving the POD creation success message, the Client go service interacts with the api-server, creates a callback listener, and adds Prometheus monitoring metrics.

[0061] S6: The api-server registers a callback listener with the Webhook;

[0062] S7: Webhook monitoring of Prometheus metrics;

[0063] S8: The Webhook returns the completed scaling request to the api-server and Client go;

[0064] S9: After all the computation tasks in Presto have been completed, Client go sends a scaling-down request to reduce the number of PODs;

[0065] S10: Client go returns the current number of PODs to Presto.

[0066] This application also provides a Presto-based elastic scaling device. It should be noted that the Presto-based elastic scaling device of this application can be used to execute the Presto-based elastic scaling method provided in this application. The following describes the Presto-based elastic scaling device provided in this application.

[0067] Figure 5 This is a schematic diagram of a Presto-based elastic scaling device according to an embodiment of this application. The above-described elastic scaling device is applied in ClientGo, such as... Figure 5 As shown, the elastic telescopic device includes:

[0068] The acquisition unit 10 is used to acquire the target performance indicators of the initial cluster and the target query information of the query request when the query request needs to run independently and is a core business. The initial cluster includes multiple PODs, the target performance indicators include at least the memory usage and CPU utilization of the initial cluster, and the target query information includes at least metadata information and context information.

[0069] The elastic scaling unit 20 is used to elastically scale the initial cluster based at least on the target performance indicators and the target query information.

[0070] In the aforementioned Presto-based elastic scaling device, the acquisition unit is used to acquire the target performance indicators of the initial cluster corresponding to the received query request and the target query information corresponding to the query request when the received query request needs to run independently and is a core business. The elastic scaling unit is used to elastically scale the initial cluster based at least on the target performance indicators and target query information of the initial cluster. Compared with the prior art method of elastically scaling the initial cluster based only on the target performance indicators corresponding to the initial cluster, the elastic scaling device of this application elastically scales the initial cluster based at least on the target performance indicators corresponding to the initial cluster and the target query information corresponding to the query request. That is, it realizes elastic scaling of the initial cluster based on the target performance indicators corresponding to the initial cluster and the custom elastic scaling indicators (i.e., the context information and metadata information corresponding to the query request). This achieves dynamic orchestration and control of resources, alleviates the waiting and queuing problem caused by resource contention, and avoids the problem of uncontrollable number of PODs caused by elastic scaling based only on CPU and memory usage (i.e., the target performance indicators corresponding to the initial cluster) as monitoring indicators. This achieves the goal of improving service stability and resource utilization.

[0071] In practical applications, the aforementioned metadata information can be data describing the object, including descriptive information about the object's attributes, operations, and relationships. The aforementioned context information can be the storage location of the query information corresponding to the query request.

[0072] Specifically, the initial cluster mentioned above is the cluster initialized to handle query requests when a query request is received from a client. This initial cluster includes multiple PODs, where a POD is the smallest resource management component in a Kubernetes cluster and also the smallest resource object for running containerized applications. One POD represents a process running in the cluster. A POD may also include Presto Workers.

[0073] In one embodiment of this application, the elastic scaling unit includes a first determining module, a second determining module, and an elastic scaling module. The first determining module determines the target number of PODs based on the target performance indicators and the target query information, wherein the target number of PODs is the number of PODs used to expand the initial cluster. The second determining module determines the target memory space required for expanding the target number of PODs based on the target number of PODs. The elastic scaling module performs elastic scaling on the initial cluster based on the HDFS address corresponding to the query request and the target memory space. In this embodiment, determining the target number of PODs based on the target performance indicators of the initial cluster and the target query information corresponding to the query request ensures a relatively accurate determination of the target number of PODs. Furthermore, based on the target memory space occupied by the PODs within the target number of PODs and the HDFS address corresponding to the query request, the creation location of the PODs can be determined relatively easily, achieving a more reasonable elastic scaling of the initial cluster.

[0074] To further improve the utilization of cluster resources, in another embodiment of this application, the elastic scaling module includes a sub-splitting module, a traversal sub-module, a first pop-out sub-module, and a second pop-out sub-module. The sub-splitting module is used to split the HDFS address into multiple HDFS sub-addresses. The traversal sub-module is used to traverse the multiple HDFS sub-addresses to determine the remaining memory space corresponding to each HDFS sub-address. The first pop-out sub-module is used to pop the target number of PODs from the corresponding HDFS sub-address when there is a remaining memory space greater than or equal to the target memory space. The second pop-out sub-module is used to pop the target number of PODs from the target cluster when all remaining memory spaces are less than the target memory space. The target cluster is the cluster where the initially reserved spare PODs are located.

[0075] In another embodiment of this application, the above-mentioned elastic scaling method includes a first adjustment unit, a first submission unit, and a second adjustment unit. The first adjustment unit is used to add the target number of PODs to a candidate queue after elastically scaling the initial cluster based at least on the target performance indicators and the target query information, and to determine whether the number of tasks processed by each POD in the candidate queue reaches a target value, wherein the target value is an upper limit for the number of tasks processed by a target POD. The first submission unit is used to submit unassigned tasks in the task queue to the target POD if the number of tasks processed by the target POD does not reach the target value, wherein the target POD is one of a plurality of PODs, and the unassigned tasks are unassigned tasks corresponding to the query request. The second adjustment unit is used to add the unassigned tasks to the task queue if the number of tasks run by each POD reaches the target value. Specifically, if the number of tasks processed by the target POD in the candidate queue has not reached its upper limit, the unassigned tasks are submitted to the target POD so that they can be processed relatively quickly. If the number of tasks processed by each POD in the candidate queue has reached the target value, the unassigned tasks are added to the task queue to wait for processing. This can further alleviate the queuing problem of unassigned tasks caused by resource contention and further ensure the stability of service operation.

[0076] In one specific embodiment of this application, the assigned unassigned tasks can be processed based on the corresponding Workers on each POD.

[0077] To more easily determine whether a query request needs to be run independently, in another embodiment of this application, the acquisition unit includes a third determining module, a fourth determining module, and a fifth determining module. The third determining module is used to determine the priority of the query request; the fourth determining module is used to determine that the query request runs independently if the priority is higher than the target level and / or the amount of data corresponding to the query request is greater than or equal to the target quantity; and the fifth determining module is used to determine that the query request does not run independently if the priority is lower than the target level and / or the amount of data corresponding to the query request is less than the target quantity.

[0078] In practical applications, the priority of a query request can also be interpreted as the business timeliness of the corresponding query request, i.e., the urgency of the corresponding query request.

[0079] To further improve resource utilization and alleviate queuing issues caused by resource competition, in one embodiment of this application, the elastic scaling device further includes a first determining unit, a second submitting unit, and a third adjusting unit. The first determining unit is used to determine whether the target POD in the candidate queue has reached a target value after determining that the query request is not running independently, in cases where the priority is lower than the target level and / or the data volume corresponding to the query request is less than the target quantity. The target POD is one of a plurality of PODs, and the target value is an upper limit on the number of tasks processed by a single target POD. The second submitting unit is used to submit unassigned tasks in the task queue to the target POD if the number of tasks processed by the target POD has not reached the target value. The unassigned tasks are the unassigned tasks corresponding to the query request. The third adjusting unit is used to add the unassigned tasks to the task queue if the number of tasks run by each POD has reached the target value.

[0080] In another embodiment of this application, the aforementioned elastic scaling device further includes a second determining unit, a third determining unit, and a fourth adjusting unit. The second determining unit is used to determine whether to enable a forced scheduling strategy before obtaining the target performance indicators of the initial cluster and the target query information of the query request, when the query request needs to run independently and is a core business. The forced scheduling strategy characterizes whether to perform elastic scaling based on the target performance indicators and the target query information. The third determining unit is used to determine the HDFS address corresponding to the query request when the forced scheduling strategy is not enabled. The fourth adjusting unit is used to determine the corresponding target rack location based on the HDFS address and add the POD corresponding to the target rack location to the candidate queue. In this embodiment, when the forced scheduling strategy is not enabled, the POD corresponding to the target rack location is determined based on the HDFS address corresponding to the query request, and the corresponding POD is added to the candidate queue. This achieves more flexible service deployment and ensures lower costs.

[0081] In another embodiment of this application, the aforementioned elastic scaling device further includes a sending unit, configured to send the query results to the client after all tasks corresponding to the query request have been completed. The query results are the query results corresponding to the query request. This ensures that the client can receive the query results corresponding to the query request in a timely manner, guaranteeing a better user experience.

[0082] In one specific embodiment of this application, such as Figure 2As shown, this application also provides a containerized deployment scheme for computing and storage. Specifically, this application utilizes container technology to achieve online deployment of images and orchestration scripts. It mainly involves containerized deployment of a Presto-based computing layer, a Hadoop component-based read-write separation layer, and a containerized cloud and IaaS (Infrastructure as a Service)-based storage separation layer. Specifically, the Presto-based computing layer consists of a Presto hybrid real-time data analysis cluster, a Presto data quality analysis cluster, a Presto elastic business dedicated cluster, and a Presto elastic business computing dedicated cluster, with reserved space for elastic scaling for Pod expansion. The Hadoop component-based read-write separation layer extracts and encapsulates software functions from big data, uses containerized management software for cluster deployment design, and configures Hadoop read-write separation, delegating read-write consistency issues to big data and providing read-write services. Storage separation layer based on container cloud and IaaS: The storage separation design provides storage for upper-layer services through Persistent Volumes, shielding the differences between different storage solutions.

[0083] In another specific embodiment of this application, such as Figure 3 As shown, a Presto-based elastic scaling technology solution is provided. The specific implementation steps are as follows:

[0084] S1: The SQL query statement corresponding to the client's query request is parsed by the Coordinator in Presto, and the target query information corresponding to the query request, namely Query Context and Query Metadata, is sent to Client Go through the Restful API interface;

[0085] S2: Client Go uses the Kubernetes cluster's api-server to obtain the initial cluster's target performance metrics (including memory usage and CPU utilization) from ETCD. Based on these metrics and the target query information provided by the Coordinator, it manages the number of PODs in the initial cluster.

[0086] S3: Based on the K8S cluster's Client go, elastically scale the initial cluster and send the query results to the client.

[0087] In one specific embodiment of this application, such as Figure 4 As shown, Figure 3The corresponding elastic scaling technology solution's sequence diagram is shown. Specifically, Client Go obtains the initial cluster's target performance metrics from ETCD, as well as context and metadata information provided by Presto (Coordinator), such as the number of query requests currently running and queued in the Presto cluster. The specific steps are as follows:

[0088] S1: Presto interacts with the K8S Client go, providing elastic scaling metric data customized through the Operator, i.e., the target query information corresponding to the query request, and sending POD scaling requests to the Client go;

[0089] S2: Client Go determines the target number of PODs to be expanded based on the target performance metrics of the initial cluster and the target query information corresponding to the query request. Then, it sends the expansion request with the target number of PODs to the api-server corresponding to the initial cluster so that the api-server can create PODs.

[0090] S3: After receiving a scaling request with a target number of PODs, the api-server revises the ETCD and creates the PODs;

[0091] S4: After the api-server has created the target number of PODs, it returns a message to the Client go indicating that the PODs have been successfully created.

[0092] S5: After receiving the POD creation success message, the Client go service interacts with the api-server, creates a callback listener, and adds Prometheus monitoring metrics.

[0093] S6: The api-server registers a callback listener with the Webhook;

[0094] S7: Webhook monitoring of Prometheus metrics;

[0095] S8: The Webhook returns the completed scaling request to the api-server and Client go;

[0096] S9: After all the computation tasks in Presto have been completed, Client go sends a scaling-down request to reduce the number of PODs;

[0097] S10: Client go returns the current number of PODs to Presto.

[0098] The aforementioned Presto-based elastic scaling device includes a processor and a memory. The aforementioned acquisition unit and elastic scaling unit are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0099] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the uncontrollable number of PODs caused by elastic scaling based on CPU and memory usage as monitoring metrics in existing technologies.

[0100] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0101] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the aforementioned elastic scaling method based on Presto scheduling.

[0102] This invention provides a processor for running a program, wherein the program executes the aforementioned elastic scaling method based on Presto scheduling.

[0103] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0104] Step S101: When the query request needs to run independently and is a core business, obtain the target performance indicators of the initial cluster and the target query information of the query request. The initial cluster includes multiple PODs, the target performance indicators include at least the memory usage and CPU utilization of the initial cluster, and the target query information includes at least metadata information and context information.

[0105] Step S102: Based at least on the aforementioned target performance indicators and the aforementioned target query information, perform elastic scaling on the aforementioned initial cluster.

[0106] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0107] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0108] Step S101: When the query request needs to run independently and is a core business, obtain the target performance indicators of the initial cluster and the target query information of the query request. The initial cluster includes multiple PODs, the target performance indicators include at least the memory usage and CPU utilization of the initial cluster, and the target query information includes at least metadata information and context information.

[0109] Step S102: Based at least on the aforementioned target performance indicators and the aforementioned target query information, perform elastic scaling on the aforementioned initial cluster.

[0110] To enable those skilled in the art to better understand the technical solution of this application, the technical solution and technical effects of this application will be described below in conjunction with specific embodiments.

[0111] Example

[0112] This embodiment relates to a technical solution for elastic scaling based on Presto scheduling, specifically as follows: Figure 6 As shown. The specific process is as follows:

[0113] S1: Determine whether to enable the forced scheduling policy;

[0114] S2: When the forced scheduling policy is enabled, determine whether the corresponding query request needs to be run independently based on the priority and data volume of the query request; when the forced scheduling policy is not enabled, determine the HDFS address corresponding to the query request, and determine the corresponding target rack location based on the HDFS address, and add the POD corresponding to the target rack location to the candidate queue.

[0115] S3: When the query request needs to run independently and is a core business, obtain the target performance indicators of the initial cluster corresponding to the query request and the target query information corresponding to the query request; when the query request does not need to run independently and / or is not a core business, determine whether there is a target POD in the candidate queue whose number of tasks processed has not reached the target value.

[0116] S4: Based on the target performance metrics of the initial cluster and the target query information corresponding to the query request, determine the target number of PODs that need to be expanded, and determine the target memory space required by the PODs corresponding to the target number of PODs.

[0117] S5: Split the HDFS address corresponding to the query request into multiple HDFS sub-addresses, and traverse each HDFS sub-address. If there is a remaining memory space greater than or equal to the target memory space, pop the target number of PODs from the corresponding HDFS sub-address; if the remaining memory space corresponding to each HDFS sub-address is less than the target memory space, pop the target number of PODs from the target cluster.

[0118] S6: Add the expanded target number of PODs to the candidate queue;

[0119] S7: Determine whether there is a target POD in the candidate queue whose number of tasks processed has not reached the target value;

[0120] S8: If the number of tasks processed by the target POD has not reached the target value, submit the unassigned tasks in the task queue to the target POD. If the number of tasks run by each POD has reached the target value, add the unassigned tasks to the task queue.

[0121] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units described above can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0123] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0126] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0127] 1) In the Presto-based elastic scaling method of this application, when the received query request needs to run independently and is a core business, the target performance indicators of the initial cluster corresponding to the query request and the target query information corresponding to the query request are obtained. Then, the initial cluster is elastically scaled based at least on the target performance indicators and the target query information of the initial cluster. Compared with the prior art method that only uses the target performance indicators corresponding to the initial cluster for elastic scaling, the elastic scaling method of this application uses at least the target performance indicators corresponding to the initial cluster and the target query information corresponding to the query request for elastic scaling. That is, it realizes elastic scaling of the initial cluster based on the target performance indicators corresponding to the initial cluster and the custom elastic scaling indicators (i.e., the context information and metadata information corresponding to the query request). This achieves dynamic orchestration and control of resources, alleviates the waiting and queuing problem caused by resource contention, and avoids the problem of uncontrollable number of PODs caused by elastic scaling based only on CPU and memory usage (i.e., the target performance indicators corresponding to the initial cluster) as monitoring indicators. This achieves the goal of improving service stability and resource utilization.

[0128] 2) In the Presto-based elastic scaling device of this application, the acquisition unit is used to acquire the target performance indicators of the initial cluster corresponding to the query request and the target query information corresponding to the query request when the received query request needs to run independently and is a core business; the elastic scaling unit is used to elastically scale the initial cluster at least based on the target performance indicators and target query information of the initial cluster. Compared with the prior art method of elastically scaling the initial cluster based only on the target performance indicators corresponding to the initial cluster, the elastic scaling device of this application elastically scales the initial cluster at least based on the target performance indicators corresponding to the initial cluster and the target query information corresponding to the query request. That is, it realizes elastic scaling of the initial cluster based on the target performance indicators corresponding to the initial cluster and the custom elastic scaling indicators (i.e., the context information and metadata information corresponding to the query request). This realizes dynamic orchestration and control of resources, and alleviates the waiting and queuing problem caused by resource contention. It avoids the problem of uncontrollable number of PODs caused by elastic scaling based only on CPU and memory usage (i.e., the target performance indicators corresponding to the initial cluster) as monitoring indicators, and achieves the purpose of improving service stability and resource utilization.

[0129] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for elastic scaling based on Presto scheduling, characterized in that, include: When a query request needs to run independently and is a core business, the target performance metrics of the initial cluster and the target query information of the query request are obtained. The initial cluster includes multiple PODs, the target performance metrics include at least the memory usage and CPU utilization of the initial cluster, and the target query information includes at least metadata information and context information. The initial cluster is elastically scaled based at least on the target performance metrics and the target query information; The initial cluster is elastically scaled based on at least the target performance metrics and the target query information, including: determining the target number of PODs based on the target performance metrics and the target query information, wherein the target number of PODs is the number of PODs to be used to expand the initial cluster; determining the target memory space required by the PODs to expand the target number of PODs based on the target number of PODs; and elastically scaling the initial cluster based on the HDFS address corresponding to the query request and the target memory space. Based on the HDFS address corresponding to the query request and the target memory space, the initial cluster is elastically scaled, including: splitting the HDFS address to obtain multiple HDFS sub-addresses; traversing the multiple HDFS sub-addresses to determine the remaining memory space corresponding to each HDFS sub-address; if there is a remaining memory space greater than or equal to the target memory space, popping the target number of PODs from the corresponding HDFS sub-address; if all remaining memory spaces are less than the target memory space, popping the target number of PODs from the target cluster, wherein the target cluster is the cluster where the initially reserved spare PODs are located.

2. The elastic stretching method according to claim 1, characterized in that, After elastically scaling the initial cluster based at least on the target performance metric and the target query information, the elastic scaling method includes: The target number of PODs is added to the candidate queue, and it is determined whether the number of tasks processed by each POD in the candidate queue reaches the target value, wherein the target value is the upper limit of the number of tasks processed by a target POD. If the number of tasks processed by the target POD does not reach the target value, unassigned tasks are submitted to the target POD, wherein the target POD is one of the multiple PODs, and the unassigned tasks are the unassigned tasks corresponding to the query request; Once the number of tasks run by each of the aforementioned PODs reaches the target value, the unassigned tasks are added to the task queue.

3. The elastic stretching method according to claim 1, characterized in that, The process of determining whether the query request is executed independently includes: Determine the priority of the query request; If the priority is higher than the target level and / or the amount of data corresponding to the query request is greater than or equal to the target quantity, the query request is determined to run independently; If the priority is lower than the target level and / or the amount of data corresponding to the query request is less than the target quantity, it is determined that the query request will not run independently.

4. The elastic stretching method according to claim 3, characterized in that, When the priority is lower than the target level and / or the amount of data corresponding to the query request is less than the target quantity, after determining that the query request will not run independently, the elastic scaling method further includes: Determine whether the target POD in the candidate queue has reached the target value, wherein the target POD is one of a plurality of PODs, and the target value is an upper limit on the number of tasks processed by a target POD; If the number of tasks processed by the target POD does not reach the target value, unassigned tasks are submitted to the target POD, wherein the unassigned tasks are the unassigned tasks corresponding to the query request; Once the number of tasks run by each of the aforementioned PODs reaches the target value, the unassigned tasks are added to the task queue.

5. The elastic stretching method according to claim 1, characterized in that, In cases where the query request needs to run independently and is a core business function, before obtaining the target performance metrics of the initial cluster and the target query information of the query request, the elastic scaling method further includes: Determine whether to enable the forced scheduling strategy, wherein the forced scheduling strategy is used to characterize whether to perform elastic scaling based on the target performance indicators and the target query information; Without enabling the forced scheduling policy, determine the HDFS address corresponding to the query request; Based on the HDFS address, the corresponding target rack location is determined, and the POD corresponding to the target rack location is added to the candidate queue.

6. The elastic stretching method according to claim 2, characterized in that, The elastic stretching method further includes: Once all tasks corresponding to the query request have been completed, the query results will be sent to the client. The query results are those corresponding to the query request.

7. A resilient scaling device based on Presto scheduling, characterized in that, include: The acquisition unit is used to acquire the target performance indicators of the initial cluster and the target query information of the query request when the query request needs to run independently and is a core business. The initial cluster includes multiple PODs, the target performance indicators include at least the memory usage and CPU utilization of the initial cluster, and the target query information includes at least metadata information and context information. An elastic scaling unit is used to elastically scale the initial cluster based at least on the target performance metric and the target query information; The elastic scaling unit includes a first determining module, a second determining module, and an elastic scaling module. The first determining module is used to determine the target number of PODs based on the target performance indicators and the target query information, wherein the target number of PODs is the number of PODs used to expand the initial cluster. The second determining module is used to determine the target memory space required by the PODs to expand the target number of PODs based on the target number of PODs. The elastic scaling module is used to elastically scale the initial cluster based on the HDFS address corresponding to the query request and the target memory space. The elastic scaling module includes a sub-splitting module, a traversal sub-module, a first pop-out sub-module, and a second pop-out sub-module. The sub-splitting module is used to split the HDFS address into multiple HDFS sub-addresses. The traversal sub-module is used to traverse the multiple HDFS sub-addresses to determine the remaining memory space corresponding to each HDFS sub-address. The first pop-out sub-module is used to pop the target number of PODs from the corresponding HDFS sub-address when there is a remaining memory space greater than or equal to the target memory space. The second pop-out sub-module is used to pop the target number of PODs from the target cluster when all remaining memory spaces are less than the target memory space, wherein the target cluster is the cluster where the initially reserved spare PODs are located.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes the Presto-based elastic scaling method according to any one of claims 1 to 6.

Images