Presto automated telescoping system, method and medium

Abstract

The application provides a Presto automatic expansion system, method and medium, the system comprises a front-end configuration module, a service management module and a platform base module; the front-end configuration module encapsulates the information submitted by a user and submits the information to the service management module through an interface; the service management module converts the parameters transmitted by the front-end configuration module into a Presto containerized component arrangement file acceptable by a kubernetes cluster, and submits the Presto containerized component arrangement file to the underlying containerized platform base module to start a Presto cluster; and the platform base module is a containerized platform base.The application solves the problems of slow travel query and service stop caused by the incapability of self-adaptation to high concurrency and large data volume by using a containerized Presto big data calculation engine component; and the application solves the problem of chaotic cluster resource configuration and management by using a tenant instance management mode to allocate different tenant permissions to different tenants.

Description

Technical Field

[0001] This invention relates to the technical field of big data computing, and more specifically, to a system, method, and medium for Presto automated scaling. Background Technology

[0002] Big data, also known as massive data, refers to data volumes so large that mainstream software tools cannot capture, manage, process, and organize it into information that helps businesses make more proactive decisions within a reasonable timeframe. In Viktor Mayer-Schönberger and Kenneth Cukier's book, *Big Data: A Revolution That Will Transform How We Live, Work, and Think*, big data refers to analyzing and processing all data without resorting to shortcuts like random sampling. The 5Vs of big data (proposed by IBM) are: Volume, Velocity, Variety, Value (low value density), and Veracity.

[0003] Patent document CN109934361A discloses an automated operation and maintenance platform model based on containers and big data, including continuous building, deployment, and automated operation and maintenance of business systems. Based on the principles of containers and big data, it utilizes Kubernetes container scheduling to ensure container scheduling and analysis, ensuring that scheduled containers are on appropriate nodes, guaranteeing disaster recovery and backup of business systems, and enhancing system security. It monitors business systems and servers through indicator monitoring components and pushes the data to a big data platform for analysis. Through automatic configuration of elastic scaling algorithms, it analyzes business system monitoring data to achieve elastic scaling of the business system. It distributes access traffic through load balancing strategies to ensure normal user access. The platform system makes intelligent decisions based on business application scenarios, such as restarting, scaling up, and scaling down.

[0004] Regarding the aforementioned technologies, the inventors believe that they suffer from problems such as slow query speeds and service interruptions due to the inability to adapt to high-concurrency, large-volume data; and also from chaotic cluster resource configuration and management. Therefore, a new technical solution is needed to improve these technical problems. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this invention is to provide a system, method and medium for automated Presto scaling.

[0006] According to the present invention, a Presto automated scaling system includes a front-end configuration module, a service management module, and a platform base module.

[0007] The front-end configuration module encapsulates the information submitted by the user and submits it to the service management module through an interface;

[0008] The service management module converts the parameters transmitted by the front-end configuration module into a Presto containerized component orchestration file that the Kubernetes cluster can accept, and submits it to the underlying containerized platform base module to start the Presto cluster.

[0009] The platform base module is a containerized platform base.

[0010] Preferably, the front-end configuration module includes functions for workplace configuration, instance configuration, and template configuration;

[0011] The Workplace configuration creates different resource usage spaces for different tenants; it constructs the namespace of the Kubernetes platform through the API provided by the Kubernetes cluster, and assigns labels and taints to different nodes through the API.

[0012] The instance configuration management allows Presto instances to be created, updated, and deleted within a specified workplace via a front-end API in the Kubernetes cluster.

[0013] The template management system saves workspace management and instance management information.

[0014] Preferably, the service management module converts the parameters transmitted by the front-end configuration module into a Presto containerized component orchestration file that can be accepted by the Kubernetes cluster. This file includes a task distribution component Coordinator and a task execution component Worker module. The module then constructs detailed orchestration file details based on the user permission information and computing resource application information transmitted by the front-end configuration module and submits them to the underlying containerized platform base module to start the Presto cluster.

[0015] Preferably, the platform base module is a containerized platform base; it generates instances of specific Presto components Coordinator and Worker based on the information transmitted by the service management module, starts and initializes them, monitors the Presto platform status, starts Presto instances to run on designated machines according to user information transmitted by the service management module, and isolates user computing resources; through the log management module, it collects Presto log content to the storage cluster to provide content for subsequent log auditing; and uses the resource monitoring component to collect resource usage details of Presto components and then expands or reduces the number of Presto cluster instances according to the configured management rules.

[0016] The present invention also provides a method for automated scaling of Presto, the method applying the Presto automated scaling system as described in any one of claims 1-4, the method comprising the following steps:

[0017] Step S1: The configuration management module encapsulates the information submitted by the user and submits it to the service management module through the interface;

[0018] Step S2: The service management module converts the parameters transmitted by the front-end configuration module into a Presto containerization component orchestration file that the Kubernetes cluster can accept. Based on the user permission information and computing resource application information transmitted by the front-end configuration module, it constructs detailed orchestration file details and submits them to the underlying containerization platform base module to start the Presto cluster.

[0019] Step S3: The platform base module generates instances of the specific Presto components Coordinator and Worker based on the information transmitted by the service management module, starts and initializes them, monitors the Presto platform status, and starts the Presto instance to run on the specified machine according to the user information transmitted by the service management module, thus isolating the user's computing resources.

[0020] Preferably, the front-end configuration module in step S1 includes functions for workplace configuration, instance configuration, and template configuration;

[0021] The Workplace configuration creates different resource usage spaces for different tenants; it constructs the namespace of the Kubernetes platform through the API provided by the Kubernetes cluster, and assigns labels and taints to different nodes through the API.

[0022] The instance configuration management allows Presto instances to be created, updated, and deleted within a specified workplace via a front-end API in the Kubernetes cluster.

[0023] The template configuration management can save workspace management and instance management information.

[0024] Preferably, the Presto containerization component orchestration file in step S2 includes a task distribution component Coordinator and a task execution component Worker module.

[0025] Preferably, the platform base module in step S3 is a containerized platform base; the log management module collects Presto log content to the storage cluster to provide content for log auditing; the resource monitoring component collects resource usage details of the Presto component and then expands or reduces the number of Presto cluster instances according to the configured management rules.

[0026] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. This invention solves the problems of slow query speed and service interruption caused by the inability to adapt to high-concurrency large data volumes by adopting containerized Presto big data computing engine components;

[0029] 2. This invention solves the problem of chaotic cluster resource configuration and management by adopting a tenant instance management mode, assigning different tenant permissions to different tenants, and reducing the difficulty of supervising job scheduling and component instance operation. Attached Figure Description

[0030] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0031] Figure 1 This is a system architecture diagram of the present invention;

[0032] Figure 2 This is a diagram of the tenant isolation architecture of the present invention;

[0033] Figure 3 This is a flowchart illustrating the principle of the present invention. Detailed Implementation

[0034] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0035] Example 1:

[0036] Reference Figure 1 and Figure 2According to the present invention, a Presto automated scaling system is provided, comprising a front-end configuration module, a service management module, and a platform base module. The front-end configuration module encapsulates the information submitted by the user and submits it to the service management module through an interface. The service management module converts the parameters transmitted by the front-end configuration module into a Presto containerized component orchestration file that can be accepted by the Kubernetes cluster and submits it to the underlying containerized platform base module to start the Presto cluster. The platform base module is a containerized platform base.

[0037] The front-end configuration module includes functions for workplace configuration, instance configuration, and template configuration. Workplace configuration allows different tenants to build different resource usage spaces. It builds namespaces for the Kubernetes platform through the API provided by the Kubernetes cluster and assigns tags and taints to different nodes through the API. Instance configuration management allows the creation, updating, and deletion of Presto instances in the Kubernetes cluster through the front-end API in the specified workplace. Template management saves workspace management and instance management information.

[0038] The service management module transforms the parameters transmitted by the front-end configuration module into Presto containerized component orchestration files that can be accepted by the Kubernetes cluster. These files include the task distribution component Coordinator and the task execution component Worker module. Based on the user permission information and computing resource request information transmitted by the front-end configuration module, the module constructs detailed orchestration files and submits them to the underlying containerized platform base module to start the Presto cluster.

[0039] The platform foundation module is a containerized platform foundation. Based on the information transmitted by the service management module, it generates instances of specific Presto components, such as the Coordinator and Workers, and starts and initializes them. It also monitors the Presto platform status, starts Presto instances on designated machines based on user information transmitted by the service management module, and isolates user computing resources. Through the log management module, it collects Presto log content and stores it in the storage cluster, providing content for subsequent log auditing. The resource monitoring component collects resource usage details of Presto components and then expands or reduces the number of Presto cluster instances according to the configured management rules.

[0040] Example 2:

[0041] Example 2 is a preferred embodiment of Example 1, and is used to illustrate the present invention in more detail.

[0042] Reference Figure 3The present invention also provides a method for automated Presto scaling, the method applying the Presto automated scaling system as described in any one of claims 1-4, the method comprising the following steps:

[0043] Step S1: The configuration management module encapsulates the information submitted by the user and submits it to the service management module through the interface; the front-end configuration module includes functions for workplace configuration, instance configuration, and template configuration; workplace configuration builds different resource usage spaces for different tenant users; it builds the namespace of the Kubernetes platform through the API provided by the Kubernetes cluster, and assigns tags and taints to different nodes through the API; instance configuration management creates, updates, and deletes Presto instances in the specified workplace through the front-end API in the Kubernetes cluster; template configuration management can save workplace management and instance management information.

[0044] Step S2: The service management module converts the parameters transmitted by the front-end configuration module into a Presto containerized component orchestration file that can be accepted by the Kubernetes cluster. Based on the user permission information and computing resource request information transmitted by the front-end configuration module, it constructs a detailed orchestration file and submits it to the underlying containerized platform base module to start the Presto cluster. The Presto containerized component orchestration file includes the task distribution component Coordinator and the task execution component Worker module.

[0045] Step S3: The platform base module generates instances of the specific Presto components Coordinator and Worker based on the information transmitted by the service management module, starts and initializes them, monitors the Presto platform status, and starts the Presto instance to run on the specified machine according to the user information transmitted by the service management module, isolating user computing resources. The Presto containerization component orchestration file in Step S2 includes the task distribution component Coordinator and the task execution component Worker module; the platform base module is the containerized platform base; through the log management module, Presto log content is collected to the storage cluster to provide content for log auditing; the resource monitoring component collects the resource usage details of Presto components and then expands or reduces the number of Presto cluster instances according to the configured management rules.

[0046] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0047] Example 3:

[0048] Example 3 is a preferred example of Example 1, and is used to illustrate the present invention in more detail.

[0049] This invention provides a method, medium, and system for automated scaling of Presto, which containerizes the big data computing engine Presto to create a stable, efficient, and scalable big data computing platform; deploys a Presto cluster according to tenant application requirements, configures different computing resources as needed, and ensures that each tenant's use of resources is independent, thereby achieving resource isolation.

[0050] An autoscaling method, medium, and system for the big data computing component Presto, comprising:

[0051] The front-end configuration module includes functions for workplace configuration, instance configuration, and template configuration. Workplace configuration creates different resource usage spaces for different tenants. It builds the Kubernetes platform's namespace through the API provided by the Kubernetes cluster, and assigns tags and taints to different nodes via the API. Instance configuration management allows the creation, updating, and deletion of Presto instances within a specified workplace in the Kubernetes cluster via the front-end API. Template management saves information such as workspace and instance management, facilitating future deployment and startup of the Presto cluster. The configuration management module encapsulates user-submitted information and submits it to the service management module via an interface.

[0052] The service management module transforms the parameters transmitted by the front-end configuration module into Presto containerized component orchestration files that can be accepted by the Kubernetes cluster. These files include the task distribution component Coordinator and the task execution component Worker module. The module then constructs detailed orchestration files based on the user permission information and computing resource request information transmitted by the front-end configuration module. Finally, the module submits these files to the underlying containerized platform base module to start the Presto cluster.

[0053] The platform foundation module serves as the containerized platform foundation. Based on information transmitted from the service management module, it generates instances of the specific Presto components, Coordinator and Worker, and starts and initializes them. It also monitors the Presto platform status and, based on user information transmitted from the service management module, starts Presto instances running on designated machines to isolate user computing resources. The log management module collects Presto log content and stores it in the storage cluster, providing data for subsequent log auditing. The resource monitoring component collects resource usage details of Presto components and then expands or reduces the number of Presto cluster instances based on configured management rules.

[0054] This invention solves the problems of slow query speed and service interruption caused by the inability to adapt to high-concurrency large data volumes by adopting containerized Presto big data computing engine components. By adopting a tenant instance management mode, different tenant permissions are assigned to different tenants, which solves the problem of chaotic cluster resource configuration and management, and reduces the difficulty of supervising job scheduling and component instance operation.

[0055] Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0056] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0057] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A Presto automated telescoping system, characterized by, The system comprises a front-end configuration module, a service management module and a platform base module; The front-end configuration module encapsulates the information submitted by the user and submits it to the service management module through an interface; The service management module converts the parameters transmitted by the front-end configuration module into a Presto containerized component orchestration file acceptable to the kubernetes cluster, and submits it to the underlying containerized platform base module to start the Presto cluster; The platform base module is a containerized platform base; The front-end configuration module comprises the functions of workplace configuration, instance configuration and template configuration; The workplace configuration builds different resource use spaces for different tenant users; the namespace of the kubernetes platform is built through the API provided by the kubernetes cluster, and the tags and stains are assigned to different nodes through the API; The instance configuration manages the creation, update and deletion of Presto instances in the specified workplace through the front-end API in the kubernetes cluster; The template management saves the workspace management and instance management information; The platform base module is a containerized platform base; According to the information transmitted by the service management module, the specific Presto component Coordinator and Worker instances are generated, and they are started and initialized, and the Presto platform state is supervised, and the Presto instance is started and run to the specified machine according to the user information content transmitted by the service management module, and the user computing resources are isolated; Through the log management module, the log content of Presto is collected to the storage cluster to provide content for subsequent log auditing; the resource supervision component is used to collect the resource use details of the Presto component, and then the Presto cluster instances are expanded or reduced according to the configured management rules.

2. The Presto automated telescoping system of claim 1, wherein, The service management module converts the parameters transmitted by the front-end configuration module into a Presto containerized component orchestration file acceptable to the kubernetes cluster, which includes the task distribution component Coordinator and the task execution component Worker module, and constructs detailed orchestration file details according to the user permission information and computing resource application related content transmitted by the front-end configuration module, and submits them to the underlying containerized platform base module to start the Presto cluster.

3. A method of Presto automated telescoping, characterized by, The method applies the Presto automatic scaling system according to any one of claims 1-2, and the method comprises the following steps: Step S1: The configuration management module encapsulates the information submitted by the user and submits it to the service management module through an interface; Step S2: The service management module converts the parameters transmitted by the front-end configuration module into a Presto containerized component orchestration file acceptable to the kubernetes cluster, and constructs detailed orchestration file details according to the user permission information and computing resource application related content transmitted by the front-end configuration module, and submits them to the underlying containerized platform base module to start the Presto cluster; Step S3: The platform base module generates specific Presto component instances Coordinator and Worker according to the information transmitted by the service management module, and starts and initializes them, and supervises the Presto platform state, and starts the Presto instance to run to the specified machine according to the user information content transmitted by the service management module, and isolates the user computing resources.

4. The method of Presto automation telescoping of claim 3, wherein, The front-end configuration module in the step S1 includes the functions of workplace configuration, instance configuration and template configuration; The Workplace configures different resource use spaces for different tenant users; the namespace of the kubernetes platform is constructed through the API provided by the kubernetes cluster, and the tags and stains are given to different nodes through the API to construct; The instance configuration management creates, updates and deletes the Presto instance in the specified workplace through the front-end API in the kubernetes cluster; The template configuration management can save the workspace management and instance management information.

5. The method of Presto automation telescoping of claim 3, wherein, The Presto containerized component orchestration file in the step S2 includes the task distribution component Coordinator and the task execution component Worker module.

6. The method of Presto automation and telescoping of claim 3, wherein, The platform base module in the step S3 is a containerized platform base; through the log management module, the log content of Presto is collected to the storage cluster to provide content for log auditing; the resource supervision component is used to collect the resource use details of the Presto component, and then the instances of the Presto cluster are expanded or reduced according to the configured management rules.

7. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of claim 3.

Images