Impala automated scaling system, method, medium, and apparatus

Abstract

The application provides an Impala automatic scaling system, method, medium and equipment, comprising: a configuration management module: managing a workspace, an instance and a template; an application control module: receiving information submitted by the configuration management module, and translating the information into a format acceptable by a container platform, and generating API information by arranging each component of the Impala; and a container platform module: generating instance initialization of internal components of the Impala according to the API information, collecting resource information of the Impala components through a monitoring component, and automatically scaling the instance according to a preset configuration rule. By using the containerized impala big data computing service component, the application solves the problem that high concurrency and large data volume queries are too slow and cannot be automatically scaled.

Description

Technical Field

[0001] This invention relates to the field of Impala automated scaling technology, specifically to a system, method, medium, and device for Impala automated scaling. Background Technology

[0002] Impala, as a mainstream big data query component, is widely used in scenarios such as ad-hoc big data queries and real-time data warehouses. However, with the increase in data volume and user base, issues such as how to handle high-concurrency access and tenant management are becoming increasingly apparent.

[0003] Patent document CN113138976A (application number: CN202010064306.9) discloses an implementation scheme to improve Impala's query capacity. It collects and processes data from data sources and stores it in a Hadoop cluster. Impala is associated with the Hadoop cluster, and by caching metadata in Redis, when a user submits a query request for the first time and the SQL engine creates an execution plan, it retrieves the metadata from Redis, deserializes it, and stores it in Lrucache. Subsequent requests retrieve metadata directly from Lrucache when creating execution plans. This patent primarily focuses on vertical improvements to Impala nodes but does not address horizontal node expansion. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the purpose of this invention is to provide a system, method, medium and device for automated scaling of Impala.

[0005] The Impala automated scaling system provided by the present invention includes:

[0006] Configuration Management Module: Manages workspaces, instances, and templates;

[0007] Application Control Module: Receives information submitted by the Configuration Management Module, translates it into a format acceptable to the containerization platform, and orchestrates Impala's various components to generate API information;

[0008] Container Platform Module: Generates instance initializations of Impala internal components based on API information, collects Impala component resource information by monitoring components, and automatically scales instances according to preset configuration rules.

[0009] Preferably, the configuration management module includes: a workspace management module, which establishes different resource isolation space logical plans for different tenants, creates a workspace through the Kubernetes interface, the workspace corresponds to the Namespace in the k8s cluster, and tags and taints it through the interface.

[0010] Preferably, the configuration management module includes an instance management module that provides functions for creating, modifying, recycling, and querying Impala instances in a specified workspace through a front-end or API in different workspaces of the Kubernetes cluster.

[0011] Preferably, the configuration management module includes: a template management module, which stores information from the workspace management module and the instance management module, and encapsulates the information submitted by the user and submits it to the application control module through an interface.

[0012] Preferably, the logs of each component are collected by a log collector and stored on a log storage platform for audit analysis.

[0013] Preferably, based on the tenant information configuration content submitted by the application control module, the Impala instance is started and run on the specified node to perform multi-tenant resource isolation and monitor the Impala cluster status.

[0014] Preferably, the application control module includes: defining components that support the Impala type for resource control, generating specific configuration file information based on the tenant generation permissions and resource isolation content submitted by the configuration management module, and finally submitting it to the container platform module to generate an Impala cluster and writing the configuration into the generated instance.

[0015] The Impala automated scaling method provided by the present invention includes:

[0016] Step 1: Manage workspaces, instances, and templates;

[0017] Step 2: Receive information submitted by the configuration management module, translate it into a format acceptable to the containerization platform, and orchestrate Impala's various components to generate API information;

[0018] Step 3: Generate instance initialization of Impala internal components based on API information, collect Impala component resource information by monitoring components, and automatically scale instances according to preset configuration rules.

[0019] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, it implements the steps of the method.

[0020] The Impala automated scaling device provided by the present invention includes: a controller;

[0021] The controller includes the computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the Impala automated scaling method; or, the controller includes the Impala automated scaling system.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) This invention solves the problem of slow response and inability to automatically scale up or down when dealing with high-concurrency large data volume queries by using containerized Impala big data computing service components;

[0024] (2) By adopting the tenant instance management mode, this invention establishes different tenants and permissions for different departments and users, which solves the management problem of sub-clusters and resource usage, and also simplifies the difficulty of monitoring and managing job scheduling and operation. Attached Figure Description

[0025] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0026] Figure 1 System architecture diagram;

[0027] Figure 2 For deployment plan diagram;

[0028] Figure 3 A diagram showing the tenant isolation architecture. Detailed Implementation

[0029] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0030] Example:

[0031] This invention proposes an automatic scaling system for the big data component Impala, comprising:

[0032] The configuration management module includes functions such as workspace management, instance management, and template management.

[0033] Workspace management establishes different resource isolation space logical plans for different tenants (which can be divided according to departments or groups of users). A workspace is created through the Kubernetes interface. The workspace corresponds to the namespace in the Kubernetes cluster and is tagged and tainted through the interface.

[0034] Instance management provides the ability to create, modify, reclaim, and query Impala instances in different workspaces within a Kubernetes cluster via a front-end or API.

[0035] Template management saves information such as workspace and instance management, providing a quick and convenient way to redeploy and start the Impala cluster in the future. The configuration management module encapsulates the information submitted by the user and submits it to the application control module through an interface.

[0036] The application control module receives information submitted by the configuration management module based on the template engine, translates it into a format acceptable to the containerization platform, and generates orchestration files for various Impala components, including metadata information Catalog, state information Statestore, and the Impala computing component.

[0037] Based on the Impala module definition, the Impala control resources are abstracted into objects that support Impala type component definitions. The configuration management module generates specific configuration file information based on the tenant generation permissions and resource isolation content submitted by the configuration management module. Finally, the underlying container platform module is submitted to generate the Impala cluster and write the configuration into the generated instance.

[0038] The container platform module serves as the foundation of the containerization platform. It generates instance initializations for specific Impala internal components such as Statestore, Catalog, and Impala based on API information, and monitors the Impala cluster status. Furthermore, it starts Impala instances on designated nodes based on tenant configuration information submitted by the application control module, thus achieving multi-tenant resource isolation. A log collector gathers runtime logs from each component and stores them on a log storage platform, providing a basis for future auditing and analysis. Monitoring components collect Impala component resource information and automatically scale instances according to configuration rules.

[0039] The Impala automated scaling method provided by the present invention includes: Step 1: managing workspaces, instances, and templates; Step 2: receiving information submitted by the configuration management module, translating it into a format acceptable to the containerization platform, and orchestrating various Impala components to generate API information; Step 3: generating instance initialization of Impala internal components based on the API information, collecting Impala component resource information through monitoring components, and automatically scaling instances according to preset configuration rules.

[0040] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, it implements the steps of the method.

[0041] The Impala automated scaling device provided by the present invention includes: a controller; the controller includes a computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the Impala automated scaling method; or, the controller includes the Impala automated scaling system.

[0042] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0043] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. An Impala automated telescoping system, characterized by, Comprise: A configuration management module: manage workspace, instance and template; An application control module: receive information submitted by the configuration management module, and translate it into a format that the containerization platform can accept, and generate API information for the orchestration of Impala components; A container platform module: generate instance initialization of Impala internal components according to API information, collect Impala component resource information through monitoring components, and automatically scale instances according to preset configuration rules; The configuration management module comprises: a workspace management module, which establishes different resource isolation space logical plans for different tenants, creates a workspace through a Kubnates interface, and the workspace corresponds to a Namespace namespace in a k8s cluster, and is labeled and stained through an interface; The configuration management module comprises: an instance management module, which provides creation, change, recovery and query functions of Impala instances in a specified workspace through a front end or API in different workspaces of a k8s cluster; The configuration management module comprises: a template management module, which saves information of the workspace management module and the instance management module, encapsulates information submitted by a user, and submits it to the application control module through an interface; The application control module comprises: defining components supporting Impala type for resource control, generating specific configuration file information according to tenant information submitted by the configuration management module, and finally submitting the container platform module to generate Impala cluster and write the configuration into the generated instance.

2. The Impala automated telescoping system of claim 1, wherein, Collect the running logs of each component through a log collector to a log storage platform, and perform audit analysis.

3. The Impala automated telescoping system of claim 1, wherein, According to the tenant information configuration content submitted by the application control module, start the Impala instance to run in the specified node, perform multi-tenant resource isolation, and monitor the state of the Impala cluster.

4. A method of Impala automated scaling, characterized by, The system for automatically scaling Impala according to any one of claims 1-3, comprising: Step 1: manage workspace, instance and template; Step 2: receive information submitted by the configuration management module, and translate it into a format that the containerization platform can accept, and generate API information for the orchestration of Impala components; Step 3: generate instance initialization of Impala internal components according to API information, collect Impala component resource information through monitoring components, and automatically scale instances according to preset configuration rules.

5. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of claim 4.

6. An Impala automated telescoping device characterized by, Comprise: A controller; The controller comprises the computer readable storage medium of claim 5, wherein the computer program is executed by the processor to realize the steps of the method of claim 4; or the controller comprises the system for automatically scaling Impala of any one of claims 1-3.

Images