Resource dynamic adjustment method and device, equipment and medium

Abstract

The invention relates to a data processing technology, can be applied to business system platforms of medical health, financial science and technology and the like, and discloses a resource dynamic adjustment method, device and equipment and a medium, and the method comprises the steps: obtaining the use condition and weight information of each resource group in a current database; generating resource configuration information corresponding to each resource group based on the use condition and the weight information of each resource group; submitting a resource adjustment request to a database front-end node according to the resource configuration information in a hard restriction isolation mode; and forwarding the resource configuration information to a database rear-end node based on the database front-end node, executing the resource configuration information based on the database rear-end node, and storing a resource use condition. Dynamic adjustment of resources is supported, the system resources are fully utilized, the query response time is shortened, and meanwhile the stability of the system can be guaranteed; and for tasks with high weights, resources can be preferentially allocated or allocated more, and the operation and maintenance efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for dynamic resource adjustment. Background Technology

[0002] Apache Doris is a high-performance, real-time analytical database based on an MPP architecture. It boasts excellent query performance, supporting sub-second response times for massive datasets. It is widely used in high-concurrency point query scenarios and high-throughput complex analytical scenarios. Common application scenarios include report analysis, ad-hoc queries, and lakeware warehousing. Users can build dashboards, user behavior analysis, A / B testing platforms, log retrieval and analysis, user profiling, order analysis, and other applications on top of it.

[0003] In the medical field, medical imaging data is characterized by high timeliness, high repetition, and multi-source correlation. Traditional Hadoop systems, due to their complex components, high operational costs, and insufficient real-time performance, struggle to meet clinical needs. By replacing the traditional Hadoop system with Apache Doris in the medical imaging platform, the number of dependent components has been reduced from 6 to 3, the data model from 15 to 2, and 6 new views have been added, thus reducing operational complexity. It supports real-time writing and querying of data ranging from MB to GB, resulting in significant savings in physical resources.

[0004] In the financial sector, Doris is used to build an integrated indicator data service platform, solving problems such as inconsistent indicator definitions, redundant calculations, and low delivery efficiency in traditional report creation. Through materialized views and indexing, it achieves millisecond-level query response times in multi-table join scenarios, supporting the analysis needs of bank operating performance indicators and customer profiles. In lending operations, Doris replaces the Kettle+MySQL offline data warehouse and Trino federated query architecture, achieving seamless metadata integration, improved query performance, and reduced operational costs, supporting scenarios such as potential customer credit assessment and post-loan management.

[0005] In the medical and financial fields, Doris supports resource isolation to fully utilize server resources, ensuring the stability of the Doris cluster and reducing response time. Resource isolation is mainly achieved by dividing the system into multiple groups, allocating certain resources to each group and limiting them, thereby ensuring that tasks from different users do not interfere with each other.

[0006] Doris can use Workload Group hard limits to achieve resource isolation. Specifically, the resources (CPU, memory, IO) of a Be node are divided into multiple resource groups through Cgroups to achieve more granular resource allocation. No resource group can exceed the resource limit at any time.

[0007] The existing workload group hard-limit method prevents each group from exceeding the resource limit, but it also cannot dynamically adjust the resources. Even if there are idle resources in the system, they cannot be used by other resource-intensive groups, resulting in resource waste. Summary of the Invention

[0008] In view of the shortcomings of the prior art, the present invention provides a method, apparatus, device and medium for dynamic resource adjustment, which aims to solve the problem that the existing Workload Group hard limit method makes it impossible for each group to break through the resource limit, but it is impossible to dynamically adjust the resources. Even if there are idle resources in the system, they cannot be used by other resource-intensive groups, resulting in resource waste.

[0009] The technical solution of the present invention is as follows:

[0010] The first embodiment of the present invention provides a method for dynamic resource adjustment, the method comprising:

[0011] Retrieve the usage and weight information of each resource group in the current database;

[0012] Based on the usage and weight information of each resource group, resource configuration information corresponding to each resource group is generated.

[0013] The resource configuration information is used to submit a resource adjustment request to the database front-end node in a hard-limit isolation manner;

[0014] The database front-end node forwards the resource configuration information to the database back-end node, executes the resource configuration information based on the database back-end node, and stores the resource usage information.

[0015] Another embodiment of the present invention provides a resource dynamic adjustment device, the device comprising:

[0016] The data acquisition module is used to obtain the usage and weight information of each resource group in the current database;

[0017] The resource configuration module is used to generate resource configuration information corresponding to each resource group based on the usage and weight information of each resource group;

[0018] The data forwarding module is used to submit resource adjustment requests to the database front-end node using hard constraint isolation of the resource configuration information;

[0019] The resource usage recording module is used to forward the resource configuration information to the database backend node based on the database frontend node, execute the resource configuration information based on the database backend node, and store the resource usage information.

[0020] Another embodiment of the present invention provides a computer device, the computer device including at least one processor; and,

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the steps of the above-described resource dynamic adjustment method.

[0023] Another embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the above-described resource dynamic adjustment method.

[0024] Beneficial Effects: The resource dynamic adjustment method, apparatus, device, and medium of this invention include: acquiring the usage and weight information of each resource group in the current database; generating resource configuration information corresponding to each resource group based on the usage and weight information; submitting a resource adjustment request to the database front-end node using a hard-limit isolation method; forwarding the resource configuration information to the database back-end node based on the database front-end node; executing the resource configuration information based on the database back-end node; and storing the resource usage information. This invention supports dynamic resource adjustment, fully utilizes system resources, improves query response time, and ensures system stability. For tasks requiring priority protection, higher weights are set, allowing for priority allocation or increased resource allocation, thus improving operational efficiency. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram illustrating the application environment of an embodiment of the resource dynamic adjustment method of the present invention;

[0027] Figure 2 This is a flowchart of a preferred embodiment of a resource dynamic adjustment method according to the present invention;

[0028] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the resource dynamic adjustment device of the present invention;

[0029] Figure 4 This is a schematic diagram of a preferred embodiment of a computer device according to the present invention;

[0030] Figure 5 This is another structural schematic diagram of a preferred embodiment of a computer device according to the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0032] The embodiments of the present invention will be described below with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0033] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Here, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0034] The resource dynamic adjustment method provided in this embodiment of the invention can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The client accesses the server's network or business platform, and the server can obtain the usage and weight information of each resource group in the current database; based on the usage and weight information of each resource group, it generates resource configuration information corresponding to each resource group; it submits a resource adjustment request to the database front-end node using a hard-limit isolation method; the database front-end node forwards the resource configuration information to the database back-end node; the database back-end node executes the resource configuration information and stores the resource usage information. In this invention, by supporting dynamic resource adjustment, system resources are fully utilized, query response time is improved, and system stability is ensured; for tasks with high weight, resources can be allocated preferentially or in greater quantities, improving operational efficiency. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster composed of multiple servers. The invention will be described in detail below through specific embodiments.

[0035] To address the above problems, embodiments of the present invention provide a method for dynamic resource adjustment. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart of a preferred embodiment of a resource dynamic adjustment method according to the present invention. Figure 2 As shown, it includes:

[0036] Step S100: Obtain the usage and weight information of each resource group in the current database.

[0037] The database used in this embodiment of the invention is the Apache Doris database. In Apache Doris, resource groups are used to implement resource isolation and allocation in a multi-tenant environment, and resources can be managed in two ways: by dividing nodes by tags or by configuring workload groups.

[0038] When configuring and managing resources in a Workload Group, monitoring resource group usage can be achieved through several methods. One method is via SHOW PROC. For example: `sql SHOW PROC ' / workload_groups'`; key fields include: QueryCount (current active queries), CpuUsage (CPU utilization), and MemoryUsage (memory usage). Doris Metrics can also be used. Example monitoring metrics:

[0039] doris_fe_workload_group_query_count{group="high_priority_group"}

[0040] Current number of queries for the resource group.

[0041] doris_fe_workload_group_cpu_usage_percent{group="low_priority_group"}

[0042] CPU utilization of resource group.

[0043] The weight information can be set in advance by administrators. Weights are set according to the type of task. Workload Group is a mechanism for grouping and managing workloads, aiming to achieve resource isolation and fine-grained control, and improve system resource utilization and stability.

[0044] A Workload Group is a logical entity that represents one or more workloads. A workload refers to the combination of a dataset to be processed and the operations performed on it, describing the type and scale of tasks that a big data system needs to perform. When sharing the same cluster, multiple business units or tenants may face concurrent queries from various analytical workloads. Under limited resource conditions, resource contention between query tasks will lead to performance degradation or even cluster instability.

[0045] Workload Groups enable fine-grained control over memory and CPU resources by grouping workloads. By associating user-executed queries with Workload Groups, the percentage of CPU and memory resources allocated to a single query on a single BE node can be limited.

[0046] For example, in Apache Doris, Workload Groups implement soft and hard limits for CPU resources. Soft limits allow for flexible allocation of resources when they are idle, improving resource utilization; while hard limits ensure that groups do not interfere with each other due to load changes, guaranteeing performance stability.

[0047] Step S100, which involves obtaining the usage and weight information of each resource group in the current database, includes:

[0048] Step S101: Obtain the usage status of each resource group in the current database at first predetermined time intervals;

[0049] Step S102: Based on the usage of each resource group, a first task group, a second task group, a third task group, and a fourth task group are obtained. The first task group is a timed task group whose timed task start time is less than a first time threshold; the second task group is a timed task group whose timed task end time is less than a second time threshold; the third task group is a task group whose resource utilization rate is greater than a first utilization rate threshold; and the fourth task group is a task group whose resource utilization rate is less than a second utilization rate threshold, and the first utilization rate threshold is greater than the second utilization rate threshold.

[0050] Step S103: Obtain the weight information of each resource group in the current database at first predetermined time intervals.

[0051] The usage and weight information of each resource group are periodically analyzed at a first predetermined time interval. Within each first predetermined time interval, the following are identified: task groups about to start, task groups about to end, task groups with high resource utilization, and task groups with low resource utilization. Specifically, the first task group is defined as those whose scheduled task start time is less than a first time threshold; the second task group is defined as those whose scheduled task end time is less than a second time threshold; the third task group is defined as those whose resource utilization is greater than a first utilization threshold; and the fourth task group is defined as those whose resource utilization is less than a second utilization threshold, where the first utilization threshold is greater than the second utilization threshold. The first predetermined time interval can be selected as 30 seconds, or can be set as needed.

[0052] Specifically, step S101, which involves obtaining the usage status of each resource group in the current database at first predetermined time intervals, includes:

[0053] Step S11: Pre-set a first utilization threshold, wherein the first utilization threshold is one or more of a first CPU utilization threshold, a first memory threshold, and a first I / O transfer rate threshold.

[0054] Step S12: Pre-set the second utilization rate, wherein the second utilization rate threshold is one or more of the following combinations: second CPU utilization threshold, second memory threshold, and second I / O transfer rate threshold.

[0055] Step S13: Set the first time threshold and the second time threshold in advance.

[0056] Groups exceeding the first utilization threshold are classified as high resource utilization groups. In Apache Doris, high resource utilization in a resource group or the cluster as a whole can lead to query latency, task backlog, and even service unavailability. Common causes of high resource utilization include: CPU overload: complex queries (such as multi-table joins, high cardinality GroupBy), data skew, and excessive concurrent queries; Insufficient memory: large query cache, spill to disk failures, and memory leaks (such as unreleased UDF resources); High disk I / O load: frequent compaction, large data import / export, and cold data access; Imbalanced workload group weight distribution: high-priority groups consume too many resources, while low-priority groups are starved; Uneven distribution of node label resources: some groups have too few BE nodes or low hardware configuration; Lack of resource limits: a single query may monopolize all resources, causing other queries to be blocked; Sudden data surges, such as: a sudden increase in data volume: a surge in data volume during peak business periods (such as log systems); and slow query backlog: unoptimized SQL or missing indexes cause queries to run for a long time. Insufficient cluster size: Hardware resources (CPU, memory, disk) cannot meet business needs.

[0057] When setting up high resource utilization groups (this is a reference and can be adjusted according to the actual situation): High CPU threshold: above 90% of the group; High memory threshold: above 90% of the group; High IO: tentatively set at 100MB / s;

[0058] Low resource utilization grouping criteria (this is a reference and can be adjusted according to actual conditions): Low CPU threshold: below 60% of the group; Low memory threshold: below 60% of the group; Low IO threshold: below 50MB / s.

[0059] Step S200: Generate resource configuration information corresponding to each resource group based on the usage and weight information of each resource group.

[0060] In Apache Doris, resource configuration information for each workload group can be dynamically generated or adjusted based on the usage (such as real-time metrics like CPU, memory, and query concurrency) and weight information (such as configurations like cpu_share and memory_limit) to achieve resource isolation, priority scheduling, and load balancing.

[0061] Step S200, which generates resource configuration information corresponding to each resource group based on the usage and weight information of each resource group, includes:

[0062] Step S201: Trigger a resource allocation instruction every second predetermined time interval;

[0063] Step S202: In response to the resource allocation instruction, generate the resource adjustment priority corresponding to the resource group based on the usage and weight information of each resource group;

[0064] Step S203: Generate resource configuration information corresponding to each resource group based on resource adjustment priority.

[0065] In this invention, the second predetermined time interval is greater than the first predetermined time interval. For example, if the first predetermined time interval is set to 30 seconds, then the second predetermined time interval can be set to 180 seconds. Within 180 seconds, a resource allocation instruction is triggered. Based on this instruction, in Apache Doris, the core objective of generating resource adjustment priorities is to prioritize the stability of high-priority resource groups during resource scarcity, while preventing low-priority resource groups from being "starved." Resource configuration information corresponding to each resource group is then generated based on the resource adjustment priorities.

[0066] Step S202, which is to respond to the resource allocation instruction and generate the resource adjustment priority corresponding to the resource group based on the usage and weight information of each resource group, includes:

[0067] Step S221: In response to the resource allocation instruction, the resource adjustment is pre-divided into four priorities, namely the first priority, the second priority, the third priority and the fourth priority;

[0068] Step S222: Set the priority of the first task group whose weight is greater than the preset first weight threshold to the first priority;

[0069] Step S223: Set the priority of the third task group whose weight is greater than the preset second weight threshold to the second priority;

[0070] Step S224: Set the priority of the second task group, whose timed task has ended and whose weight is less than the preset third weight threshold, to the third priority.

[0071] Step S225: Set the priority of the fourth task group to the fourth priority.

[0072] Resources are prioritized. For example, they can be divided into four priorities. The first task, which has the highest weight and is about to start its scheduled task, is set to the first priority. The task group with high usage but lower weight is set to the second priority. The task group with low weight and about to end its scheduled task is set to the third priority. The task group with low usage is set to the fourth priority. The first weight threshold is higher than the second weight threshold, and the second weight threshold is higher than the third weight threshold.

[0073] Step S203, which generates resource configuration information corresponding to each resource group based on resource adjustment priority, includes:

[0074] Step S231: Configure the first priority task group as the first in the resource addition order;

[0075] Step S232: Configure the second priority task group as the second position in the resource addition order;

[0076] Step S233: Configure the third priority task as the first priority task in the resource reduction order;

[0077] Step S234: Configure the fourth priority task group as the second priority in the resource reduction order.

[0078] Resource allocation is periodically adjusted using a period (e.g., 180 seconds). The principle for resource allocation is: prioritize allocating resources to scheduled task groups with upcoming tasks and high weights, followed by resource groups with high utilization and high weights. The principle for resource acquisition is: prioritize acquiring resources from scheduled task groups with completed tasks and low weights, followed by resource groups with low resource utilization. The goal is to allocate resources with maximum efficiency to ensure full resource utilization.

[0079] Step S300: Submit a resource adjustment request to the database front-end node using the resource configuration information in a hard-restriction isolation manner.

[0080] After the resource configuration is adjusted, the resource configuration requests for each group will be submitted to the fe node using the workload group hard limit method. The workload group hard limit method uses CGroup technology to set absolute upper limits on CPU, memory, and IO resources within the BE process, ensuring that tenants cannot exceed the limits, preventing resource exhaustion or preemption, and ensuring cluster stability and performance.

[0081] The core mechanisms of hard limits include absolute resource caps, which set insurmountable thresholds for resource usage. For example, a CPU hard limit restricts the CPU usage of a workload group; even if the system is idle, tenants cannot exceed this limit. Memory hard limits are set via the `memory_limit` parameter, which sets the percentage of BE memory that a resource group can use. If the system detects that the limit has been exceeded, it immediately terminates the task with the highest memory usage within the group to release resources. I / O hard limits are set via the `read_bytes_per_second` and `remote_read_bytes_per_second` parameters, which limit the read rates of local and remote files, respectively.

[0082] Resource isolation and stability. Hard limits achieve intra-process resource isolation through CGroup technology, ensuring that different workload groups do not interfere with each other due to load changes. In scenarios with high stability requirements, such as financial risk control and real-time analysis, hard limits can prevent sudden traffic from exhausting resources and ensure the response speed of critical queries.

[0083] The CPU hard limit configuration method relies on CGroup: Doris version 2.1 and above require CGroup to implement CPU hard limit. First, ensure that CGroup is installed on the BE node, and confirm the version via the path / sys / fs / cgroup / cpu / (CGroup v1) or / sys / fs / cgroup / cgroup.controllers (CGroup v2).

[0084] Create the CGroup directory: Create a new directory called doris under the CGroup path and grant the Doris process read, write and execute permissions.

[0085] The memory hard limit configuration method is to set the percentage of BE memory that a resource group can use through the memory_limit parameter, for example:

[0086] SQL

[0087] CREATE WORKLOAD GROUP IF NOT EXISTSg1 PROPERTIES("memory_limit"="30%");

[0088] Disable soft memory isolation (enable_memory_overcommit=false) to enable hard limits. In this case, the system will immediately terminate the task if it detects that the memory has exceeded the limit.

[0089] The hardware-based IO limit configuration method limits the read rate of local and remote files using the `read_bytes_per_second` and `remote_read_bytes_per_second` parameters, respectively. For example:

[0090] SQL

[0091] CREATE WORKLOAD GROUP IF NOT EXISTSg1 PROPERTIES("read_bytes_per_second"="10485760";-- Limit local I / O to 10MB / s.

[0092] Step S400: Based on the database front-end node, forward the resource configuration information to the database back-end node, execute the resource configuration information based on the database back-end node, and store the resource usage information.

[0093] Resource configuration information is distributed from the database frontend fe node to the database frontend be node. The resource usage of the Doris cluster is recorded at intervals (e.g., 30 seconds).

[0094] In Apache Doris, the FE (Frontend) node is the control center of the cluster, responsible for core functions such as metadata management, query parsing, and cluster coordination, while the BE (Backend) node is the core unit for data storage and computation. The two work together to achieve efficient data analysis.

[0095] The FE stores and manages metadata information for all databases, tables, partitions, materialized views, and other objects, including table structure (column names, types, indexes, etc.), data distribution information (mapping from Tablet to BE), and user permission information. Metadata is stored in the FE's memory and persisted to disk via BDB (Berkeley DB), employing a RAFT-like protocol to ensure high availability. The FE receives SQL query requests from clients, parses them into an Abstract Syntax Tree (AST), and performs semantic analysis (such as checking table existence and permissions). The query optimizer optimizes the execution plan based on a cost model, including logical optimizations such as predicate pushdown, column pruning, and partition pruning, as well as selecting the optimal execution plan based on cost.

[0096] The FE (Functional Environment) is responsible for managing the registration and heartbeat detection of BE (Browser / Enterprise) nodes, monitoring cluster status, and triggering replica repair when a BE node fails. It adjusts data distribution through load balancing mechanisms to ensure queries are evenly distributed across BE nodes, avoiding hotspots. It coordinates data import jobs, assigns import tasks to appropriate BE nodes, monitors import progress, and maintains the atomicity of import transactions. The FE manages user permissions and authentication, ensuring only authorized users can access cluster resources.

[0097] FE contains several core components that work together to accomplish its functions:

[0098] Catalog: The core of metadata storage, recording information about all database objects, such as the definitions of databases, tables, partitions, materialized views, etc.

[0099] Query Planner: An SQL parser that converts SQL into a logical execution plan.

[0100] Optimizer: The query optimizer optimizes the execution plan based on the cost model and selects the optimal query execution path.

[0101] Query Coordinator: The query coordinator distributes the execution plan to multiple BEs for parallel execution and collects the results to return to the client.

[0102] Load Manager: Manages data import jobs, coordinates data writing to BE nodes, and ensures data consistency.

[0103] Admin Console: Provides a management interface and API to facilitate cluster management, monitoring, and debugging.

[0104] In the Apache Doris architecture, the Backend (BE) node is the core computing and storage unit, responsible for critical tasks such as actual data storage, query execution, replica management, and resource isolation. BE nodes employ a columnar storage engine, supporting efficient data compression and encoding (such as dictionary encoding and bitmap indexes) to reduce storage space consumption and improve query performance. Its storage structure is divided into a data area (storing columnar data), an index area (accelerating data retrieval), and a footer (metadata information). A paginated loading mechanism enables on-demand reading, reducing I / O overhead. Data is stored in multiple replicas in units of Tablets (data shards), with a default of three replicas distributed across different BE nodes. BE uses a two-phase commit protocol to ensure the atomicity of data writes and combines this with the Paxos protocol to achieve strong consistency between replicas, ensuring no data loss in the event of a single machine failure. BE supports multi-tenant resource isolation through resource groups. Administrators can divide BE nodes into multiple resource groups, allocating independent computing resources (CPU, memory) to each group and binding specific data replicas with tags. User queries can only access data within authorized resource groups, avoiding resource contention.

[0105] The core functions of a BE node include: Data storage and management. Partition-bucketing model: Data is horizontally partitioned by a partition key (e.g., time), and each partition is further hashed into buckets called Tablets. A Tablet is the smallest unit for data movement and replication. Replica management: The BE node maintains local Tablet replicas, reports replica status to the FE via a heartbeat mechanism, and triggers automatic repair in case of failure. Data compression and merging: Background threads periodically perform Base Compaction (merging small files) and Cumulative Compaction (merging incremental data) to optimize query performance and reduce storage fragmentation. Query execution and parallel computing: MPP architecture: The BE node executes query plan fragments in parallel based on the MPP model, prioritizing local data reading through DataLocality to reduce network transmission. Vectorized execution engine: Employs SIMD instruction sets to optimize columnar data processing, supporting predicate pushdown, column pruning, and other optimization techniques to significantly improve the performance of complex queries. Dynamic partition pruning: Automatically skips irrelevant partitions based on query conditions, reducing data scanning volume. Data import and transaction support. Two-phase import: Data is first written to a memory buffer, then asynchronously flushed to the Segment file. This two-phase commit ensures the atomicity of the import transaction. Streaming and batch import: Supports streaming import from real-time data sources such as Flink CDC and Kafka, as well as batch import from HDFS and S3.

[0106] Collaboration between BE and FE nodes. Query process: FE parses the SQL and generates a distributed execution plan, distributing plan fragments to relevant BE nodes; BEs perform local computations and return intermediate results, while FE is responsible for merging the final results. Data distribution: FE distributes data evenly to BE nodes according to bucketing rules. BEs periodically report load information, and FE dynamically adjusts data distribution to achieve load balancing. Fault recovery: When a BE node fails, FE detects a loss of heartbeat, marks the failed node as unavailable, and schedules other BE replicas to repair the data, ensuring service continuity. Resource management and scheduling: Resource group isolation: BE nodes are divided into different resource groups using tags, with each group allocated an independent query queue and resource quota to avoid cross-tenant resource contention. Dynamic resource adjustment: Supports online scaling up / down of BE nodes, with automatic data rebalancing without downtime maintenance.

[0107] Step S400, which involves forwarding the resource configuration information to the database backend node based on the database frontend node, executing the resource configuration information based on the database backend node, and storing the resource usage information, includes:

[0108] Step S401: Forward the resource configuration information to the database backend node based on the database frontend node;

[0109] Step S402: Obtain resources from the third priority task group and the fourth priority task group based on the database backend node, and allocate the obtained resources to the first priority task group and the second priority task group.

[0110] Step S403: Store the resource usage information.

[0111] The database front-end node forwards the resource configuration information to the database back-end node, prioritizing resource allocation to scheduled task groups with high weights and tasks about to start, followed by resource allocation to resource groups with high utilization and high weights. The principle for resource acquisition is: prioritizing resources from scheduled task groups with low weights that have finished, followed by resource allocation from task groups with low resource utilization; making every effort to allocate resources to ensure full utilization. Resource usage information is also stored.

[0112] This embodiment of the invention also requires pre-configuration of the cgrouping method for each node. An example of this configuration is as follows:

[0113] Configure the cgrouping method for each BE node (taking cgroup v1 as an example).

[0114] # Create a directory named doris under the CGroup path. The user can specify this directory name as they wish.

[0115] mkdir / sys / fs / cgroup / cpu / doris

[0116] #Change the permissions of this directory to read, write, and execute.

[0117] chmod 770 / sys / fs / cgroup / cpu / doris

[0118] #Assign ownership of this directory to Doris's account.

[0119] chown-R doris:doris / sys / fs / cgroup / cpu / doris

[0120] # Modify the BE configuration file be.conf, add the following configuration, specifying the cgroup path doris_cgroup_cpu_path = / sys / fs / cgroup / cpu / doris

[0121] # Navigate to the root directory of the be node and restart the be node.

[0122] . / be / bin / start_be.sh --daemon

[0123] Configure Doris resource isolation method to workload group hard limit.

[0124] #Creating Groups

[0125] create workload group if not exists g1_soft_test

[0126] properties

[0127] "cpu_hard_limit" = "20%",

[0128] "memory_limit" = 20%,

[0129] "read_bytes_per_second"="100000000",

[0130] "remote_read_bytes_per_second"="100000000",

[0131] "max_concurrency" = "100",

[0132] "max_queue_size" = "100";

[0133] "queue_timeout" = "600000" );

[0135] #Assign group permissions to users

[0136] GRANT USAGE_PRIV ON WORKLOAD GROUP'g1_soft_test'TO'user_g1_soft_test';

[0137] # Bind users to groups

[0138] set property for'user_g1_soft_test''default_workload_group'='g1_soft_test';

[0139] Deploy Doris resource dynamic adjustment tool

[0140] The main logic of the program is as follows:

[0141]

[0142]

[0143] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0144] It supports dynamic resource adjustment, which can make full use of system resources, improve query response time, and ensure system stability.

[0145] By introducing the concept of weights, higher weights are assigned to tasks that require priority protection, so that resources can be allocated to them first or in greater quantities.

[0146] It supports dynamic adjustment of resources for scheduled tasks. Scheduled tasks have a tidal nature, running only during a specified time period. During this period, more resources can be allocated, while during other time periods, very few resources need to be allocated.

[0147] This tool can improve operational efficiency by automatically adjusting and adapting resources, thereby greatly reducing labor costs.

[0148] It should be noted that there is no necessary order between the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0149] Another embodiment of the present invention provides a resource dynamic adjustment device, which corresponds one-to-one with the resource dynamic adjustment method of the above embodiments. For example... Figure 3 As shown, device 1 includes:

[0150] Data acquisition module 100 is used to acquire the usage and weight information of each resource group in the current database;

[0151] Resource configuration module 200 is used to generate resource configuration information corresponding to each resource group based on the usage and weight information of each resource group;

[0152] The data forwarding module 300 is used to submit a resource adjustment request to the database front-end node using a hard constraint isolation method to transmit the resource configuration information.

[0153] The resource usage recording module 400 is used to forward the resource configuration information to the database backend node based on the database frontend node, execute the resource configuration information based on the database backend node, and store the resource usage information.

[0154] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0155] In one embodiment, the data acquisition module 100 is specifically used for:

[0156] Retrieve the usage status of each resource group in the current database at a first predetermined time interval;

[0157] Based on the usage of each resource group, a first task group, a second task group, a third task group, and a fourth task group are obtained. The first task group is a scheduled task group whose time to start is less than a first time threshold; the second task group is a scheduled task group whose time to end is less than a second time threshold; the third task group is a task group whose resource utilization rate is greater than a first utilization rate threshold; and the fourth task group is a task group whose resource utilization rate is less than a second utilization rate threshold, and the first utilization rate threshold is greater than the second utilization rate threshold.

[0158] The weight information of each resource group in the current database is obtained at the first predetermined time interval.

[0159] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0160] In one embodiment, the resource configuration module 200 is specifically used for:

[0161] A resource allocation command is triggered every second predetermined time interval;

[0162] In response to the resource allocation instruction, a resource adjustment priority corresponding to the resource group is generated based on the usage and weight information of each resource group;

[0163] Resource configuration information for each resource group is generated based on resource adjustment priority.

[0164] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0165] In one embodiment, the resource configuration module 200 is further configured to:

[0166] In response to the resource allocation instruction, the resource adjustment is pre-divided into four priorities: first priority, second priority, third priority, and fourth priority.

[0167] Set the priority of the first task group whose weight is greater than the preset first weight threshold to the first priority;

[0168] Set the priority of the third task group whose weight is greater than the preset second weight threshold to the second priority;

[0169] Set the priority of the second task group, whose scheduled task has ended and whose weight is less than the preset third weight threshold, to the third priority.

[0170] Set the priority of the fourth task group to the fourth priority.

[0171] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0172] In one embodiment, the resource configuration module 200 is further configured to:

[0173] Configure the first priority task group as the first in the resource addition order;

[0174] Configure the second priority task group as the second priority group in the resource addition order;

[0175] Configure the third priority task as the first in the resource reduction order;

[0176] Configure the fourth priority task group as the second priority in the resource reduction order.

[0177] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0178] In one embodiment, the apparatus further includes a threshold setting module, which is specifically used for:

[0179] A first utilization threshold is preset, which is one or more of a first CPU utilization threshold, a first memory threshold, and a first I / O transfer rate threshold.

[0180] A second utilization rate is preset, and the second utilization rate threshold is one or more combinations of a second CPU utilization threshold, a second memory threshold, and a second I / O transfer rate threshold;

[0181] The first and second time thresholds are set in advance.

[0182] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0183] In one embodiment, the resource usage recording module 400 is specifically used for:

[0184] The resource configuration information is forwarded to the database backend node based on the database frontend node;

[0185] Based on the database backend node, resources are obtained from the third priority task group and the fourth priority task group, and the obtained resources are allocated to the first priority task group and the second priority task group.

[0186] It also stores information on resource usage.

[0187] For specific implementation details, please refer to the method embodiment; they will not be repeated here.

[0188] This invention provides a resource dynamic adjustment device that supports dynamic adjustment of resources, makes full use of system resources, improves query response time, and ensures system stability. For tasks with high weight, resources can be allocated preferentially or in greater quantities, thereby improving operational efficiency.

[0189] Another embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as follows. Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a dynamic resource adjustment method on the server side.

[0190] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a dynamic resource adjustment method.

[0191] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0192] Retrieve the usage and weight information of each resource group in the current database;

[0193] Based on the usage and weight information of each resource group, resource configuration information corresponding to each resource group is generated.

[0194] The resource configuration information is used to submit a resource adjustment request to the database front-end node in a hard-limit isolation manner;

[0195] The database front-end node forwards the resource configuration information to the database back-end node, executes the resource configuration information based on the database back-end node, and stores the resource usage information.

[0196] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0197] Retrieve the usage and weight information of each resource group in the current database;

[0198] Based on the usage and weight information of each resource group, resource configuration information corresponding to each resource group is generated.

[0199] The resource configuration information is used to submit a resource adjustment request to the database front-end node in a hard-limit isolation manner;

[0200] The database front-end node forwards the resource configuration information to the database back-end node, executes the resource configuration information based on the database back-end node, and stores the resource usage information.

[0201] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0202] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0203] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can exist in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0205] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

[0206] Among other things, conditional language such as “can,” “may,” “may,” or “may,” unless otherwise specifically stated or otherwise understood as in the context in which they are used, is generally intended to convey that a particular implementation may include (but not others) certain features, elements, and / or operations. Therefore, such conditional language is also generally intended to imply that features, elements, and / or operations are necessary for one or more implementations in any way, or that one or more implementations must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular implementation.

[0207] The contents already described herein in this specification and accompanying drawings include examples of methods and apparatuses capable of providing dynamic resource adjustment. It is certainly not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of this disclosure, but it will be appreciated that many other combinations and substitutions of the disclosed features are possible. Therefore, it will be apparent that various modifications can be made to this disclosure without departing from the scope or spirit of this disclosure. Furthermore, or in alternatives, other embodiments of this disclosure may become apparent from consideration of this specification and accompanying drawings and from practice of this disclosure as presented herein. It is intended that the examples presented in this specification and accompanying drawings be considered illustrative rather than restrictive in all respects. Although specific terminology is used herein, it is used in a general and descriptive sense and is not intended for limiting purposes.

Claims

1. A method for dynamic resource adjustment, characterized in that... The method includes: Retrieve the usage and weight information of each resource group in the current database; Based on the usage and weight information of each resource group, resource configuration information corresponding to each resource group is generated. The resource configuration information is used to submit a resource adjustment request to the database front-end node in a hard-limit isolation manner; The database front-end node forwards the resource configuration information to the database back-end node, executes the resource configuration information based on the database back-end node, and stores the resource usage information.

2. The resource dynamic adjustment method according to claim 1, characterized in that, The step of obtaining the usage and weight information of each resource group in the current database includes: Retrieve the usage status of each resource group in the current database at a first predetermined time interval; Based on the usage of each resource group, a first task group, a second task group, a third task group, and a fourth task group are obtained. The first task group is a scheduled task group whose time to start is less than a first time threshold; the second task group is a scheduled task group whose time to end is less than a second time threshold; the third task group is a task group whose resource utilization rate is greater than a first utilization rate threshold; and the fourth task group is a task group whose resource utilization rate is less than a second utilization rate threshold, and the first utilization rate threshold is greater than the second utilization rate threshold. The weight information of each resource group in the current database is obtained at the first predetermined time interval.

3. The resource dynamic adjustment method according to claim 2, characterized in that, The process of generating resource configuration information corresponding to each resource group based on the usage and weight information of each resource group includes: A resource allocation command is triggered every second predetermined time interval; In response to the resource allocation instruction, a resource adjustment priority corresponding to the resource group is generated based on the usage and weight information of each resource group; Resource configuration information for each resource group is generated based on resource adjustment priority.

4. The resource dynamic adjustment method according to claim 3, characterized in that, The step of responding to the resource allocation instruction by generating a resource adjustment priority corresponding to each resource group based on the usage and weight information of each resource group includes: In response to the resource allocation instruction, the resource adjustment is pre-divided into four priorities: first priority, second priority, third priority, and fourth priority. Set the priority of the first task group whose weight is greater than the preset first weight threshold to the first priority; Set the priority of the third task group whose weight is greater than the preset second weight threshold to the second priority; Set the priority of the second task group, whose scheduled task has ended and whose weight is less than the preset third weight threshold, to the third priority. Set the priority of the fourth task group to the fourth priority.

5. The resource dynamic adjustment method according to claim 4, characterized in that, The process of generating resource configuration information for each resource group based on resource adjustment priority includes: Configure the first priority task group as the first in the resource addition order; Configure the second priority task group as the second priority group in the resource addition order; Configure the third priority task as the first in the resource reduction order; Configure the fourth priority task group as the second priority in the resource reduction order.

6. The resource dynamic adjustment method according to claim 1, characterized in that, Before obtaining the usage status of each resource group in the current database at each first predetermined time interval, the process includes: A first utilization threshold is preset, which is one or more of a first CPU utilization threshold, a first memory threshold, and a first I / O transfer rate threshold. A second utilization rate is preset, and the second utilization rate threshold is one or more combinations of a second CPU utilization threshold, a second memory threshold, and a second I / O transfer rate threshold; The first and second time thresholds are set in advance.

7. The resource dynamic adjustment method according to claim 5, characterized in that, The process of forwarding the resource configuration information to the database backend node based on the database frontend node, executing the resource configuration information based on the database backend node, and storing the resource usage information includes: The resource configuration information is forwarded to the database backend node based on the database frontend node; Based on the database backend node, resources are obtained from the third priority task group and the fourth priority task group, and the obtained resources are allocated to the first priority task group and the second priority task group. It also stores information on resource usage.

8. A resource dynamic adjustment device, characterized in that, The device includes: The data acquisition module is used to obtain the usage and weight information of each resource group in the current database; The resource configuration module is used to generate resource configuration information corresponding to each resource group based on the usage and weight information of each resource group; The data forwarding module is used to submit resource adjustment requests to the database front-end node using hard constraint isolation of the resource configuration information; The resource usage recording module is used to forward the resource configuration information to the database backend node based on the database frontend node, execute the resource configuration information based on the database backend node, and store the resource usage information.

9. A computer device, characterized in that, The computer device includes at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the resource dynamic adjustment method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the resource dynamic adjustment method according to any one of claims 1-7.

Images