A defragmentation method, device, and medium for GoldenDB database
By uniformly collecting metadata from the GoldenDB database and filtering tables based on dual thresholds, combined with strategies for processing large tables individually or grouping all tables, the problem of incompatibility with distributed architectures and low defragmentation efficiency in existing technologies is solved, achieving a fragment defragmentation effect with controllable resource consumption and minimal business impact.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG CITY COMMERCIAL BANK COOP ALLIANCE CO LTD
- Filing Date
- 2026-03-24
- Publication Date
- 2026-05-26
AI Technical Summary
Existing defragmentation solutions are not compatible with the GoldenDB distributed architecture. They are inefficient and lack flexibility, and cannot plan tasks differently based on table size, resulting in improper resource consumption and excessive impact on online business.
Cluster metadata is collected uniformly through management nodes, tables to be processed are filtered based on dual thresholds, and a strategy of processing large tables separately or processing all tables in groups is adopted. Combined with an improved greedy algorithm, tasks are grouped and executed in parallel, adapting to the GoldenDB distributed architecture, so as to achieve controllable resource consumption and minimize the impact on online business.
It improved the accuracy and efficiency of defragmentation of the GoldenDB database, reduced invalid defragmentation, reduced the impact on online business, and improved the overall defragmentation efficiency.
Smart Images
Figure CN121901212B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a defragmentation method, device and medium for GoldenDB database. Background Technology
[0002] GoldenDB is a distributed relational database designed for financial-grade scenarios. It employs a "computation and storage separation" and "horizontal sharding" architecture, splitting a large table across multiple shard nodes for storage. It is widely used in core business systems such as banking and securities. During long-term operation, GoldenDB generates significant table fragmentation due to data insertion, deletion, and update operations. This fragmentation severely reduces I / O efficiency for data queries and wastes storage space: firstly, this free space is difficult to fully utilize for subsequent insertion operations, leading to table physical file bloat; secondly, under the sharding architecture, the uneven distribution of table fragments across nodes further reduces database query performance and storage utilization. Existing defragmentation solutions are mainly designed for single-machine databases (such as MySQL) and have the following drawbacks: 1. Incompatible with GoldenDB's distributed architecture: Traditional solutions only support defragmentation of single-node tables, cannot collect table fragmentation information from multiple sharded nodes in batches, and cannot coordinate defragmentation tasks across nodes; 2. Low defragmentation efficiency: They do not differentiate between table sizes, mixing large and small tables for defragmentation. The time-consuming defragmentation of large tables can block the defragmentation of small tables, resulting in an excessively long overall defragmentation cycle; 3. Lack of flexibility: They do not support user-defined threshold filtering or manual selection of large tables for defragmentation, and cannot adapt to the needs of different business scenarios.
[0003] The core problem lies in the failure to fully consider the decisive role of "table size" as a key dimension in the resource consumption and system impact of the sorting task. Sorting a 1TB large table requires significantly different levels of I / O, CPU, network bandwidth, and lock holding time compared to sorting a 1MB small table.
[0004] Therefore, a defragmentation method is needed that is compatible with the GoldenDB distributed architecture and can perform differentiated task planning and execution based on table size, so as to achieve controllable resource consumption, minimize the impact on online business, and improve overall defragmentation efficiency. Summary of the Invention
[0005] This application provides a defragmentation method, device, and medium for GoldenDB databases to solve the above-mentioned problems.
[0006] On the one hand, this application provides a defragmentation method for a GoldenDB database, the method comprising the following steps:
[0007] Step S1: Obtain metadata information of all tables in the cluster from the management node of the GoldenDB distributed database, where the management node is the metadata center of the GoldenDB cluster;
[0008] Step S2: Based on preset dual thresholds, filter out the tables that meet the sorting conditions from all tables to complete the accurate determination of the sorting objects;
[0009] Step S3: Intelligently group the table to be sorted according to the preset defragmentation strategy, and generate corresponding defragmentation tasks based on the grouping results;
[0010] Step S4: Distribute the defragmentation task to the compute nodes of the GoldenDB database for execution, and complete the data defragmentation operation within the cluster.
[0011] In one implementation of this application, in step S1, the metadata information includes the table physical size, number of data blocks, fragmentation rate, percentage of free pages, table name, database to which it belongs, creation time, and sharding distribution information of the table on each data node; the metadata information is periodically pushed to the management node by the DB node's DBAgent, and the management node stores the metadata information in the metadata management service MDS. The GoldenDB distributed database adopts a share-nothing architecture.
[0012] In one implementation of this application, in step S2, the dual thresholds are a table size threshold and a fragmentation rate threshold, and the consolidation condition is that the physical size of the table is greater than or equal to the table size threshold and the table fragmentation rate is greater than or equal to the fragmentation rate threshold. Step S2 also includes data conversion and fragmentation rate calculation steps. The data conversion converts the byte unit of the table size in the metadata to MB unit and retains two decimal places. The fragmentation rate is determined for InnoDB engine tables based on the ratio of logical space to physical space in the data file, and for MyISAM engine tables based on the proportion of holes in the data file.
[0013] In one implementation of this application, in step S3, the preset defragmentation strategy includes a large table separate processing strategy and a full table group processing strategy, which can be selected according to the business scenario and system status of the GoldenDB database; before selecting the defragmentation strategy, the tables to be defragmented are sorted in descending order according to their physical size.
[0014] In one implementation of this application, when the large table processing strategy is selected, the specific operations of step S3 include: setting a large table threshold, marking tables whose physical size exceeds the large table threshold in the sorted tables to be processed as large tables, and generating a separate fragmentation task for each large table; for the remaining non-large tables to be processed, calculating the number of groups according to the preset number of tables in each group, performing balanced grouping of the non-large tables, and generating a fragmentation task for each group; the fragmentation tasks corresponding to the large tables and the fragmentation tasks corresponding to the non-large table groups are executed in parallel.
[0015] In one implementation of this application, when the full table grouping processing strategy is selected, the specific operations of step S3 include: calculating the number of groups of all tables to be sorted according to the preset number of tables in each group; using an improved greedy algorithm to group the sorted tables to be sorted so that the difference in the total physical size of the tables in each group is minimized, thereby achieving system load balancing; generating a corresponding defragmentation task for each group, and executing the defragmentation tasks of all groups simultaneously.
[0016] In one implementation of this application, the improved greedy algorithm takes as input a list of tables to be organized, sorted in descending order of physical size, and a preset number of tables in each group; the output is the grouping result of the tables to be organized. The specific implementation steps of the improved greedy algorithm are as follows: calculate the number of groups, group_count = ceil(len(list of tables to be organized) / preset number of tables in each group); initialize an empty list of grouping results groups, and initialize the total size array sums, with the length of sums being group_count and all elements initialized to 0; traverse the list of tables to be organized, and assign each table to the group with the smallest total size in sums, and update the total size of the group in sums after assignment; after the traversal is complete, output the grouping result groups.
[0017] In one implementation of this application, the specific execution process of step S4 includes: the management node distributes the generated defragmentation tasks to stateless computing nodes, wherein the computing nodes support horizontal scaling and can process multiple defragmentation tasks in parallel; the computing nodes first identify the storage engine type of the table to be defragmented, and if it is an InnoDB engine, they send an OPTIMIZE TABLE tablename statement to the data node corresponding to the table to be defragmented to execute defragmentation; the data nodes use row-level locks or table-level locks when performing defragmentation operations to avoid affecting online business operations; during the execution process, the start time and end time of each defragmentation task, as well as the defragmentation time of each table to be defragmented, are recorded to achieve real-time monitoring of the defragmentation progress.
[0018] Secondly, this application also provides a defragmentation device for a GoldenDB database, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the aforementioned defragmentation method for a GoldenDB database.
[0019] Finally, this application also provides a non-volatile computer storage medium for defragmenting a GoldenDB database, storing computer-executable instructions that are executed by a processor to implement the aforementioned defragmentation method for a GoldenDB database.
[0020] This application provides a defragmentation method, device, and medium for GoldenDB databases, which have the following beneficial effects:
[0021] 1. This application is adapted to the GoldenDB distributed architecture, and collects full table metadata from the management node. It accurately filters tables to be reorganized based on dual thresholds of table size and fragmentation rate, abandoning the limitations of traditional single-machine reorganization, achieving precise positioning of reorganization objects, effectively avoiding ineffective reorganization, and improving the targeting and effectiveness of fragment reorganization.
[0022] 2. This application supports two differentiated strategies: processing large tables separately and processing all tables in groups. First, the tables are sorted in descending order of physical size. Then, an improved greedy algorithm is used to achieve load balancing for groups. Large and small tables are planned separately and executed in parallel, which solves the blocking problem of mixed defragmentation in traditional solutions and greatly improves the overall defragmentation efficiency of the cluster.
[0023] 3. This application distributes the sorting tasks to horizontally scalable computing nodes for parallel execution. It adapts the corresponding sorting statements according to the storage engine of the table, and the data nodes use row-level / table-level locks to perform operations. At the same time, it records the task execution time and consumption in real time, which makes full use of system resources and minimizes the impact on online business, so as to achieve controllable sorting process. Attached Figure Description
[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0025] Figure 1 A flowchart of a defragmentation method for a GoldenDB database provided in this application embodiment;
[0026] Figure 2This is a schematic diagram of a defragmentation device for a GoldenDB database provided in an embodiment of this application. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0028] This application provides a method, device, and medium for defragmenting a GoldenDB database. The technical solution proposed in this application will be described in detail below with reference to the accompanying drawings.
[0029] Figure 1 This document provides a flowchart of a defragmentation method for a GoldenDB database, as illustrated in an embodiment of this application. Figure 1 As shown, the method mainly includes the following steps:
[0030] Step S1: Obtain metadata information of all tables in the cluster from the management node of the GoldenDB distributed database, where the management node is the metadata center of the GoldenDB cluster;
[0031] Step S2: Based on preset dual thresholds, filter out the tables that meet the sorting conditions from all tables to complete the accurate determination of the sorting objects;
[0032] Step S3: Intelligently group the table to be sorted according to the preset defragmentation strategy, and generate corresponding defragmentation tasks based on the grouping results;
[0033] Step S4: Distribute the defragmentation task to the compute nodes of the GoldenDB database for execution, and complete the data defragmentation operation within the cluster.
[0034] In this application, in step S1, the metadata information includes the table physical size, number of data blocks, fragmentation rate, proportion of free pages, table name, database to which it belongs, creation time, and sharding distribution information of the table on each data node; the metadata information is pushed to the management node by the DB node's DBAgent on a regular basis, and the management node stores the metadata information in the metadata management service MDS. The GoldenDB distributed database adopts a share-nothing architecture.
[0035] In this application, in step S2, the dual thresholds are a table size threshold and a fragmentation rate threshold. The consolidation condition is that the physical size of the table is greater than or equal to the table size threshold and the table fragmentation rate is greater than or equal to the fragmentation rate threshold. Step S2 also includes data conversion and fragmentation rate calculation steps. Data conversion converts the byte unit of the table size in the metadata to MB unit and retains two decimal places. Fragmentation rate calculation is determined for InnoDB engine tables based on the ratio of logical space to physical space in the data file, and for MyISAM engine tables based on the proportion of holes in the data file.
[0036] In this application, in step S3, the preset defragmentation strategy includes a large table separate processing strategy and a full table group processing strategy, which can be selected according to the business scenario and system status of the GoldenDB database; before selecting the defragmentation strategy, the tables to be defragmented are sorted in descending order according to their physical size.
[0037] In this application, when the large table processing strategy is selected, the specific operations of step S3 include: setting a large table threshold, marking tables whose physical size exceeds the large table threshold in the sorted tables to be processed as large tables, and generating a separate defragmentation task for each large table; for the remaining non-large tables to be processed, calculating the number of groups according to the preset number of tables in each group, performing balanced grouping of the non-large tables, and generating a defragmentation task for each group; the defragmentation tasks corresponding to the large tables and the defragmentation tasks corresponding to the non-large table groups are executed in parallel.
[0038] In this application, when the full table grouping processing strategy is selected, the specific operations of step S3 include: calculating the number of groups of all tables to be processed according to the preset number of tables in each group; using an improved greedy algorithm to group the sorted tables to be processed, so that the difference in the total physical size of the tables in each group is minimized, thereby achieving system load balancing; generating a corresponding defragmentation task for each group, and the defragmentation tasks of all groups are executed simultaneously.
[0039] In this application, the improved greedy algorithm takes as input a list of tables to be organized, sorted in descending order of physical size, and a preset number of tables in each group; the output is the grouping result of the tables to be organized; the specific implementation steps of the improved greedy algorithm are as follows: calculate the number of groups, group_count = ceil(len(list of tables to be organized) / preset number of tables in each group); initialize an empty list of grouping results groups, and initialize the total size array sums, with the length of sums being group_count and all elements initialized to 0; traverse the list of tables to be organized, and assign each table to the group with the smallest total size in sums, and update the total size of that group in sums after assignment; after the traversal is complete, output the grouping result groups.
[0040] In this application, the specific execution process of step S4 includes: the management node distributes the generated defragmentation tasks to stateless compute nodes, which support horizontal scaling and can process multiple defragmentation tasks in parallel; the compute nodes first identify the storage engine type of the table to be defragmented, and if it is an InnoDB engine, they send an OPTIMIZE TABLE tablename statement to the data node corresponding to the table to be defragmented to execute defragmentation; the data nodes use row-level locks or table-level locks when performing defragmentation operations to avoid affecting online business operations; during the execution process, the start time and end time of each defragmentation task, as well as the defragmentation time of each table to be defragmented, are recorded to achieve real-time monitoring of the defragmentation progress.
[0041] The GoldenDB distributed database involved in this application adopts a share-nothing architecture, which mainly consists of four core components: management node, compute node, data node and global transaction management node (GTM).
[0042] The management node is responsible for cluster management, metadata management, and operations and maintenance management. Internally, the management node contains several service modules: the Metadata Management Service (MDS) stores all the system's metadata and serves as the metadata center for the entire GoldenDB database cluster; Insight provides a unified, visual operations and maintenance management interface; and the Cluster Management Service (CM) is responsible for high availability management and task scheduling for the cluster.
[0043] Compute nodes are stateless computing modules responsible for receiving user SQL operations, performing logical and physical optimizations, and generating distributed query plans that satisfy distributed transaction consistency.
[0044] Data nodes are responsible for storing actual business data and provide functions such as data sharding, backup and recovery, and master-slave synchronization. Data nodes adopt a one-master-multiple-slave deployment mode and achieve data synchronization through fast synchronization replication technology.
[0045] The Global Transaction Management Node (GTM) is specifically designed to handle distributed transactions, maintain the entire lifecycle of global transactions, and provide functions for allocating, reclaiming, and querying global transaction IDs.
[0046] In this invention, the management node acquires and maintains metadata information through the cluster. As a distributed key-value storage system, it stores metadata information and maintains metadata consistency. After the DB node is configured, DBAgent automatically starts and periodically pushes metadata information from tables within the cluster to the management node.
[0047] Regarding metadata acquisition and storage, the metadata information obtained by the management node from the cluster includes: basic table information such as table name, database to which it belongs, and creation time; table size information such as data size, index size, and total size (unit: bytes); fragmentation rate information determined by calculating the ratio of logical space to physical space of the data file; and the sharding status of the table on each data node. The metadata acquisition mechanism is as follows: after the DB node is installed and configured, DBAgent automatically starts and periodically pushes the metadata information of the tables in the cluster to the management node.
[0048] In this application, the system provides two key threshold parameters: Table size threshold: in MB, only tables with a total size greater than or equal to this threshold will be included in the filtering range. Fragmentation rate threshold: in percentage, only tables with a fragmentation rate greater than or equal to this threshold will be included in the filtering range.
[0049] The specific steps for table filtering are as follows: First, convert the table size (in bytes) obtained from the metadata to MB units, rounded to two decimal places. Then, calculate the fragmentation rate based on the logical and physical space of the data file. For InnoDB tables, the logical space is the total size of the data and indexes, and the physical space is the actual disk space used. For MyISAM tables, the fragmentation rate is determined by the percentage of holes in the statistics file. Only tables that simultaneously meet the criteria of table size ≥ table size threshold and fragmentation rate ≥ fragmentation rate threshold will be selected for defragmentation.
[0050] Strategy 1: Separate Processing for Large Tables. The core idea of this strategy is to process large tables and small tables separately, avoiding the impact of fragmentation on other tables caused by the defragmentation of large tables. The specific process is as follows:
[0051] The system presets or dynamically calculates a large table threshold (e.g., 100GB). Tables exceeding this threshold are considered large tables. All filtered tables are sorted in descending order of size. The sorted table list is traversed, and tables exceeding the large table threshold are marked as large tables. The large table task and the grouping task can be executed in parallel. A separate defragmentation task is generated for each large table; the remaining non-large tables are evenly grouped according to table size, with each group containing a preset number of tables (e.g., 5); a defragmentation task is generated for each group. The large table task and the grouping task can be executed in parallel.
[0052] Strategy 2: Full table grouping processing strategy.
[0053] The full-table grouping strategy groups all tables requiring defragmentation to ensure a balanced distribution of system load. The specific process is as follows: Sort all selected tables in descending order of size; calculate the required number of groups based on the total number of tables and the preset number of tables per group. For example, if there are 20 tables, with 5 tables per group, then 4 groups are needed. Generate a defragmentation task for each group, with each task containing all tables within that group. A greedy algorithm is used for grouping, with the following steps: Initialize an array to store the total size of each group, with an initial value of 0; sequentially assign each table to the group with the smallest current total size, updating the total size of that group; generate a defragmentation task for each group, with each task containing all tables within that group. All grouping tasks are executed simultaneously to ensure full utilization of system resources.
[0054] This application employs an improved greedy algorithm for grouping, ensuring that the total size of each group is as balanced as possible. The specific implementation of the algorithm is as follows: The input to the algorithm is a list of tables to be processed, sorted in descending order of physical size, and a pre-defined number of tables that each group can hold. The output is the final grouping result of the tables to be processed. First, the total number of groups to be divided is calculated, which is the total number of tables to be processed divided by the number of tables in each group, rounded up. Next, an empty list of grouping results is initialized, along with a total size array. The array length is the same as the total number of groups, and all elements are initially set to 0. Then, each table in the list of tables to be processed is traversed sequentially. The group index corresponding to the smallest element in the total size array is found, and the current table is assigned to the group corresponding to that index. The physical size of the current table is then added to the value of that group in the total size array. After all tables have been assigned, the final grouping result list is output. This grouping method ensures that the difference in the total size of each group is minimized, thereby achieving load balancing.
[0055] The execution flow of the defragmentation task in this application is as follows: The management node distributes the generated defragmentation task to each compute node. Compute nodes are stateless, horizontally scalable, and capable of processing multiple tasks in parallel. For InnoDB tables, the compute node sends an `OPTIMIZE TABLE table_name` statement to the relevant data nodes; the data nodes perform table rebuilding operations, using row-level locks during which other operations are not affected; the data nodes also perform table optimization operations, using table-level locks during which the table's storage engine type needs to be identified before defragmentation. The `OPTIMIZE TABLE table_name` statement is then used for defragmentation. This method eliminates fragmentation by rebuilding the table and does not lock the entire table (supports online DDL). Execution process: The logs for each task are reviewed, and the time spent defragmenting each table is recorded during the execution of each task.
[0056] The following is a specific example:
[0057] Assume there are 100 tables in the GoldenDB cluster, of which 20 tables meet the defragmentation criteria (table size ≥ 100MB and fragmentation rate ≥ 10%).
[0058] Scenario 1: Using a separate large table processing strategy. The preset large table threshold is 100GB; 3 large tables are identified (150GB, 120GB, and 110GB respectively); the remaining 17 tables are grouped into 4 groups of 5 tables each (the first 3 groups of 5 tables each, and the last group of 2 tables each); a total of 7 defragmentation tasks are generated (3 large table tasks + 4 group tasks).
[0059] Scenario 2: Using a full table grouping strategy. All 20 tables are grouped; each group is pre-defined to contain 5 tables, forming 4 groups; the total size of each group is calculated using a greedy algorithm to ensure load balancing across groups; a total of 4 defragmentation tasks are generated.
[0060] The above describes a defragmentation method for a GoldenDB database provided by an embodiment of this application. Based on the same inventive concept, this application also provides a defragmentation device for a GoldenDB database. Figure 2 A schematic diagram of a defragmentation device for a GoldenDB database provided in this application embodiment is shown below. Figure 2 As shown, the device mainly includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor; wherein the memory 202 stores instructions that can be executed by the at least one processor 201, and the instructions are executed by the at least one processor 201 to enable the at least one processor 201 to complete the aforementioned defragmentation method for the GoldenDB database.
[0061] In addition, embodiments of this application also provide a non-volatile computer storage medium for defragmenting a GoldenDB database, storing computer-executable instructions, which are executed by a processor to implement the aforementioned defragmentation method for a GoldenDB database.
[0062] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0065] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0066] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0067] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0068] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A defragmentation method for GoldenDB databases, characterized in that, The method includes the following steps: Step S1: Obtain metadata information for all tables within the cluster from the management node of the GoldenDB distributed database. The management node is the metadata center of the GoldenDB cluster. The metadata information includes table physical size, number of data blocks, fragmentation rate, free page ratio, table name, database to which it belongs, creation time, and sharding distribution information of the table on each data node. The metadata information is periodically pushed to the management node by the DB node's DBAgent, and the management node stores the metadata information in the Metadata Management Service (MDS). The GoldenDB distributed database adopts a share-nothing architecture. Step S2: Based on preset dual thresholds, select tables that meet the reorganization conditions from all tables to accurately determine the reorganization targets; the dual thresholds are the table size threshold and the fragmentation rate threshold, and the reorganization conditions are that the physical size of the table is greater than or equal to the table size threshold and the fragmentation rate of the table is greater than or equal to the fragmentation rate threshold; Step S2 also includes data conversion and fragmentation rate calculation steps. Data conversion converts the byte unit of the table size in the metadata to MB unit and retains two decimal places. Fragmentation rate calculation is determined for InnoDB engine tables based on the ratio of logical space to physical space in the data file, and for MyISAM engine tables based on the proportion of holes in the data file; Step S3: Intelligently group the tables to be processed according to the preset defragmentation strategy, and generate corresponding defragmentation tasks for the grouping results; the preset defragmentation strategy includes a large table separate processing strategy and a full table group processing strategy, which can be selected according to the business scenario and system status of the GoldenDB database; before selecting the defragmentation strategy, sort the tables to be processed in descending order according to the physical size of the tables. Step S4: Distribute the defragmentation task to the compute nodes of the GoldenDB database for execution, completing the data defragmentation operation within the cluster. The specific execution process includes: the management node distributes the generated defragmentation task to stateless compute nodes, which support horizontal scaling and can process multiple defragmentation tasks in parallel; the compute nodes first identify the storage engine type of the table to be defragmented. If it is an InnoDB engine, the compute nodes send an OPTIMIZETABLE tablename statement to the data node corresponding to the table to be defragmented to execute the defragmentation; the data nodes use row-level locks or table-level locks when performing the defragmentation operation to avoid affecting online business operations; during the execution process, the start time and end time of each defragmentation task, as well as the defragmentation time of each table to be defragmented, are recorded to achieve real-time monitoring of the defragmentation progress.
2. The defragmentation method for GoldenDB database according to claim 1, characterized in that, When the strategy of processing large tables separately is selected, the specific operations of step S3 include: setting a large table threshold, marking tables whose physical size exceeds the large table threshold in the sorted tables to be processed as large tables, and generating a separate fragmentation task for each large table; for the remaining non-large tables to be processed, calculating the number of groups according to the preset number of tables in each group, performing balanced grouping of the non-large tables, and generating a fragmentation task for each group; the fragmentation tasks corresponding to the large tables and the fragmentation tasks corresponding to the non-large table groups are executed in parallel.
3. The defragmentation method for GoldenDB database according to claim 1, characterized in that, When the full table grouping processing strategy is selected, the specific operations of step S3 include: calculating the number of groups for all tables to be processed according to the preset number of tables in each group; using an improved greedy algorithm to group the sorted tables to be processed, so as to minimize the difference in the total physical size of the tables in each group and achieve system load balancing; generating a corresponding defragmentation task for each group, and the defragmentation tasks of all groups are executed simultaneously.
4. The defragmentation method for GoldenDB database according to claim 3, characterized in that, The improved greedy algorithm takes as input a list of tables to be processed, sorted in descending order by physical size, and a preset number of tables in each group; the output is the grouping result of the tables to be processed; the specific implementation steps of the improved greedy algorithm are as follows: Calculate the number of groups; Initialize an empty list of grouping results (groups), and also initialize a total size array (sums) with a length equal to the number of groups and all elements initialized to 0. Iterate through the list of tables to be sorted, and assign each table to the group with the smallest total size in sums. After the assignment, update the total size of that group in sums. After the traversal is complete, output the grouping results: groups.
5. A defragmentation device for a GoldenDB database, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a defragmentation method for a GoldenDB database as described in any one of claims 1-4.
6. A non-volatile computer storage medium for defragmenting a GoldenDB database, storing computer-executable instructions, characterized in that, The computer-executable instructions are executed by a processor to implement a defragmentation method for a GoldenDB database as described in any one of claims 1-4.