Data management method and apparatus, program product, device, and storage medium

Abstract

Provided in the present description are a data management method and apparatus, a program product, a device, and a storage medium, wherein a data table in a database is divided into a plurality of partitions; data in the data table is organized into a log-structured merge tree; a merging mode is set for the data table, and different merging modes represent different probabilities of triggering adaptive merging; and at least one sorted string table in the log-structured merge tree is stored in each partition. The method comprises: on the basis of a merging threshold function in a merging mode of a data table to which a target partition belongs, and disk statistical information of the target partition, determining whether the target partition satisfies an adaptive merging condition, wherein the disk statistical information is used for recording storage-related information of data in the target partition; and if the target partition satisfies the adaptive merging condition, performing a merging operation on a sorted string table in the target partition.

Description

Data management methods and apparatus, software products, equipment and storage media Technical Field

[0001] This specification relates to the field of database technology, and more particularly to a data management method and apparatus, program product, device and storage medium. Background Technology

[0002] In today's era of rapid development in the internet and information technology, data generation is exploding, thus placing increasingly higher demands on databases and their management. Distributed databases based on the LSM-Tree storage engine can perform data insertion, update, and deletion operations during Data Manipulation Language (DML) execution. However, deletion operations are only logically marked for deletion; data items are not actually physically deleted until a merge operation is performed. This can severely impact data query performance when there are a large number of insert and delete operations in the database.

[0003] While merging data items can reduce or overcome the performance degradation caused by the aforementioned factors, it also consumes computational and transmission resources. Therefore, finding a balance between resource consumption and improved query performance is crucial. In related technologies, merging data items in the database is performed periodically or initiated based on fixed metrics such as the number of ordered string tables. This avoids resource consumption but fails to improve data query performance. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide a data management method and apparatus, program product, device and storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data management method is proposed; wherein a data table in a database is divided into multiple partitions; data in the data table is organized into a log structure merge tree; a merge mode is set for the data table, and different merge modes indicate different probabilities of triggering adaptive merging; each partition stores at least one ordered string table in the log structure merge tree; the method includes:

[0007] Based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics of the target partition, it is determined whether the target partition meets the adaptive merge conditions; wherein, the disk statistics are used to record storage-related information of the data in the target partition;

[0008] If the target partition meets the adaptive merge condition, perform a merge operation on the ordered string table in the target partition.

[0009] In one or more embodiments of this specification, determining whether the target partition meets the adaptive merge condition based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition, includes:

[0010] If the target partition meets the iteration requirements, it is determined whether the target partition meets the adaptive merging conditions based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition.

[0011] In one or more embodiments of this specification, determining whether the target partition meets the adaptive merge condition based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition, includes:

[0012] If the memory data table in the log structure merge tree is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition, the target partition is determined to meet the adaptive merge condition based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition.

[0013] In one or more embodiments of this specification, the method further includes:

[0014] Determine whether the memory size occupied by the memory data table has reached the memory threshold;

[0015] If the memory size occupied by the memory data table reaches the memory threshold, the memory data table is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

[0016] In one or more embodiments of this specification, different merging modes also indicate different probabilities of triggering a fast freeze;

[0017] The method further includes:

[0018] Based on the freeze threshold function in the merge mode of the data table to which the target partition belongs, and the memory statistics of the memory data table, it is determined whether the memory data table meets the fast freeze condition; or based on the freeze threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics of the target partition and the memory statistics of the memory data table, it is determined whether the memory data table meets the fast freeze condition; wherein, the memory statistics are used to record storage-related information of the data in the memory data table;

[0019] If the memory data table meets the fast freeze condition, the memory data table is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

[0020] In one or more embodiments of this specification, the method further includes:

[0021] After organizing the data in the frozen memory data table into an ordered string table and storing it in the target partition, the disk statistics of the target partition are updated based on the memory statistics of the memory data table.

[0022] In one or more embodiments of this specification, the disk statistics include at least one of the following:

[0023] The data includes the number of rows deleted, the number of rows updated, the proportion of update operations within a preset time period, the number of ordered string tables, the number of slow queries, and the proportion of slow queries. The slow query is a data query operation whose execution time exceeds a time threshold, and the slow query proportion is the proportion of the number of slow queries in the total number of data query operations.

[0024] The memory statistics include at least one of the following:

[0025] The active duration of the in-memory data table, the number of rollback transactions, the number of deleted rows, the number of updated rows, the proportion of update operations within the preset duration, the number of slow queries, and the slow query ratio.

[0026] In one or more embodiments of this specification, the disk statistics are stored in memory, and the memory statistics are stored in memory; and / or,

[0027] The disk statistics include the latest disk statistics within a preset time period and all disk statistics since the last merge operation.

[0028] In one or more embodiments of this specification, the step of performing a merge operation on the ordered string table in the target partition if the target partition meets the adaptive merge condition includes:

[0029] If the target partition meets the adaptive merging conditions, the ordered string table in the target partition is merged according to the merging strategy in the merging mode of the data table to which the target partition belongs. The merging mode of the data table to which the target partition belongs is one of a variety of preset merging modes, wherein the merging threshold function, freezing threshold function and merging strategy are different in different merging modes.

[0030] In one or more embodiments of this specification, among the various preset merge modes, at least one of the following merge strategies is provided: basic merge strategy, advanced merge strategy, and aggressive merge strategy;

[0031] The basic merge strategy includes performing partition merge only, wherein the partition merge is to merge the baseline ordered string table and the incremental ordered string table at the same position in all database instances involved in the partition;

[0032] The advanced merge strategy includes performing a partition merge when the time elapsed since the last partition merge reached a preset period, and performing a basic merge when the time elapsed since the last partition merge did not reach the preset period. The basic merge is to merge the baseline ordered string table and the incremental ordered string table within the database instance.

[0033] The aggressive merger strategy includes performing only the basic merger.

[0034] In one or more embodiments of this specification, the method further includes:

[0035] Using the database as the environment, the disk statistics of the partition and the memory statistics of the memory data table as the state, the merging mode change of the partition as the action, and the reinforcement learning training with a preset reward function until the convergence condition is met, the merging mode of each data table in the database is obtained.

[0036] In one or more embodiments of this specification, the partition merging mode change includes:

[0037] Change the merging mode of the partitions from one preset merging mode to another preset merging mode; and,

[0038] Adjust at least one of the merge threshold function and the freeze threshold function within the merge mode of the partition.

[0039] In one or more embodiments of this specification, the reward function is used to characterize at least one of the following:

[0040] If the execution time of the data query operation decreases after the action is performed, a positive reward will be given; if the execution time of the data query operation increases after the action is performed, a negative reward will be given.

[0041] If the proportion of slow queries decreases after the action is performed, a positive reward is given; if the proportion of slow queries increases after the action is performed, a negative reward is given.

[0042] If the ratio between the logical rows scanned and the physical rows scanned in the data query operation increases after the action is executed, a positive reward will be given; if the ratio between the logical rows scanned and the physical rows scanned in the data query operation decreases after the action is executed, a negative reward will be given.

[0043] If the amount of data read / written by the disk, the number of write operations, or the number of ordered string tables decreases within a preset time period after the action is executed, a positive reward will be given. If the amount of data read / written by the disk, the number of write operations, or the number of ordered string tables increase within a preset time period after the action is executed, a negative reward will be given.

[0044] According to a second aspect of one or more embodiments of this specification, a data management apparatus is provided; wherein a data table in a database is divided into multiple partitions; data in the data table is organized into a log structure merge tree; a merge mode is set for the data table, and different merge modes indicate different probabilities of triggering adaptive merging; each partition stores at least one ordered string table in the log structure merge tree; the apparatus includes:

[0045] The determination module is used to determine whether the target partition meets the adaptive merging conditions based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition; wherein, the disk statistics information is used to record storage-related information of the data in the target partition;

[0046] The merge module is used to perform a merge operation on the ordered string table in the target partition if the target partition meets the adaptive merge conditions.

[0047] In one or more embodiments of this specification, the determining module is specifically used for:

[0048] If the target partition meets the iteration requirements, it is determined whether the target partition meets the adaptive merging conditions based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition.

[0049] In one or more embodiments of this specification, the determining module is specifically used for:

[0050] If the memory data table in the log structure merge tree is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition, the target partition is determined to meet the adaptive merge condition based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition.

[0051] In one or more embodiments of this specification, the apparatus further includes a freezing module for:

[0052] Determine whether the memory size occupied by the memory data table has reached the memory threshold;

[0053] If the memory size occupied by the memory data table reaches the memory threshold, the memory data table is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

[0054] In one or more embodiments of this specification, different merging modes also indicate different probabilities of triggering a fast freeze;

[0055] The device also includes a rapid freezing module for:

[0056] Based on the freeze threshold function in the merge mode of the data table to which the target partition belongs, and the memory statistics of the memory data table, it is determined whether the memory data table meets the fast freeze condition; or based on the freeze threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics of the target partition and the memory statistics of the memory data table, it is determined whether the memory data table meets the fast freeze condition; wherein, the memory statistics are used to record storage-related information of the data in the memory data table;

[0057] If the memory data table meets the fast freeze condition, the memory data table is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

[0058] In one or more embodiments of this specification, the apparatus further includes an updating module for:

[0059] After organizing the data in the frozen memory data table into an ordered string table and storing it in the target partition, the disk statistics of the target partition are updated based on the memory statistics of the memory data table.

[0060] In one or more embodiments of this specification, the disk statistics include at least one of the following:

[0061] The data includes the number of rows deleted, the number of rows updated, the proportion of update operations within a preset time period, the number of ordered string tables, the number of slow queries, and the proportion of slow queries. The slow query is a data query operation whose execution time exceeds a time threshold, and the slow query proportion is the proportion of the number of slow queries in the total number of data query operations.

[0062] The memory statistics include at least one of the following:

[0063] The active duration of the in-memory data table, the number of rollback transactions, the number of deleted rows, the number of updated rows, the proportion of update operations within the preset duration, the number of slow queries, and the slow query ratio.

[0064] In one or more embodiments of this specification, the disk statistics are stored in memory, and the memory statistics are stored in memory; and / or,

[0065] The disk statistics include the latest disk statistics within a preset time period and all disk statistics since the last merge operation.

[0066] In one or more embodiments of this specification, the step of performing a merge operation on the ordered string table in the target partition if the target partition meets the adaptive merge condition includes:

[0067] If the target partition meets the adaptive merging conditions, the ordered string table in the target partition is merged according to the merging strategy in the merging mode of the data table to which the target partition belongs. The merging mode of the data table to which the target partition belongs is one of a variety of preset merging modes, wherein the merging threshold function, freezing threshold function and merging strategy are different in different merging modes.

[0068] In one or more embodiments of this specification, among the various preset merge modes, at least one of the following merge strategies is provided: basic merge strategy, advanced merge strategy, and aggressive merge strategy;

[0069] The basic merge strategy includes performing partition merge only, wherein the partition merge is to merge the baseline ordered string table and the incremental ordered string table at the same position in all database instances involved in the partition;

[0070] The advanced merge strategy includes performing a partition merge when the time elapsed since the last partition merge reached a preset period, and performing a basic merge when the time elapsed since the last partition merge did not reach the preset period. The basic merge is to merge the baseline ordered string table and the incremental ordered string table within the database instance.

[0071] The aggressive merger strategy includes performing only the basic merger.

[0072] In one or more embodiments of this specification, the apparatus further includes a training module for:

[0073] Using the database as the environment, the disk statistics of the partition and the memory statistics of the memory data table as the state, the merging mode change of the partition as the action, and the reinforcement learning training with a preset reward function until the convergence condition is met, the merging mode of each data table in the database is obtained.

[0074] In one or more embodiments of this specification, the partition merging mode change includes:

[0075] Change the merging mode of the partitions from one preset merging mode to another preset merging mode; and,

[0076] Adjust at least one of the merge threshold function and the freeze threshold function within the merge mode of the partition.

[0077] In one or more embodiments of this specification, the reward function is used to characterize at least one of the following:

[0078] If the execution time of the data query operation decreases after the action is performed, a positive reward will be given; if the execution time of the data query operation increases after the action is performed, a negative reward will be given.

[0079] If the proportion of slow queries decreases after the action is performed, a positive reward is given; if the proportion of slow queries increases after the action is performed, a negative reward is given.

[0080] If the ratio between the logical rows scanned and the physical rows scanned in the data query operation increases after the action is executed, a positive reward will be given; if the ratio between the logical rows scanned and the physical rows scanned in the data query operation decreases after the action is executed, a negative reward will be given.

[0081] If the amount of data read / written by the disk, the number of write operations, or the number of ordered string tables decreases within a preset time period after the action is executed, a positive reward will be given. If the amount of data read / written by the disk, the number of write operations, or the number of ordered string tables increase within a preset time period after the action is executed, a negative reward will be given.

[0082] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0083] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0084] processor;

[0085] Memory used to store processor-executable instructions;

[0086] The processor implements the method as described in the first aspect by running the executable instructions.

[0087] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0088] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0089] The data management method provided in this specification involves dividing a database table into multiple partitions. The data in the data table is organized into a log structure merge tree. A merge mode is set for the data table, with different merge modes indicating different probabilities of triggering adaptive merge. Each partition stores at least one ordered string table from the log structure merge tree. In this case, it can be determined whether the target partition meets the adaptive merge condition based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition. If the target partition meets the adaptive merge condition, a merge operation is performed on the ordered string table in the target partition. This method allows for different merging modes to be set on the data table. The merging threshold function within each mode sets the conditions for merging data items, and the merging strategy within the mode sets the method for merging data items. This allows the method to adapt to different data tables, initiating data item merging under different conditions and executing it in different ways. Different conditions result in different data information being considered during merging, different thresholds, and different methods consuming different resources. Therefore, this method can, to a certain extent, find a balance between resource consumption and query performance improvement, thus significantly improving database query performance without significantly increasing resource consumption. Attached Figure Description

[0090] Figure 1 is a schematic diagram of the structure of a distributed database provided in an exemplary embodiment.

[0091] Figure 2 is a flowchart illustrating a data management method provided in an exemplary embodiment.

[0092] Figure 3 is a flowchart illustrating a rapid freezing method provided in an exemplary embodiment.

[0093] Figure 4 is a control logic diagram of a data management method provided in an exemplary embodiment.

[0094] Figure 5 is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0095] Figure 6 is a block diagram of a data management device provided in an exemplary embodiment. Detailed Implementation

[0096] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0097] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0098] First, some of the concepts involved in this instruction manual will be explained.

[0099] Log Structure Merge-Tree (LSM-Tree).

[0100] Memory Table (MemTable): Typically comes in two forms: B-tree and hash table. It is commonly used as a data structure organization form to represent key-value storage in memory.

[0101] Sorted String Table (SSTable) represents static read-only data in the LSM-Tree storage engine.

[0102] Freeze: Converts an active MemTable into a frozen MemTable, preventing new writes.

[0103] Dump: Organize the frozen MemTable data into an SSTable structure and write it to disk.

[0104] Compaction: Merges multiple SSTable files, removes duplicate keys, deletes keys, updates expired keys, and generates a new SSTable file.

[0105] Buffer table: Specifically refers to a type of table in the LSM-Tree architecture that is frequently inserted and deleted (including updates, which can be abstracted as first deleting and then inserting), similar to the replacement of data items in a buffer; it is also commonly seen in the scenario of continuous dequeueing and enqueueing in message queues, hence it is also called a queuing table.

[0106] Baseline SSTable (Major SSTable): An SSTable on disk that stores the committed data that has been merged.

[0107] Incremental SSTable (Mini SSTable): An SSTable that stores incremental data that has not yet been merged on disk. It is obtained by freezing and dumping MemTable or merging multiple incremental SSTables.

[0108] SQL execution plan: The sequence of steps a database designs for executing SQL (Structured Query Language) queries.

[0109] In today's era of rapid development in the internet and information technology, data generation is exploding, placing increasingly higher demands on databases and their management. Databases can perform operations such as insertion, update, and deletion during the execution of Data Manipulation Language (DML). However, deletion is only a logical marker of deletion; data items are not physically deleted until a merge operation is performed. This can severely impact data query performance when there are a large number of insertion and deletion operations, resulting in the buffer table effect: a table may only have a few dozen rows of data, but millions of insertion and deletion operations have actually been performed, generating multiple versions of incremental data. This leads to severe space amplification and read amplification, making data query speeds extremely slow. Simultaneously, the severe mismatch between the number of physical rows and the number of logical rows affects the SQL execution plan obtained by the optimizer based on the cost calculation model, resulting in a significant drop in SQL performance.

[0110] While merging data items can reduce or overcome the performance degradation in data queries caused by the aforementioned factors (i.e., the buffer table effect), it also consumes computational and transmission resources. Therefore, finding a balance between resource consumption and improved query performance is crucial. In related technologies, merging data items in the database is performed periodically or initiated based on fixed metrics such as the number of ordered string tables. This avoids resource consumption but fails to improve data query performance.

[0111] Based on this, at least one embodiment of this specification provides a data management method that can configure a suitable merge mode for each data table in the database. This allows the partitions of these data tables to initiate merge operations at the time specified by the merge mode and execute the merge operations in the manner or strategy specified by the merge mode. In this way, a balance can be found between resource consumption and query performance improvement, so that the impact of the decrease in query performance on each data table is greater than the impact of the resource consumption caused by the merge operation on that data table. This allows for timely optimization of query performance through merge operations without causing excessive resource consumption.

[0112] This method can be applied to database management systems, such as the management system running for each database instance in the distributed database based on the LSM-Tree storage engine shown in Figure 1. It can also be applied to management systems running for standalone databases based on the LSM-Tree storage engine.

[0113] Please refer to Figure 2, which exemplarily illustrates the flow of a data management method, including steps S201 to S203.

[0114] First, it should be noted that for databases managed using this method, the data tables in the database can be divided into multiple partitions; the data in each data table can be organized into an LSM-Tree, which typically includes one MemTable and multiple SSTables. The MemTable is usually stored in memory, while the SSTables are stored on disk; a merge mode is set for each data table, and different merge modes indicate different probabilities of triggering adaptive merge; each partition can store at least one SSTable from the LSM-Tree.

[0115] In practical applications, a merge mode can be set for a data table when it is created or modified. The merge modes set for different data tables within the same database can be the same or different.

[0116] In step S201, based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics of the target partition, it is determined whether the target partition meets the adaptive merge conditions; wherein, the disk statistics are used to record storage-related information of the data in the target partition.

[0117] In this embodiment, the target partition is any partition of any table in the database. For example, the method can target each partition of each table in the database, so this step can obtain the disk statistics information of each partition of each table. In practical applications, the disk statistics information of the target partition can be used to record storage-related information of the data in the target partition. Of course, if there are conditions for obtaining the disk statistics information of the partition, the disk statistics information of the target partition will only be obtained when the target partition meets the conditions.

[0118] For example, the disk statistics can be stored in memory for easy retrieval. For instance, in a distributed database based on the LSM-Tree storage engine, during the execution of Data Manipulation Language (DML), operations such as INSERT, DELETE, UPDATE, and SELECT on the in-memory data table, as well as the number of physical rows, logical rows, and the smallest storage unit (e.g., micro-blocks or pages) scanned for each query, are recorded in memory and added to the disk statistics when the in-memory data table is frozen and dumped into an ordered string table.

[0119] For another example, the disk statistics include the latest disk statistics within a preset time period and all disk statistics since the last merge operation. The latest disk statistics within the preset time period can be referred to as recent values, while all disk statistics since the last merge operation can be referred to as cumulative values.

[0120] For example, the disk statistics information is used to record at least the statistical results of data operations performed within the partition; for example, the disk statistics information includes at least one of the following: number of deleted rows, number of updated rows, proportion of update operations within a preset time period, number of ordered string tables, number of slow queries, and slow query proportion. Here, the number of deleted rows refers to the number of data rows deleted by all delete operations, the number of updated rows refers to the number of data rows updated by all update operations, the proportion of update operations refers to the proportion of the number of update operations in the total number of data operations, the slow query is a data query operation whose execution time exceeds a time threshold, and the slow query proportion is the proportion of the number of slow queries in the total number of data query operations.

[0121] For example, freezing the data in a MemTable and dumping it to a target partition—that is, freezing the MemTable—and then organizing the data in the frozen MemTable into an SSTable and storing it in the target partition, can update the disk statistics of the target partition. In other words, freezing and dumping a memory data table to disk to form an ordered string table will cause changes to the data on the disk, and therefore the disk statistics can be updated based on these changes.

[0122] Specifically, memory statistics for the MemTable can be obtained. In practical applications, MemTable memory statistics can be used to record storage-related information for the data in the MemTable. After organizing the data in the frozen MemTable into an SSTable and storing it in the target partition, the disk statistics of the target partition can be updated based on the MemTable memory statistics. For example, the MemTable memory statistics can be added to the original disk statistics of the target partition.

[0123] For example, the memory statistics are stored in memory for easy access.

[0124] For another example, the memory statistics include at least one of the following: the active duration of the memory data table, the number of rollback transactions, the number of deleted rows, the number of updated rows, the proportion of update operations within a preset duration, the number of slow queries, and the slow query ratio. Here, the number of deleted rows refers to the number of data rows deleted by all delete operations, the number of updated rows refers to the number of data rows updated by all update operations, the proportion of update operations refers to the proportion of update operations to the total number of data operations, the slow query is a data query operation whose execution time exceeds a time threshold, and the slow query ratio is the proportion of the number of slow queries to the total number of data query operations.

[0125] In this embodiment, the database has multiple preset table modes, each representing a different probability of triggering adaptive merging. Specifically, each merge mode includes at least a merge threshold function, which sets the conditions for merging data items in the table. Different conditions result in different data information being considered during the merge, and different thresholds, etc. Among the preset merge modes, the merge threshold functions differ, meaning the conditions for merging data items are different. The merge mode of the data table to which the partition belongs is one of these preset merge modes.

[0126] For example, the merge threshold function is represented as T_compact(m, Y→), where m identifies the merge mode, such as m1, m2, m3, m4, m5, etc., and Y→ represents disk statistics. When T_compact(m, Y→)... → When ) > 0, the partition is determined to meet the adaptive merging condition.

[0127] For example, the merge threshold function can characterize the scenarios that trigger adaptive merging, such as the tombstone scenario, where the ratio of updated to deleted rows exceeds a certain range within a preset time period, or the inefficient read scenario, where the number of logical rows scanned is less than a certain percentage of the number of physical rows scanned within a preset time period.

[0128] In step S202, if the target partition meets the adaptive merging condition, a merging operation is performed on the ordered string table in the target partition.

[0129] In this embodiment, if the target partition meets the adaptive merge condition, the adaptive merge of the target partition can be triggered, that is, the merge operation can be performed on the ordered string table in the target partition.

[0130] For example, the merge mode can also specify a merge strategy. The merge strategy sets how data items in the data table are merged; different strategies consume different resources. Among the preset merge modes, the merge strategies differ, meaning the ways data items are merged differ. The merge mode of the data table to which the target partition belongs is one of the preset merge modes.

[0131] Among the various preset merge modes, at least one of the following merge strategies is provided: basic merge strategy, advanced merge strategy, and aggressive merge strategy. The basic merge strategy may include performing only partition merge (Medium Compaction), whereby partition merge merges the baseline ordered string table and the incremental ordered string table at the same position across all database instances (i.e., database instances) involved in the partition. The advanced merge strategy may include performing partition merge when the time elapsed since the last partition merge reaches a preset period, and performing basic merge (Meta Compaction) when the time elapsed since the last partition merge does not reach the preset period. Basic merge merges the baseline ordered string table and the incremental ordered string table within the database instance, generating a temporary basic ordered string table (Meta SSTable), which replaces the baseline ordered string table during queries. The aggressive merge strategy may include performing only the basic merge.

[0132] In the various preset merging modes, the difficulty of the conditions represented by the merging threshold function is adapted to the merging strategy. That is, when the conditions represented by the merging threshold function are difficult, the merging strategy is a basic merging strategy; when the conditions represented by the merging threshold function are of moderate difficulty, the merging strategy is an advanced merging strategy; and when the conditions represented by the merging threshold function are easy, the merging strategy is an aggressive merging strategy. For example, in merging modes m1, m2, m3, m4, and m5, the difficulty of the conditions represented by the merging threshold function decreases sequentially. In merging modes m1 and m2, the merging strategy is a basic merging strategy; in merging mode m3, the merging strategy is an advanced merging strategy; and in merging modes m4 and m5, the merging strategy is an aggressive merging strategy.

[0133] Based on the above, if the merge strategy in the merge mode of the data table to which the target partition belongs is a merge strategy, then a partition merge will be performed on the target partition in this step; if the merge strategy in the merge mode of the data table to which the target partition belongs is an advanced merge strategy, and the time elapsed since the last time the partition merge was performed reaches a preset period, then a partition merge will be performed on the target partition in this step; if the merge strategy in the merge mode of the data table to which the target partition belongs is an advanced merge strategy, and the time elapsed since the last time the partition merge was performed does not reach a preset period, then a basic merge will be performed on the target partition in this step; if the merge strategy in the merge mode of the data table to which the target partition belongs is an aggressive merge strategy, then a basic merge will be performed on the target partition in this step.

[0134] This method allows for different merging modes to be set on the data table. The merging threshold function within each mode sets the conditions for merging data items, and the merging strategy within the mode sets the method for merging data items. This allows the method to adapt to different data tables, initiating data item merging under different conditions and executing it in different ways. Different conditions result in different data information being considered during merging, different thresholds, and different methods consuming different resources. Therefore, this method can, to a certain extent, find a balance between resource consumption and query performance improvement, thus significantly improving database query performance without significantly increasing resource consumption.

[0135] In some embodiments of this disclosure, the method flow shown in Figure 2 can be initiated in at least one of the following ways:

[0136] Method 1: If the data in the MemTable is frozen and dumped to the target partition, i.e., the MemTable is frozen, and the data in the frozen MemTable is organized into an SSTable and stored in the target partition, the target partition's suitability for adaptive merging can be determined based on the merge threshold function in the merge mode of the target partition's data table and the target partition's disk statistics. Specifically, each time the data in the MemTable is frozen and dumped to the target partition to form an ordered string table, the target partition's disk statistics are first updated, and then the method flow shown in Figure 2 is executed based on the updated disk statistics to determine whether the ordered string table in the target partition needs to be merged.

[0137] Method 2: If the target partition meets the iteration requirements, it can be determined whether the target partition meets the adaptive merge condition based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition. That is, the thread used to execute the method flow shown in Figure 2 can continuously traverse each partition. When the traversal reaches the target partition, the target partition meets the iteration requirements, and then the method flow shown in Figure 2 is executed on the target partition to determine whether it is necessary to merge the ordered string table in the target partition.

[0138] In this embodiment, the method flow shown in Figure 2 is initiated by at least one of the above methods to determine whether the partitions need to be adaptively merged at an appropriate time, thereby further improving the accuracy and targeting of the merging operation and further enhancing the data query performance of the database.

[0139] In some embodiments of this disclosure, the method further includes the rapid freezing process shown in Figure 3, including steps S301 to S303.

[0140] In step S301, it is determined whether the memory data table meets the fast-freeze condition based on the freeze threshold function in the merge mode of the data table to which the target partition belongs, and the memory statistics of the memory data table; or it is determined whether the memory data table meets the fast-freeze condition based on the freeze threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics of the target partition and the memory statistics of the memory data table; wherein, the memory statistics are used to record storage-related information of the data in the memory data table.

[0141] In this embodiment, the target partition is any partition of any data table in the database. For example, the method can target each partition of each data table in the database, so this step can obtain the memory statistics information of each partition of each data table.

[0142] The database has multiple preset merge modes (Table Modes), each with a different probability of triggering a fast freeze. Specifically, each merge mode includes at least a freeze threshold function, which sets the freezing conditions for the MemTable in memory. Different conditions result in different data information being considered for freezing the MemTable (e.g., some merge modes only consider information from memory statistics, while others consider both memory and disk statistics), and different thresholds. Among the preset merge modes, the freeze threshold functions differ, meaning the freezing conditions for the MemTable vary. The merge mode of the data table to which the partition belongs is one of these preset merge modes.

[0143] For example, the merging threshold function is represented as T_freeze(m, X). → ), where m is used to identify the merging mode, for example, it can be m1, m2, m3, m4, m5, etc., X → This provides memory statistics; when f_reeze(m, X) → If ) > 0, then MemTable meets the fast freeze condition.

[0144] In step S302, if the memory data table meets the fast freeze condition, the memory data table is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

[0145] In this embodiment, if the MemTable meets the conditions for fast freezing, the MemTable can be frozen, thereby organizing the data in the frozen MemTable into an ordered string table and storing it in a specific partition (e.g., the target partition).

[0146] This embodiment can adapt to different data tables and initiate rapid freezing under different conditions, thereby promptly freezing and dumping memory data tables that are prone to buffer table effects to the disk, so that they can be merged into the baseline ordered string table as soon as possible during the merging process, thereby reducing the decline in data query performance caused by excessive deletion operations from the source.

[0147] It's understandable that this method could also freeze and dump the MemTable when its memory usage reaches a certain threshold. In other words, it can determine whether the MemTable's memory usage has reached the threshold; if so, it can freeze the MemTable, organizing the data within it into an ordered string table and storing it in a specific partition.

[0148] In some embodiments of this disclosure, a merging mode can be configured for each data table in the database in the following manner: using the database as the environment, the disk statistics of the partition and the memory statistics of MemTable as the state, the merging mode change of the partition as the action, and performing reinforcement learning training with a preset reward function until the convergence condition is met, so as to obtain the merging mode of each data table in the database.

[0149] The partition merge mode change can include: changing the partition merge mode from one preset merge mode to another preset merge mode; and adjusting at least one of the merge threshold function and freeze threshold function within the partition merge mode, or adjusting at least one of the merge threshold function, freeze threshold function, and merge strategy within the partition merge mode. For example, lowering / raising the merge cooldown time threshold, lowering / raising the number of rows deleted threshold, lowering / raising the validity period of statistics, the elimination rate, the buffer pool size, and raising / lowering the ratio of triggering basic merges during a merge, etc., are essentially adjusting the freeze threshold function f_reeze(m, X). → ) and the merging threshold function T_compact(m, Y) → The strategy involves selecting points for a walk, but the direction is random.

[0150] The reward function is used to characterize at least one of the following:

[0151] A positive reward is given if the execution time of the data query operation decreases after the action is performed, and a negative reward is given if the execution time of the data query operation increases after the action is performed; rewards are preferably calculated for data query operations with high execution frequency (e.g., operations with an execution rate of more than 80%).

[0152] If the proportion of slow queries decreases after the action is performed, a positive reward is given; if the proportion of slow queries increases after the action is performed, a negative reward is given.

[0153] If the ratio between the logical rows scanned and the physical rows scanned in the data query operation increases after the action is executed, a positive reward will be given; if the ratio between the logical rows scanned and the physical rows scanned in the data query operation decreases after the action is executed, a negative reward will be given.

[0154] If the disk read / write data volume (I / O volume), disk write operation volume, or ordered string table volume decreases within a preset time period after the action is executed, a positive reward is given; if the disk read / write data volume, disk write operation volume, or ordered string table volume increases within a preset time period after the action is executed, a negative reward is given.

[0155] For example, a smart agent is pre-defined to generate actions based on the state in the environment. In reinforcement learning, actions can be generated based on the state in each iteration, and a reward value can be determined using a reward function based on the impact of the actions on the state. The agent's parameters are then tuned based on the reward value until the obtained reward value meets the convergence requirement. The merging pattern of the action corresponding to the reward value that meets the convergence requirement is then determined as the merging pattern of the data table.

[0156] This embodiment uses reinforcement learning to determine the merging pattern of each data table in the database, thereby improving the accuracy of the merging pattern and enabling the method to achieve good results in different databases and different data tables, thus expanding the applicability of the method.

[0157] It is understandable that the user can determine and configure the merge mode for each data table in this method.

[0158] Please refer to Figure 4, which exemplarily illustrates a data management method obtained by combining the above-described embodiments.

[0159] Among them, the frozen memory check thread can freeze the MemTable when the MemTable reaches the memory threshold, that is, convert the active MemTable into a frozen MemTable, dump the Mini SSTable, update the disk statistics of the partition, and then attempt to initiate a merge. That is, the process shown in Figure 2 determines whether to perform an adaptive merge on the SSTable in the partition.

[0160] The fast freeze scheduling thread can determine whether MemTable meets the freezing conditions based on memory statistics and the freeze threshold function. That is, it can determine whether to perform fast freeze on MemTable through the process shown in Figure 3. If it meets the conditions, it freezes MemTable, dumps Mini SSTable, updates the disk statistics of the partition, and then attempts to initiate a merge. That is, it can determine whether to perform adaptive merge on SSTable in the partition through the process shown in Figure 2.

[0161] Among them, the partition merge scheduling thread can determine whether to perform adaptive merging on the SSTables in the partition based on the disk statistics of the partition and the merge threshold function when iterating to each partition, through the process shown in Figure 2.

[0162] Figure 5 is a schematic structural diagram of a device provided in an exemplary embodiment. Referring to Figure 5, at the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include other hardware required for tasks. One or more embodiments of this specification can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into the memory 508 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0163] Please refer to Figure 6. The data management device can be applied to the device shown in Figure 5 to implement the technical solution of this specification. The data tables in the database are divided into multiple partitions; the data in the data tables is organized into a log structure merge tree; merge modes are set for the data tables, with different merge modes indicating different probabilities of triggering adaptive merging; each partition stores at least one ordered string table from the log structure merge tree; the data management device may include:

[0164] The determining module 601 is used to determine whether the target partition meets the adaptive merging conditions based on the merge threshold function in the merge mode of the data table to which the target partition belongs and the disk statistics information of the target partition; wherein, the disk statistics information is used to record storage-related information of the data in the target partition;

[0165] The merging module 602 is used to perform a merging operation on the ordered string table in the target partition if the target partition meets the adaptive merging conditions.

[0166] In one or more embodiments of this specification, the determining module is specifically used for:

[0167] If the target partition meets the iteration requirements, it is determined whether the target partition meets the adaptive merging conditions based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition.

[0168] In one or more embodiments of this specification, the determining module is specifically used for:

[0169] If the memory data table in the log structure merge tree is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition, the target partition is determined to meet the adaptive merge condition based on the merge threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics information of the target partition.

[0170] In one or more embodiments of this specification, the apparatus further includes a freezing module for:

[0171] Determine whether the memory size occupied by the memory data table has reached the memory threshold;

[0172] If the memory size occupied by the memory data table reaches the memory threshold, the memory data table is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

[0173] In one or more embodiments of this specification, different merging modes also indicate different probabilities of triggering a fast freeze;

[0174] The device also includes a rapid freezing module for:

[0175] Based on the freeze threshold function in the merge mode of the data table to which the target partition belongs, and the memory statistics of the memory data table, it is determined whether the memory data table meets the fast freeze condition; or based on the freeze threshold function in the merge mode of the data table to which the target partition belongs, and the disk statistics of the target partition and the memory statistics of the memory data table, it is determined whether the memory data table meets the fast freeze condition; wherein, the memory statistics are used to record storage-related information of the data in the memory data table;

[0176] If the memory data table meets the fast freeze condition, the memory data table is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

[0177] In one or more embodiments of this specification, the apparatus further includes an updating module for:

[0178] After organizing the data in the frozen memory data table into an ordered string table and storing it in the target partition, the disk statistics of the target partition are updated based on the memory statistics of the memory data table.

[0179] In one or more embodiments of this specification, the disk statistics include at least one of the following:

[0180] The data includes the number of rows deleted, the number of rows updated, the proportion of update operations within a preset time period, the number of ordered string tables, the number of slow queries, and the proportion of slow queries. The slow query is a data query operation whose execution time exceeds a time threshold, and the slow query proportion is the proportion of the number of slow queries in the total number of data query operations.

[0181] The memory statistics include at least one of the following:

[0182] The active duration of the in-memory data table, the number of rollback transactions, the number of deleted rows, the number of updated rows, the proportion of update operations within the preset duration, the number of slow queries, and the slow query ratio.

[0183] In one or more embodiments of this specification, the disk statistics are stored in memory, and the memory statistics are stored in memory; and / or,

[0184] The disk statistics include the latest disk statistics within a preset time period and all disk statistics since the last merge operation.

[0185] In one or more embodiments of this specification, the step of performing a merge operation on the ordered string table in the target partition if the target partition meets the adaptive merge condition includes:

[0186] If the target partition meets the adaptive merging conditions, the ordered string table in the target partition is merged according to the merging strategy in the merging mode of the data table to which the target partition belongs. The merging mode of the data table to which the target partition belongs is one of a variety of preset merging modes, wherein the merging threshold function, freezing threshold function and merging strategy are different in different merging modes.

[0187] In one or more embodiments of this specification, among the various preset merge modes, at least one of the following merge strategies is provided: basic merge strategy, advanced merge strategy, and aggressive merge strategy;

[0188] The basic merge strategy includes performing partition merge only, wherein the partition merge is to merge the baseline ordered string table and the incremental ordered string table at the same position in all database instances involved in the partition;

[0189] The advanced merge strategy includes performing a partition merge when the time elapsed since the last partition merge reached a preset period, and performing a basic merge when the time elapsed since the last partition merge did not reach the preset period. The basic merge is to merge the baseline ordered string table and the incremental ordered string table within the database instance.

[0190] The aggressive merger strategy includes performing only the basic merger.

[0191] In one or more embodiments of this specification, the apparatus further includes a training module for:

[0192] Using the database as the environment, the disk statistics of the partition and the memory statistics of the memory data table as the state, the merging mode change of the partition as the action, and the reinforcement learning training with a preset reward function until the convergence condition is met, the merging mode of each data table in the database is obtained.

[0193] In one or more embodiments of this specification, the partition merging mode change includes:

[0194] Change the merging mode of the partitions from one preset merging mode to another preset merging mode; and,

[0195] Adjust at least one of the merge threshold function and the freeze threshold function within the merge mode of the partition.

[0196] In one or more embodiments of this specification, the reward function is used to characterize at least one of the following:

[0197] If the execution time of the data query operation decreases after the action is performed, a positive reward will be given; if the execution time of the data query operation increases after the action is performed, a negative reward will be given.

[0198] If the proportion of slow queries decreases after the action is performed, a positive reward is given; if the proportion of slow queries increases after the action is performed, a negative reward is given.

[0199] If the ratio between the logical rows scanned and the physical rows scanned in the data query operation increases after the action is executed, a positive reward will be given; if the ratio between the logical rows scanned and the physical rows scanned in the data query operation decreases after the action is executed, a negative reward will be given.

[0200] If the amount of data read / written by the disk, the number of write operations, or the number of ordered string tables decreases within a preset time period after the action is executed, a positive reward will be given. If the amount of data read / written by the disk, the number of write operations, or the number of ordered string tables increase within a preset time period after the action is executed, a negative reward will be given.

[0201] One or more embodiments of this specification also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method provided in any of the above embodiments.

[0202] One or more embodiments of this specification also provide a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the method provided in any of the above embodiments.

[0203] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0204] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0205] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0206] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0207] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0208] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0209] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0210] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0211] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0212] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

Claims

1. A data management method; wherein, The data table in the database is divided into multiple partitions; data in the data table is organized into a log-structured merge tree; A merge mode is set for the data table, and different merge modes represent different probabilities of triggering adaptive merge; At least one ordered string table in the log-structured merge tree is stored in each partition; The method comprises: According to the merge threshold function in the merge mode of the data table to which the target partition belongs and the disk statistical information of the target partition, it is determined whether the target partition meets the adaptive merge condition; wherein the disk statistical information is used to record the storage related information of the data in the target partition; If the target partition meets the adaptive merge condition, a merge operation is performed on the ordered string table in the target partition.

2. The data management method of claim 1, wherein the determination of whether the target partition meets the adaptive merge condition according to the merge threshold function in the merge mode of the data table to which the target partition belongs and the disk statistical information of the target partition comprises: If the target partition meets the iteration requirement, the determination of whether the target partition meets the adaptive merge condition according to the merge threshold function in the merge mode of the data table to which the target partition belongs and the disk statistical information of the target partition.

3. The data management method of claim 1, wherein the determination of whether the target partition meets the adaptive merge condition according to the merge threshold function in the merge mode of the data table to which the target partition belongs and the disk statistical information of the target partition comprises: If the memory data table in the log-structured merge tree is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition, the determination of whether the target partition meets the adaptive merge condition according to the merge threshold function in the merge mode of the data table to which the target partition belongs and the disk statistical information of the target partition.

4. The data management method of claim 3, further comprising: Determining whether the memory size occupied by the memory data table reaches a memory threshold value; If the memory size occupied by the memory data table reaches the memory threshold value, the memory data table is frozen, and the data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

5. The data management method of claim 3, wherein different merge modes also represent different probabilities of triggering fast freezing; The method further comprises: According to the freeze threshold function in the merge mode of the data table to which the target partition belongs and the memory statistical information of the memory data table, it is determined whether the memory data table meets the fast freezing condition, or according to the freeze threshold function in the merge mode of the data table to which the target partition belongs and the disk statistical information of the target partition and the memory statistical information of the memory data table, it is determined whether the memory data table meets the fast freezing condition; wherein the memory statistical information is used to record the storage related information of the data in the memory data table. If the memory data table meets the quick freezing condition, the memory data table is frozen, and data in the frozen memory data table is organized into an ordered string table and stored in the target partition.

6. The data management method of any one of claims 3-5, further comprising: After the data in the frozen memory data table is organized into an ordered string table and stored in the target partition, updating disk statistical information of the target partition according to memory statistical information of the memory data table.

7. The data management method of claim 5, wherein the disk statistical information comprises at least one of: the number of deleted rows, the number of updated rows, the proportion of update operations in a preset time period, the number of ordered string tables, the number of slow queries, and the proportion of slow queries, wherein the slow query is a data query operation with an execution time exceeding a time threshold, and the slow query ratio is a ratio of the number of slow queries to the number of all data query operations; the memory statistical information comprises at least one of: an active duration of a memory data table, a number of rollback transactions, a number of deleted rows, a number of updated rows, a ratio of a number of update operations in a preset duration, a number of slow queries, and a slow query ratio.

8. The data management method of claim 5, wherein the disk statistical information is stored in memory, and the memory statistical information is stored in memory; and / or, the disk statistical information comprises the latest disk statistical information in a preset duration and all disk statistical information after a last execution of a merge operation.

9. The data management method of claim 1, wherein if the target partition meets the adaptive merge condition, performing a merge operation on the ordered string table in the target partition comprises: if the target partition meets the adaptive merge condition, performing a merge operation on the ordered string table in the target partition according to a merge strategy in a merge mode of a data table to which the target partition belongs; and the merge mode of the data table to which the target partition belongs is one of a plurality of preset merge modes, wherein in the plurality of preset merge modes, a merge threshold function, a freezing threshold function, and a merge strategy are different in different merge modes.

10. The data management method of claim 9, wherein in the plurality of preset merge modes, at least one of the following merge strategies is provided: a basic merge strategy, an advanced merge strategy, and an aggressive merge strategy; wherein the basic merge strategy comprises only performing a partition merge, and the partition merge is a merge of a baseline ordered string table and a delta ordered string table at the same site in all database instances involved in the partition; the advanced merge strategy comprises performing a partition merge when a duration from a time of a last execution of the partition merge to a current time reaches a preset period, and performing a basic merge when the duration from the time of the last execution of the partition merge to the current time does not reach the preset period, and the basic merge is a merge of the baseline ordered string table and the delta ordered string table in the database instance; the aggressive merge strategy comprises only performing the basic merge.

11. The data management method of claim 5, further comprising: The reinforcement learning training is performed with the database as an environment, with disk statistics of the partition and memory statistics of the memory data table as a state, with a change of the merge mode of the partition as an action, and with a preset reward function until a convergence condition is met, to obtain a merge mode of each data table in the database. 12.The data management method of claim 11, wherein the change of the merge mode of the partition comprises: changing the merge mode of the partition from a preset one to another; and adjusting at least one of a merge threshold function and a freeze threshold function in the merge mode of the partition. 13.The data management method of claim 12, wherein the reward function is used to represent at least one of: a positive reward is given if an execution time of a data query operation is reduced after the action is performed, and a negative reward is given if the execution time of the data query operation is increased after the action is performed; a positive reward is given if a proportion of slow queries is reduced after the action is performed, and a negative reward is given if the proportion of slow queries is increased after the action is performed; a positive reward is given if a proportion of scanned logical rows to scanned physical rows in a data query operation is increased after the action is performed, and a negative reward is given if the proportion of scanned logical rows to scanned physical rows in the data query operation is reduced after the action is performed; a positive reward is given if a disk read-write data volume in a preset time length, a write operation number of the disk in the preset time length, or an ordered string table number is reduced after the action is performed, and a negative reward is given if the disk read-write data volume in the preset time length, the write operation number of the disk in the preset time length, or the ordered string table number is increased after the action is performed. data tables in the database are divided into a plurality of partitions; data in the data tables is organized into log-structured merge trees; 14. A data management apparatus; wherein, a merge mode is set for the data tables, and different merge modes represent different probabilities of triggering adaptive merge; at least one ordered string table in the log-structured merge trees is stored in each partition; the apparatus comprises: a determination module configured to determine whether a target partition meets an adaptive merge condition according to a merge threshold function in a merge mode of a data table to which the target partition belongs and disk statistics of the target partition, wherein the disk statistics are used to record storage-related information of data in the target partition; a merge module configured to perform a merge operation on an ordered string table in the target partition if the target partition meets the adaptive merge condition. 15.A computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 13. 16.An electronic device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of any one of claims 1 to 13 by running the executable instructions. 17.A computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 13. ​

Images