Database dynamic sampling and partition compression method and device, equipment and medium

Abstract

The invention relates to the field of distributed databases, and discloses a database dynamic sampling and partition compression method which comprises the following steps: acquiring a target data table, and analyzing the target data table to generate table configuration information; generating a dynamic sampling task and a partition compression task according to the table configuration information, and packaging the dynamic sampling task and the partition compression task into an operation unit; creating a dynamic thread pool, and issuing the operation unit to the dynamic thread pool; and executing the dynamic sampling task and the partition compression task by the dynamic thread pool, and feeding back an execution result to the processing tool. According to the method, table configuration and packaging tasks are read through the processing tool, sampling and partition compression are completed by the dynamic thread pool without interaction between the coordination node and the storage node, stepped reduction of the data size is achieved, and the storage efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed databases, in particular to a database dynamic sampling and partition compression method, device, equipment and storage medium. BACKGROUND

[0002] Under the background of the rapid development of industrial internet and distributed systems, the running state time series data generated by massive device components presents exponential growth. Taking a typical scenario as an example, a single component generates monitoring data once per second, and when the system scale reaches the level of thousands of nodes, the daily incremental data volume can exceed 1TB. Such data has significant time series strong correlation, only additional writing characteristics and cold and hot layered access characteristics, which poses a serious challenge to traditional database storage architecture.

[0003] The traditional database storage architecture has many key defects when facing time series data. First, the storage efficiency bottleneck problem is prominent. The fixed granularity full storage mode leads to a sharp increase in disk space occupation, making it difficult to balance the sequential write speed of mechanical hard disks and the random read performance of SSDs. When the single table capacity exceeds 1TB, the query delay will present nonlinear deterioration. Second, the data life cycle management is missing. When using the DELETE operation to clean up useless data, it will trigger the linear growth of transaction logs, causing the peak value of master-slave synchronization delay to reach hours. At the same time, frequent I / O operations also accelerate the aging of storage media. In addition, the query performance degradation problem is serious. In the time series query scenario with high full table scan ratio, the maintenance cost of traditional indexes will increase dramatically with the growth of data volume, resulting in poor query performance. Finally, the resource scheduling is rigid. The static thread pool configuration cannot adapt to the load fluctuation of periodic data compression tasks, and the CPU utilization fluctuates greatly during task execution, with serious alternating phenomena of resource idling and overload. SUMMARY

[0004] The main purpose of the present application is to provide a database dynamic sampling and partition compression method, device, equipment and storage medium, which aims to solve the problem of low storage efficiency caused by the fixed granularity full storage mode of the existing distributed database in the massive data scenario.

[0005] To achieve the above purpose, the present application provides a database dynamic sampling and partition compression method, comprising: Obtaining a target data table and parsing the target data table to generate table configuration information; Generating a dynamic sampling task and a partition compression task according to the table configuration information, and encapsulating the dynamic sampling task and the partition compression task as a job unit; Creating a dynamic thread pool and issuing the job unit to the dynamic thread pool; The dynamic sampling task and the partition compression task are executed by the dynamic thread pool, and execution results are fed back to a processing tool.

[0006] Further, to achieve the above object, the present application provides a database dynamic sampling and partition compression device, comprising: a data analysis module, configured to obtain a target data table and analyze the target data table to generate table configuration information; a task packaging module, configured to generate a dynamic sampling task and a partition compression task according to the table configuration information, and package the dynamic sampling task and the partition compression task into a job unit; a task issuing module, configured to create a dynamic thread pool and issue the job unit to the dynamic thread pool; a task execution module, configured to execute the dynamic sampling task and the partition compression task by the dynamic thread pool, and feed execution results back to a processing tool.

[0007] Further, to achieve the above object, the present application further provides a computer device, comprising a memory, a processor, and a database dynamic sampling and partition compression program stored on the memory and executable on the processor, wherein the database dynamic sampling and partition compression program, when executed by the processor, implements the steps of the database dynamic sampling and partition compression method as described above.

[0008] Further, to achieve the above object, the present application further provides a computer readable storage medium, wherein the storage medium stores a database dynamic sampling and partition compression program, and the database dynamic sampling and partition compression program, when executed by a processor, implements the steps of the database dynamic sampling and partition compression method as described above.

[0009] Advantages: The present application relates to the field of distributed databases, and can be applied to business system platforms such as communication, medical health and financial technology, and discloses a database dynamic sampling and partition compression method, comprising: obtaining a target data table, analyzing the target data table to generate table configuration information; generating a dynamic sampling task and a partition compression task according to the table configuration information, and packaging the dynamic sampling task and the partition compression task into a job unit; creating a dynamic thread pool and issuing the job unit to the dynamic thread pool; executing the dynamic sampling task and the partition compression task by the dynamic thread pool, and feeding execution results back to a processing tool. The present application reads table configuration by a processing tool, packages tasks, executes the tasks by a dynamic thread pool, interacts with a coordination node and a storage node to complete sampling and partition compression, realizes stepwise reduction of data volume, and improves storage efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0010] The present application will be further described below with reference to the drawings and embodiments. Figure 1 An application environment schematic diagram of the database dynamic sampling and partition compression method in an embodiment of the present application; Figure 2 A flow schematic diagram of the database dynamic sampling and partition compression method in an embodiment of the present application; Figure 3 Another flow schematic diagram of the database dynamic sampling and partition compression method in an embodiment of the present application; Figure 4 A functional module schematic diagram of the database dynamic sampling and partition compression device in a preferred embodiment of the present application; Figure 5 A structure schematic diagram of a computer device in an embodiment of the present application; Figure 6 Another structure schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0011] It should be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the present application.

[0012] The database dynamic sampling and partition compression method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 , wherein a user end communicates with a service end through a network. The service end can obtain a target data table through the user end, and parse the target data table to generate table configuration information; generate a dynamic sampling task and a partition compression task according to the table configuration information, and encapsulate the dynamic sampling task and the partition compression task as a job unit; create a dynamic thread pool, and issue the job unit to the dynamic thread pool; execute the dynamic sampling task and the partition compression task by the dynamic thread pool, and feed back the execution result to a processing tool. The present application reads table configuration by the processing tool, encapsulates tasks, executes the tasks by the dynamic thread pool, and completes sampling and partition compression by interacting with a coordination node and a storage node, so as to realize stepwise reduction of data volume and improve storage efficiency. The user end can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The service end can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0013] Please refer to Figure 2 , Figure 2 A flow schematic diagram of the database dynamic sampling and partition compression method in an embodiment of the present application provided by the present application. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that herein.

[0014] As Figure 2As shown, the database dynamic sampling and partition compression method provided by the present application comprises the following steps: S100, acquiring a target data table and parsing the target data table to generate table configuration information; S200, generating a dynamic sampling task and a partition compression task according to the table configuration information, and packaging the dynamic sampling task and the partition compression task into a job unit; S300, creating a dynamic thread pool and issuing the job unit to the dynamic thread pool; S400, executing the dynamic sampling task and the partition compression task by the dynamic thread pool and feeding back the execution result to a processing tool.

[0015] In this embodiment, as shown in the figure, Figure 3 In the distributed database, the target data table is the object of dynamic sampling and partition compression. The processing tool first needs to acquire the target data table and parse it to generate table configuration information. The processing tool acquires the metadata of the target data table through a database management system (DBMS) or directly from a storage node, and the metadata includes the structure information (such as column name, data type, partition information, etc.) and storage information (such as storage location, partition strategy, etc.) of the table. Key information is extracted from the metadata of the target data table to generate table configuration information, which usually includes sampling strategy, hot and cold layering threshold, storage period, data access frequency, etc.

[0016] The processing tool generates specific dynamic sampling tasks and partition compression tasks according to the table configuration information, and packages these tasks into a job unit. According to the sampling strategy, the dynamic sampling task is generated, the historical data is aggregated according to the time window, the key statistical characteristics are retained, and the aggregated data is stored in a temporary table. According to the hot and cold layering threshold, determine which partition is a cold data partition that needs to be compressed, and generate a partition compression task for each cold data partition and select an appropriate compression strategy. The dynamic sampling task and the partition compression task are packaged into an independent job unit, which contains the specific description of the task, the execution parameter, the task priority, etc. The job unit is the basic unit that can be scheduled and executed by the dynamic thread pool.

[0017] According to the resource situation of the system, a dynamic thread pool is created and the initial thread number of the dynamic thread pool is set. The dynamic thread pool can dynamically adjust the number of threads according to the load of the system to optimize the resource utilization. The packaged job unit is issued to the dynamic thread pool, and the dynamic thread pool dynamically allocates threads to execute these tasks according to the priority of the task and the current load. The dynamic thread pool executes the dynamic sampling task and the partition compression task, and feeds back the execution result to the processing tool.

[0018] Further, the multi-level sampling strategy is a method of dynamically adjusting the sampling frequency according to the time value of the data. Specifically, 10-second raw data is retained for the last 3 days. For data within the last 3 days, since it has a higher value, more detailed information needs to be retained, so the sampling frequency is higher, retaining raw data every 10 seconds. A 30-second aggregate sampling is enabled for medium-term data (such as 3 days to 1 week), which has a relatively low value, so the sampling frequency can be appropriately reduced, with aggregate sampling every 30 seconds. The temporary table is generated by aggregating the data according to the preset time window (such as every 10 seconds or every 30 seconds). During the aggregation process, key statistical features such as the maximum value and the average value are retained.

[0019] The partition is replaced with metadata, and the aggregate data in the temporary table is exchanged with the original data in the original partition through partition exchange technology. This can avoid the log expansion and standby delay problems caused by traditional deletion operations. The compression granularity can be dynamically adjusted according to the actual use of the data. For example, when it is found that the data access frequency of a certain historical partition is further reduced, a secondary compression task can be triggered for the partition to further reduce the storage space.

[0020] The hot and cold layered compression system is a strategy for layered storage and compression according to data access frequency and importance. The hot partition (today's data) remains in the original state. The hot partition is usually data generated today, with high access frequency and real-time writing and high-frequency querying, so it remains in the original state without compression. The historical partition enables row compression algorithm. For historical partitions (such as data over a certain number of days), the access frequency is low, and the row compression algorithm can be enabled for compression. The row compression algorithm can effectively reduce the storage space, and combined with local task scheduling of distributed nodes, the compression effect is further optimized.

[0021] The adaptive task scheduling engine is a mechanism that can dynamically adjust the execution of tasks according to system resources and task status. The sampling and compression tasks are encapsulated as independent job units, and the dynamic sampling tasks and partition compression tasks are encapsulated as independent job units. Each job unit contains specific descriptions of tasks, execution parameters, priority, and other information. Based on the dynamic thread pool elastic resource allocation, according to the system resource situation (such as CPU utilization, memory occupation, etc.), the dynamic thread pool can elastically allocate resources and allocate appropriate number of threads for each job unit. The timeout fuse mechanism is introduced to avoid long-term resource occupation or abnormality of some tasks. When the task execution time exceeds the preset threshold, the task is automatically terminated, and the metadata of the failed partition is recorded. The task status tracking table records the processed partition granularity to avoid repeated calculation. The task status tracking table records the processing status and granularity information of each partition. In this way, the processed partitions can be skipped in subsequent task scheduling.

[0022] The metadata-driven management framework is a mechanism for task management and scheduling through metadata. Table-level control parameters are provided: control parameters are provided for each table, such as storage period, sampling switch, hot and cold layering threshold, etc. These parameters can be configured according to specific needs to adapt to different business scenarios. Cross-node partition compression task coordination is supported: in a distributed database, data may be distributed across multiple nodes. The metadata-driven management framework supports cross-node partition compression task coordination to ensure distributed consistency of the entire system.

[0023] Through the above steps and mechanisms, the processing tool can efficiently perform dynamic sampling and partition compression on the target data table in the distributed database. The introduction of dynamic thread pool makes the execution of tasks more flexible and efficient, and can dynamically adjust the number of threads according to the system resource situation to optimize resource utilization. Finally, through the feedback mechanism, the processing tool can monitor the execution of tasks in real time to ensure stable operation of the system.

[0024] In one embodiment, the step S200 includes: S201, parsing the table configuration information to obtain the sampling strategy and the hot and cold layering threshold; S202, generating a dynamic sampling task according to the sampling strategy; S203, generating a partition compression task according to the hot and cold layering threshold; S204, encapsulating the dynamic sampling task and the partition compression task as a job unit.

[0025] In this embodiment, in the distributed database, table configuration information is the core of managing and optimizing data storage and query. By parsing these configuration information, the sampling strategy can be extracted, which is an optimization mechanism for time series data. Sampling strategy can effectively reduce the amount of data while retaining key information. For example, time decay rules can be used, that is, gradually reduce the sampling frequency of data over time. Specifically, the original data of 10 seconds can be retained for the last 3 days, while the medium-term enables 30-second aggregate sampling. In addition, the hot and cold layering threshold is an important part of the table configuration information, which is used to distinguish the access frequency and importance of data, so as to realize the hierarchical storage of hot and cold data. Hot data is usually recently generated and frequently accessed data, which needs to respond to queries quickly; while cold data is historical data with low access frequency, which can be optimized for storage through compression and other methods.

[0026] Dynamic sampling tasks are specific operation tasks generated according to the sampling strategy, used for down-sampling processing of data. Partition compression tasks are optimization operations for cold data, aiming to reduce the occupation of storage space. In order to facilitate the management and scheduling of tasks, the generated dynamic sampling tasks and partition compression tasks can be encapsulated as independent job units. Each job unit contains specific task information and execution parameters. These job units can be submitted to the dynamic thread pool, which is scheduled and executed by the thread pool according to the system's resource situation and task priority.

[0027] Through the above steps, effective management and optimization of massive time series data in a distributed database can be achieved. This not only improves storage efficiency, but also significantly improves query performance, thereby better meeting the needs of distributed databases in handling large-scale time series data.

[0028] In one embodiment, the step S300 comprises: S301, obtaining the resource situation of the system; S302, creating a dynamic thread pool according to the resource situation and setting the initial number of threads; S303, analyzing the target data table to obtain the storage period and data access frequency; S304, generating a task allocation priority according to the storage period and data access frequency; S305, assigning a number of threads to each dynamic sampling task and partition compression task according to the task allocation priority, task load situation and task state tracking table by the dynamic thread pool.

[0029] In this embodiment, in a distributed database environment, the resource conditions of the system (such as CPU usage, memory occupancy, and disk I / O, etc.) have an important influence on the performance and efficiency of the thread pool. By obtaining these resource information in real time, the size of the thread pool can be dynamically adjusted to adapt to different load requirements. The initial number of threads of the dynamic thread pool is usually set according to the number of CPU cores of the system, for example, it can be set to 1 to 2 times the number of CPU cores, so that the CPU resources of the system can be fully utilized at startup.

[0030] The storage period and data access frequency of the target data table are key factors in determining the priority of the task. The storage period refers to the length of time for which the data needs to be retained, while the data access frequency reflects the frequency of data queries. By analyzing this information, hot data (frequently accessed data) and cold data (less frequently accessed data) can be distinguished. The priority of task allocation is determined according to the storage period and access frequency of the data: hot data needs to be processed first due to its high access frequency, so it is assigned a higher priority; while cold data can be assigned a lower priority due to its low access frequency. In this way, resources in the thread pool can be reasonably allocated, and important tasks can be prioritized.

[0031] The dynamic thread pool dynamically allocates the number of threads according to the priority of the task, the current load condition, and the task state tracking table. The task state tracking table records the execution status of each task, including whether it is completed, whether it fails, and other information. Through this information, the dynamic thread pool can reasonably allocate the number of threads to ensure efficient execution of tasks.

[0032] Through the above steps, effective management and optimization of massive time series data in a distributed database can be achieved, thereby improving storage efficiency and query performance.

[0033] In one embodiment, the step S400 includes: S4011, receiving a data sampling command through a coordination node, and parsing the data sampling command to generate a sampling parsing result; S4012, issuing the sampling parsing result and the data sampling command to a storage node by the coordination node; S4013, the storage node executes a dynamic sampling task according to the allocated number of threads, obtains an execution result, and feeds back the sampling result to a processing tool.

[0034] In this embodiment, in a distributed database system, the coordination node (CN) is the core node responsible for managing and scheduling tasks. It receives data sampling commands from processing tools (such as management tools or user interfaces), which usually contain table names (specifying the data table that needs to be sampled), sampling strategies (defining how to sample data, such as sampling frequency, time window, aggregation method, etc.), and partition information (specifying the data partition range that needs to be sampled). After receiving the command, the coordination node parses it and extracts the key information to generate a sampling analysis result. The analysis result includes specific sampling tasks (such as performing time window aggregation on a certain partition of a certain table), task priorities (according to the system's resource situation and the importance of the task to allocate priorities), and task execution parameters (such as the size of the aggregation time window, the retention of key statistical features such as maximum value, average value, etc.).

[0035] After parsing, the coordination node will issue the sampling analysis result together with the original data sampling command to the specific storage node (GDB). The storage node is the node that actually stores the data and is responsible for executing specific sampling tasks. The content of the issue usually includes the specific description of the task (such as sampling a certain partition of a certain table), the execution parameters (such as the size of the time window, the aggregation method, etc.), and the priority of the task (used by the storage node to allocate resources when executing the task). After receiving the sampling task, the storage node will execute the task according to the number of threads allocated by the dynamic thread pool. The specific steps include: reading data (reading data from the specified partition), aggregating by time window (aggregating data by time window according to the sampling strategy, such as aggregating data every 10 seconds into one data point, retaining key statistical features such as maximum value, average value, etc.), generating aggregated data (storing aggregated data into a temporary table for subsequent processing), and recording execution results (including aggregated data and task execution status information such as success, failure, time consumption, etc.).

[0036] After the storage node completes the task, it will feed back the execution result to the coordination node, which will then forward the result to the processing tool. The feedback content includes aggregated data (data stored in the temporary table), task execution status (such as whether the task is successfully completed, if there is an error, it will also contain error information), and key statistical features (such as the maximum value and average value of the aggregated data points). Through the above steps, the coordination node and the storage node work together to achieve dynamic sampling of data in a distributed database. The coordination node is responsible for receiving and analyzing commands, the storage node is responsible for executing specific sampling tasks, and the results are fed back to the processing tool. This mechanism can effectively reduce the amount of data while retaining key information, thereby improving storage efficiency and query performance.

[0037] In one embodiment, the step S400 comprises: S4021, receiving a partition compression command by a coordination node, and parsing the partition compression command to generate a compression parsing result; S4022, issuing the compression parsing result and the partition compression command to a storage node by the coordination node; S4023, the storage node executes a partition compression task according to the number of threads allocated, obtains an execution result, and feeds back a compression result to a processing tool.

[0038] In this embodiment, after the coordination node receives the partition compression command, it parses it, extracts key information, and generates a compression parsing result. The compression parsing result includes specific compression tasks (e.g., compressing a certain partition of a certain table), task priorities (assigning priorities according to system resource conditions and task importance), and task execution parameters (e.g., selection of compression algorithm, compression granularity such as row compression, etc.).

[0039] After parsing, the coordination node issues the compression parsing result and the original partition compression command to the specific storage node (GDB). The storage node is the node that actually stores the data and is responsible for executing the specific compression task. The content of the issue usually includes the specific description of the task (e.g., compressing a certain partition of a certain table), the execution parameters (e.g., compression algorithm, compression granularity, etc.), and the priority of the task (used by the storage node to allocate resources when executing the task). After receiving the compression task issued by the coordination node, the storage node will execute the task according to the number of threads allocated by the dynamic thread pool. The specific steps include: counting partition information (determining which partitions belong to cold data and need to be compressed), selecting a compression strategy (selecting an appropriate compression strategy according to the cold and hot layering threshold and the compression parsing result, e.g., using row compression algorithm for cold data partition), executing compression operation (according to the number of threads allocated, compressing the data of the target partition to reduce the storage space occupation of the data while preserving the integrity of the data), updating metadata (after compression, updating the metadata of the partition to record the status and related information after compression), and recording the execution result (including the status of the compressed data, the status information of the task execution such as success, failure, time consumption, etc., and the compression effect such as the storage space saving situation after compression).

[0040] After the storage node completes the task, it feeds back the execution result to the coordination node, which then forwards the result to the processing tool. The feedback content includes the compressed data state (e.g., the compression state of the partition, the storage space occupation after compression), the task execution state (e.g., whether the task is successfully completed, and if there is an error, the error information is also included), and the compression effect (e.g., the storage space saving after compression). Through the above steps, the coordination node and the storage node work together to achieve the partition compression of the data in the distributed database. The coordination node is responsible for receiving and analyzing the command, the storage node is responsible for executing the specific compression task, and the result is fed back to the processing tool. This mechanism can effectively reduce the storage space occupation of cold data, and through the resource allocation of the dynamic thread pool, it ensures the efficient execution of the task and improves the overall performance of the system.

[0041] In one embodiment, the step S4013 comprises: Obtaining a statistical table in the target data table; According to the sampling strategy of the metadata configuration information, the historical data in the statistical table is aggregated according to the time window to generate aggregated data; Store the aggregated data to a temporary table and extract key statistical features; Through the partition exchange technology, the aggregated data in the temporary table and the original data in the original partition are exchanged in units of partitions, and the result is fed back to the processing tool.

[0042] In this embodiment, in the distributed database, the statistical table is a table used to store the metadata and aggregation information of the target data table, and its main fields include timestamp (time information of recording data), data value (value of original data), and aggregation information (such as maximum value, minimum value, average value, etc. Statistical characteristics).

[0043] The sampling strategy is a rule defined in advance according to the metadata configuration information, which is used to reduce the sampling of historical data. The specific steps are as follows: first, time window division is performed, and historical data is grouped according to the preset time window (such as every 10 seconds, every 30 seconds). Then, in each time window, use aggregation functions (such as maximum value, average value, count, etc.) to calculate the data to generate aggregated data. In this way, the aggregated data retains the key statistical characteristics of the original data while reducing the data volume. The aggregated data will be stored in a temporary table, which is used to temporarily store the aggregated data for subsequent processing. The key statistical features (such as maximum value, minimum value, average value, etc.) extracted from the aggregated data can reflect the main information of the original data.

[0044] The partition exchange technique is an optimization technique that exchanges aggregated data in temporary tables with original data in original partitions. By exchanging data in partitions, the problem of log inflation and master-slave synchronization delay caused by traditional deletion operations can be avoided. The partition exchange technique ensures data integrity and consistency while improving operational efficiency. The results of the exchange are fed back to the processing tool, which processes the results for subsequent processing.

[0045] Through the above steps, dynamic sampling and partition compression of historical data in a distributed database are achieved. This process effectively reduces the amount of data, improves storage efficiency and query performance.

[0046] In one embodiment, the step S4023 comprises: The processing tool performs statistics on the partitions of the statistics table, distinguishes hot partitions and historical partitions according to data access frequency, and determines the historical partitions to be compressed; According to the historical region to be compressed, a compression strategy is selected According to the compression strategy, the compression operation is performed on the historical partitions to be compressed, and the compression results are fed back to the processing tool.

[0047] In this embodiment, in a distributed database, data partitions are divided according to time or other logic, and each partition contains data within a certain time period. The processing tool needs to analyze the statistics of the partitions of the statistics table to determine which partitions are hot and which are historical. For this purpose, the processing tool reads the metadata of the statistics table to obtain detailed information of each partition, including the time range of the partition, the amount of data, the access frequency, etc. The access frequency of each partition can be obtained by querying the log or the access counter. Hot partitions are usually recently generated and frequently accessed data partitions, and these data need to maintain high access performance, so they are not suitable for compression. Historical partitions are data partitions that are generated earlier and have lower access frequency, and these data are suitable for compression to save storage space.

[0048] The processing tool divides the partitions into hot and historical partitions according to the data access frequency and the preset hot and cold stratification threshold, and selects the historical partitions with access frequency lower than the preset threshold as the partitions to be compressed, and the data of these partitions will be marked as needing compression. Selecting the appropriate compression strategy is the key to ensuring the compression effect and performance. For this purpose, the characteristics of the data to be compressed, such as data type, data distribution, data volume, etc. need to be checked. For example, for numerical data, row-wise compression algorithm can be used; for text data, dictionary encoding and other compression algorithms can be used. According to the data characteristics and system resource conditions, the appropriate compression algorithm is selected. Common compression strategies include row-wise compression, column-wise compression, dictionary encoding, etc. For example, for historical partitions, row-wise compression algorithm can be selected, because it can effectively reduce storage space while maintaining data integrity.

[0049] The last step of the whole process is to perform the compression operation and feedback the results. The storage node performs the compression operation on each historical partition to be compressed according to the number of threads allocated. The compression operation includes reading partition data, applying a compression algorithm, generating a compressed data file, and the like. During the compression process, information such as the progress of compression, time consumption, compression ratio, and the like is recorded. After compression is completed, the metadata of the partition is updated to record the state after compression, including the compression algorithm, compression time, size of compressed data, and the like. The update of the metadata ensures the data consistency and queryability of the system. The storage node feeds back the compression results to the coordination node, and the coordination node forwards the results to the processing tool. The feedback content includes the state of the compressed data (for example, the compression state of the partition, the storage space occupation after compression), the task execution state (for example, whether the task is successfully completed, and if there is an error, the error information is also included), and the compression effect (for example, the storage space saving after compression).

[0050] Through the above steps, the processing tool can effectively compress the historical partition to save storage space. At the same time, through the resource allocation of the dynamic thread pool, the efficient execution of the task is ensured, thereby improving the overall performance of the system.

[0051] In an embodiment, a database dynamic sampling and partition compression device is provided, which corresponds to the database dynamic sampling and partition compression method in the above embodiment. Referring to Figure 4 , Figure 4 A functional module schematic diagram of a preferred embodiment of the database dynamic sampling and partition compression device of the present application is shown. The data analysis module 10, the task packaging module 20, the task issuing module 30, and the task issuing module 30. The detailed description of each functional module is as follows: The data analysis module 10 is used to obtain a target data table and analyze the target data table to generate table configuration information; The task packaging module 20 is used to generate dynamic sampling tasks and partition compression tasks according to the table configuration information, and package the dynamic sampling tasks and partition compression tasks into a job unit; The task issuing module 30 is used to create a dynamic thread pool and issue the job unit to the dynamic thread pool; The task execution module 40 is used to execute the dynamic sampling tasks and partition compression tasks by the dynamic thread pool and feed back the execution results to the processing tool.

[0052] In an embodiment, the task packaging module 20 includes: The information analysis unit is used to analyze the table configuration information to obtain a sampling strategy and a hot and cold layering threshold; A sampling task unit is configured to generate a dynamic sampling task according to the sampling strategy. A compression task unit is configured to generate a partition compression task according to the cold-hot layering threshold. A task packaging unit is configured to package the dynamic sampling task and the partition compression task into a job unit.

[0053] In one embodiment, the task issuing module 30 comprises: A resource condition unit is configured to obtain resource conditions of a system. A dynamic thread pool unit is configured to create a dynamic thread pool according to the resource conditions and set an initial thread number. A data table analysis unit is configured to analyze the target data table to obtain a storage period and a data access frequency. A priority unit is configured to generate a task allocation priority according to the storage period and the data access frequency. An allocated thread number unit is configured to allocate a thread number for each dynamic sampling task and partition compression task by the dynamic thread pool according to the task allocation priority, a task load condition, and a task state tracking table.

[0054] In one embodiment, the task execution module 40 comprises: A sampling analysis unit is configured to receive a data sampling command by a coordination node, analyze the data sampling command, and generate a sampling analysis result. A task issuing unit is configured to issue the sampling analysis result and the data sampling command to a storage node by the coordination node. A sampling execution unit is configured to execute a dynamic sampling task according to the allocated thread number by the storage node, obtain an execution result, and feed back a sampling result to a processing tool.

[0055] In one embodiment, the task execution module 40 comprises: A compression analysis unit is configured to receive a partition compression command by a coordination node, analyze the partition compression command, and generate a compression analysis result. A task issuing unit is configured to issue the compression analysis result and the partition compression command to a storage node by the coordination node. A compression execution unit is configured to execute a partition compression task according to the allocated thread number by the storage node, obtain an execution result, and feed back a compression result to a processing tool.

[0056] In one embodiment, the sampling execution unit comprises: Obtain a statistical table in the target data table. According to the sampling strategy of the metadata configuration information, historical data in the statistical table is aggregated according to a time window to generate aggregated data; The aggregated data is stored in a temporary table, and key statistical features are extracted; Through a partition exchange technique, the aggregated data in the temporary table is exchanged with original data in an original partition in a partition unit, and the result is fed back to a processing tool.

[0057] In one embodiment, the compression execution unit comprises: The processing tool performs statistics on the partitions of the statistical table, distinguishes hot partitions and historical partitions according to data access frequency, and determines a historical partition to be compressed; According to the historical partition to be compressed, a compression strategy is selected According to the compression strategy, a compression operation is performed on the historical partition to be compressed, and the compression result is fed back to the processing tool.

[0058] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 5 The computer device comprises a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. The computer program is executed by the processor to implement the functions or steps of the database dynamic sampling and partition compression method on the server side.

[0059] In one embodiment, a computer device is provided, which can be a user terminal, and an internal structure diagram thereof can be as shown in Figure 6 The computer device comprises a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. The computer program is executed by the processor to implement the functions or steps of the database dynamic sampling and partition compression method on the user terminal side In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: obtaining a target data table, and parsing the target data table to generate table configuration information; generating a dynamic sampling task and a partition compression task according to the table configuration information, and encapsulating the dynamic sampling task and the partition compression task as a job unit; creating a dynamic thread pool, and issuing the job unit to the dynamic thread pool; executing the dynamic sampling task and the partition compression task by the dynamic thread pool, and feeding back an execution result to a processing tool.

[0060] In one embodiment, a computer readable storage medium is provided, having a computer program stored thereon, the computer program implementing the following steps when executed by a processor: obtaining a target data table, and parsing the target data table to generate table configuration information; generating a dynamic sampling task and a partition compression task according to the table configuration information, and encapsulating the dynamic sampling task and the partition compression task as a job unit; creating a dynamic thread pool, and issuing the job unit to the dynamic thread pool; executing the dynamic sampling task and the partition compression task by the dynamic thread pool, and feeding back an execution result to a processing tool.

[0061] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the related descriptions of the server side and the user side in the foregoing method embodiments, and will not be described again here to avoid repetition.

[0062] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0063] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0064] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for database dynamic sampling and partition compression, the method comprising: The method comprises the following steps: ​ obtaining a target data table and parsing the target data table to generate table configuration information; generating a dynamic sampling task and a partition compression task according to the table configuration information, and encapsulating the dynamic sampling task and the partition compression task into a job unit; creating a dynamic thread pool and assigning the job unit to the dynamic thread pool; executing the dynamic sampling task and the partition compression task by the dynamic thread pool, and feeding back the execution result to a processing tool.

2. The method for database dynamic sampling and partition compression of claim 1, wherein, The generating of the dynamic sampling task and the partition compression task according to the table configuration information and the encapsulating of the dynamic sampling task and the partition compression task into the job unit comprise: parsing the table configuration information to obtain a sampling strategy and a hot and cold layering threshold value; generating the dynamic sampling task according to the sampling strategy; generating the partition compression task according to the hot and cold layering threshold value; encapsulating the dynamic sampling task and the partition compression task into the job unit.

3. The method for database dynamic sampling and partition compression of claim 1, wherein, The creating of the dynamic thread pool and the assigning of the dynamic sampling task and the partition compression task to the dynamic thread pool comprise: obtaining resource conditions of a system; creating the dynamic thread pool according to the resource conditions and setting an initial thread number; analyzing the target data table to obtain a storage period and a data access frequency; generating a task allocation priority according to the storage period and the data access frequency; allocating a thread number for each dynamic sampling task and partition compression task according to the task allocation priority, a task load condition and a task state tracking table by the dynamic thread pool.

4. The method for database dynamic sampling and partition compression of claim 1, wherein, The executing of the dynamic sampling task by the dynamic thread pool and the feeding back of the execution result to the processing tool comprise: receiving a data sampling command by a coordination node and parsing the data sampling command to generate a sampling parsing result; assigning the sampling parsing result and the data sampling command to a storage node by the coordination node; executing the dynamic sampling task according to the allocated thread number by the storage node, obtaining an execution result and feeding back the sampling result to the processing tool.

5. The method for database dynamic sampling and partition compression of claim 1, wherein, The executing of the partition compression task by the dynamic thread pool and the feeding back of the execution result to the processing tool comprise: receiving a partition compression command by a coordination node and parsing the partition compression command to generate a compression parsing result; assigning the compression parsing result and the partition compression command to a storage node by the coordination node; executing the partition compression task according to the allocated thread number by the storage node, obtaining an execution result and feeding back the compression result to the processing tool.

6. The method for database dynamic sampling and partition compression of claim 4, wherein, The executing of the dynamic sampling task according to the allocated thread number by the storage node, the obtaining of an execution result and the feeding back of the sampling result to the processing tool comprise: obtaining a statistical table in the target data table; aggregating historical data in the statistical table according to a sampling strategy of metadata configuration information and a time window to generate aggregated data; storing the aggregated data to a temporary table and extracting key statistical features; swapping the aggregated data in the temporary table with original data in an original partition by a partition exchange technology in a partition unit, and feeding back the execution result to the processing tool.

7. The method for database dynamic sampling and partition compression of claim 5, wherein, The storage node executes the partition compression task according to the number of threads allocated, obtains an execution result, and feeds back a compression result to the processing tool, including: statistical tables are counted by the processing tool, hot partitions and historical partitions are distinguished according to data access frequency, and a historical partition to be compressed is determined; a compression strategy is selected according to the historical partition to be compressed a compression operation is performed on the historical partition to be compressed according to the compression strategy, and a compression result is fed back to the processing tool.

8. A database dynamic sampling and partition compression apparatus, characterized by, The database dynamic sampling and partition compression device comprises: a data analysis module configured to obtain a target data table and analyze the target data table to generate table configuration information; a task packaging module configured to generate a dynamic sampling task and a partition compression task according to the table configuration information, and package the dynamic sampling task and the partition compression task into a job unit; a task issuing module configured to create a dynamic thread pool and issue the job unit to the dynamic thread pool; a task execution module configured to execute the dynamic sampling task and the partition compression task by the dynamic thread pool, and feed back an execution result to a processing tool.

9. A computer device, comprising: The computer device comprises a memory, a processor, and a database dynamic sampling and partition compression program stored on the memory and executable on the processor, and the database dynamic sampling and partition compression program, when executed by the processor, implements the steps of the database dynamic sampling and partition compression method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a database dynamic sampling and partition compression program, and the database dynamic sampling and partition compression program, when executed by the processor, implements the steps of the database dynamic sampling and partition compression method according to any one of claims 1-7.

Images