Data processing method and device, computer equipment and storage medium

Abstract

The invention discloses a data processing method and device, computer equipment and a storage medium, relates to the technical field of data processing and database optimization, and can solve the problem that in the prior art, a compression strategy cannot be dynamically adjusted according to the characteristics of partitions, so that the data storage speed and the storage space utilization rate are affected. The method comprises the following steps: inputting a target data feature of a target partition in a current time period into a preset prediction model for prediction to obtain a query frequency corresponding to the target partition in a next time period; based on the frequency range to which the query frequency belongs, determining a compression strategy corresponding to the target partition in the next time period; and adjusting the compression ratio of the target partition based on the compression strategy.

Description

Technical Field

[0001] The present application relates to the technical field of data processing and database optimization, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid growth of data volume, data compression has become an important means to improve data storage efficiency and reduce storage costs. In the StarRocks database system, compressing the data of each partition can significantly reduce the storage space requirements and also help improve query performance. However, in existing database systems, each partition usually uses a fixed compression strategy, and it is impossible to dynamically adjust the compression strategy according to the characteristics of the partition, which affects the data storage speed and storage space utilization. Summary of the invention

[0003] The present application provides a data processing method, apparatus, computer device and storage medium to solve the problem in the prior art that the compression strategy cannot be dynamically adjusted according to the characteristics of the partition, thereby affecting the data storage speed and storage space utilization.

[0004] In a first aspect, a data processing method is provided, the method comprising:

[0005] Input the target data features of the target partition in the current time period into a preset prediction model for prediction, and obtain the query frequency corresponding to the target partition in the next time period;

[0006] Determine, based on the frequency range to which the query frequency belongs, a compression strategy corresponding to the target partition in the next time period;

[0007] The compression ratio of the target partition is adjusted based on the compression strategy.

[0008] In one embodiment, before inputting the target data features of the target partition in the current time period into a preset prediction model for prediction, the method further includes:

[0009] Get the historical query logs and partition information of each partition;

[0010] Extracting historical data features corresponding to each partition from each historical query log and each partition information;

[0011] Obtaining first compression strategy labels corresponding to respective historical data features;

[0012] The initial prediction model is trained based on each of the historical data features and the first compression strategy label corresponding to each of the historical data features to obtain the preset prediction model.

[0013] In one embodiment, the historical data characteristics include at least one or more of query partition, query frequency, query complexity, query data range, data volume, and data update frequency.

[0014] In one embodiment, the frequency range includes at least: a first frequency range and a second frequency range, wherein a minimum value of the second frequency range is greater than a maximum value of the first frequency range.

[0015] In one embodiment, determining the compression strategy corresponding to the target partition in the next time period based on the frequency range to which the query frequency belongs includes:

[0016] When the query frequency belongs to the first frequency range, determining that the compression strategy corresponding to the target partition in the next time period is that the compression rate belongs to the first compression rate range;

[0017] When the query frequency belongs to the second frequency range, determine that the compression strategy corresponding to the target partition in the next time period is a compression rate that belongs to the second compression rate range; wherein the maximum value of the second compression rate range is less than the minimum value of the first compression rate range.

[0018] In one embodiment, adjusting the compression rate of the target partition based on the compression strategy includes:

[0019] Get the partition list;

[0020] A management command of the database system is called, and a compression ratio corresponding to the target partition in the partition list is adjusted based on the management command and the compression strategy.

[0021] In one embodiment, the method further comprises:

[0022] In response to receiving an update instruction of the preset prediction model, reacquiring the second compression strategy label corresponding to each historical data feature;

[0023] The preset prediction model is trained based on each of the historical data features and the second compression strategy label corresponding to each of the historical data features to obtain an updated preset prediction model.

[0024] In a second aspect, the present application provides a data processing device, the device comprising:

[0025] A prediction module, used to input the target data features of the target partition in the current time period into a preset prediction model for prediction, and obtain the query frequency corresponding to the target partition in the next time period;

[0026] A determination module, configured to determine a compression strategy corresponding to the target partition in the next time period based on a frequency range to which the query frequency belongs;

[0027] An adjustment module is used to adjust the compression rate of the target partition based on the compression strategy.

[0028] In one embodiment, the prediction module is further used to:

[0029] Get the historical query logs and partition information of each partition;

[0030] Extracting historical data features corresponding to each partition from each historical query log and each partition information;

[0031] Obtaining first compression strategy labels corresponding to respective historical data features;

[0032] The initial prediction model is trained based on each of the historical data features and the first compression strategy label corresponding to each of the historical data features to obtain the preset prediction model.

[0033] In one embodiment, the historical data characteristics include at least one or more of query partition, query frequency, query complexity, query data range, data volume, and data update frequency.

[0034] In one embodiment, the frequency range includes at least: a first frequency range and a second frequency range, wherein a minimum value of the second frequency range is greater than a maximum value of the first frequency range.

[0035] In one embodiment, the determining module is specifically used to:

[0036] When the query frequency belongs to the first frequency range, determining that the compression strategy corresponding to the target partition in the next time period is that the compression rate belongs to the first compression rate range;

[0037] When the query frequency belongs to the second frequency range, determine that the compression strategy corresponding to the target partition in the next time period is a compression rate that belongs to the second compression rate range; wherein the maximum value of the second compression rate range is less than the minimum value of the first compression rate range.

[0038] In one embodiment, the adjustment module is specifically used to:

[0039] Get the partition list;

[0040] A management command of the database system is called, and a compression ratio corresponding to the target partition in the partition list is adjusted based on the management command and the compression strategy.

[0041] In one embodiment, the prediction module is further used to:

[0042] In response to receiving an update instruction of the preset prediction model, reacquiring the second compression strategy label corresponding to each historical data feature;

[0043] The preset prediction model is trained based on each of the historical data features and the second compression strategy label corresponding to each of the historical data features to obtain an updated preset prediction model.

[0044] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method of the first aspect when executing the computer program.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method of the first aspect.

[0046] The above-mentioned data processing method, device, computer equipment and storage medium dynamically select an appropriate compression rate according to data characteristics, which can ensure data access speed while minimizing storage space, and adjust the compression rate by adjusting the frequency range to which the query frequency belongs, which can improve the efficiency and response speed of data query. At the same time, the compression rate is automatically adjusted using a preset prediction model without manual intervention, thereby reducing operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a data processing method in one embodiment;

[0048] Figure 2 A schematic diagram of a flow chart of a model training method in one embodiment;

[0049] Figure 3 is a flow chart of a data processing method in another embodiment;

[0050] Figure 4 is a structural block diagram of a data processing device in one embodiment;

[0051] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, and A and B exist, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0053] In order to facilitate understanding by those skilled in the art, the technical terms involved in the embodiments of the present application are first explained.

[0054] (1) StarRocks database system: It is a new generation of extremely fast full-scenario Massively Parallel Processing (MPP) database. StarRocks uses partitions to trim data. For example, when partitioning by day, the data for each day will be stored in a separate partition. Another purpose of partitioning is to use partitions as separate management units, and to set separate compression strategies for each partition.

[0055] (2) Compression ratio: The ratio of the size of the compressed data to the size of the data before compression. For example, if 100MB of data is compressed to 40MB, the compression ratio is 40%. A higher compression ratio will save disk storage space, but at the same time, the data needs to be decompressed when it is queried, which will increase the query response time. Uncompressed data will take up more storage space, but at the same time the query response speed will be maximized.

[0056] The following is a brief introduction to the background technology of the embodiments of the present application.

[0057] With the rapid growth of data volume, data compression has become an important means to improve data storage efficiency and reduce storage costs. In the StarRocks database system, compressing the data of each partition can significantly reduce the storage space requirements and also help improve query performance. However, in existing database systems, each partition usually uses a fixed compression strategy, and it is impossible to dynamically adjust the compression strategy according to the characteristics of the partition, which affects the data storage speed and storage space utilization.

[0058] In order to solve the above problems, the present application provides a data processing method, apparatus, computer equipment and storage medium, which dynamically select an appropriate compression rate according to data characteristics, can ensure data access speed while minimizing storage space, and adjust the compression rate by the frequency range to which the query frequency belongs, which can improve the efficiency and response speed of data query. At the same time, the compression rate is automatically adjusted using a preset prediction model without manual intervention, thereby reducing operation and maintenance costs.

[0059] To further illustrate the technical solution provided by the embodiments of the present application, a detailed description is given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] In one embodiment, Figure 1 As shown, a data processing method is provided, comprising the following steps:

[0061] S102, inputting the target data features of the target partition in the current time period into a preset prediction model for prediction, and obtaining the query frequency corresponding to the target partition in the next time period;

[0062] The current time period may be the same as the partition time of the target partition. For example, if the target partition is partitioned by day, the current time period is today, and the corresponding next time period is tomorrow. The current time period and the next time period may also be specified by the user. The specific time period depends on the situation and is not limited here.

[0063] Specifically, the target query log and partition information of the target partition in the current time period are obtained, and the corresponding target data features are extracted from the target query log and partition information, wherein the target data features at least include query partition, query frequency, query complexity, query data range, data volume, and data update frequency. Further, the target data features are input into a preset prediction model for prediction, and the query frequency corresponding to the target partition in the next time period can be obtained.

[0064] S104, determining a compression strategy corresponding to the target partition in the next time period based on the frequency range to which the query frequency belongs;

[0065] Specifically, the frequency range includes at least a first frequency range and a second frequency range, wherein a minimum value of the second frequency range is greater than a maximum value of the first frequency range.

[0066] When the query frequency belongs to the first frequency range, the compression strategy corresponding to the target partition in the next time period is determined to have a compression rate within the first compression rate range; when the query frequency belongs to the second frequency range, the compression strategy corresponding to the target partition in the next time period is determined to have a compression rate within the second compression rate range; wherein the maximum value of the second compression rate range is less than the minimum value of the first compression rate range.

[0067] In one embodiment, in order to meet different data and application requirements, the frequency range may further include a first frequency range, a second frequency range, a third frequency range and a fourth frequency range, wherein the minimum value of the second frequency range is greater than the maximum value of the first frequency range, the minimum value of the third frequency range is greater than the maximum value of the second frequency range, and the minimum value of the fourth frequency range is greater than the maximum value of the third frequency range. For example, the first frequency range is (0, 20], the second frequency range is (20, 50], the third frequency range is (50, 80], and the fourth frequency range is (80, 100]. It should be noted that the specific frequency range values ​​depend on the circumstances and are not limited here.

[0068] When the query frequency belongs to the first frequency range, the compression strategy corresponding to the target partition in the next time period is determined to be a compression rate belonging to the first compression rate range (i.e., high compression rate); when the query frequency belongs to the second frequency range, the compression strategy corresponding to the target partition in the next time period is determined to be a compression rate belonging to the second compression rate range (i.e., medium compression rate); wherein the maximum value of the second compression rate range is less than the minimum value of the first compression rate range; when the query frequency belongs to the third frequency range, the compression strategy corresponding to the target partition in the next time period is determined to be a compression rate belonging to the third compression rate range (i.e., low compression rate); wherein the maximum value of the third compression rate range is less than the minimum value of the second compression rate range; when the query frequency belongs to the fourth frequency range, the compression strategy corresponding to the target partition in the next time period is determined to be no compression. For example, the high compression rate is [70%, 80%], the medium compression rate is [40-70%), and the low compression rate is (0, 40%). It should be noted that the specific compression rate range depends on the situation and is not limited here.

[0069] For example, financial statements are frequently queried at the end of each month. Using the preset prediction model, the compression strategy for the corresponding partition can be determined to be low compression rate or no compression at the end of the month. In other time periods, the compression strategy for the corresponding partition can be determined to be high compression rate to cope with low-frequency data access.

[0070] In one embodiment, in order to improve the efficiency of determining the compression strategy, characters 0, 1, 2, and 3 may be used to represent no compression, low compression rate, medium compression rate, and high compression rate, respectively.

[0071] Through the above method, multiple compression rate options are supported, which can improve the flexibility of data compression.

[0072] S106: Adjust the compression rate of the target partition based on the compression strategy.

[0073] Specifically, a partition list is obtained, and further, a management command of the database system is called to adjust the compression ratio corresponding to the target partition in the partition list based on the management command and the compression strategy, or an adjustment function is called to adjust the compression ratio corresponding to the target partition in the partition list based on the adjustment function and the compression strategy.

[0074] In the above data processing method, the appropriate compression rate is dynamically selected according to the data characteristics, which can ensure the access speed of the data while minimizing the storage space. The compression rate is adjusted by the frequency range to which the query frequency belongs, which can improve the efficiency and response speed of data query. At the same time, the compression rate is automatically adjusted using a preset prediction model without manual intervention, thereby reducing operation and maintenance costs.

[0075] In one embodiment, the preset prediction model can be trained based on the historical data features of each partition. Figure 2 As shown, a model training method is provided, comprising the following steps:

[0076] S202, obtaining historical query logs and partition information of each partition;

[0077] S204, extracting historical data features corresponding to each partition from each historical query log and each partition information;

[0078] The historical data characteristics include at least: query partition, query frequency, query complexity, query data range, data volume, and data update frequency;

[0079] S206, obtaining first compression strategy labels corresponding to respective historical data features;

[0080] Among them, the first compression strategy label can be obtained from user feedback and / or expert experience. The first compression strategy label can be a first compression rate range and a second compression rate range, or it can be no compression, low compression rate, medium compression rate and high compression rate. The specific label depends on the situation and is not limited here.

[0081] S208: Training the initial prediction model based on each historical data feature and the first compression strategy label corresponding to each historical data feature to obtain a preset prediction model.

[0082] Specifically, the compression strategies corresponding to the respective data features are determined based on the initial training model, and the compression strategies corresponding to the respective data features are compared with the first compression strategy labels corresponding to the respective data features. When the ratio of the respective compression strategies to the corresponding first compression strategy labels exceeds a preset threshold, a preset prediction model is obtained.

[0083] In the embodiment of the present application, the initial training model is preferably a random forest model. Of course, the initial training model can also be other machine learning algorithms, such as decision trees, support vector machines, neural networks, etc., which can also be used to predict and adjust data compression rates. These alternatives can be selected according to actual needs and data characteristics to achieve the best compression effect and system performance.

[0084] In the above data processing steps, the preset prediction model is trained based on the historical data features of each partition, so that the preset prediction model can be used to predict the compression strategy, and the compression rate of each partition can be dynamically adjusted according to the model prediction results to ensure the efficient operation of the system. At the same time, model training based on historical data features of multiple dimensions can improve the robustness of the preset prediction model.

[0085] In another embodiment, in order to ensure the accuracy of compression strategy prediction, the adjustment effect of the compression strategy can also be regularly evaluated, for example, by observing data query performance and storage space utilization, to continuously optimize the preset prediction model and compression strategy. Figure 3 As shown, another data processing method is provided, comprising the following steps:

[0086] S302, in response to receiving an update instruction of a preset prediction model, reacquiring a second compression strategy label corresponding to each historical data feature;

[0087] Specifically, when the data query performance and storage space utilization do not meet the preset requirements (for example, the data query efficiency is lower than the first threshold and / or the storage space utilization is lower than the second threshold), an update instruction of the preset prediction model is initiated, and in response to receiving the update instruction of the preset prediction model, the second compression strategy label corresponding to each historical data feature is re-acquired.

[0088] It should be noted that the specific values ​​of the first threshold and the second threshold depend on the circumstances and are not limited here.

[0089] S304: Train the preset prediction model based on each historical data feature and the second compression strategy label corresponding to each historical data feature to obtain an updated preset prediction model.

[0090] In the above data processing method, in response to receiving an update instruction for the preset prediction model, the second compression strategy label corresponding to each historical data feature is re-acquired, and the preset prediction model is updated and optimized based on the second compression strategy label corresponding to each historical data feature, which can further ensure the accuracy of the preset prediction model and the efficient operation of the system.

[0091] It should be understood that although Figure 1-3The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0092] In one embodiment, Figure 4 As shown, a data processing device is provided, including: a prediction module 401, a determination module 402 and an adjustment module 403, wherein:

[0093] Prediction module 401, used to input the target data features of the target partition in the current time period into a preset prediction model for prediction, and obtain the query frequency corresponding to the target partition in the next time period;

[0094] A determination module 402, configured to determine a compression strategy corresponding to the target partition in the next time period based on the frequency range to which the query frequency belongs;

[0095] The adjustment module 403 is used to adjust the compression rate of the target partition based on the compression strategy.

[0096] In one embodiment, the prediction module 401 is further configured to:

[0097] Get the historical query logs and partition information of each partition;

[0098] Extract the historical data features corresponding to each partition from each historical query log and each partition information;

[0099] Obtaining first compression strategy labels corresponding to respective historical data features;

[0100] The initial prediction model is trained based on each historical data feature and the first compression strategy label corresponding to each historical data feature to obtain a preset prediction model.

[0101] In one embodiment, the historical data characteristics include at least one or more of query partition, query frequency, query complexity, query data range, data volume, and data update frequency.

[0102] In one embodiment, the frequency range includes at least: a first frequency range and a second frequency range, wherein a minimum value of the second frequency range is greater than a maximum value of the first frequency range.

[0103] In one embodiment, the determination module 402 is specifically configured to:

[0104] When the query frequency belongs to the first frequency range, determining that the compression strategy corresponding to the target partition in the next time period is that the compression rate belongs to the first compression rate range;

[0105] When the query frequency belongs to the second frequency range, the compression strategy corresponding to the target partition in the next time period is determined to be a compression rate within the second compression rate range; wherein the maximum value of the second compression rate range is less than the minimum value of the first compression rate range.

[0106] In one embodiment, the adjustment module 403 is specifically used to:

[0107] Get the partition list;

[0108] Call the management command of the database system, and adjust the compression ratio corresponding to the target partition in the partition list based on the management command and the compression strategy.

[0109] In one embodiment, the prediction module 401 is further configured to:

[0110] In response to receiving an update instruction of a preset prediction model, reacquiring a second compression strategy label corresponding to each of the historical data features;

[0111] The preset prediction model is trained based on each historical data feature and the second compression strategy label corresponding to each historical data feature to obtain an updated preset prediction model.

[0112] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0113] It should be noted here that the above-mentioned device provided in the present application can implement all the method steps in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0114] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data processing data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0115] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0117] Input the target data features of the target partition in the current time period into the preset prediction model for prediction, and obtain the query frequency corresponding to the target partition in the next time period;

[0118] Determine the compression strategy corresponding to the target partition in the next time period based on the frequency range to which the query frequency belongs;

[0119] Adjust the compression ratio of the target partition based on the compression strategy.

[0120] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0121] Get the historical query logs and partition information of each partition;

[0122] Extract the historical data features corresponding to each partition from each historical query log and each partition information;

[0123] Obtaining first compression strategy labels corresponding to respective historical data features;

[0124] The initial prediction model is trained based on each historical data feature and the first compression strategy label corresponding to each historical data feature to obtain a preset prediction model.

[0125] In one embodiment, the historical data characteristics include at least one or more of query partition, query frequency, query complexity, query data range, data volume, and data update frequency.

[0126] In one embodiment, the frequency range includes at least: a first frequency range and a second frequency range, wherein a minimum value of the second frequency range is greater than a maximum value of the first frequency range.

[0127] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0128] When the query frequency belongs to the first frequency range, determining that the compression strategy corresponding to the target partition in the next time period is that the compression rate belongs to the first compression rate range;

[0129] When the query frequency belongs to the second frequency range, the compression strategy corresponding to the target partition in the next time period is determined to be a compression rate within the second compression rate range; wherein the maximum value of the second compression rate range is less than the minimum value of the first compression rate range.

[0130] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0131] Get the partition list;

[0132] Call the management command of the database system, and adjust the compression ratio corresponding to the target partition in the partition list based on the management command and the compression strategy.

[0133] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0134] In response to receiving an update instruction of a preset prediction model, reacquiring a second compression strategy label corresponding to each of the historical data features;

[0135] The preset prediction model is trained based on each historical data feature and the second compression strategy label corresponding to each historical data feature to obtain an updated preset prediction model.

[0136] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0137] Input the target data features of the target partition in the current time period into the preset prediction model for prediction, and obtain the query frequency corresponding to the target partition in the next time period;

[0138] Determine the compression strategy corresponding to the target partition in the next time period based on the frequency range to which the query frequency belongs;

[0139] Adjust the compression ratio of the target partition based on the compression strategy.

[0140] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0141] Get the historical query logs and partition information of each partition;

[0142] Extract the historical data features corresponding to each partition from each historical query log and each partition information;

[0143] Obtaining first compression strategy labels corresponding to respective historical data features;

[0144] The initial prediction model is trained based on each historical data feature and the first compression strategy label corresponding to each historical data feature to obtain a preset prediction model.

[0145] In one embodiment, the historical data characteristics include at least one or more of query partition, query frequency, query complexity, query data range, data volume, and data update frequency.

[0146] In one embodiment, the frequency range includes at least: a first frequency range and a second frequency range, wherein a minimum value of the second frequency range is greater than a maximum value of the first frequency range.

[0147] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0148] When the query frequency belongs to the first frequency range, determining that the compression strategy corresponding to the target partition in the next time period is that the compression rate belongs to the first compression rate range;

[0149] When the query frequency belongs to the second frequency range, the compression strategy corresponding to the target partition in the next time period is determined to be a compression rate within the second compression rate range; wherein the maximum value of the second compression rate range is less than the minimum value of the first compression rate range.

[0150] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0151] Get the partition list;

[0152] Call the management command of the database system, and adjust the compression ratio corresponding to the target partition in the partition list based on the management command and the compression strategy.

[0153] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0154] In response to receiving an update instruction of a preset prediction model, reacquiring a second compression strategy label corresponding to each of the historical data features;

[0155] The preset prediction model is trained based on each historical data feature and the second compression strategy label corresponding to each historical data feature to obtain an updated preset prediction model.

[0156] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0157] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: include: Input the target data features of the target partition in the current time period into a preset prediction model for prediction, and obtain the query frequency corresponding to the target partition in the next time period; Determine, based on the frequency range to which the query frequency belongs, a compression strategy corresponding to the target partition in the next time period; The compression ratio of the target partition is adjusted based on the compression strategy.

2. The method according to claim 1, characterized in that: Before inputting the target data features of the target partition in the current time period into the preset prediction model for prediction, the method further includes: Get the historical query logs and partition information of each partition; Extracting historical data features corresponding to each partition from each historical query log and each partition information; Obtaining first compression strategy labels corresponding to respective historical data features; An initial prediction model is trained based on each of the historical data features and the first compression strategy label pairs corresponding to each of the historical data features to obtain the preset prediction model.

3. The method according to claim 2, characterized in that The historical data characteristics include at least one or more of query partition, query frequency, query complexity, query data range, data volume and data update frequency.

4. The method according to claim 1, characterized in that The frequency range includes at least: a first frequency range and a second frequency range, wherein a minimum value of the second frequency range is greater than a maximum value of the first frequency range.

5. The method according to claim 4, characterized in that The determining, based on the frequency range to which the query frequency belongs, a compression strategy corresponding to the target partition in the next time period includes: When the query frequency belongs to the first frequency range, determining that the compression strategy corresponding to the target partition in the next time period is that the compression rate belongs to the first compression rate range; When the query frequency belongs to the second frequency range, determine that the compression strategy corresponding to the target partition in the next time period is a compression rate that belongs to the second compression rate range; wherein the maximum value of the second compression rate range is less than the minimum value of the first compression rate range.

6. The data processing method according to claim 1, characterized in that: The adjusting the compression rate of the target partition based on the compression strategy includes: Get the partition list; A management command of the database system is called, and a compression ratio corresponding to the target partition in the partition list is adjusted based on the management command and the compression strategy.

7. The data processing method according to claim 2, characterized in that: The method further comprises: In response to receiving an update instruction of the preset prediction model, reacquiring the second compression strategy label corresponding to each historical data feature; The preset prediction model is trained based on each of the historical data features and the second compression strategy label corresponding to each of the historical data features to obtain an updated preset prediction model.

8. A data processing device, characterized in that: The device comprises: A prediction module, used to input the target data features of the target partition in the current time period into a preset prediction model for prediction, and obtain the query frequency corresponding to the target partition in the next time period; A determination module, configured to determine a compression strategy corresponding to the target partition in the next time period based on a frequency range to which the query frequency belongs; An adjustment module is used to adjust the compression rate of the target partition based on the compression strategy.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images