Data compression method, device, equipment, computer storage medium and program product

Abstract

The application discloses a data compression method and device, equipment, a computer storage medium and a program product, and relates to the technical field of big data. The data compression method comprises the following steps: obtaining target information of N data of the same storage level from a database, wherein the target information comprises the access frequency of the data in a target time period, and N is a positive integer; and based on the target information, the data with the access frequency lower than a preset threshold in the N data is compressed. According to the embodiment of the application, the compression of the data in the database can be efficiently and reasonably realized without affecting the data service processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of big data, and particularly relates to a data compression method and device, equipment, computer storage medium and program product. BACKGROUND

[0002] At present, in order to improve the storage efficiency of data, for example, a time series database usually encodes and compresses data in the process of data writing, so as to reduce the amount of disk space used. In this way, after encoding and compression, the amount of data of I / O operation can be reduced in the subsequent process of writing and reading data, thereby improving performance,

[0003] However, the essence of the above compression scheme is to encode and compress all data in the database to exchange CPU (Central Processing Unit / Processor) time for disk space or network I / O transmission. In this way, when the data throughput is large, a large amount of write data needs to be encoded and compressed, which slows down the disk reading rate and affects the business.

[0004] In view of the above, a new data compression method is still urgently needed to efficiently and reasonably compress data in the database without affecting the efficiency of data business processing. SUMMARY

[0005] The embodiments of the present application provide a data compression method, device, equipment, computer storage medium and program product, which can efficiently and reasonably compress data in the database without affecting the efficiency of data business processing.

[0006] In a first aspect, the embodiments of the present application provide a data compression method, which comprises:

[0007] obtaining target information of N data of the same storage level from the database, the target information comprising the access frequency of the data in a target time period, and N being a positive integer;

[0008] based on the target information, compressing the data with an access frequency lower than a preset threshold in the N data.

[0009] In some possible implementation manners, before obtaining the target information of the N data of the same storage level from the database, the data compression method further comprises:

[0010] respectively adding a target frequency field in the N data;

[0011] The target frequency field is used to record the access frequency of the data in the target time period in chronological order.

[0012] In some possible implementation manners, the N data further comprises a compression identifier field, a first state of the compression identifier field is used to represent that the data does not need to be compressed, and a second state of the compression identifier field is used to represent that the data needs to be compressed.

[0013] Based on the target information, the data with the access frequency lower than the preset threshold in the N data is compressed, including:

[0014] In the case of reaching an end node of the target time period, the access frequency of the N data in the target time period is counted to obtain a statistical result.

[0015] Based on the statistical result, the compression identifier field included in the target data is switched from the first state to the second state, and the target data is the data with the access frequency lower than the preset threshold in the target time period in the N data.

[0016] The target data with the compression identifier field in the second state is compressed.

[0017] In some possible implementation manners, the database is a time-series database, the N data in the same storage level comprises N pages, and the target frequency field is added in the N data, including:

[0018] The target frequency field is added in the PageHeader structure in the N pages.

[0019] In some possible implementation manners, the database is a time-series database, the N data in the same storage level comprises N storage partitions, and the target frequency field is added in the N data, including:

[0020] The target frequency field is added in the partition index file in the N storage partitions.

[0021] In some possible implementation manners, before the target frequency field is added in the partition index file in the N storage partitions, the data compression method further comprises:

[0022] The storage group in the database is partitioned to obtain the N storage partitions.

[0023] In a second aspect, an embodiment of the present application provides a data compression device, which comprises:

[0024] A first acquisition module is configured to acquire target information of N data in the same storage level from a database, the target information comprising an access frequency of data in a target time period, and N is a positive integer.

[0025] A first compression module is configured to compress data with an access frequency lower than a preset threshold in the N data based on the target information.

[0026] In a third aspect, an embodiment of the present application provides a data compression device, the data compression device comprising:

[0027] a processor and a memory storing computer program instructions;

[0028] The processor implements the data compression method provided in any one of the above embodiments of the present application when executing the computer program instructions.

[0029] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the data compression method provided in any one of the above embodiments of the present application.

[0030] In a fifth aspect, an embodiment of the present application provides a computer program product, and instructions in the computer program product are executed by a processor of an electronic device to enable the electronic device to perform the data compression method provided in any one of the above embodiments of the present application.

[0031] The data compression method, device, equipment, computer storage medium and program product provided in the embodiments of the present application can obtain the access frequency of N data in a target time period from a database of the same storage level, so as to filter out data with an access frequency lower than a preset threshold in the N data according to the access frequency of the N data in the target time period. The data compression method, device, equipment, computer storage medium and program product provided in the embodiments of the present application can balance the contradiction between the container storage space demand and the data fast reading by fully combining the access frequency of the data in the same storage level of the database to compress the data, and preferentially compressing the data with less access, so as to efficiently and reasonably realize the compression of the data in the database without affecting the data business processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. For those skilled in the art, other drawings can also be obtained without creative labor on the premise of the drawings.

[0033] Figure 1 is a flowchart of the data compression method provided in an embodiment of the present application;

[0034] Figure 2 is a user access frequency diagram of the data compression method provided in an embodiment of the present application;

[0035] Figure 3 is a structural diagram of a data compression ratio example provided in an embodiment of the present application;

[0036] Figure 4 is a structural schematic diagram of a data compression device provided by an embodiment of the present application;

[0037] Figure 5 is a structural schematic diagram of a data compression device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] The features and exemplary embodiments of various aspects of the present application will be described in detail below with reference to the drawings. To make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of the specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0039] It should be noted that, in this document, relational terms such as first and second and the like can only be used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations exist in any such actual relationship or order. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the elements defined by the statement "comprising" do not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the elements.

[0040] As described in the foregoing background section, in the prior art, when compressing database data, the data in the database is usually fully encoded and compressed, so as to exchange CPU time for disk space or network I / O transmission. However, when the data throughput is large, this compression scheme will slow down the disk reading rate due to the need to encode and compress a large amount of write data, which will affect the business.

[0041] To solve the problems in the prior art, the embodiments of the present application provide a data compression method, device, equipment, storage medium and computer program product. It should be noted that the embodiments provided by the present application are not intended to limit the scope of the present application.

[0042] First, the data compression method provided by the embodiments of the present application will be introduced.

[0043] Figure 1A flowchart of a data compression method provided by an embodiment of the present application is shown. The data compression method is applied to an electronic device, which can include a server or a user terminal, etc. As shown in the figure, the data compression method includes the following steps: Figure 1

[0044] S110, obtaining target information of N data in the same storage level from a database, the target information including access frequency of the data in a target time period, N being a positive integer;

[0045] S120, compressing data with access frequency lower than a preset threshold in the N data based on the target information.

[0046] The data compression method of the embodiment of the present application can obtain access frequency of N data in the same storage level in a target time period from a database, so as to filter out data with access frequency lower than a preset threshold in the N data according to the access frequency of the N data in the target time period for compression. The data compression method provided by the embodiment of the present application can balance the contradiction between container storage space demand and fast data reading by sufficiently combining access frequency of data in the same storage level in a database to compress the data, preferentially compressing less accessed data, so as to efficiently and reasonably compress data in the database without affecting data business processing efficiency.

[0047] The specific implementation of the above steps 110 to 120 is described in detail below.

[0048] In S110, the target information of N data in the same storage level is obtained from a database in the specific implementation, the target information including access frequency of the data in a target time period, N being a positive integer.

[0049] It should be noted that in a time series database, data is usually stored in a certain tree directory structure, and the size of data under different target structures is different. For example, in the field of time series databases, the smallest storage unit in the disk is a data block chunk, and each data block stores one or more data.

[0050] Based on this, the same storage level involved in the embodiment of the present application is essentially about the same target structure of data storage. Considering that the storage target structures set in different databases are not consistent, and a database includes multiple storage target structures, the present application does not limit the above data to which storage level in this regard.

[0051] The above target time period can be set according to actual data compression needs, for example, the above target time period can be a certain week, a certain month, etc., and the present application does not make specific limitations here.​

[0052] In the present application, the above-mentioned database can be a time-series database, a relational database, or the like, and the present application does not make specific limitations thereto.

[0053] Specifically, the target information of the N data of the same storage level obtained from the database can be the number of times each data in the N data is accessed in the target time period, so that the access frequency of the N data in the target time period can be obtained in the case that the target time period reaches the end node.

[0054] In some possible implementation manners, specifically, in order to more accurately obtain the access frequency of the N data in the target time period, before obtaining the target information of the N data of the same storage level from the database, the data compression method can further include:

[0055] adding a target frequency field in each of the N data respectively;

[0056] The target frequency field can be used to record the access frequency of the data in the target time period in chronological order.

[0057] Specifically, the target frequency field can be added in each of the N data, and the target frequency field is incremented by 1 when a user accesses a certain data until the target time period reaches the end node.

[0058] In S120, based on the target information, the data with an access frequency lower than a preset threshold in the N data is compressed.

[0059] Specifically, after obtaining the access frequency of the N data in the target time period, the N data can be sorted according to the access frequency of the N data, and compared with the preset threshold. In this way, the data lower than the preset threshold in the N data can be screened out for compression, so that the data with less access can be compressed preferentially, thereby balancing the contradiction between the container storage space demand and the fast reading of data, and further realizing more reasonable and efficient data compression.

[0060] In some possible implementation manners, further, in order to more reasonably and efficiently realize data compression, the N data can further include a compression identification field, a first state of the compression identification field can be used to represent that the data does not need to be compressed, and a second state of the compression identification can be used to represent that the data needs to be compressed.

[0061] The step 120 of compressing the data with an access frequency lower than a preset threshold in the N data based on the target information can specifically include:

[0062] In the case of reaching the end node of the target time period, the access frequency of the N data in the target time period is counted to obtain a statistical result;

[0063] Based on the statistical result, the compression identification field included in the target data is switched from the first state to the second state; the target data is the data in the N data whose access frequency in the target time period is lower than a preset threshold;

[0064] The target data with the compression identification field in the second state is compressed.

[0065] Exemplarily, the compression identification field can be switched between 0 and 1, 0 is the first state of the compression identification field, and 1 can be the second state of the compression identification field, which is not specifically limited in the present application.

[0066] In a specific implementation, at the start node of the target time period, the compression identification fields included in the N data are all in the first state. When the end node of the target time period is reached, the access frequency of the N data in the target time period is counted through the target frequency field in the N data, so as to obtain the statistical result of the N data about the access frequency.

[0067] After obtaining the statistical result, based on the statistical result, the data whose access frequency in the target time period is lower than the preset threshold is determined from the N data, and the compression identification field in the data whose access frequency in the target time period is lower than the preset threshold is switched from the first state to the second field.

[0068] In this way, when the data is finally compressed, it is only necessary to determine whether the compression identification field in the N data in the database is in the second state, and the data with the compression identification field in the second state is determined as the data that needs to be compressed. In this way, the prior compression processing of the data with less access frequency can be more quickly and accurately realized.

[0069] In some more specific embodiments, in order to more reasonably realize the addition of the target frequency field, and in order to guarantee the accuracy of the compressed data, the database can be a time series database, and the N data in the same storage level can include N pages; adding the target frequency field in the N data can include:

[0070] The target frequency field is added in the PageHeader structure in the N pages, respectively.

[0071] Specifically, in the time series database, the TsFile is a file format specially customized for time series data storage, and is organized in a tree directory structure as a whole. The TsFile is a multi-level mapping table as a whole, and the metadata index structure is TsFileMetaData->TimeSeriesMetadata->ChunkMetadata->Chunk. The Chunk is a data block / storage unit, and stores data of a physical quantity (Measurement) for a period of time. The Chunk can be further divided, and the Chunk can include multiple pages.

[0072] In the embodiment, the N pieces of data can be N pages. The target frequency field is added to the PageHeader structure in the N pages to record the number of times that the data in the target time period is accessed by the user.

[0073] More specifically, the compression identifier field can be added to the PageHeader structure in the N pages to identify whether the data is data that needs to be compressed. In addition, to more reasonably implement the statistics of the data access frequency in the target time period, the target time period field can be included in the PageHeader structure in the N pages. The target time period field is specifically the length of time for which the number of accesses is counted, and the time scale can be accurate to milliseconds.

[0074] To more intuitively reflect the fields added to the PageHeader structure in the N pages, the types of the fields, and the roles of the fields in the present application, please refer to Table 1.

[0075]

[0076] In Table 1, the frequency field is the target frequency field, the field type is long, and the frequency field specifically represents the number of times that the database accesses the page in a specified period (target time period); the period field is the target time period field, the field type is String, and the period field is used to count the length of time for which the number of accesses is counted; and the compression field is the compression identifier field, the field type is String, and the compression field specifically identifies whether compression is performed.

[0077] In a complete data compression process, the database starts the data compression function, and the user inputs related parameters to customize the period field (target time period), the data compression mode, the data compression ratio, and the preset threshold.

[0078] After the above configuration is completed, when a user accesses any one of the N pages, the frequency field is automatically incremented by 1 until the period time ends.

[0079] In the case where the period time ends, the compression field of the page data below the threshold is identified as needing compression (i.e., switched to the second state) according to the frequency field ranking. Finally, the page data for which the compression field is identified as needing compression is compressed. The compression manner, compression ratio, etc. of specific data can be determined according to the compression parameters configured by the user in advance.

[0080] In this embodiment, by recording the access times of different page data during data access, page data with low access frequency is counted and compressed. For details, please refer to Figure 2 , Figure 2 is a user access frequency diagram of the data compression method provided by an embodiment of the present application. As shown in the following Figure 2 , the horizontal axis is time and the vertical axis is access times, it can be known from Figure 2 that this embodiment can sufficiently compress data in time periods with low access times.

[0081] In some more specific embodiments, similarly, in order to more reasonably achieve the addition of the target frequency field and, in order to quickly achieve compression of data in batches and large scales, the above database can be a time series database, and the N data in the same storage level can include N storage partitions; adding the target frequency field in the N data respectively can include:

[0082] Adding the target frequency field in the partition index file in the N storage partitions respectively.

[0083] In actual implementation, since the data is located in different storage levels, the corresponding data sizes are also different, and therefore, in the present application, the user can set the storage level of the data according to actual compression requirements. In the time series database, when data compression in batches and large scales needs to be achieved, the above N data can directly correspond to N storage partitions.

[0084] In this way, by counting the access frequencies of data in different storage partitions, data compression in large batches in the database can be efficiently achieved.

[0085] In some possible embodiments, before adding the target frequency field in the partition index file in the N storage partitions respectively, the data compression method can further include:

[0086] The storage groups in the database are partitioned to obtain N storage partitions.

[0087] Specifically, in order to facilitate data expiration, the data stored by the time series database is stored in different storage groups using different directories. The tsfile files and resource files exist in different directories, and the resource files store the metadata information of the tsfile files. The directory organization of the time series database is generally data directory: storage group->partition number (currently one partition by default)->data file.tsfile and partition index file resource.

[0088] In this embodiment, in order to more reasonably realize batch statistics and compression of data, the subdirectory storage of time series data is further refined to partitions, and multiple partitions are allocated for the same storage group. In this way, by adding the fields shown in Table 1 to the partition index file resource in each partition, the frequency of user access to different partitions can be effectively recorded, so that the data blocks corresponding to the storage partitions with low access frequency are preferentially compressed.

[0089] In another complete data compression process, the time series database starts the data compression function, and the user inputs related parameters to customize the configuration of the above-mentioned period field (target time period), data compression mode, data compression ratio, and the above-mentioned preset threshold value.

[0090] After the above-mentioned configuration is completed, when a user accesses any one of the N storage partitions, the data accessed by the user is located in which storage partition, and after the location, the frequency field in the partition index file resource of the corresponding storage partition is incremented by 1, until the period ends.

[0091] In the case where the period ends, according to the frequency field sorting, the page data whose compression field is lower than the threshold value is identified as needing compression (i.e., switching to the second state). Finally, the data in the storage partition corresponding to the partition index file with the compression field identified as needing compression is compressed. The compression mode, compression ratio, etc. of the specific data can be determined according to the compression parameters configured by the user in advance.

[0092] After the data compression is completed, the information after the data compression (such as compression algorithm, compression ratio, data size before and after compression, etc.) is counted and updated to the metadata TimeseriesMetadata.

[0093] In order to facilitate more intuitive understanding of the data compression scheme provided in the above embodiment, please refer to Figure 3 , Figure 3is a structural schematic diagram of a data compression ratio example provided by an embodiment of the present application. Figure 3 As shown, for Figure 3 The data blocks of each storage partition on the top have a user-configured compression ratio of 50%. At this time, the storage partitions that need to be compressed are determined according to the access frequencies of the data blocks of each storage partition, and the data blocks of the storage partitions with access frequencies of 10, 30, and 60 need to be compressed. Figure 3 As shown, for

[0094] The data blocks of each storage partition on the top have a user-configured compression ratio of 50%. At this time, the storage partitions that need to be compressed are determined according to the access frequencies of the data blocks of each storage partition, and the data blocks of the storage partitions with access frequencies of 10, 30, and 60 need to be compressed. Figure 3 As shown, for Figure 3 The data blocks of each storage partition on the top have a user-configured compression ratio of 50%. At this time, the storage partitions that need to be compressed are determined according to the access frequencies of the data blocks of each storage partition, and the data blocks of the storage partitions with access frequencies of 10, 30, and 60 need to be compressed.

[0095] In this embodiment, the storage groups are regionally divided, thereby reducing the time consumed for reading data by using the multi-partition form. In this way, by using the access frequencies of each storage partition, the data corresponding to the storage partitions with low read frequencies is preferentially compressed, thereby improving the execution efficiency and system resource utilization, and effectively balancing the contradiction between the container storage space demand and the fast data reading.

[0096] Based on the data compression method provided in the above embodiment, the present application further provides a data compression device corresponding to the above data compression method, which will be described in detail below. Figure 4 The data compression device will be described in detail.

[0097] Figure 4 A structural schematic diagram of a data compression device provided by an embodiment of the present application is shown. Figure 4 The data compression device 400 shown includes:

[0098] The first obtaining module 410 is configured to obtain target information of N data of the same storage level from a database, the target information including an access frequency of the data in a target time period, and N is a positive integer.

[0099] The first compression module 420 is configured to compress the data with an access frequency lower than a preset threshold value in the N data based on the target information.

[0100] The data compression device provided in the embodiment of the present application can obtain the access frequency of N data in the same storage level in the target time period from the database by setting the corresponding function module, so as to filter the data with the access frequency lower than the preset threshold in the N data according to the access frequency of the N data in the target time period, and compress the data. The data compression device provided in the embodiment of the present application can compress the data by fully combining the access frequency of the user to the data in the same storage level in the database, and preferentially compress the data with less access, so as to balance the contradiction between the container storage space requirement and the fast reading of data, and efficiently and reasonably compress the data in the database without affecting the data business processing efficiency.

[0101] In some possible implementation manners, before obtaining the target information of the N data in the same storage level from the database, the data compression device can further include:

[0102] The adding module can be used to add the target frequency field in the N data respectively.

[0103] The target frequency field can be used to record the access frequency of the data in the target time period in chronological order.

[0104] In some possible implementation manners, the N data can further include a compression identification field, and the first state of the compression identification field can be used to represent that the data does not need to be compressed, and the second state of the compression identification field can be used to represent that the data needs to be compressed.

[0105] The first compression module 420 can specifically include:

[0106] The obtaining submodule can be used to obtain the statistical result of the access frequency of the N data in the target time period when the end node of the target time period is reached.

[0107] The switching submodule can be used to switch the compression identification field included in the target data from the first state to the second state based on the statistical result, and the target data is the data with the access frequency lower than the preset threshold in the N data in the target time period.

[0108] The compression submodule can be used to compress the target data with the second state of the compression identification field.

[0109] In some possible implementation manners, the database can be a time series database, and the N data in the same storage level can include N pages. The adding module and the target frequency field added in the N data respectively can include:

[0110] The target frequency field is added in the PageHeader structure in the N pages respectively.

[0111] In some possible implementation manners, the database is a time-series database, and the N data in the same storage level can include N storage partitions; and the increasing module and the increasing target frequency field in the N data respectively can include:

[0112] increasing the target frequency field in the partition index file in the N storage partitions respectively.

[0113] In some possible implementation manners, before the target frequency field is increased in the partition index file in the N storage partitions respectively, the data compression apparatus can further include:

[0114] a partitioning module, configured to perform partitioning on the storage groups in the database to obtain the N storage partitions.

[0115] Figure 5 FIG. 1 is a structural schematic diagram of a data compression device according to an embodiment of the present application.

[0116] The data compression device can include a processor 501 and a memory 502 storing computer program instructions.

[0117] Specifically, the processor 501 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application.

[0118] The memory 502 can include a mass storage for data or instructions. By way of example and not limitation, the memory 502 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. The memory 502 can include removable or non-removable (or fixed) media, where appropriate. The memory 502 can be integral to, or external to, the integrated gateway disaster recovery device, as appropriate. In particular embodiments, the memory 502 is non-volatile, solid-state memory.

[0119] The memory can include read-only memory (ROM), random access memory (RAM), magnetic disk storage mediums devices, optical storage mediums devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage mediums (e.g., memory devices) encoded with software including computer-executable instructions that, when executed (e.g., by one or more processors), are operable to perform the operations described with reference to the methods according to an aspect of the present disclosure.

[0120] The processor 501 implements any one of the data compression methods in the above-described embodiments by reading and executing computer program instructions stored in the memory 502.

[0121] In one example, the data compression device can further include a communication interface 503 and a bus 510. As shown, the processor 501, the memory 502, and the communication interface 503 are connected through the bus 510 and complete communication with each other. Figure 5

[0122] The communication interface 503 is mainly used to realize the communication between the modules, devices, units, and / or apparatuses in the embodiments of the present application.

[0123] The bus 510 includes hardware, software, or both, that couples components of the data compression device to each other. By way of example, and not limitation, the bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, the bus 510 can include one or more buses. Although the present application describes and illustrates a particular bus, the present application contemplates any suitable bus or interconnect.

[0124] The data compression device executes the data compression method in the embodiments of the present application, thereby realizing Figure 1 the data compression method described.

[0125] In addition, in combination with the data compression method in the above-described embodiments, the embodiments of the present application can provide a computer storage medium to realize. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to realize any one of the data compression methods in the above-described embodiments. ​

[0126] Based on the data compression method in the above embodiments, an embodiment of the present application provides a computer program product. Instructions in the computer program product are executed by a processor of an electronic device, so that the electronic device performs the data compression method provided in any one of the above embodiments of the present application.

[0127] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0128] The functional blocks shown in the structural block diagrams described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of the machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via a computer network such as the Internet, an intranet, etc.

[0129] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0130] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0131] The above only is a specific implementation of the present application, and those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, module and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described herein. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements shall be covered within the protection scope of the present application.

Claims

1. A data compression method, characterized in that, include: Retrieve target information for N data items at the same storage level from the database. The target information includes the access frequency of the data within a target time period, where N is a positive integer. Based on the target information, the data among the N data whose access frequency is lower than a preset threshold is compressed, wherein the compression method and compression ratio are determined according to the pre-configured compression parameters; Before retrieving the target information of N data points at the same storage level from the database, the method further includes: Add a target frequency field to each of the N data points; The target frequency field is used to record the access frequency of data within the target time period in chronological order. The N data also include a compression identifier field. The first state of the compression identifier field is used to indicate that the data does not need to be compressed, and the second state of the compression identifier field is used to indicate that the data needs to be compressed. The step of compressing data with an access frequency lower than a preset threshold among the N data based on the target information includes: Upon reaching the end of the target time period, the access frequency of the N data points within the target time period is statistically analyzed to obtain the statistical results. Based on the statistical results, the compression identifier field included in the target data is switched from the first state to the second state; the target data is the data among the N data that has an access frequency lower than the preset threshold within the target time period; The target data whose compression identifier field is in the second state is compressed; The database is a time-series database, and the N data at the same storage level include N storage partitions; adding a target frequency field to each of the N data includes: Add the target frequency field to the partition index file of each of the N storage partitions.

2. The method according to claim 1, characterized in that, The database is a time-series database, and the N data items at the same storage level comprise N pages; adding a target frequency field to each of the N data items includes: Add the target frequency field to the PageHeader structure of each of the N pages.

3. The method according to claim 1, characterized in that, Before adding the target frequency field to the partition index files in the N storage partitions respectively, the method further includes: The storage groups in the database are partitioned to obtain the N storage partitions.

4. A data compression device, characterized in that, The device includes: The first acquisition module is used to acquire target information of N data at the same storage level from the database. The target information includes the access frequency of the data within the target time period, where N is a positive integer. The first compression module is used to compress data with an access frequency lower than a preset threshold among the N data based on the target information, wherein the compression method and compression ratio are determined according to pre-configured compression parameters; The first acquisition module is further configured to add a target frequency field to each of the N data; wherein, the target frequency field is used to record the access frequency of the data within the target time period in chronological order; the N data also include a compression identifier field, wherein a first state of the compression identifier field is used to indicate that the data does not need to be compressed, and a second state of the compression identifier field is used to indicate that the data needs to be compressed. The first compression module is further configured to, upon reaching the end of the target time period, statistically analyze the access frequency of the N data within the target time period to obtain statistical results; based on the statistical results, switch the compression identifier field included in the target data from a first state to a second state; the target data is the data among the N data whose access frequency within the target time period is lower than the preset threshold; compress the target data whose compression identifier field is in the second state; the database is a time-series database, and the N data at the same storage level include N storage partitions; add the target frequency field to the partition index file of each of the N storage partitions.

5. A data compression device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data compression method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the data compression method as described in any one of claims 1-3.

7. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the data compression method as described in any one of claims 1-3.

Images