Data compression method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of data processing technology, and discloses a data compression method, apparatus, computer equipment, and storage medium. The present invention determines parameters of a compression object and a compression window based on current read / write performance and current processor utilization, and configures the compression object and compression window; obtains data to be written, and performs a compression operation on the data to be written based on the compression object and compression window to obtain compressed data corresponding to the data to be written; and writes the compressed data into a data block. In this way, the data to be written is compressed in sections using the compression object and compression window, reducing the size of the space used to store intermediate data, thereby reducing the occupation of memory resources; at the same time, the data is compressed before being written into the data block, and the compressed data is directly written into the data block, eliminating the need for repeated reading and erasing, thereby not only improving the efficiency of data compression but also reducing the occupation of processor and memory resources.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data compression method, device, computer equipment and storage medium. Background Art

[0002] To save storage space and improve transmission efficiency, currently, after writing data from an I / O queue to a data block, the data in the block is typically compressed by traversing the physical addresses within the block. During data compression, the newly generated compressed data is written to another continuous storage space. Once the compression is complete, the original data space is reclaimed. This method requires additional, relatively long storage space to store intermediate data and requires repeated reading and erasing, resulting in low data compression efficiency and a waste of processor and memory resources. Summary of the Invention

[0003] In view of this, the present invention provides a data compression method, apparatus, computer device and storage medium to solve the problems of low data compression efficiency and waste of processor and memory resources.

[0004] In a first aspect, the present invention provides a data compression method, the method comprising:

[0005] Determine the parameters of the compression object and the compression window based on the current read / write performance and the current processor performance, and configure the compression object and the compression window;

[0006] Obtain the data to be written, and perform a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written;

[0007] Write compressed data into data blocks.

[0008] In this way, before the data is written into the data block, it is segmented and compressed using compression objects and compression windows to reduce the space used to store intermediate data, thereby reducing the occupation of memory resources; at the same time, the data is compressed before being written into the data block, and the compressed data is directly written into the data block, eliminating the need for repeated reading and erasing, which not only improves the efficiency of data compression, but also reduces the occupation of processor and memory resources.

[0009] In an optional implementation, determining parameters of a compression object and a compression window based on current read / write performance and current processor performance includes:

[0010] Obtaining preset parameters of the compression object and compression window; the preset parameters include a preset field length of the compression object, a preset window length of the compression window, and a preset number of windows;

[0011] Based on the current read and write performance, adjust and determine the preset field length and the preset window length to obtain the field length of the compression object and the window length of the compression window;

[0012] Based on the current processor performance, the preset number of windows is adjusted to obtain the number of compressed windows.

[0013] In this way, the parameters of the compression object and the compression window are dynamically adjusted according to the current read and write performance and the current processor performance, thereby comprehensively considering the current read and write performance and the current processor performance to achieve a balance between compression efficiency and space utilization.

[0014] In an optional implementation, the current read / write performance includes current disk performance and current queue waiting time. Adjusting and determining a preset field length and a preset window length based on the current read / write performance to obtain a field length of a compression object and a window length of a compression window includes:

[0015] If the current disk placement performance is lower than the preset disk placement performance within the first time period, the preset field length is increased by the unit field length, and the preset window length is increased by the unit window length to obtain the field length and window length;

[0016] If the current queue waiting time is greater than the preset queue waiting time within the second time period, the preset field length is reduced by the unit field length, and the preset window length is reduced by the unit window length to obtain the field length and the window length.

[0017] In this way, when the current disk placement performance is less than the preset disk placement performance, the field length of the compression object and the window length of the compression window are increased, thereby making more full use of the disk placement performance and improving the compression efficiency and space utilization; when the current queue waiting time is greater than the preset queue waiting time, the field length of the compression object and the window length of the compression window are reduced, thereby reducing the queue waiting time and improving the compression efficiency.

[0018] In an optional implementation, adjusting the preset number of windows based on current processor performance to obtain the number of compressed windows includes:

[0019] If the current processor performance is less than the preset processor performance during the third time period, the preset number of windows is increased by the number of unit windows to obtain the number of windows;

[0020] If the current processor performance is greater than the preset processor performance during the fourth time period, the preset window number is reduced by the unit window number to obtain the window number.

[0021] In this way, when the current processor performance is less than the preset processor performance, the number of compression windows is increased, thereby improving processor utilization and improving compression efficiency; when the current processor performance is greater than the preset processor performance, the number of compression windows is reduced to improve compression efficiency.

[0022] In an optional embodiment, performing a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written includes:

[0023] Write the data to be written into the compression queue;

[0024] Create a compression window at the initial position of the compression window corresponding to the compression queue;

[0025] The data to be written in the compression window corresponding to the compression queue is used as the data to be compressed, and the data to be compressed is traversed;

[0026] If the traversal of the data to be compressed is not completed, a compressed object is obtained from the data to be compressed based on the field length of the compressed object, and the compressed object is compared with the data objects in the data object set;

[0027] If the data object set does not contain the data object corresponding to the compressed object, construct the data object based on the compressed object and the object code and physical address of the compressed object, and write it into the data object set;

[0028] Writing the compressed object into the compressed data set based on the data object set and the physical address of the compressed object;

[0029] If the traversal of the data to be compressed has been completed, the compression window will be slid by the data length corresponding to the window length of the compression window to update the compression window, and the data to be written corresponding to the updated compression window in the compression queue will be used as the updated data to be compressed. The updated data to be compressed will be traversed until all the data to be written in the compression queue are traversed, and the data in the compressed data set will be used as the compressed data.

[0030] In this way, the data to be written in the compression queue is compressed by the data objects in the data object set, and the data object set is updated according to the data to be written in the compression queue, thereby completing the data compression.

[0031] In an optional embodiment, writing the compressed object into the compressed data set based on the data object set and the physical address of the compressed object includes:

[0032] Determine, based on the data object set, an object code and a physical address of a data object corresponding to the compressed object;

[0033] Determine an offset distance corresponding to the compressed object based on a physical address of the compressed object and a physical address of a data object corresponding to the compressed object;

[0034] The object code and offset distance of the data object corresponding to the compressed object are written into the compressed data set.

[0035] In this way, the object code and offset distance of the data object corresponding to the compressed object are written into the compressed data set, and the data is converted into the object code and offset distance, thereby reducing the data volume of the compressed data set and achieving data compression.

[0036] In an optional implementation, after writing the compressed data into the data block, the method further includes:

[0037] Acquire multiple data object sets, and compare the data objects contained in the multiple data object sets;

[0038] If there is a second data object that is consistent with the first data object, the second data object is compressed based on the object code corresponding to the first data object; the first data object is a data object that appears for the first time in the set of multiple data objects.

[0039] In this way, the data object set can be further compressed to achieve secondary data compression, reduce the space required for data compression, and improve the efficiency of data compression.

[0040] In a second aspect, the present invention provides a data compression device, comprising:

[0041] A parameter determination module, configured to determine parameters of a compression object and a compression window based on current read / write performance and current processor performance, and to configure the compression object and the compression window;

[0042] A data compression module is used to obtain the data to be written and perform a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written;

[0043] The data writing module is used to write the compressed data into the data block.

[0044] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data compression method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data compression method of the first aspect or any corresponding embodiment thereof.

[0046] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data compression method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 is a flow chart of a data compression method according to an embodiment of the present invention;

[0049] Figure 2 is a schematic structural diagram of a data block in a data compression method according to an embodiment of the present invention;

[0050] Figure 3 is a flow chart of another data compression method according to an embodiment of the present invention;

[0051] Figure 4 is a flow chart of another data compression method according to an embodiment of the present invention;

[0052] Figure 5 is a schematic diagram of data compression in a data compression method according to an embodiment of the present invention;

[0053] Figure 6 is a structural block diagram of a data compression device according to an embodiment of the present invention;

[0054] Figure 7 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0056] In order to save storage space and improve transmission efficiency, it is usually necessary to traverse the physical addresses in the data block after the data in the IO queue is written into the data block, and compress the data in the data block. When the data is compressed, the newly generated compressed data will be written into another continuous storage space. When the compression is complete, the original data space is recycled. This method requires an additional relatively long space to store intermediate data, and requires repeated reading and erasing, resulting in low data compression efficiency and a waste of processor and memory resources. At the same time, in the related art, when compressing data, it is necessary to traverse all the data in the storage space at one time, requiring a very large space to store the data dictionary for data compression, and each traversed object needs to be compared with the data dictionary, which makes parallel compression impossible and has low compression efficiency.

[0057] Based on this, an embodiment of the present invention provides a data compression method. This method determines the parameters of a compression object and a compression window based on current read / write performance and current processor performance, and configures the compression object and compression window. The method then obtains the data to be written and, based on the compression object and compression window, compresses the data to be written to obtain compressed data corresponding to the data to be written. The compressed data is then written into a data block. Thus, before the data is written into the data block, the compression object and compression window are used to compress the data to be written in segments, reducing the space used to store intermediate data and thus reducing the use of memory resources. Furthermore, the data is compressed before being written into the data block, and the compressed data is directly written into the data block, eliminating the need for repeated reading and erasing. This not only improves data compression efficiency but also reduces the use of processor and memory resources.

[0058] According to an embodiment of the present invention, an embodiment of a data compression method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] In this embodiment, a data compression method is provided, which can be used for data storage devices such as magnetic disks and hard disks. Figure 1 : is a flow chart of a data compression method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0060] Step S101 : determining parameters of a compression object and a compression window based on current read / write performance and current processor performance, and configuring the compression object and the compression window.

[0061] In the embodiments of the present invention, a compression object is the smallest unit of data to be compressed during the data compression process. Specifically, data of a corresponding size to the compression object is obtained and compressed each time. By adjusting the length of the compression object, the compression ratio can be adjusted. A compression window is the data covered by a single data compression process, i.e., the total data compressed during a single data compression process. By setting the number of compression windows, parallel data compression can be achieved.

[0062] In an embodiment of the present invention, current read / write performance is determined based on the data storage device's current data read / write rate, the wait time required to read / write data, and the performance of data writes to disk; current processor performance is determined based on the data storage device's processor utilization. Current read / write performance and current processor performance are used as parameters to measure the data storage device's operating performance. Based on these parameters, the parameters of the compression object and compression window are determined. Thus, the parameters of the compression object and compression window are dynamically adjusted based on the data storage device's operating performance, ensuring that data compression efficiency matches the data storage device's operating performance.

[0063] Step S102 : obtaining the data to be written, and performing a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written.

[0064] In an embodiment of the present invention, the data to be written is data in a data block that needs to be written to a data storage device. After obtaining the data to be written, a compression operation is performed on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written. In other words, in an embodiment of the present invention, the data is compressed before being written to the data block, and the data written to the data block is already compressed data. This avoids the need for repeated reading and erasing of the data after it is written to the data block, which would cause unnecessary occupation of processor and memory resources.

[0065] Step S103: writing the compressed data into the data block.

[0066] In the embodiment of the present invention, the compressed data is written into a data block. Specifically, the compressed data is written into a grain in the block. Figure 2 FIG. 1 is a schematic diagram of the structure of a data block in a data compression method according to an embodiment of the present invention. Figure 2As shown, the data area of ​​a data storage device can be divided into several blocks. Each block is of equal size, typically 128Mb. A block consists of a data block header and multiple grains, with data stored in the grains within the block. A bitmap is recorded in the data block header to track the status of each block. The block status can include used or available. When allocating space, the bitmap is scanned to find available blocks. The bitmap is a module that stores block configuration information. A bitmap block contains 262,144 bits, meaning that a block can contain up to 262,144 grains. The bitmap block records the block's sequence number, status, address range, number of grains, status of each grain, and grain address. When compressed data is written to a block, the compressed data is written to the grains within the block, and the block's configuration information is also written to the bitmap block.

[0067] In an optional embodiment, the data compression method provided by the embodiment of the present invention can be implemented by an intelligent online compression management module (Smart Online Compression, SOC). The intelligent online compression management module can be set on the board to dynamically manage the detection module, front-end module, main control module, back-end module and disk array module. Among them, the front-end module is used to implement IO queue management, which can obtain the read and write performance of each main control under different business data models in real time, that is, the current read and write performance; the detection module is used to detect the performance of the processor, that is, to obtain the current processor performance; the main control module is used to manage the compression queue, which is the queue used when compressing the data to be written in the above step S102; the back-end module is used to complete the data writing in the block data block and the background compression management; the disk array module is used to manage the storage resources such as the hard disk of the data storage device.

[0068] The data compression method provided by the embodiment of the present invention uses compression objects and compression windows to compress the data to be written in segments before the data is written into the data block, thereby reducing the size of the space used to store intermediate data and thus reducing the occupation of memory resources; at the same time, the data is compressed before being written into the data block, and the compressed data is directly written into the data block, eliminating the need for repeated reading and erasing, which not only improves the efficiency of data compression but also reduces the occupation of processor and memory resources.

[0069] In this embodiment, a data compression method is provided, which can be used for data storage devices such as magnetic disks and hard disks. Figure 3 is a flow chart of another data compression method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0070] Step S301 : determining parameters of a compression object and a compression window based on current read / write performance and current processor performance, and configuring the compression object and the compression window.

[0071] Specifically, in step S301, based on the current read / write performance and the current processor performance, determining the parameters of the compression object and the compression window may include the following steps:

[0072] Step S3011: Obtain preset parameters of the compression object and compression window.

[0073] In the embodiment of the present invention, the preset parameters include a preset field length of a compression object, a preset window length of a compression window, and a preset number of windows.

[0074] In an optional embodiment, the preset window number of the compression window can be set to 1, that is, the data to be written is compressed with 1 compression window by default; the preset window length of the compression window can be adjusted according to actual needs, for example, it can be set to 10000 bits. Accordingly, the parameters corresponding to the compression window also include a sliding step size, and the sliding step size is usually set to be consistent with the window length of the compression window, that is, after completing the compression of the data currently corresponding to the compression window, the compression window is slid to the end of the current position and starts to compress the new data.

[0075] In one optional embodiment, the preset field length is related to the business load and data model loaded by the data storage device. Specifically, the intelligent online compression management module pre-sets a length relationship table, which contains the preset field lengths of compressed objects that achieve theoretically optimal compression efficiency under various business loads and data models. Based on this length relationship table and the current data model of the data storage device, the preset field length of the compressed object is determined from the length relationship table.

[0076] In one optional embodiment, the preset window length and the preset number of windows are related to the business load and data model loaded by the data storage device. Specifically, the intelligent online compression management module pre-sets a window relationship table, which contains the preset window lengths and preset number of windows for compression objects that achieve theoretically optimal compression efficiency under various business loads and data models. Based on this window relationship table and the current data model of the data storage device, the preset window length and preset number of windows for the compression window are determined from the window relationship table.

[0077] Step S3012: Based on the current read and write performance, the preset field length and the preset window length are adjusted and determined to obtain the field length of the compression object and the window length of the compression window.

[0078] In an embodiment of the present invention, the current read and write performance includes the current disk write performance and the current queue waiting time. The current disk write performance refers to the efficiency and speed of the data storage device or the IO queue in the data storage device when performing the disk write operation at the current moment. The current queue waiting time refers to the time required for a data IO task in the current IO queue to wait from entering the current IO queue to being written to the block data block.

[0079] In an embodiment of the present invention, based on the current read and write performance, that is, based on the current disk performance and the current queue waiting time, the preset field length and the preset window length are adjusted to meet the current performance and working status of the data storage device.

[0080] In an optional implementation, the adjustment of the preset field length and the preset window length can be divided into the following two cases:

[0081] In the first case, if the current disk performance is less than the preset disk performance during the first time period, the preset field length is increased by the unit field length, and the preset window length is increased by the unit window length, to obtain the field length and window length. In this case, the current disk performance is less than the preset disk performance during the first time period, indicating that the read and write performance of the current data storage device exceeds the performance required for data compression, and the read and write performance of the data storage device is not being maximized. Therefore, the preset field length can be increased by the unit field length, and the preset window length can be increased by the unit window length, to obtain the current field length and window length.

[0082] In the second case, if the current queue waiting time is greater than the preset queue waiting time within the second time period, the preset field length is reduced by the unit field length, and the preset window length is reduced by the unit window length, to obtain the field length and window length. In this case, the current queue waiting time is greater than the preset queue waiting time within the second time period, indicating that the read and write performance of the current data storage device is lower than the performance required for data compression, resulting in a longer waiting time for data writing. Therefore, the preset field length can be reduced by the unit field length, and the preset window length can be reduced by the unit window length, to obtain the field length and window length in the current case.

[0083] In an optional implementation, the first time period and the second time period may be periods calculated based on different time nodes and different period lengths.

[0084] In an optional embodiment, the preset disk drop performance may be obtained by multiplying the theoretical disk drop performance by a first coefficient, and the preset queue waiting time may be obtained by multiplying the theoretical queue waiting time by a second coefficient. The first coefficient is the coefficient of the disk drop performance corresponding to the data storage device under the current service model; the second coefficient is the coefficient of the queue waiting time corresponding to the data storage device under the current service model.

[0085] In an optional implementation, different theoretical disk-push performance and theoretical queue wait times can be set for different data models, different storage media, and different processors. For example, data models may include 8k, 16k, 32k, 1M, etc., and storage media may include HDD, SSD, NVME, etc. Optionally, a data relationship table can be established to store the theoretical disk-push performance and theoretical queue wait times corresponding to different data models, different storage media, and different processors. During data compression, the current theoretical disk-push performance and theoretical queue wait time can be directly obtained from the data relationship table.

[0086] Therefore, through the above method, when the current disk placement performance is less than the preset disk placement performance, the field length of the compression object and the window length of the compression window are increased, thereby making more effective use of the disk placement performance and improving compression efficiency and space utilization; when the current queue waiting time is greater than the preset queue waiting time, the field length of the compression object and the window length of the compression window are reduced, thereby reducing the queue waiting time and improving compression efficiency.

[0087] Step S3013: Based on the current processor performance, the preset number of windows is adjusted to obtain the number of compressed windows.

[0088] In the embodiment of the present invention, the current processor performance is the utilization of the processor at the current moment. Based on the current processor performance, the preset number of windows is adjusted to match the current performance and working state of the data storage device.

[0089] In an optional implementation, the adjustment of the preset number of windows can be divided into the following two cases:

[0090] In the first case, if the current processor performance is less than the preset processor performance during the third time period, the preset number of windows is increased by the unit window number to obtain the window number. In this case, the current processor performance is less than the preset processor performance during the third time period, indicating that the processor performance is not being maximized. Therefore, the preset number of windows can be increased by the unit window number to obtain the current window number.

[0091] In the second case, if the current processor performance exceeds the preset processor performance during the fourth time period, the preset number of windows is reduced by the unit window number to obtain the window number. In this case, the current processor performance exceeds the preset processor performance during the fourth time period, indicating that the processor performance occupied by the current data compression has exceeded the processor's capacity. Therefore, the preset number of windows can be reduced by the unit window number to obtain the current window number.

[0092] In an optional implementation, the third time period and the fourth time period may be periods calculated based on different time nodes and different period lengths.

[0093] In an optional implementation, the preset processor performance may be obtained by multiplying the theoretical processor performance by a third coefficient, wherein the third coefficient is a coefficient of the processor performance corresponding to the data storage device under the current service model.

[0094] Therefore, through the above method, when the current processor performance is less than the preset processor performance, the number of compression windows is increased, thereby improving processor utilization and improving compression efficiency; when the current processor performance is greater than the preset processor performance, the number of compression windows is reduced to improve compression efficiency.

[0095] Step S302: Obtain the data to be written, and perform compression on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0096] Step S303: Write the compressed data into the data block. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0097] The data compression method provided by the embodiment of the present invention dynamically adjusts the parameters of the compression object and the compression window according to the current read and write performance and the current processor performance, thereby comprehensively considering the current read and write performance and the current processor performance to achieve a balance between compression efficiency and space utilization.

[0098] In this embodiment, a data compression method is provided, which can be used for data storage devices such as magnetic disks and hard disks. Figure 4 is a flow chart of another data compression method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0099] Step S401: Determine the parameters of the compression object and compression window based on the current read / write performance and the current processor performance, and configure the compression object and compression window. Figure 1Step S101 of the illustrated embodiment will not be described in detail here.

[0100] Step S402 : obtaining data to be written, and performing a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written.

[0101] Figure 5 This is a schematic diagram of data compression in a data compression method according to an embodiment of the present invention. Figure 5 , further explanation is given to step S402. Specifically, step S402 may include the following steps:

[0102] Step S4021: Write the data to be written into the compression queue.

[0103] In the embodiment of the present invention, the data to be written is written from the IO queue into the compression queue, and subsequent compression operations on the data to be written are completed in the compression queue.

[0104] Step S4022: Create a compression window at the initial position of the compression window corresponding to the compression queue.

[0105] In this embodiment of the present invention, a compression window is created at the initial position of the compression window corresponding to the compression queue. If the number of compression windows is 1, the initial position of the compression window is the starting position of the compression queue. If the number of compression windows is not 1, the initial positions of different compression windows are located at different positions in the compression queue.

[0106] In an optional embodiment, if the number of compression windows is n, the initial positions of the compression windows are respectively the starting position of the compression queue, the 1 / n position of the compression queue, the 2 / n position of the compression queue, ..., and the (n-1) / n position of the compression queue, thereby dividing the compression queue evenly and performing parallel compression at each compression window, thereby improving data compression efficiency.

[0107] Step S4023: The data to be written in the compression window corresponding to the compression queue is used as the data to be compressed, and the data to be compressed is traversed.

[0108] In an embodiment of the present invention, the data to be written in the compression queue corresponding to the compression window is used as the data to be compressed. Specifically, the data to be compressed corresponding to the compression window is traversed within the compression window to compress the data to be compressed. If the traversal of the data to be compressed is completed, it indicates that the compression of the data to be compressed has been completed.

[0109] Step S4024: If the traversal of the data to be compressed is not completed, a compressed object is obtained from the data to be compressed based on the field length of the compressed object, and the compressed object is compared with the data objects in the data object set.

[0110] In this embodiment of the present invention, if traversal of the data to be compressed is not complete, it indicates that compression of the data to be compressed has not yet been completed. In this case, based on the field length of the compression object, a compression object is obtained from the data to be compressed. That is, the data corresponding to the field length of the compression object is compressed each time. The compressed object is compared with the data objects in the data object set to confirm whether the data object corresponding to the compressed object exists in the data object set.

[0111] In embodiments of the present invention, a data object set, or data dictionary, is used to compress data to be compressed. The data object set contains multiple data objects, each of which includes data information, an object code, and a physical address. The data information represents the original data content corresponding to the data object; the object code, serving as a code for the data object, uniquely identifies the data object within the data object set; and the physical address represents the compressed object corresponding to the data object, i.e., the physical address where the data information first appears.

[0112] Step S4025: If the data object set does not include the data object corresponding to the compressed object, a data object is constructed based on the compressed object and the object code and physical address of the compressed object, and written into the data object set.

[0113] In an embodiment of the present invention, if the data object set does not contain the data object corresponding to the compressed object, that is, the data information of each data object in the data object set is different from that of the compressed object, then it indicates that the data dictionary corresponding to the compressed object is not recorded in the data object set. Therefore, the data dictionary corresponding to the compressed object needs to be updated to the data object set so that when a compressed object with the same data content as the compressed object is encountered again in the future, it can be compressed. Specifically, the compressed object, that is, the data content of the compressed object, is used as the data information of the corresponding data object; the object code of the compressed object is used as the object code of the corresponding data object, and its object code can also be determined according to the order of the data object in the data object set; the physical address of the compressed object is used as the physical address of the corresponding data object, thereby constructing the data object corresponding to the compressed object and writing the data object into the data object set.

[0114] Step S4026: Based on the data object set and the physical address of the compressed object, write the compressed object into the compressed data set.

[0115] In the embodiment of the present invention, after the data object corresponding to the compressed object is written into the data object set, or the data object set contains the data object corresponding to the compressed multiple write, the compressed object is written into the compressed data set based on the data object set and the physical address of the compressed object; wherein the compressed data set is the data object corresponding to the compressed multiple write. Figure 5The compressed data in is a data set obtained by compressing the data to be compressed.

[0116] In an optional embodiment, writing a compressed object into a compressed data set based on a data object set and a physical address of the compressed object may include: determining an offset distance corresponding to the compressed object based on the physical address of the compressed object and the physical address of the data object corresponding to the compressed object; and writing the object code and offset distance of the data object corresponding to the compressed object into the compressed data set. In this way, the object code and offset distance of the data object corresponding to the compressed object are written into the compressed data set, and the data is converted into the object code and offset distance, thereby reducing the amount of data in the compressed data set and achieving data compression. Figure 5 For example, Figure 5 The data in the compression queue is abcdabcd, so after compression processing, the data object set obtained is:<a,1,1> ;<b,2,2> ;<c,3,3> ;<d,4,4> , the compressed data set is: <1, 0>; <2, 0>; <3, 0>; <4, 0>; <1, 4>; <2, 4>; <3, 4>; <4, 4>.

[0117] Step S4027, if the traversal of the data to be compressed has been completed, the compression window is slid by the data length corresponding to the window length of the compression window to update the compression window, and the data to be written corresponding to the updated compression window in the compression queue is used as the updated data to be compressed, and the updated data to be compressed is traversed until all the data to be written in the compression queue are traversed, and the data in the compressed data set is used as the compressed data.

[0118] In an embodiment of the present invention, if the traversal of the data to be compressed has been completed, the compression window is slid by the data length corresponding to the window length of the compression window, and the position of the compression window is updated, thereby updating the data to be compressed. The above steps S4023 to S4026 are repeated to traverse and compress all the data to be written in the compression queue. At this time, the data in the compressed data set is the compressed data corresponding to the data to be written.

[0119] Step S403: Write the compressed data into the data block. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0120] In an optional embodiment, to improve data compression efficiency and reduce the space required after data compression, after writing the compressed data into the data block, a second compression step is also included for the data object set. Specifically, the steps include: obtaining multiple data object sets and comparing the data objects contained in the multiple data object sets; if a second data object is consistent with the first data object, compressing the second data object based on the object code corresponding to the first data object; the first data object is the data object that first appears in the multiple data object sets. In other words, the data information of the data objects is compared, and if there are data objects with identical data information, the data information of the second data object recorded later is modified to the object code of the first data object recorded earlier or the first appearing in the data object set, thereby further compressing the data objects.

[0121] In an optional embodiment, each compression window corresponds to a data object set, and the object codes of the data objects in the data object set corresponding to each compression window are different. Alternatively, the object code of a data object can consist of two bits, with the first bit representing the code of the data object set to which the data object belongs, and the second bit representing the code of the data object within the data object set to which the data object belongs. Therefore, the data object set to which the data object belongs can be determined based on the object code of the data object. This not only avoids compression errors caused by the same object codes in different data object sets, but also allows for rapid location of data objects, improving the efficiency of secondary compression.

[0122] In this embodiment, a data compression device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0123] This embodiment provides a data compression device, such as Figure 6 Shown, including:

[0124] Parameter determination module 601, for determining parameters of a compression object and a compression window based on current read / write performance and current processor performance, and configuring the compression object and the compression window;

[0125] The data compression module 602 is used to obtain the data to be written and perform a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written;

[0126] The data writing module 603 is used to write the compressed data into the data block.

[0127] In an optional implementation, the parameter determination module 601 includes:

[0128] A preset parameter acquisition unit, configured to acquire preset parameters of a compression object and a compression window; the preset parameters include a preset field length of the compression object, a preset window length of the compression window, and a preset number of windows;

[0129] A length adjustment unit, configured to adjust and determine a preset field length and a preset window length based on current read and write performance, to obtain a field length of a compression object and a window length of a compression window;

[0130] The quantity adjustment unit is used to adjust the preset number of windows based on the current processor performance to obtain the number of compressed windows.

[0131] In an optional implementation, the current read / write performance includes the current disk flushing performance and the current queue waiting time; and the length adjustment unit includes:

[0132] A length increasing subunit is configured to increase the preset field length by a unit field length and the preset window length by a unit window length if the current disk placement performance is less than the preset disk placement performance within the first time period, thereby obtaining the field length and the window length;

[0133] The length reduction subunit is used to reduce the preset field length by the unit field length and the preset window length by the unit window length if the current queue waiting time is greater than the preset queue waiting time within the second time period, so as to obtain the field length and the window length.

[0134] In an optional embodiment, the quantity adjustment unit includes:

[0135] a window number increasing subunit, configured to increase the preset window number by the unit window number to obtain the window number if the current processor performance is less than the preset processor performance within the third time period;

[0136] The window number reducing subunit is configured to reduce the preset window number by the unit window number to obtain the window number if the current processor performance is greater than the preset processor performance in the fourth time period.

[0137] In an optional implementation, the data compression module 602 includes:

[0138] A data writing unit, used for writing the data to be written into the compression queue;

[0139] A compression window creating unit, configured to create a compression window at an initial position of the compression window corresponding to the compression queue;

[0140] a data traversal unit, configured to take the to-be-written data corresponding to the compression window in the compression queue as the to-be-compressed data, and traverse the to-be-compressed data;

[0141] a data comparison unit, configured to obtain a compressed object from the data to be compressed based on the field length of the compressed object if the traversal of the data to be compressed is not completed, and compare the compressed object with the data objects in the data object set;

[0142] a data object updating unit, configured to construct the data object based on the compressed object and the object code and physical address of the compressed object and write the data object into the data object set if the data object set does not contain the data object corresponding to the compressed object;

[0143] a compressed data set updating unit, configured to write the compressed object into the compressed data set based on the data object set and the physical address of the compressed object;

[0144] The compressed data determination unit is used to slide the compression window by the data length corresponding to the window length of the compression window to update the compression window if the traversal of the data to be compressed has been completed, and use the data to be written corresponding to the updated compression window in the compression queue as the updated data to be compressed, traverse the updated data to be compressed, and use the data in the compressed data set as the compressed data.

[0145] In an optional embodiment, the compressed data set updating unit includes:

[0146] A data object determination subunit, configured to determine, based on the data object set, an object code and a physical address of a data object corresponding to the compressed object;

[0147] an offset distance determining subunit, configured to determine an offset distance corresponding to the compressed object based on a physical address of the compressed object and a physical address of a data object corresponding to the compressed object;

[0148] The compressed data set updating subunit is used to write the object code and offset distance of the data object corresponding to the compressed object into the compressed data set.

[0149] In an optional embodiment, the device further includes:

[0150] A data object comparison module is used to obtain multiple data object sets and compare the data objects contained in the multiple data object sets;

[0151] The data object compression module is used to compress the second data object based on the object code corresponding to the first data object if there is a second data object consistent with the first data object; the first data object is the data object that appears for the first time in the multiple data object sets.

[0152] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0153] The data compression device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0154] The embodiment of the present invention also provides a computer device having the above Figure 6 The data compression device shown.

[0155] See also Figure 7 , Figure 7 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0156] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0157] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0158] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0160] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0161] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0162] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0163] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data compression method, characterized in that: The method comprises: Based on current read / write performance and current processor performance, parameters of a compression object and a compression window are determined, and the compression object and the compression window are configured; the determining of the parameters of the compression object and the compression window based on the current read / write performance and current processor performance includes: obtaining preset parameters of the compression object and the compression window; the preset parameters include a preset field length of the compression object, a preset window length of the compression window, and a preset number of windows; based on the current read / write performance, adjusting and determining the preset field length and the preset window length to obtain the field length of the compression object and the window length of the compression window; based on the current processor performance, adjusting the preset number of windows to obtain the number of windows of the compression window; Obtaining data to be written, and performing a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written; performing a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written, including: writing the data to be written into a compression queue; creating the compression window at an initial position of the compression queue corresponding to the compression window; using the data to be written corresponding to the compression window in the compression queue as data to be compressed, and traversing the data to be compressed; if the traversal of the data to be compressed is not completed, obtaining the compression object from the data to be compressed based on the field length of the compression object, and comparing the compression object with the data objects in the data object set. if the data object set does not contain the data object corresponding to the compressed object, constructing a data object based on the compressed object and the object code and physical address of the compressed object, and writing it into the data object set; writing the compressed object into the compressed data set based on the data object set and the physical address of the compressed object; if the traversal of the data to be compressed has been completed, sliding the compression window by the data length corresponding to the window length of the compression window to update the compression window, and using the data to be written in the compression queue corresponding to the updated compression window as the updated data to be compressed, traversing the updated data to be compressed until all the data to be written in the compression queue are traversed, and using the data in the compressed data set as the compressed data; The compressed data is written into a data block.

2. The method according to claim 1, characterized in that The current read / write performance includes a current disk performance and a current queue waiting time; and adjusting and determining the preset field length and the preset window length based on the current read / write performance to obtain the field length of the compression object and the window length of the compression window includes: If the current disk placement performance is less than the preset disk placement performance within the first time period, the preset field length is increased by a unit field length, and the preset window length is increased by a unit window length to obtain the field length and the window length; If the current queue waiting time is greater than the preset queue waiting time within the second time period, the preset field length is reduced by the unit field length, and the preset window length is reduced by the unit window length to obtain the field length and the window length.

3. The method according to claim 1, characterized in that The adjusting the preset number of windows based on the current processor performance to obtain the number of windows of the compression window includes: If the current processor performance is less than the preset processor performance within the third time period, increasing the preset number of windows by the number of unit windows to obtain the number of windows; If the current processor performance is greater than the preset processor performance during the fourth time period, the preset window quantity is reduced by the unit window quantity to obtain the window quantity.

4. The method according to claim 1, wherein The step of writing the compressed object into the compressed data set based on the data object set and the physical address of the compressed object comprises: Determining, based on the data object set, an object code and a physical address of a data object corresponding to the compressed object; Determining an offset distance corresponding to the compressed object based on a physical address of the compressed object and a physical address of a data object corresponding to the compressed object; The object code of the data object corresponding to the compressed object and the offset distance are written into the compressed data set.

5. The method according to claim 1, wherein After writing the compressed data into the data block, the method further includes: Acquire multiple data object sets, and compare the data objects contained in the multiple data object sets; If there is a second data object that is consistent with the first data object, the second data object is compressed based on the object code corresponding to the first data object; the first data object is the data object that appears for the first time in the plurality of data object sets.

6. A data compression device, characterized in that: The device comprises: A parameter determination module is configured to determine parameters of a compression object and a compression window based on current read / write performance and current processor performance, and configure the compression object and the compression window; determining the parameters of the compression object and the compression window based on the current read / write performance and current processor performance comprises: obtaining preset parameters of the compression object and the compression window; the preset parameters include a preset field length of the compression object, a preset window length of the compression window, and a preset number of windows; adjusting and determining the preset field length and the preset window length based on the current read / write performance to obtain the field length of the compression object and the window length of the compression window; and adjusting the preset number of windows based on the current processor performance to obtain the number of windows of the compression window; A data compression module, for obtaining data to be written, and performing a compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written; the compression operation on the data to be written based on the compression object and the compression window to obtain compressed data corresponding to the data to be written includes: writing the data to be written into a compression queue; creating the compression window at an initial position of the compression queue corresponding to the compression window; using the data to be written corresponding to the compression window in the compression queue as data to be compressed, and traversing the data to be compressed; if the traversal of the data to be compressed is not completed, obtaining the compression object from the data to be compressed based on the field length of the compression object, and comparing the compression object with the data in the data object set. the compressed data set; if the data object set does not contain the data object corresponding to the compressed object, constructing a data object based on the compressed object and the object code and physical address of the compressed object, and writing it into the data object set; writing the compressed object into the compressed data set based on the data object set and the physical address of the compressed object; if the traversal of the data to be compressed has been completed, sliding the compression window by the data length corresponding to the window length of the compression window to update the compression window, and using the to-be-written data corresponding to the updated compression window in the compression queue as the updated data to be compressed, traversing the updated data to be compressed until all the to-be-written data in the compression queue have been traversed, and using the data in the compressed data set as the compressed data; The data writing module is used to write the compressed data into the data block.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data compression method according to any one of claims 1 to 5 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data compression method according to any one of claims 1 to 5.

Images