Data compression methods, apparatus, equipment and storage media

Abstract

This invention provides a data compression method, apparatus, device, and storage medium. The data compression method includes: compressing original data to obtain a first compression result; based on the first compression result, obtaining first skip frame data, which is used to locate the original data; and compressing the first skip frame data to obtain compressed first skip frame data; concatenating the compressed first skip frame data after the first compression result; based on the first compression result and the compressed first skip frame data, obtaining second skip frame data, which is used to locate the first skip frame data and the original data; compressing the second skip frame data to obtain compressed second skip frame data, concatenating the compressed second skip frame data before the first compression result, and outputting a second compression result. The compressed file obtained by the data compression method of this invention allows random access to a segment of data within the compressed file without needing to decompress the entire compressed file.

Description

Technical Field

[0001] This invention belongs to the field of Field Programmable Gate Array (FPGA) integrated circuit technology, and particularly relates to data compression methods, devices, equipment and storage media. Background Technology

[0002] Compression algorithms typically consider two factors: compression ratio and compression efficiency. Some compression algorithms have very high compression ratios but poor compression efficiency; others have good compression efficiency but low compression ratios.

[0003] ZStandard (zstd) is a fast, lossless compression algorithm open-sourced by Facebook, which strikes a good balance between compression ratio and compression efficiency. However, compressed files obtained using the zstd algorithm do not support random access. When accessing a specific section of data in a compressed file, it can only be read sequentially from the beginning of the compressed file, which reduces the efficiency of reading compressed files in certain application scenarios and limits the application scope of the zstd algorithm. Summary of the Invention

[0004] Based on this, the present invention provides a data compression method, apparatus, device and storage medium, which solves the problem that compressed files obtained by the existing zstd algorithm do not support random access to specific data in the compressed file, thus reducing the efficiency of reading compressed files in certain application scenarios and reducing the application scope of the zstd algorithm.

[0005] This invention provides a data compression method, the method comprising:

[0006] Compress the original data to obtain the first compression result;

[0007] Based on the first compression result, first skip frame data is obtained, which is used to locate the original data; and the first skip frame data is compressed to obtain compressed first skip frame data; the compressed first skip frame data is appended to the first compression result.

[0008] Based on the first compression result and the compressed first skip frame data, second skip frame data is obtained, and the second skip frame data is used to locate the first skip frame data and the original data.

[0009] The second skip frame data is compressed to obtain compressed second skip frame data. The compressed second skip frame data is concatenated before the first compression result, and the second compression result is output.

[0010] Further, the compression of the original data to obtain the first compression result includes:

[0011] The core data in the original data is split into frame data, and the frame data is compressed to obtain compressed frame data. The core data is the data that needs to be accessed.

[0012] Record the first information corresponding to the compressed frame data, obtain index data, compress the index data to obtain compressed index data, and concatenate the compressed index data after the compressed frame data;

[0013] Record the index data, compress it into the second information corresponding to the compressed index data, obtain the data header data, compress the data header data to obtain the compressed data header data, and concatenate the compressed data header data before the compressed frame data to obtain the first compression result.

[0014] Further, obtaining the first skipped frame data based on the first compression result includes:

[0015] Record the frame data compressed into third information corresponding to the compressed frame data; record the index data compressed into fourth information corresponding to the compressed index data; and generate first metadata based on the third information and the fourth information.

[0016] The first metadata is stored in the first skip frame structure corresponding to the first skip frame data. The first identifier of the skip frame storing the first metadata is identified and stored in the first skip frame structure where the first metadata is located to obtain the first skip frame data.

[0017] Further, the recording of the frame data compressed into third information corresponding to the compressed frame data; the recording of the index data compressed into fourth information corresponding to the compressed index data; and the generation of first metadata based on the third information and the fourth information, including:

[0018] Record the first position and first data size of the compressed frame data, and generate the first positioning information of the frame data based on the first position and the first data size;

[0019] Record the second position of the index data corresponding to the frame data in the compressed index data, and generate the second positioning information of the index data based on the second position;

[0020] Based on the first location information and the second location information, the first metadata is obtained.

[0021] Further, obtaining the second skip frame data based on the first compression result and the compressed first skip frame data includes:

[0022] Record the fifth information corresponding to the compressed first skip frame data, record the sixth information corresponding to the compressed data header data, and generate second metadata based on the fifth information and the sixth information;

[0023] The second metadata is stored in the second skip frame structure corresponding to the second skip frame data. The second identifier of the skip frame storing the second metadata is identified and stored in the second skip frame structure where the second metadata is located to obtain the second skip frame data.

[0024] Further, the recording of the first skipped frame data compressed into the fifth information corresponding to the compressed first skipped frame data, the recording of the sixth information corresponding to the compressed data header data, and the generation of second metadata based on the fifth information and the sixth information, including:

[0025] Record the first skipped frame data corresponding to the frame data, compress it into the third position and third data size in the compressed first skipped frame data, and generate the third positioning information of the first skipped frame data based on the third position and the third data size;

[0026] Record the data header data corresponding to the frame data, compress it into the fourth position and the fourth data size in the compressed data header data, and generate the fourth positioning information of the data header data based on the fourth position and the fourth data size;

[0027] The second metadata is obtained based on the third and fourth location information.

[0028] Furthermore, the first skip frame data and the second skip frame data are skip frame data provided in the Zstd standard.

[0029] The present invention also provides a data compression apparatus, the apparatus comprising:

[0030] The first acquisition module is used to compress the original data and obtain the first compression result;

[0031] The second acquisition module is used to acquire first skip frame data based on the first compression result, wherein the first skip frame data is used to locate the original data; and to compress the first skip frame data to obtain compressed first skip frame data; wherein the compressed first skip frame data is appended to the first compression result.

[0032] The third acquisition module is used to acquire second skip frame data based on the first compression result and the compressed first skip frame data, wherein the second skip frame data is used to locate the first skip frame data and the original data;

[0033] The compression output module is used to compress the second skip frame data to obtain compressed second skip frame data. The compressed second skip frame data is concatenated before the first compression result and the second compression result is output.

[0034] The present invention also provides a computer device, the computer device comprising:

[0035] Memory: Used to store computer programs;

[0036] Processor: Used to execute the computer program to implement the data compression method described in any of the preceding claims.

[0037] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data compression method described in any of the preceding claims.

[0038] The data compression method provided by this invention involves compressing original data to obtain a first compression result; based on the first compression result, obtaining first skip frame data and compressing it to obtain compressed first skip frame data; the compressed first skip frame data is appended to the first compression result; based on the first compression result and the compressed first skip frame data, obtaining second skip frame data, compressing it to obtain compressed second skip frame data, which is appended to the first compression result, thus obtaining the final compressed data result. The compressed file obtained using the data compression method provided in this application allows random access to a segment of data within the compressed file without needing to decompress the entire file. This improves the efficiency of reading compressed files in certain application scenarios and broadens the application scope of the zstd algorithm. The data compression device, equipment, and storage medium provided by this invention can also achieve the above effects. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0041] Figure 2 for Figure 1 A schematic diagram of a possible hardware structure for a device in the application scenario shown;

[0042] Figure 3A schematic diagram showing the composition of the original data that needs to be compressed;

[0043] Figure 4 A schematic flowchart of a data compression method provided in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the data format and index relationship compressed using the data compression method provided in this embodiment of the invention;

[0045] Figure 6 A schematic diagram illustrating the implementation process of the data compression method provided in this embodiment of the invention;

[0046] Figure 7 A flowchart illustrating a data decompression method provided by the present invention;

[0047] Figure 8 This is a block diagram of a data compression device provided in an embodiment of the present invention. Detailed Implementation

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0049] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0051] In the following description, suffixes such as “module,” “part,” or “unit” used to denote elements are used only for the purposes of this application and have no specific meaning in themselves.

[0052] Figure 1 This diagram illustrates an application scenario provided by an embodiment of this application. The scenario includes a source device 100 and a target device 200, which transmit data via a network 300. The source device 100 and the target device 200 can be two independent devices. In other scenarios, the source device 100 and the target device 200 can be in the same device, transmitting data via lines within the device. To save transmission bandwidth, the data to be transmitted can be compressed before being transmitted to the target device 200. Upon receiving the compressed data, the target device 200 decompresses it to obtain the original data.

[0053] In this embodiment, the source end 100 uses the Zstandard (zstd) compression algorithm to compress data before transmission. Specifically, during compression, a second skip frame is added before the original data, and a first skip frame is added after the original data. This Zstandard (zstd) compression algorithm is stored simultaneously in both the source end 100 and the target end 200, ensuring that when the target end 200 accesses the compressed file, it can randomly access a segment of data within the compressed file without needing to decompress the entire file.

[0054] The devices involved in this application may include computer equipment, handheld devices, vehicle-mounted equipment, wearable devices or other devices connected to a wireless modem, as well as cloud equipment, terminals, terminal equipment, monitoring equipment, servers, etc.

[0055] Figure 2 for Figure 1 The diagram illustrates a possible hardware structure of the device in the application scenario shown. Figure 2 As shown, the device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will understand that... Figure 2The structure shown does not constitute a limitation on the device and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Specifically: Processor 10 is the control center of the device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in memory 20, and by calling data stored in memory 20, thereby providing overall control of the device. Processor 10 can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor.

[0056] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. The program storage area may store the operating system 21, the first acquisition module 22, the second acquisition module 23, the third acquisition module 24, the compression output module 25, and one or more application programs 26 required for each function (such as compression applications, etc.). The data storage area may store data created according to the usage of the UE or the target server. The memory 20 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). Accordingly, memory 20 may also include a memory controller to provide processor 10 with access to memory 20.

[0057] The memory 20 performs the following functions by running the first acquisition module 22: compressing the original data to obtain a first compression result; the memory 20 performs the following functions by running the second acquisition module 23: based on the first compression result, acquiring first skip frame data, which is used to locate the original data; and compressing the first skip frame data to obtain compressed first skip frame data; the compressed first skip frame data is appended to the first compression result; the memory 20 performs the following functions by running the third acquisition module 24: based on the first compression result and the compressed first skip frame data, acquiring second skip frame data, which is used to locate the first skip frame data and the original data; the memory 20 performs the following functions by running the compression output module 25: compressing the second skip frame data to obtain compressed second skip frame data, which is appended to the first compression result, and outputting the second compression result.

[0058] Please see Figure 3 This is a schematic diagram illustrating the composition of the original data that needs to be compressed, such as... Figure 3 As shown, `Data` is the core data that needs to be accessed, `Header` is the data header used to store the index information of `Segmententry`, and `Segmententry` is an array used to index specific data within `Data`. Each element in the `Segmententry` array corresponds to a segment of data within `Data`. Before compression, when accessing a segment of data within `Data` is needed, the index information of `Segmententry` in the `Header` is read, the `Segmententry` array is read, and then the desired segment of data within `Data` is found based on the `Data` index information stored in the target element of the read `Segmententry` array, achieving fast random access to `Data` data. After compressing the original data using the `zstd` algorithm, only after decompressing the entire compressed file can random access to `Data` data be achieved in the same way as before compression. However, in some application scenarios where only specific data needs to be accessed, decompressing the entire compressed file before achieving random access to `Data` data is obviously inefficient.

[0059] To achieve fast random access to specific data, such as Figure 4 The diagram shown is a flowchart of a data compression method provided in an embodiment of the present invention. The method includes:

[0060] Step S1: Compress the original data to obtain the first compression result;

[0061] Step S2: Based on the first compression result, obtain the first skip frame data, which is used to locate the original data; and compress the first skip frame data to obtain compressed first skip frame data; the compressed first skip frame data is appended to the first compression result.

[0062] Step S3: Based on the first compression result and the compressed first skip frame data, obtain the second skip frame data, which is used to locate the first skip frame data and the original data;

[0063] Step S4: Compress the second skip frame data to obtain compressed second skip frame data. The compressed second skip frame data is concatenated before the first compression result, and the second compression result is output.

[0064] Specifically, Zstd compressed data consists of one or more frames, each of which is independent and can be decompressed independently of other frames. The Zstd standard provides two types of frames: standard frames and skipped frames. Standard frames contain compressed data, while skipped frames contain user-defined metadata. The Zstd standard decoder can correctly decode standard frames but ignores skipped frames, thus ensuring data consistency before and after decoding. Based on this characteristic, in this embodiment, the Zstandard compression algorithm first compresses the original data to be compressed. Then, based on the result of compressing the original data, the first skipped frame data is obtained. Next, the first skipped frame data is compressed, and the compressed first skipped frame data is appended to the result of compressing the original data. Then, based on the result of compressing the original data and the first skipped frame data, the second skipped frame data is obtained. The second skipped frame data is then compressed, and the compressed second skipped frame data is appended to the result of compressing the original data, resulting in the final second compression result. When accessing specific data in a Zstd compressed file, the specific data to be accessed can be directly located using the first and second skipped frame data. Only the compressed portion of that specific data needs to be decompressed, enabling fast access to specific data in the Zstd compressed file.

[0065] like Figure 5 The diagram shows the data format and index relationship compressed using the data compression method provided in this embodiment of the invention. Skippable_Aux is the newly added second skipped frame data, Skippable_Record[] is the newly added second skipped frame data, and Skippable_Record[] is an array of data used to index Data. Each frame in the Data data corresponds to one element. For example... Figure 6 The diagram shown illustrates the implementation process of the data compression method provided in this embodiment of the invention.

[0066] In some embodiments, step S1 includes:

[0067] Step S11: Split the core data in the original data into frame data, compress the frame data to obtain compressed frame data, and the core data is the data that needs to be accessed;

[0068] Step S12: Record the first information corresponding to the compressed frame data, obtain index data, compress the index data to obtain compressed index data, and concatenate the compressed index data after the compressed frame data;

[0069] Step S13: Record the second information corresponding to the compressed index data, obtain the data header data, compress the data header data to obtain the compressed data header data, and concatenate the compressed data header data before the compressed frame data to obtain the first compression result.

[0070] Specifically, the original data to be compressed is first split into frames, and the frame data is compressed using the Zstandard compression algorithm. The characteristics of each frame data are recorded during the compression process to obtain the index data of the frame data. Then, the index data is compressed, and the compression result is concatenated after the compressed frame data. The position of the compressed index data is obtained to get the data header data corresponding to the frame data. The data header data is compressed, and the compression result is concatenated before the compressed frame data to obtain the first compression result.

[0071] In some embodiments, step S2, based on the first compression result, obtains the first skipped frame data, including:

[0072] Record the frame data compressed into third information corresponding to the compressed frame data; record the index data compressed into fourth information corresponding to the compressed index data; and generate first metadata based on the third information and the fourth information.

[0073] The first metadata is stored in the first skip frame structure corresponding to the first skip frame data. The first identifier of the skip frame storing the first metadata is identified and stored in the first skip frame structure where the first metadata is located to obtain the first skip frame data.

[0074] Further, the recording of the frame data compressed into third information corresponding to the compressed frame data; the recording of the index data compressed into fourth information corresponding to the compressed index data; and the generation of first metadata based on the third information and the fourth information, including:

[0075] Record the first position and first data size of the compressed frame data, and generate the first positioning information of the frame data based on the first position and the first data size;

[0076] Record the second position of the index data corresponding to the frame data in the compressed index data, and generate the second positioning information of the index data based on the second position;

[0077] Based on the first location information and the second location information, the first metadata is obtained.

[0078] Specifically, when compressing frame data, the first position and first data size of the frame data in the compressed frame data are recorded when the frame data is compressed into the compressed frame data, and the second position of the index data corresponding to the frame data in the compressed index data is recorded when the index data is compressed into the compressed index data. Then, based on the first position and first data size of the frame data in the compressed frame data, and the second position of the index data corresponding to the frame data in the compressed index data, first metadata for locating the frame data is generated. The first metadata is then stored in the first skipped frame structure corresponding to the first skipped frame data. The first identifier of the first skipped frame storing the first metadata is identified, and the first identifier and the first metadata are combined to obtain the first skipped frame data. The first skipped frame structure corresponding to the first skipped frame data can be pre-established. The following is a design illustration of a first skipped frame structure:

[0079] Magic_Number: Skip frame identifier.

[0080] Frame_Size:4 Skip frames and customize data size.

[0081] User_Data: n Custom data.

[0082] Custom data includes:

[0083] Data_n_pos: The data position corresponding to this skipped frame.

[0084] Data_n_size: The size of the data corresponding to this skipped frame.

[0085] Segmententry_pos: The Segmententry position corresponding to this skipped frame.

[0086] In some embodiments, step S3, which involves obtaining second skip frame data based on the first compression result and the compressed first skip frame data, includes:

[0087] Record the fifth information corresponding to the compressed first skip frame data, record the sixth information corresponding to the compressed data header data, and generate second metadata based on the fifth information and the sixth information;

[0088] The second metadata is stored in the second skip frame structure corresponding to the second skip frame data. The second identifier of the skip frame storing the second metadata is identified and stored in the second skip frame structure where the second metadata is located to obtain the second skip frame data.

[0089] Further, the recording of the first skipped frame data compressed into the fifth information corresponding to the compressed first skipped frame data, the recording of the sixth information corresponding to the compressed data header data, and the generation of second metadata based on the fifth information and the sixth information, including:

[0090] Record the first skipped frame data corresponding to the frame data, compress it into the third position and third data size in the compressed first skipped frame data, and generate the third positioning information of the first skipped frame data based on the third position and the third data size;

[0091] Record the data header data corresponding to the frame data, compress it into the fourth position and the fourth data size in the compressed data header data, and generate the fourth positioning information of the data header data based on the fourth position and the fourth data size;

[0092] The second metadata is obtained based on the third and fourth location information.

[0093] Specifically, when compressing the first skip frame data, the third position and third size of the first skip frame data when compressed into compressed first skip frame data are recorded. When compressing the header data, the fourth data size of the fourth position when compressed into compressed header data is recorded. Second metadata is generated for locating the first skip frame data and locating the header data. Then, the second metadata is stored in the second skip frame structure corresponding to the second skip frame data. The second identifier of the second skip frame storing the second metadata is identified. The second identifier and the second metadata are combined to obtain the second skip frame data. The second skip frame structure corresponding to the second skip frame data can be pre-established. The following is a design illustration of a second skip frame structure:

[0094] Magic_Number: Skip frame identifier.

[0095] Frame_Size: Skip frame and customize data size.

[0096] User_Data: Custom data.

[0097] Custom data includes:

[0098] Header_pos: The position of the header data in the file.

[0099] Header_size: Size of the header data

[0100] Skippalbe_Record_pos: The position of Skippalbe_Record data in the file.

[0101] Skippalbe_Record_size: Size of Skippalbe_Record data

[0102] Preferably, the first skip frame data and the second skip frame data are skip frame data provided in the Zstd standard.

[0103] like Figure 7 The diagram shown is a flowchart illustrating a data decompression method provided by the present invention. Figure 7 As shown, this method can be derived from... Figure 1 The method can be executed on the target end in the application scenario shown. Figure 4 The provided data compression method is then executed, and this method includes the following steps:

[0104] Step S51: Obtain the second compression result, that is, the compressed original data.

[0105] After obtaining the second compression result, the source end transmits the second compression result to the target end.

[0106] Step S52: Receive a request to access specific data, determine the compressed second skip frame data corresponding to the specific data; decompress the compressed second skip frame data corresponding to the specific data, and locate the compressed data header data and compressed first skip frame data corresponding to the specific data;

[0107] Step S53: Decompress the compressed data header data and the compressed first skip frame data corresponding to the specific data, locate the index data corresponding to the specific data, decompress the index data, and locate the frame data corresponding to the specific data based on the decompressed index data;

[0108] Step S54: Decompress the frame data corresponding to the specific data.

[0109] Specifically, taking accessing Data_1 data as an example, the process involves finding the corresponding Skippable_Aux based on the Data_1 data. Then, using Header_pos and Skippable_Record_pos in Skippable_Aux, the positions of the Header and Skippable_Record are located, and the Skippable_Record data is decompressed. The element with the corresponding characteristic is found in the Segmententry array. Based on the found Segmententry element, the corresponding Skippable_record element is found in the Skippable_Record array. Based on the position and size of the Data_1 frame recorded in the file by the found Skippable_record, the frame data is decompressed, thus obtaining the decompressed data_1. This allows for the rapid retrieval of data_1 without decompressing data_2 to data_n.

[0110] It should be understood that the number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0111] like Figure 8 The diagram shown is a block diagram of a data compression device 800 provided in an embodiment of the present invention. The device 800 includes:

[0112] The first acquisition module 801 is used to compress the original data and obtain the first compression result;

[0113] The second acquisition module 802 is used to acquire first skip frame data based on the first compression result, wherein the first skip frame data is used to locate the original data; and to compress the first skip frame data to obtain compressed first skip frame data; wherein the compressed first skip frame data is appended to the first compression result.

[0114] The third acquisition module 803 is used to acquire second skip frame data based on the first compression result and the compressed first skip frame data, wherein the second skip frame data is used to locate the first skip frame data and the original data;

[0115] The compression output module 804 is used to compress the second skip frame data to obtain compressed second skip frame data. The compressed second skip frame data is concatenated before the first compression result to output the second compression result.

[0116] Optionally, the first acquisition module 801 includes:

[0117] The compressed frame data acquisition module 8011 is used to split the core data in the original data into frame data, compress the frame data to obtain compressed frame data, wherein the core data is the data that needs to be accessed.

[0118] The compressed index data acquisition module 8012 is used to record the first information corresponding to the compressed frame data when the frame data is compressed, to obtain index data, to compress the index data, to obtain compressed index data, and to concatenate the compressed index data after the compressed frame data.

[0119] The first compression result acquisition unit 8013 is used to record the second information corresponding to the compressed index data when the index data is compressed, to obtain data header data, to compress the data header data, to obtain compressed data header data, and to concatenate the compressed data header data before the compressed frame data to obtain the first compression result.

[0120] Optionally, the second acquisition module 802 includes

[0121] The first metadata acquisition module 8021 is used to record the third information corresponding to the compressed frame data when the frame data is compressed; record the fourth information corresponding to the compressed index data when the index data is compressed; and generate the first metadata based on the third information and the fourth information.

[0122] The first skip frame data acquisition module 8022 is used to store the first metadata to the first skip frame structure corresponding to the first skip frame data, identify the first identifier of the skip frame storing the first metadata, store the first identifier to the first skip frame structure where the first metadata is located, and obtain the first skip frame data.

[0123] Optionally, the first metadata acquisition module 8021 is configured to record the first position and first data size of the frame data compressed into the compressed frame data, and generate first positioning information of the frame data based on the first position and the first data size; to record the second position of the index data corresponding to the frame data in the compressed index data, and generate second positioning information of the index data based on the second position; and to obtain the first metadata based on the first positioning information and the second positioning information.

[0124] Optionally, the third acquisition module 803 includes

[0125] The second metadata acquisition module 8031 ​​is used to record the fifth information corresponding to the first skip frame data being compressed into the compressed first skip frame data, record the sixth information corresponding to the compressed data header data, and generate the second metadata based on the fifth information and the sixth information;

[0126] The second skip frame data module 8032 is used to store the second metadata to the second skip frame structure corresponding to the second skip frame data, identify the second identifier of the skip frame storing the second metadata, store the second identifier to the second skip frame structure where the second metadata is located, and obtain the second skip frame data.

[0127] Optionally, the second metadata acquisition module 8031 ​​is configured to record the third position and third data size of the first skipped frame data corresponding to the frame data compressed into the compressed first skipped frame data, and generate third positioning information of the first skipped frame data based on the third position and the third data size; and to record the fourth position and fourth data size of the header data corresponding to the frame data compressed into the compressed header data, and generate fourth positioning information of the header data based on the fourth position and the fourth data size; and to obtain the second metadata based on the third positioning information and the fourth positioning information.

[0128] Preferably, the first skip frame data and the second skip frame data are skip frame data provided in the Zstd standard.

[0129] Specific limitations regarding the data compression device can be found in the limitations of the data compression method described above, and will not be repeated here. Each module in the aforementioned data compression device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the corresponding operations of each module.

[0130] In this embodiment of the application, a computer-readable storage medium is also provided, which stores computer instructions. When the computer instructions stored in the computer-readable storage medium are executed by a computer device, the computer device performs the data compression method provided above.

[0131] In this embodiment of the application, a computer program product containing instructions is also provided, which, when run on a computer device, causes the computer device to execute the data compression method provided above.

[0132] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0133] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0134] The above description is merely an optional embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data compression method, characterized in that, The method includes: Compress the original data to obtain the first compression result; Based on the first compression result, first skip frame data is obtained, which is used to locate the original data; and the first skip frame data is compressed to obtain compressed first skip frame data; the compressed first skip frame data is appended to the first compression result. Based on the first compression result and the compressed first skip frame data, second skip frame data is obtained, and the second skip frame data is used to locate the first skip frame data and the original data. The second skipped frame data is compressed to obtain compressed second skipped frame data, which is then appended before the first compression result to output the second compression result. The compression of the original data to obtain the first compression result includes: The core data in the original data is split into frame data, and the frame data is compressed to obtain compressed frame data. The core data is the data that needs to be accessed. Record the first information corresponding to the compressed frame data, obtain index data, compress the index data to obtain compressed index data, and concatenate the compressed index data after the compressed frame data; Record the index data and compress it into the second information corresponding to the compressed index data to obtain data header data. Compress the data header data to obtain compressed data header data. The compressed data header data is concatenated before the compressed frame data to obtain the first compression result. The step of obtaining the first skipped frame data based on the first compression result includes: Record the frame data compressed into third information corresponding to the compressed frame data; record the index data compressed into fourth information corresponding to the compressed index data; and generate first metadata based on the third information and the fourth information. Store the first metadata in the first skip frame structure corresponding to the first skip frame data, identify the first identifier of the skip frame storing the first metadata, store the first identifier in the first skip frame structure where the first metadata is located, and obtain the first skip frame data; The step of obtaining the second skip frame data based on the first compression result and the compressed first skip frame data includes: Record the fifth information corresponding to the compressed first skip frame data, record the sixth information corresponding to the compressed data header data, and generate second metadata based on the fifth information and the sixth information; The second metadata is stored in the second skip frame structure corresponding to the second skip frame data. The second identifier of the skip frame storing the second metadata is identified and stored in the second skip frame structure where the second metadata is located to obtain the second skip frame data.

2. The data compression method according to claim 1, characterized in that, The record compresses the frame data into third information corresponding to the compressed frame data; Record the index data compressed into fourth information corresponding to the compressed index data, and generate first metadata based on the third information and the fourth information, including: Record the first position and first data size of the compressed frame data, and generate the first positioning information of the frame data based on the first position and the first data size; Record the second position of the index data corresponding to the frame data in the compressed index data, and generate the second positioning information of the index data based on the second position; Based on the first location information and the second location information, the first metadata is obtained.

3. The data compression method according to claim 1, characterized in that, The process involves recording the first skipped frame data compressed into the fifth information corresponding to the compressed first skipped frame data, recording the sixth information corresponding to the compressed data header data, and generating second metadata based on the fifth and sixth information, including: Record the first skipped frame data corresponding to the frame data, compress it into the third position and third data size in the compressed first skipped frame data, and generate the third positioning information of the first skipped frame data based on the third position and the third data size; Record the data header data corresponding to the frame data, compress it into the fourth position and the fourth data size in the compressed data header data, and generate the fourth positioning information of the data header data based on the fourth position and the fourth data size; The second metadata is obtained based on the third and fourth location information.

4. The data compression method according to claim 1, characterized in that, The first skip frame data and the second skip frame data are skip frame data provided in the Zstd standard.

5. A data compression device, characterized in that, The device includes: The first acquisition module is used to compress the original data and obtain the first compression result; The second acquisition module is used to acquire first skip frame data based on the first compression result, wherein the first skip frame data is used to locate the original data; and to compress the first skip frame data to obtain compressed first skip frame data; wherein the compressed first skip frame data is appended to the first compression result. The third acquisition module is used to acquire second skip frame data based on the first compression result and the compressed first skip frame data, wherein the second skip frame data is used to locate the first skip frame data and the original data; A compression output module is used to compress the second skip frame data to obtain compressed second skip frame data, which is then appended to the first compression result to output the second compression result. The compression of the original data to obtain the first compression result includes: The core data in the original data is split into frame data, and the frame data is compressed to obtain compressed frame data. The core data is the data that needs to be accessed. Record the first information corresponding to the compressed frame data, obtain index data, compress the index data to obtain compressed index data, and concatenate the compressed index data after the compressed frame data; Record the index data and compress it into the second information corresponding to the compressed index data to obtain data header data. Compress the data header data to obtain compressed data header data. The compressed data header data is concatenated before the compressed frame data to obtain the first compression result. The step of obtaining the first skipped frame data based on the first compression result includes: Record the frame data compressed into third information corresponding to the compressed frame data; record the index data compressed into fourth information corresponding to the compressed index data; and generate first metadata based on the third information and the fourth information. Store the first metadata in the first skip frame structure corresponding to the first skip frame data, identify the first identifier of the skip frame storing the first metadata, store the first identifier in the first skip frame structure where the first metadata is located, and obtain the first skip frame data; The step of obtaining the second skip frame data based on the first compression result and the compressed first skip frame data includes: Record the fifth information corresponding to the compressed first skip frame data, record the sixth information corresponding to the compressed data header data, and generate second metadata based on the fifth information and the sixth information; The second metadata is stored in the second skip frame structure corresponding to the second skip frame data. A second identifier of the skip frame storing the second metadata is identified, and the second identifier is stored in the second skip frame structure containing the second metadata to obtain the second skip frame data.

6. A computer device, the computer device comprising: Memory: Used to store computer programs; Processor: for executing the computer program to implement the data compression method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data compression method as described in any one of claims 1-4.

Images