A lossless data compression method, system and device for high-speed transparent transmission

Abstract

The application discloses a lossless data compression method, system and device for high-speed transparent transmission, belongs to the field of data processing, and is characterized in that: sample data is analyzed and counted in advance to obtain preconfigured parameters meeting compression requirements, then online real-time lossless compression is performed on received data according to the preconfigured parameters, context information is not required, a higher processing speed and lower delay can be supported, the lossless data compression method, system and device for high-speed transparent transmission are simple in compression process, low in resource occupation, flexible in bandwidth expansion, can meet the requirement of online lossless compression for high-speed transmission, and are low in occupied hardware circuit resources.

Description

Technical Field

[0001] This invention belongs to the field of data processing, and in particular relates to a lossless data compression method, system and apparatus for high-speed transparent transmission. Background Technology

[0002] Digital technology is impacting every aspect of our lives. Communication signals, images, and voice are all being collected, transmitted, and recorded in digital form, and the amount of data to be processed is growing exponentially. A significant challenge today is the enormous space required for the transmission and storage of this information, coupled with stringent requirements for transmission speed and real-time performance. Furthermore, in most scenarios, the collected data is not entirely random but exhibits certain patterns: for example, only a small portion of the image data changes between adjacent frames in a video image; or in audio signal data, only a portion is valid audio, with the majority likely being background noise. Current compression methods for this type of data typically involve comparing and calculating contextual data to find patterns, then encoding and compressing based on these patterns. However, current methods for processing this data often suffer from high CPU load, complex compression circuits, high latency, insufficient bandwidth, and inflexible scalability. Summary of the Invention

[0003] The present invention aims to solve the above problems and provides a lossless data compression method, system and device for high-speed transparent transmission based on preset parameters.

[0004] In a first aspect, the present invention provides a lossless data compression method for high-speed transparent transmission. First, sample data is pre-analyzed, and pre-configured parameters are obtained based on the pre-analysis results. Then, the data to be compressed is compressed according to the pre-configured parameters.

[0005] Furthermore, in the high-speed transparent transmission lossless data compression method of the present invention, the process of pre-analyzing the sample data includes:

[0006] The sample data is divided into several data units with a preset granularity;

[0007] Perform statistical analysis on the data units to calculate the probability of different types of data codes appearing in the sample data;

[0008] Data patterns are classified according to their probability of occurrence.

[0009] According to different classifications, a fixed-length classification identifier is set for each type of data code.

[0010] According to different classifications, the code length of the compressed code is set for the data corresponding to each type of data code according to the encoding rules;

[0011] The pre-configured parameters include preset granularity, data code type classification, classification identifier, and code length.

[0012] Furthermore, in the high-speed transparent transmission lossless data compression method of the present invention, the preset granularity is one or more complete bytes; the value of the data code type classification is 2 to the power of n; and the length of the classification identifier is n binary digits.

[0013] Furthermore, in the lossless data compression method for high-speed transparent transmission described in this invention, the encoding rule is as follows: based on the probability of data code occurrence from high to low, the code length of the compressed and encoded data is from short to long; wherein, the shortest code length is 0, and the longest code length is the original data length.

[0014] Furthermore, the lossless data compression method for high-speed transparent transmission described in this invention includes the following steps in the process of compressing the data to be compressed according to pre-configured parameters:

[0015] The data to be compressed is first converted to a bit width multiple of the aforementioned preset granularity.

[0016] Based on the classification of data codes, identify the category of the data to be compressed and generate the corresponding category identifier;

[0017] Based on the classification of data code types, the data to be compressed is compressed according to the set code length after compression encoding, and the compressed data is output.

[0018] The classification identifier and the compressed data are aligned, and a new compressed data is output in a preset order, thus completing the data compression.

[0019] Furthermore, the lossless data compression method for high-speed transparent transmission described in this invention includes the following data alignment process: if the data bit width of the new compressed data is less than the output data bit width, the data is temporarily stored and concatenated with the next received data; if the bit width after concatenation is greater than or equal to the required output data bit width, a valid compressed data with bit width alignment is sent out; the excess bits are continued to be stored and concatenated with the next received data; if the bit width of the concatenated data is still less than the required output data bit width, it is latched and waited for merging with the next data.

[0020] Furthermore, in the lossless data compression method for high-speed transparent transmission described in this invention, the output speed of the new compressed data can be adjusted by inserting invalid codes that do not conform to the aforementioned encoding rules into the data stream.

[0021] In a second aspect, the present invention provides a high-speed transparent transmission lossless data compression system, including a data receiving module, a data bit width conversion module, a data compression encoding module, a data alignment module, and a data transmission control module;

[0022] The data receiving module is used to obtain the data to be compressed from the interface or cache;

[0023] The data bit width conversion module is used to first convert the bit width of the data to be compressed, converting the bit width into a multiple of the aforementioned preset granularity;

[0024] The data compression encoding module is used to identify the category of the data to be compressed according to the classification of data code type, generate the corresponding category identifier; and compress the data to be compressed according to the classification of data code type and the set code length after compression encoding, and output the compressed data.

[0025] The data alignment module is used to perform data alignment processing on the classification identifier and the compressed data, and output a new compressed data composed in a preset order;

[0026] The data transmission control module is used to output compressed data.

[0027] If the width of the new compressed data is less than the width of the output data, the data is temporarily stored and concatenated with the next received data. If the width of the concatenated data is greater than or equal to the width of the output data, a valid compressed data set with aligned width is sent out, and the portion exceeding the width is latched and concatenated with the next received data. If the width of the concatenated data is still less than the required output data width, it is latched and awaits concatenation with the next data.

[0028] At the same time, invalid codes can be inserted into the data stream to control the output speed of the compressed data.

[0029] When the received data bit width exceeds the set processing data unit width, the received data to be compressed can be striped, and then multiple compression modules can be invoked for parallel and synchronous processing to improve processing bandwidth. The compression ratios of the multiple stripes may differ; by inserting invalid codes into the data stream, the overall balance between the amount of data input before and after compression across the stripes is achieved.

[0030] Thirdly, the present invention provides a lossless data compression device for high-speed transparent transmission, comprising a processor and a memory electrically connected to each other; the memory is used to store a computer program; when the processor executes the aforementioned computer program, it can realize the lossless data compression method for high-speed transparent transmission as described in the first aspect above.

[0031] Fourthly, the present invention provides a computer-readable storage medium storing a computer program thereon; when the computer program is executed, it can implement the lossless data compression method for high-speed transparent transmission as described in the first aspect above.

[0032] The high-speed transparent transmission lossless data compression method, system, and apparatus of this invention obtains pre-configured parameters that meet compression requirements by analyzing and statistically processing sample data in advance. Then, the received data is compressed online in real time according to the pre-configured parameters. No context information is required, and it can support high processing speed and low latency. The high-speed transparent transmission lossless data compression method, system, and apparatus of this invention have a simple compression process, low resource consumption, flexible bandwidth expansion, can meet the requirements of high-speed transmission online lossless compression, and occupy few hardware circuit resources. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the lossless data compression method for high-speed transparent transmission according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the high-speed transparent transmission lossless data compression system structure described in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the high-speed transparent transmission lossless data compression system structure described in an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the lossless data compression method for high-speed transparent transmission according to an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram illustrating the alignment of data in the lossless data compression method for high-speed transparent transmission described in an embodiment of the present invention. Detailed Implementation

[0038] The lossless data compression method, system, and apparatus for high-speed transparent transmission described in this invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0039] Example 1

[0040] This embodiment discloses a lossless data compression method for high-speed transparent transmission, such as... Figure 1 As shown, by performing pre-analysis on the sample data, pre-configured parameters are obtained based on the pre-analysis results; then, the data to be compressed is compressed according to the pre-configured parameters.

[0041] The pre-analysis process for sample data includes: dividing the sample data into several data units with a preset granularity; in this embodiment, the preset granularity is one or more complete bytes; the classification value of the data code type is 2 to the power of n; the classification identifier length is n binary digits. Statistical analysis is performed on the data units to calculate the probability of different types of data code types appearing in the sample data; the data code types are classified according to their probability of occurrence; a fixed-length classification identifier is set for each type of data code type according to different classifications; the compressed code length is set for each type of data code type according to different classifications, based on encoding rules; the encoding rules are: according to the probability of occurrence of the data code type from high to low, the code length of the compressed encoded data is from short to long; wherein the shortest code length is 0, and the longest code length is the original data length. The pre-configured parameters include the preset granularity, the classification of the data code type, the classification identifier, and the code length.

[0042] The data compression process based on pre-configured parameters includes: first, converting the bit width of the data to be compressed to a multiple of the aforementioned preset granularity; identifying the category of the data to be compressed based on the data code type and generating a corresponding category identifier; compressing the data to be compressed according to the set code length after compression encoding based on the data code type and outputting the compressed data; aligning the category identifier and the compressed data and outputting a new compressed data in a preset order to complete the data compression.

[0043] In this embodiment of the disclosure, the data alignment process includes: if the data bit width of the new compressed data is less than the output data bit width, the data is temporarily stored and concatenated with the next received data; if the bit width after concatenation is greater than or equal to the required output data bit width, a valid compressed data with bit width alignment is sent out; the excess bits are continued to be stored and concatenated with the next received data; if the bit width of the concatenated data is still less than the required output data bit width, it is latched and waited for merging with the next data.

[0044] During the data compression output process, when the output speed of the final new compressed data needs to be adjusted, it can be adjusted by inserting invalid codes that do not conform to the aforementioned encoding rules into the data stream. At the same time, if the bit width of the data to be compressed is greater than the set processing data unit width, the received data to be compressed can be striped and multiple parallel steps can be used for data compression processing, thereby effectively improving the processing bandwidth.

[0045] In this embodiment of the disclosure, the compression ratio of the data to be compressed can be calculated based on the probability of different data code patterns and the code length. When the calculated compression ratio does not meet the compression requirements, the preset granularity of the data and the classification of the data code patterns can be modified, and the above process can be iterated until the parameters that meet the compression ratio requirements are calculated.

[0046] like Figure 4 As shown, in this embodiment of the disclosure, the sample data is statistically analyzed at a preset granularity of 8 bits to calculate the probability of different data code patterns occurring; and then classified according to the probability. For example... Figure 4 The 402 module is classified into four data code types, with each classification identifier being 2 bits long and described in detail. The 8-bit data 00000000 is assigned to the identifier 00, and 11111111 is assigned to the identifier 01. The data with the high 5 bits of 8 binary numbers being 0 (i.e., 0000_0xxx) and the data with the high 5 bits of 8 binary numbers being 1 (i.e., 1111_1xxx) are assigned to the identifier 10, and the other data is assigned to the identifier code 11.

[0047] Data is categorized by data encoding type and then trimmed and compressed without loss. Figure 4 Figure 403 shows the data lengths after lossless compression for different data categories. Figure 4 The data shown in Figure 404 is the lossless compressed data.

[0048] Based on the aforementioned code type classification in this embodiment, when the identifier is 00 or 01, the data code length after compression is 0, and the compression rate of this type of data is 25%; when the identifier is 10, the data code length after compression is 4, and after adding the identifier, the compression rate of this type of data is 75%; when the identifier code is 11, the data code length after compression is 8, and the compression rate of this type of data is 125%.

[0049] Using the above encoding rules, an invalid code that does not conform to the rules can be set as padding; a custom invalid code can be defined, for example, the identifier is 11, and the corresponding data is 00000000; during data compression, the compression ratio is calculated based on the probability of different data code types and the length after compression. If the above encoding rules meet the compression ratio requirements, the parameters to be pre-configured for the compression module are determined: the data processing granularity is 8 bits, the number of data categories is 4, the data identifier length is 2 bits, and the correspondence between the identifier and the actual compressed data is as follows. Figure 4 As shown.

[0050] Example 2

[0051] Based on Embodiment 1, this disclosure provides a decompression method; after the decompression process is initiated, the data to be decompressed can be processed, as follows:

[0052] Upon receiving the data to be compressed, the identifier sent first is extracted. Based on the correspondence between the identifier and the data code length, the effective data length of the data to be decompressed is obtained. The effective data is extracted, and the data bits that were clipped during compression are added to form the original data, which is then sent out. The bit width of the decompressed data is the same as the unit bit width of the corresponding compression module.

[0053] If the extracted compressed data bits are determined to be invalid based on the identifier, they are discarded. If there are still remaining data bits after extracting the identifier and data for one cycle of data received above, the remaining data bits are temporarily stored, combined with the next data, and the decompression operation continues.

[0054] For the same compression method, the configuration parameters used for compressing and decompressing the same data remain consistent; it is a symmetrical reverse operation process.

[0055] Example 3

[0056] This disclosure presents a high-speed, transparent, lossless data compression system, such as... Figure 2 As shown, it includes a data receiving module, a data bit width conversion module, a data compression encoding module, a data alignment module, and a data transmission control module;

[0057] The data receiving module is used to read the data to be compressed from the cache or input it from the interface;

[0058] The data bit width conversion module receives the data and performs bit width conversion, converting it to a pre-configured processing granularity of m bytes or multiples of m bytes. After conversion, the data is sent to the data compression encoding module.

[0059] The data compression and encoding module identifies the category of the data according to the pre-configured code pattern classification rules, and generates the corresponding data identifier and the cropped and compressed data. For example, the data 00000101 has a corresponding data category identifier of 10, and the compressed and encoded data is 0101. The data 11111111 has a corresponding data category identifier of 01, and the compressed and encoded data code length is 0, which means that all 8 bits of data can be determined using only the data category identifier.

[0060] The generated data identifier and the cropped and compressed data are sent to the data alignment module. The data alignment module combines the received identifier and the cropped and compressed data into a new data in an agreed order, and then sends it out through the data transmission control module.

[0061] If the width of the newly combined data is less than the output data width, the data is temporarily stored and concatenated with the next received data. If the width of the concatenated data is greater than or equal to the required output data width, valid data aligned with the required data width is sent out; the excess bits are latched and concatenated with the next received data. If the width of the concatenated data is still less than the required output data width, it is latched again and waited to be concatenated with the next data, and so on; Figure 5 A schematic diagram of parallel aligned data after data compression and encoding;

[0062] Depending on the specific use case, invalid codes can be inserted into the data stream when necessary to control the output speed of the compressed data.

[0063] like Figure 3 As shown, when the received data bit width is greater than the set processing data unit width, the received data to be compressed can be striped, and then multiple compression modules can be called to process it in parallel and synchronously, thereby improving the processing bandwidth. The compression ratios of multiple stripes may be different. By inserting invalid codes into the data stream, the overall balance between the amount of data before compression and the amount of data after compression among the input stripes can be achieved.

[0064] For the same compression method, the configuration parameters used for compressing and decompressing the same data remain consistent; it is a symmetrical reverse operation process.

[0065] Example 4

[0066] This disclosure presents a high-speed transparent transmission lossless data compression device, including a processor and a memory electrically connected to each other; the memory is used to store a computer program; when the processor executes the aforementioned computer program, it can implement the high-speed transparent transmission lossless data compression method described in Embodiment 1; the specific compression method steps are the same as in Embodiment 1, and will not be repeated here.

[0067] Example 5

[0068] This disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed, it can implement the lossless data compression method for high-speed transparent transmission described in Embodiment 1. The specific compression method steps are the same as in Embodiment 1 and will not be repeated here.

[0069] The computer described in this application embodiment can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. The computer-readable storage medium can be any usable medium that a computer can read, or a data storage device such as a server or data center that integrates one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile optical disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)). The software formed by the computer's stored code can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other storage media that are mature in the art.

[0070] In the various embodiments of this application, the functional modules can be integrated into one processing unit or module, or each module can exist physically separately, or two or more modules can be integrated into one unit or module. In the above embodiments, they can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated.

[0071] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A lossless data compression method for high-speed transparent transmission, characterized in that: First, the sample data is pre-analyzed, and pre-configured parameters are obtained based on the pre-analysis results; Then, the data to be compressed is performed according to the pre-configured parameters; The process of pre-analyzing the sample data includes: The sample data is divided into several data units with a preset granularity; Perform statistical analysis on the data units to calculate the probability of different types of data codes appearing in the sample data; Data patterns are classified according to their probability of occurrence. According to different classifications, a fixed-length classification identifier is set for each type of data code. According to different classifications, the code length of the compressed encoded data is set according to the encoding rules for each type of data code pattern. The encoding rules are as follows: based on the probability of the data code pattern occurring from high to low, the code length of the compressed encoded data is from short to long. Among them, the shortest code length is 0, and the longest code length is the original data length. The pre-configured parameters include preset granularity, data code type classification, classification identifier, and code length; The process of compressing the data to be compressed according to the pre-configured parameters includes: The data to be compressed is first converted to a bit width multiple of the aforementioned preset granularity. Based on the classification of data codes, identify the category of the data to be compressed and generate the corresponding category identifier; Based on the classification of data code types, the data to be compressed is compressed according to the set code length after compression encoding, and the compressed data is output. The classification identifier and the compressed data are aligned, and a new compressed data is output in a preset order to complete the data compression. The data alignment process includes: if the bit width of the new compressed data is less than the bit width of the output data, it is temporarily stored and concatenated with the next received data; if the bit width after concatenation is greater than or equal to the required output data bit width, a bit width-aligned valid compressed data is sent out; the excess bits are stored and concatenated with the next received data. If the width of the concatenated data is still less than the required output data width, then latch it and wait to merge it with the next data.

2. The lossless data compression method for high-speed transparent transmission according to claim 1, characterized in that: The preset granularity is one or more complete bytes; the classification value of the data code type is 2 to the power of n; the classification identifier is n binary digits in length.

3. The lossless data compression method for high-speed transparent transmission according to claim 1, characterized in that: The output speed of the new compressed data is adjusted by inserting invalid codes that do not conform to the aforementioned encoding rules into the data stream.

4. A high-speed transparent transmission lossless data compression system, characterized in that: It includes a data receiving module, a data bit width conversion module, a data compression encoding module, a data alignment module, and a data transmission control module; The data receiving module is used to obtain the data to be compressed from the interface or cache; The data bit width conversion module is used to first convert the bit width of the data to be compressed, converting the bit width into a multiple of a preset granularity; The data compression encoding module is used to identify the category of the data to be compressed based on the classification of the data code type, and generate the corresponding category identifier; Based on the classification of data code patterns and according to the encoding rules of the data code pattern occurrence probability from high to low, the code length of the compressed data is set from short to long. The compressed code length is then used to compress the data to be compressed and output the compressed data. The data alignment module is used to perform data alignment processing on the classification identifier and the compressed data, and output a new compressed data composed in a preset order; The data alignment process includes: if the bit width of the new compressed data is less than the bit width of the output data, it is temporarily stored and concatenated with the next received data; if the bit width after concatenation is greater than or equal to the required output data bit width, a bit width-aligned valid compressed data is sent out; the excess bits are stored and concatenated with the next received data. If the width of the concatenated data is still less than the required output data width, then latch it and wait to merge it with the next data for processing; The data transmission control module is used to output compressed data.

5. A high-speed transparent transmission lossless data compression device, comprising a processor and a memory electrically connected; the memory being used to store computer programs; characterized in that: When the processor executes the aforementioned computer program, it performs the lossless data compression method for high-speed transparent transmission as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program; when the computer program is executed, it performs the lossless data compression method for high-speed transparent transmission as described in any one of claims 1-3.

Images