Data stream compression method and system

Abstract

The invention discloses a data stream compression method and system. The method comprises the following steps: dynamically segmenting an original data stream to obtain a plurality of segmented data blocks, and calculating a characteristic value of each data block; obtaining a corresponding Hash algorithm according to the characteristic value; the Hash algorithm is used for matching historical data, and duplicated data which is duplicated with the data of the data block in the historical data are searched for; calculating a characteristic value of the duplicated data, and selecting a corresponding Huffman model according to the characteristic value of the duplicated data; encoding the repeated data by using a corresponding Huffman model; and updating the coded data to historical data. According to the scheme provided by the invention, the compression rate can be remarkably improved, the compression time is controlled within an acceptable range, and the high efficiency of interactive application in network data transmission is ensured.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to a data stream compression method and system. Background Art

[0002] In today's information age, the rapid growth of data volume has put forward higher requirements for storage and transmission. Both individual users and corporate organizations are faced with the challenge of how to efficiently manage and utilize massive amounts of data. Data compression technology, as an important solution, can store more data in limited storage space and reduce bandwidth consumption during transmission. However, traditional compression algorithms often have a contradiction between improving compression rate and compression speed, and it is difficult to optimize both at the same time. Summary of the invention

[0003] In order to solve the technical problem of the existing compression technology that there is a contradiction between improving the compression rate and the compression speed, the embodiment of the present invention provides a data stream compression method and system.

[0004] The technical solution of the embodiment of the present invention is achieved as follows: The embodiment of the present invention provides a data stream compression method, which includes: dynamically segmenting the original data stream to obtain multiple data blocks after segmentation, and calculating the characteristic value of each data block; obtaining a corresponding hash algorithm according to the characteristic value; and matching historical data with the hash algorithm to find repeated data in the historical data that is repeated with the data of the current data block; calculating the characteristic value of the repeated data, and selecting a corresponding Huffman model according to the characteristic value of the repeated data; encoding the repeated data using the corresponding Huffman model; and updating the encoded data to the historical data.

[0005] In one embodiment, after encoding the repeated data using the corresponding Huffman model, the method further includes: calculating the time consumption and the compression rate of the compressed data in real time to obtain the compression gain; and stopping encoding when the compression gain is lower than a preset threshold.

[0006] In one embodiment, calculating the characteristic value of the repeated data and selecting the corresponding Huffman model according to the characteristic value of the repeated data includes: counting the frequency and alternation interval of the repeated data; and selecting the corresponding Huffman model according to the frequency and alternation interval.

[0007] In one embodiment, the original data stream is dynamically segmented, including: obtaining a data block of a preset length from the original data stream; counting the frequency and distribution of data in the data block; judging whether the data block needs to be expanded based on the frequency and distribution; if it is judged that the data block needs to be expanded, expanding the data block and obtaining the expanded data block; counting the frequency and distribution of data in the data block; judging whether the data block needs to be expanded based on the frequency and distribution; until it is judged that the data block does not need to be expanded, a data block is obtained; and multiple data blocks are obtained from the original data to realize dynamic segmentation of the original data stream.

[0008] In one embodiment, obtaining a corresponding hash algorithm according to the characteristic value includes: selecting a corresponding hash algorithm from hash algorithms based on characters, 32-bit integers, 64-bit integers, and 128-bit integers according to the characteristic value.

[0009] An embodiment of the present invention provides a data stream compression system, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein the processor, when used to run the computer program, performs the following steps: Dynamically segment the original data stream to obtain multiple data blocks after segmentation, and calculate the characteristic value of each data block; obtain the corresponding hash algorithm according to the characteristic value; and use the hash algorithm to match the historical data to find the repeated data in the historical data that is repeated with the data of the current data block; calculate the characteristic value of the repeated data, and select the corresponding Huffman model according to the characteristic value of the repeated data; encode the repeated data using the corresponding Huffman model; and update the encoded data to the historical data.

[0010] In one embodiment, when the processor is used to run the computer program, it also performs the following steps: calculating the time consumption and the compression rate of the compressed data in real time to obtain the compression gain; when the compression gain is lower than a preset threshold, stopping encoding.

[0011] In one embodiment, when the processor is used to run the computer program, the processor further performs the following steps: counting the frequency and alternating interval of the repeated data; and selecting a corresponding Huffman model according to the frequency and alternating interval.

[0012] In one embodiment, when the processor is used to run the computer program, it also performs the following steps: obtaining a data block of a preset length from the original data stream; counting the frequency and distribution of data in the data block; judging whether the data block needs to be expanded based on the frequency and distribution; if it is judged that the data block needs to be expanded, expanding the data block and obtaining the expanded data block; counting the frequency and distribution of data in the data block; judging whether the data block needs to be expanded based on the frequency and distribution; until it is judged that the data block does not need to be expanded, a data block is obtained; and multiple data blocks are obtained from the original data to achieve dynamic segmentation of the original data stream.

[0013] In one embodiment, when the processor is used to run the computer program, it further performs the following steps: selecting a corresponding hash algorithm from hash algorithms based on characters, 32-bit integers, 64-bit integers, and 128-bit integers according to the characteristic value.

[0014] The embodiments of the present invention have the following beneficial effects: (1) Content matching based on global historical data: By using global historical data for content matching, the matching probability is improved and the data compression rate is further improved.

[0015] (2) Dynamic data block division: Dynamically divide data blocks as compression units based on the characteristics of the data stream.

[0016] (3) Select different encoding methods based on data block characteristics: Dynamically select the optimal hash algorithm and Huffman coding model to adapt to different data characteristics and further improve compression efficiency.

[0017] (4) Benefit evaluation: Through benefit evaluation, the compression rate and compression time are balanced to ensure the efficiency of the compression process.

[0018] (5) Preset Huffman model: By presetting multiple Huffman models, the space required to encode the Huffman tree into the data stream can be saved, further improving the compression rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of a data stream compression method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall flow of the data stream compression method according to an embodiment of the present invention; Figure 3 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0021] The embodiment of the present invention provides a data stream compression method, such as Figure 1 As shown, the method includes: Step 101: dynamically segment the original data stream to obtain multiple segmented data blocks, and calculate the feature value of each data block; Step 102: Obtain a corresponding hash algorithm according to the characteristic value; and use the hash algorithm to match the historical data to find duplicate data in the historical data that is duplicated with the data of the current data block; Step 103: Calculate the eigenvalue of the repeated data, and select a corresponding Huffman model according to the eigenvalue of the repeated data; Step 104: Encode the repeated data using a corresponding Huffman model; Step 105: Update the data encoded this time into the historical data.

[0022] Most existing compression methods rely on real-time data processing, ignoring the potential value of historical data. Historical data often contains a large amount of repeated and similar information. The method of this embodiment effectively utilizes historical data, which can significantly improve the compression rate and reduce the compression time. For example, in the field of video compression, there are often a lot of similarities between consecutive frames. This embodiment can significantly reduce redundant information by analyzing the data of historical frames, thereby improving compression efficiency.

[0023] This embodiment analyzes data features and uses historical data to significantly reduce compression time while improving compression rate, making the compression process more efficient and practical. It not only improves compression rate, but also reduces compression time, meeting the efficiency requirements in practical applications. (Compression can be simply understood as two parts: duplicate checking and encoding. Finding more duplicate data means a high compression rate, because duplicate data only needs a simple representation. Using historical data, the sample for duplicate checking becomes larger, and the possibility of data duplication is higher. Therefore, using historical data can improve compression rate.) The method of this embodiment analyzes different data features and establishes different statistical models, so that different encoding models are used during compression to more effectively encode high-frequency data and reduce the amount of data after compression. (Data is sent on the network based on binary 01. According to the selected length of data, the optimal model can be selected according to the statistical frequency and alternating interval of 0 and 1. For example, all 0 and all 1, more 1 and less 0, less 0 and more 1, etc. By using different encoding models, high-frequency data can be more effectively encoded, the amount of data after compression can be reduced, and the time consumption of the encoding part can be reduced.) See also Figure 2 The method of this embodiment specifically includes the following steps: 1) Preprocess the data to be encoded by reading fixed-length data first, counting the frequency and distribution of characters, and calculating whether the data segment needs to be expanded. This process is repeated until the data no longer needs to be expanded, that is, a segment is completed; Dynamic data stream segmentation: Count the distribution and frequency of 0s and 1s in fixed-length data, and calculate whether additional data is needed based on the statistical values 2) Calculate the data block feature values ​​for the data in 1) by segmenting, and select the hash algorithm based on characters, 32-bit integers, 64-bit integers, and 128-bit integers; Here, the data block feature value: a predefined value is calculated using a specific algorithm based on the input data stream; 3) Use the hash algorithm in 2) to traverse the data and query the duplicate data from the historical data; Here, the optimal hash algorithm is: a predefined hash algorithm based on the feature value and algorithm binding; 4) Calculate the eigenvalues ​​of the matching data blocks in 3) and select a predefined Huffman model based on the eigenvalues; Here, matching is done: using hash matching; 5) Use the Huffman model of 4) to encode the repeated data of 2); 6) Calculate the current time consumption and the compression ratio of the compressed data, and evaluate the compression benefit of the current data block; if the benefit is lower than the threshold, end early; Here, the compression benefit is evaluated: by the consumed time and the compression ratio of the currently compressed data; 7) Update the data encoded this time into the historical data; Here, the update is: dynamic update of the Huffman tree; 8) Repeat steps 2 to 7.

[0024] Business data generally contains a large amount of repeated data or partially repeated data. If you only look for repeated parts in the current data, the probability of repetition is low and the compression rate is low. If you can find repetitions in previously sent data, the probability of repeated data is high and the compression rate is high. Compression is a process of deduplication, describing repeated data with simple data. Compression can be simply understood as two parts: duplicate checking and encoding. Because this embodiment uses historical data and the data has high repeatability, the compression rate is improved. However, it will take more time to find duplicate data from more data. Therefore, this embodiment uses Huffman coding to reduce the time as much as possible, and selects the optimal algorithm by analyzing data features to reduce the time as much as possible.

[0025] The method of this embodiment aims to improve the data compression rate while taking into account the compression time. Through dynamic segmentation and eigenvalue calculation, the optimal hash algorithm and Huffman model are selected, the compression benefit is evaluated in real time, and the compression time and compression rate are balanced to ensure the high efficiency of compression. The method of this embodiment significantly improves the transmission efficiency of data on the network, can respond to changes in a timely manner, and has high practical value.

[0026] The method of this embodiment analyzes the characteristics of the data stream, selects the predefined optimal hash algorithm and Huffman coding model, performs content matching based on global historical data, and evaluates the compression benefits to select different strategies. This method improves the compression rate while taking into account the compression time, greatly improving the network transmission speed and user experience level under the condition of high business similarity.

[0027] The method of this embodiment has the following beneficial effects: 1) Content matching based on global historical data: By using global historical data for content matching, the matching probability is improved and the data compression rate is further improved.

[0028] 2) Dynamic data block division: Dynamically divide data blocks as compression units according to the characteristics of the data stream.

[0029] 3) Select different encoding methods according to data block characteristics: Dynamically select the optimal hash algorithm and Huffman coding model to adapt to different data characteristics and further improve compression efficiency.

[0030] 4) Benefit evaluation: Through benefit evaluation, balance the compression rate and compression time to ensure the efficiency of the compression process.

[0031] 5) Preset Huffman model: By presetting multiple Huffman models, the space for encoding the Huffman tree into the data stream is saved, further improving the compression rate.

[0032] The method of this embodiment is particularly suitable for data streams with strong coding data correlation, and can significantly improve the compression rate while controlling the compression time within an acceptable range, thereby ensuring the high efficiency of interactive applications in network data transmission.

[0033] For example, in a low-bandwidth network environment, in a remote desktop scenario, in order to improve the response time and user experience of interactive data transmission, it is effective to adopt a high data compression rate. However, excessive pursuit of compression rate will increase compression time, calculation amount and time consumption. Using the method of this embodiment, in a remote desktop scenario, the bandwidth requirements and transmission delay can be significantly reduced, thereby improving the overall performance of the system and user experience.

[0034] Compared with the existing compression method, the method of this embodiment has the following characteristics and effects: 1) It has a higher data compression rate than real-time compression; 2) The compression time is faster than real-time compression; 3) It improves the transmission efficiency of data streams on the network and improves the user experience, which has high practical value; 4) This is an innovative solution with high innovation.

[0035] In order to implement the method of an embodiment of the present invention, an embodiment of the present invention also provides a data stream compression system, including: a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, the steps of any one of the above methods are executed.

[0036] The above-mentioned system provided in this embodiment belongs to the same concept as the above-mentioned method embodiment. The specific implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0037] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0038] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0039] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0040] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0041] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0042] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0043] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0044] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0045] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0046] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0047] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0048] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0049] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0050] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data stream compression method, characterized in that: The method comprises: Dynamically segment the original data stream to obtain multiple data blocks after segmentation, and calculate the characteristic value of each data block; Obtaining a corresponding hash algorithm according to the characteristic value; and using the hash algorithm to match the historical data to find duplicate data in the historical data that is duplicated with the data of the current data block; Calculating the characteristic value of the repeated data, and selecting a corresponding Huffman model according to the characteristic value of the repeated data; Encoding the repeated data using a corresponding Huffman model; Update the data encoded this time into the historical data.

2. The data stream compression method according to claim 1, characterized in that: After encoding the repeated data using the corresponding Huffman model, the method further includes: Calculate the time consumption and compression ratio of compressed data in real time to obtain compression benefits; When the compression gain is lower than a preset threshold, encoding is stopped.

3. The data stream compression method according to claim 1, characterized in that: Calculating the characteristic value of the repeated data, and selecting the corresponding Huffman model according to the characteristic value of the repeated data, including: Counting the frequency and alternation interval of the repeated data; A corresponding Huffman model is selected according to the frequency and the alternating interval.

4. The data stream compression method according to claim 1, characterized in that: Dynamically segment the original data stream, including: Acquire a data block of a preset length from the original data stream; Counting the frequency and distribution of data in the data block; According to the frequency and distribution, determining whether the data block needs to be expanded; In the case where it is determined that the data block needs to be expanded, the data block is expanded and the expanded data block is obtained; the frequency and distribution of the data in the data block are counted; according to the frequency and distribution, whether the data block needs to be expanded is determined; until it is determined that the data block does not need to be expanded, a data block is obtained; A plurality of data blocks are obtained from the original data to achieve dynamic segmentation of the original data stream.

5. The data stream compression method according to claim 1, characterized in that: Obtaining a corresponding hash algorithm according to the characteristic value includes: A corresponding hash algorithm is selected from hash algorithms based on characters, 32-bit integers, 64-bit integers, and 128-bit integers according to the characteristic value.

6. A data stream compression system, characterized in that: include: A processor and a memory for storing a computer program that can be run on the processor; wherein the processor, when used to run the computer program, performs the following steps: Dynamically segment the original data stream to obtain multiple data blocks after segmentation, and calculate the characteristic value of each data block; Obtaining a corresponding hash algorithm according to the characteristic value; and using the hash algorithm to match the historical data to find duplicate data in the historical data that is duplicated with the data of the current data block; Calculating the characteristic value of the repeated data, and selecting a corresponding Huffman model according to the characteristic value of the repeated data; Encoding the repeated data using a corresponding Huffman model; Update the data encoded this time into the historical data.

7. The data stream compression system according to claim 6, characterized in that: When the processor is used to run the computer program, the processor further performs the following steps: Calculate the time consumption and compression ratio of compressed data in real time to obtain compression benefits; When the compression gain is lower than a preset threshold, encoding is stopped.

8. The data stream compression system according to claim 6, characterized in that: When the processor is used to run the computer program, the processor further performs the following steps: Counting the frequency and alternation interval of the repeated data; A corresponding Huffman model is selected according to the frequency and the alternating interval.

9. The data stream compression system according to claim 6, characterized in that: When the processor is used to run the computer program, the processor further performs the following steps: Acquire a data block of a preset length from the original data stream; Counting the frequency and distribution of data in the data block; According to the frequency and distribution, determining whether the data block needs to be expanded; When it is determined that the data block needs to be expanded, the data block is expanded and the expanded data block is obtained; Counting the frequency and distribution of data in the data block; judging whether the data block needs to be expanded according to the frequency and distribution; and obtaining a data block until it is judged that the data block does not need to be expanded; A plurality of data blocks are obtained from the original data to achieve dynamic segmentation of the original data stream.

10. The data stream compression system according to claim 6, characterized in that: When the processor is used to run the computer program, the processor further performs the following steps: A corresponding hash algorithm is selected from hash algorithms based on characters, 32-bit integers, 64-bit integers, and 128-bit integers according to the characteristic value.

Images