Data compression method, data compression method applied to cloud side device, computing device, computer readable storage medium and computer program product

Abstract

The embodiment of the invention provides a data compression method, a data compression method applied to cloud side equipment, computing equipment, a computer readable storage medium and a computer program product, and the data compression method comprises the following steps: receiving to-be-compressed business data, and determining at least one to-be-compressed business data segment based on the to-be-compressed business data; determining an uncompressed business data segment set and a compressed business data segment set from the business data segments to be compressed; under the condition that at least one uncompressed service data segment exists in the uncompressed service data segment set, performing compression processing on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment; and determining a service data compression result corresponding to the to-be-compressed service data according to each service data segment compression result and the compressed service data segment set. The uncompressed data segments in the to-be-compressed data are compressed, so that repeated compression of the compressed data is avoided, and resource consumption of a CPU (central processing unit) is reduced.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and more particularly to a data compression method and a data compression method for cloud-side devices. One or more embodiments of this specification also relate to a data compression device, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the widespread application of distributed storage systems, in order to save storage resources, systems often introduce compression mechanisms to reduce storage costs.

[0003] However, to control storage costs, user business layers have also begun using compression technology within their own business layers. This results in data already being compressed by the time it reaches the underlying systems. While this approach appears to improve efficiency, it also introduces new problems. The system attempts to re-compress already compressed data, which not only wastes CPU resources but also requires decompression, increasing CPU consumption. Therefore, a data compression method is needed that can effectively identify already compressed data and avoid re-compression. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data compression method and a data compression method applied to a cloud-side device. One or more embodiments of this specification also relate to a data compression apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data compression method is provided, including:

[0006] receiving service data to be compressed, and determining at least one service data segment to be compressed based on the service data to be compressed;

[0007] Determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed;

[0008] When there is at least one uncompressed service data segment in the uncompressed service data segment set, performing compression processing on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment;

[0009] The business data compression result corresponding to the business data to be compressed is determined according to the compression result of each business data segment and the set of compressed business data segments.

[0010] According to a second aspect of an embodiment of this specification, a data compression method applied to a cloud-side device is provided, including:

[0011] receiving service data to be compressed from a terminal-side device, and determining at least one service data segment to be compressed based on the service data to be compressed;

[0012] Determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed;

[0013] When there is at least one uncompressed service data segment in the uncompressed service data segment set, performing compression processing on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment;

[0014] According to the compression results of each business data segment and the set of compressed business data segments, the business data compression result corresponding to the business data to be compressed is determined, and the business data compression result is returned to the end-side device.

[0015] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0016] memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned data compression method are implemented.

[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data compression method are implemented.

[0019] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the above-mentioned data compression method when executed by a processor.

[0020] One embodiment of the present specification implements receiving business data to be compressed, and determining at least one business data segment to be compressed based on the business data to be compressed; determining a set of uncompressed business data segments and a set of compressed business data segments from each business data segment to be compressed; when there is at least one uncompressed business data segment in the set of uncompressed business data segments, performing compression processing on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment; and determining a business data compression result corresponding to the business data to be compressed based on the compression result of each business data segment and the set of compressed business data segments.

[0021] By applying the solution of the embodiments of this specification, by dividing the data to be compressed into at least one business data segment and determining a set of uncompressed business data segments in the data segment, it is possible to identify the uncompressed business data when some data in the entire data has been compressed, and subsequently compress the uncompressed data in the data segment without compressing the compressed data segment, thereby reducing the problem of CPU resource consumption caused by repeated compression of the compressed data. It also avoids the problem of a high judgment error rate caused by the failure to identify the uncompressed data in the data segment and judging the entire data segment as compressed when avoiding CPU resource consumption due to repeated compression. This achieves the reduction of CPU resource consumption while ensuring data compression accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a data compression method provided by one embodiment of this specification;

[0023] Figure 2 is a schematic diagram of a data compression process provided by an embodiment of this specification;

[0024] Figure 3 This is a flow chart of a data compression method applied to a cloud-side device provided by one embodiment of this specification;

[0025] Figure 4 This is an architectural diagram of a data compression system provided by one embodiment of this specification;

[0026] Figure 5 This is a flowchart of a processing method for storing user uploaded data provided by an embodiment of this specification;

[0027] Figure 6 This is a structural diagram of a data compression device provided by an embodiment of this specification;

[0028] Figure 7 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0029] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0030] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model. It is pre-trained on a large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as a large language model (LLM) and a multi-modal pre-training model.

[0034] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0035] First, the terms involved in one or more embodiments of this specification are explained.

[0036] LZ4: This compression algorithm prioritizes speed, offering very high compression and decompression speeds while maintaining a reasonable compression ratio. It is primarily suitable for scenarios where fast data compression and decompression are crucial, especially when processing large amounts of data. This algorithm reduces data size by identifying repeated strings within the data, making it a lossless compression technique.

[0037] Deflate: This compression method combines the LZ77 algorithm with Huffman coding. It offers a good balance of compression efficiency and speed, suitable for a wide range of file types. Deflate is a widely supported compression format, commonly used to compress data during web transmission to reduce transmission time and bandwidth consumption.

[0038] Brot LI: This compression algorithm is optimized for network transmission and offers higher compression ratios than other common compression methods. It achieves better compression by using more complex compression techniques while maintaining reasonable decompression speeds. Brot LI is particularly well-suited for compressing text data, such as web pages, scripts, and style sheets.

[0039] This specification provides a data compression method and a data compression method for cloud-side devices. One or more embodiments of this specification also involve a data compression device, a computing device, a computer-readable storage medium, and a computer program product, each of which is described in detail in the following embodiments.

[0040] Computer systems often use compression mechanisms to conserve storage resources and reduce storage costs. As user business layers control storage costs, they also begin using compression within their own business layers. Consequently, data reaches the underlying storage system already compressed. This results in the underlying layer being unaware that the user data has been compressed, leading to further compression, which wastes CPU resources. Furthermore, the compressed data must be decompressed, further wasting CPU resources without reducing storage costs.

[0041] Currently, there are two common methods for determining whether data has been compressed: first, compress the entire data and then determine whether the compression ratio is greater than a threshold. If so, the data is considered to have been compressed. However, this method of determination has already compressed the data once, which only reduces the data decompression process and still wastes a lot of CPU resources. The other method is to compress the data in a sliding window and determine whether the compression ratio of the data in the sliding window is greater than a threshold. If the compression ratio of the data in a consecutive number of sliding windows is greater than the threshold, the data is considered to have been compressed. Although this method reduces CPU resources compared to the previous method, since the data uploaded by the user may contain compressed data and uncompressed data, the error rate of using this method to determine the data is high.

[0042] See also Figure 1 , Figure 1 A flow chart of a data compression method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0043] Step 102: Receive service data to be compressed, and determine at least one service data segment to be compressed based on the service data to be compressed.

[0044] In practical applications, the business data to be compressed is business data that needs to be compressed, and the business data segment to be compressed is a segment of data in the business data to be compressed.

[0045] Specifically, the business data to be compressed can be understood as any type of data related to the business, including but not limited to, when the business is a video processing business, the business data to be compressed is video data, text data, gradient data, etc., which are data needed in the video processing business; when the business is an audio processing business, the business data to be compressed is audio data, eigenvalue data, audio processing results, etc., which are data needed to perform audio processing business.

[0046] It should be noted that the business data to be compressed is not only the data uploaded by the user or the data that the user needs to obtain, but can also be the data transmitted between devices in a distributed system, or the data generated in the same device, etc., which needs to be stored. This manual does not impose any restrictions on this.

[0047] It should be noted that the business data to be compressed can be data from multiple tasks or multiple combinations of data. Therefore, due to different data processing methods before the combination, some sections of the business data to be compressed may be compressed while other sections are not. By dividing the business data to be compressed into segments, operations can be performed on each segment of the business data to be compressed. While improving the accuracy of compression judgment, it can also compress each segment of the business data to be compressed simultaneously, thereby improving the efficiency of data compression.

[0048] The business data segments to be compressed can be understood as data segments that constitute the business data to be compressed, which are obtained after the business data to be compressed is divided. The specific method of determining the business data segments to be compressed from the business to be compressed includes but is not limited to cutting the business data to be compressed into multiple data segments of corresponding length data through a preset length, or it can be each segment composed of bytes randomly selected from the business data to be compressed, etc. This specification does not impose any restrictions on the method of cutting the business data segments to be compressed.

[0049] Taking into account the accuracy of subsequent data compression, preferably, determining at least one service data segment to be compressed based on the service data to be compressed includes:

[0050] The service data to be compressed is cut according to a preset byte length to determine at least one service data segment to be compressed.

[0051] In practical applications, the byte length is the length of the preset business data segment to be compressed, and the byte length can be understood as the way to cut the business data to be compressed.

[0052] Specifically, the byte length may be a preset length, and the method for determining the service data segment to be compressed according to the preset length is to intercept the bytes of the preset length in the service data to be compressed, and determine the intercepted data segment as the service data segment to be compressed. Figure 2 As shown, Figure 2A schematic diagram of a data compression process is provided for one embodiment of this specification. As shown in the figure, the length of the business data to be compressed is 38K, and the preset byte length is 8K. Then, 8K of the business data to be compressed is intercepted as the business data segment to be compressed. Since the remaining data after intercepting four 8K data segments is less than 8K, the length of the final business data segment to be compressed is determined to be the remaining byte length, that is, 6K. The first 3K bytes of the 8K-long business data segment to be compressed are used as the compression test data segment, and the remaining 5K bytes of the business data segment to be compressed are used as the reference data segment to be compressed, so that the compression test data segment can be compressed later. By dividing the business data to be compressed by the preset byte length as the preset length, various types of business data to be compressed can be covered, thereby improving the accuracy of the judgment of the business data to be compressed while reducing the implementation cost of the data compression method.

[0053] The byte length can also be a sequence of different lengths. The method for determining the business data to be compressed based on the preset length is to truncate the data segments of the corresponding length based on the sequence as the business data segments to be compressed. For example, if the length of the business data to be compressed is 38K and the preset byte lengths are 1, 2, 3, 5, 8, 13, and 21, then 7 business data segments to be compressed are determined, with lengths of 1K, 2K, 3K, 5K, 8K, 13K, and 6K respectively. Similarly, since the remaining bytes of the last business data to be compressed are less than the preset byte length, the length of the last business data segment to be compressed is set to the remaining byte length, that is, 6K. By segmenting the business data to be compressed according to the preset byte lengths of the sequence of different lengths, the segmentation method of the business data to be compressed can be customized according to the characteristics of each type of business data to be compressed, thereby improving the accuracy of the judgment of the business data to be compressed.

[0054] By intercepting the business data to be compressed by a preset byte length to determine each business data segment to be compressed, each segment can be judged when judging and compressing the business data to be compressed, thereby further improving the accuracy of the judgment and improving the efficiency of data compression when compressing the data.

[0055] Step 104: Determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed.

[0056] In practical applications, the uncompressed business data segment set is a set of compressible data segments, and the compressed business data segment set is a set of incompressible data segments.

[0057] Specifically, the uncompressed business data segment set can be understood as a set containing compressible data segments, the compressible data segments can be understood as data segments that have not been compressed, the uncompressed business data segment set can be understood as a set containing non-compressible data segments, and the compressible data segments can be understood as data segments that have been compressed.

[0058] By determining the data segments that can be compressed and the data segments that cannot be compressed in the business data segments to be compressed, the set of uncompressed business data segments and the set of compressed business data segments can be determined, thereby distinguishing between compressed data segments and uncompressed data segments, and reducing CPU resource consumption when compressing the data subsequently.

[0059] Considering that judging the entirety of the compressed data increases CPU resource consumption, the service data segment to be compressed includes a compression test data segment;

[0060] Furthermore, determining an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed includes:

[0061] Determining a data segment type corresponding to each of the to-be-compressed service data segments according to a compression test data segment in each of the to-be-compressed service data segments, wherein the compression test data segment is a data segment determined by a portion of bytes in the to-be-compressed service data segment;

[0062] The uncompressed service data segment set and the compressed service data segment set are determined according to the data segment type corresponding to each service data segment to be compressed.

[0063] In actual applications, the compression test data segment is a data segment in the business data segment to be compressed, and the data segment type is the type of the business data segment to be compressed.

[0064] Specifically, the compression test data segment can be understood as a data segment composed of some bytes obtained from the business data segment to be compressed, which is used to determine the type of the business data segment to be compressed. Since the overall operation of the business data segment to be compressed to determine the type of the business data segment to be compressed will cause a large CPU resource consumption, obtaining a data segment in the business data segment to be compressed as the business data segment to be compressed as a judgment feature of the type of the business data segment to be compressed can effectively determine whether the business data segment to be compressed is compressible data and reduce CPU resource consumption.

[0065] The data segment type can be understood as information representing whether the business data segment to be compressed is compressible data. Since the business data segment to be compressed includes compressible data segments and incompressible data segments, the types corresponding to each compressed data segment include uncompressed business data and compressed business data. Uncompressed business data is the type of business data segment that can be compressed, and compressed business data is the type of business data segment that cannot be compressed. A data segment of the uncompressed business data type can be understood as a data segment that has not been compressed before this data compression. A data segment of the compressed business data type can be understood as a data segment that has been compressed before this data compression.

[0066] By judging the data segment type of each service data segment to be compressed by using the compression test data segment in the service data segment to be compressed, the CPU resource consumption when judging whether the service data to be compressed is compressible data can be further reduced.

[0067] Furthermore, determining the data segment type corresponding to each of the to-be-compressed service data segments according to the compression test data segment in each of the to-be-compressed service data segments includes:

[0068] Determining a target service data segment to be compressed, wherein the target service data segment to be compressed is any one of the service data segments to be compressed;

[0069] Determine a target compression test data segment corresponding to the target service data segment to be compressed;

[0070] The data segment type corresponding to the target to-be-compressed business data segment is determined according to the target compression test data segment, wherein the data segment type includes uncompressed business data and compressed business data.

[0071] In practical applications, the target service data segment to be compressed is any service data segment to be compressed segmented from the service data to be compressed, and the target compression test data segment is a portion of bytes obtained from the target service data segment to be compressed.

[0072] Specifically, by performing target compression test data segments on each target business data segment to be compressed in the business data segment to be compressed, the data segment type of each business data segment to be compressed can be determined, thereby further reducing the CPU resource consumption when determining whether the business data to be compressed is compressible data.

[0073] Considering that both the service data to be compressed and the segments of the service data to be compressed are composed of bytes, the target service data segment to be compressed includes at least one service data segment byte data;

[0074] Further, determining a target compression test data segment corresponding to the target to-be-compressed service data segment includes:

[0075] Determining at least one target service test data segment byte data from the byte data of each service data segment corresponding to the target service data segment to be compressed, wherein the target service test data segment byte data is the byte data contained in the target compression test data segment corresponding to the target service data segment to be compressed;

[0076] The target compression test data segment is determined based on the byte data of each target service test data segment.

[0077] In actual applications, the business data segment byte data is the byte data constituting the business data segment, and the target business test data segment byte data is the byte data corresponding to the target business test segment in the target business data to be compressed.

[0078] Specifically, determining at least one target service test data segment byte data from the byte data of each service data segment corresponding to the target service data segment to be compressed can be understood as determining the byte corresponding to the target compression test data segment from each byte in the target service data segment to be compressed. In other words, the bytes corresponding to the determined target compression test data segments can be combined to form the target compression test data segment.

[0079] The method of determining the bytes corresponding to the target compression test data segment from each byte in the target to-be-compressed business data segment can be to determine the first preset number of bytes of the target to-be-compressed business data segment as the bytes corresponding to the target compression test data segment (target business test data segment byte data). The target business test data segment byte data determined in this way is combined into the target compression test data segment in a sequential combination, such as Figure 2 As shown, Figure 2 The first 3K bytes of the 8K-long business data segment to be compressed are used as the compression test data segment so that the compression test data segment can be compressed later.

[0080] The method for determining the bytes corresponding to the target compression test data segment from the bytes in the target service data segment to be compressed may also be to randomly select bytes corresponding to the target compression test data segment from the target service data segment to be compressed. The target service test data segment bytes determined in this manner may be combined to form the target compression test data segment in a random or sequential manner. This specification does not impose any limitation on the method for determining the bytes corresponding to the target compression test data segment from the bytes in the target service data segment to be compressed.

[0081] By determining the bytes corresponding to the target compression test data segment from each byte in the target business data segment to be compressed, the target compression test data segment for each target business data segment to be compressed in the business data segment to be compressed can be achieved, so that the data segment type of each business data segment to be compressed can be judged, further reducing the CPU resource consumption when judging whether the business data to be compressed is compressible data.

[0082] Furthermore, determining the uncompressed service data segment set and the compressed service data segment set according to the data segment type corresponding to each service data segment to be compressed includes:

[0083] In a case where the data segment type corresponding to the target service data segment to be compressed is uncompressed service data, inserting the target service data segment to be compressed into the uncompressed service data segment set;

[0084] In a case where the data segment type corresponding to the target service data segment to be compressed is compressed service data, the target service data segment to be compressed is inserted into the set of compressed service data segments.

[0085] In practical applications, inserting data segments of the type of uncompressed business data into a set of uncompressed business data segments, and inserting data segments of the type of compressed business data into a set of compressed business data segments, can effectively enable the computer to distinguish between compressible business data segments to be compressed and incompressible business data segments to be compressed, thereby improving the efficiency of data compression.

[0086] Further, determining the data segment type corresponding to the target to-be-compressed service data segment according to the target compression test data segment includes:

[0087] Based on a preset compression test data judgment rule, determining whether the target compression test data segment is compressible;

[0088] In a case where the target compression test data segment is compressible, determining that the data segment type corresponding to the target to-be-compressed service data segment is uncompressed service data;

[0089] In a case where the target compression test data segment is incompressible, it is determined that the data segment type corresponding to the target to-be-compressed business data segment is compressed business data.

[0090] In practical applications, the compression test data judgment rule is a rule for determining whether a compression test data segment is compressible. The compression test data judgment rule is understood to include a judgment rule for determining whether data can be compressed, including an overall judgment rule and a sliding window judgment rule. The overall judgment rule is a rule for determining whether the compression test data segment is compressed by compressing the entire segment and determining whether the compression result is normal. The sliding window judgment rule is a rule for determining whether the compression test data segment is compressible by setting a sliding window in the compression test data segment, compressing the data in the sliding window, and then determining whether the compression result of the data in the sliding window is normal.

[0091] Specifically, whether the compression test data segment in the business data segment to be compressed can be compressed represents whether the entire business data segment to be compressed can be compressed. Under the premise of ensuring the accuracy of data compression judgment, the data processing efficiency can be improved, thereby further improving the efficiency of data compression.

[0092] Further, determining whether the target compression test data segment is compressible includes:

[0093] Acquire the data length of the target compression test data segment, and determine the compression ratio corresponding to the target compression test data segment according to the data length of the target compression test data segment;

[0094] When the compression ratio is less than a preset compression threshold, determining that the target compression test data segment is compressible;

[0095] When the compression ratio is greater than or equal to a preset compression threshold, it is determined that the target compression test data segment is incompressible.

[0096] In practical applications, the compression ratio is a value indicating the degree of data compression, and the compression threshold is a value used to determine whether data is compressible.

[0097] Specifically, the method of determining the corresponding compression rate of the target compression test data segment according to the data length of the target compression test data segment can be to compress the target compression test data segment once and obtain the data length after compression, and then determine the data length before compression based on the above-mentioned data length of the target compression test data segment before compression. Thereafter, the compression rate corresponding to the target compression test data segment can be determined based on the change in the obtained data length before and after compression.

[0098] The compression ratio determined based on the data length of the target compression test data segment and the data length of the test data compression result can be understood as determining the degree of compression of the target compression test data segment after compression. When the compression ratio between the above two is greater than a preset threshold, it is considered that the target compression test data segment is incompressible, which can also be understood as the target compression test data segment being a data segment that has been compressed. Similarly, when the compression ratio between the above two is less than the threshold, it is considered that the target compression test data segment is compressible, which can also be understood as the target compression test data segment being a data segment that has not been compressed.

[0099] Through the relationship between the compression rate and the preset threshold, it is determined whether the target compression test data segment can be compressed, and then it can be realized that whether a certain data segment in the corresponding business data to be compressed can be compressed. Therefore, the business data to be compressed can be compressed only by the target compression test data segment determined in a data segment of the entire data to be compressed, thereby improving the CPU utilization rate when compressing the business data to be compressed.

[0100] Furthermore, determining the compression ratio corresponding to the target compression test data segment according to the data length of the target compression test data segment includes:

[0101] Performing compression processing on the target compressed test data segment to obtain a test data compression result corresponding to the target compressed test data segment;

[0102] Determining the data length of the target compressed test data segment as the data length before compression, and determining the data length of the test data compression result as the data length after compression;

[0103] A compression ratio is determined according to the pre-compression data length and the post-compression data length.

[0104] In practical applications, the test data compression result is the data after compression of the compressed test data, the data length before compression is the length of the original data before compression, and the data length after compression is the length of the data obtained after compression of the original data.

[0105] Specifically, there are many ways to compress data, for example, using LZ4 to compress data, using Deflate to compress data, using Brotli to compress data, etc. This specification does not impose any restrictions on this.

[0106] The compression ratio is calculated by dividing the length of the compressed data by the compressed length. Therefore, the compression ratio of compressed data can be further understood as the percentage of storage resources occupied by the compressed data compared to the original data. Therefore, the higher the compression ratio of the compressed data, the smaller the difference between the compressed data and the uncompressed data. Therefore, it can be considered that the higher the compression ratio, the less compressed the data is. When the compression ratio is greater than a preset threshold, it means that the difference between the compressed data and the original data is too small, and the original data can be further considered to be incompressible.

[0107] By compressing the target compression test data segment and then calculating the compression rate of the target compression test data segment, it is possible to determine whether the target compression test data segment is compressible based on the calculated compression rate, and then determine whether the corresponding business data segment to be compressed is compressible based on whether the target compression test data segment is compressible, thereby improving the efficiency of judging whether the business data to be compressed is compressible.

[0108] Further, determining whether the target compression test data segment is compressible includes:

[0109] Determining a sliding window of a preset size, and determining data within the sliding window as sliding window compression test data;

[0110] Performing compression processing on the sliding window compression test data to obtain a sliding window test data compression result corresponding to the sliding window compression test data;

[0111] Determining the data length of the sliding window compression test data as the data length before compression, and determining the data length of the sliding window test data compression result as the data length after compression;

[0112] Determining a compression ratio corresponding to the sliding window according to the pre-compression data length and the post-compression data length;

[0113] If the compression ratio corresponding to the sliding window is less than a preset compression threshold, continue sliding the sliding window until the sliding window slides to the end of the target compressed test data segment, generate a test data compression result corresponding to the target compressed test data segment, and determine that the target compressed test data segment is compressible;

[0114] When the compression ratio corresponding to a preset number of sliding windows is greater than or equal to a preset compression threshold, the sliding window is stopped and the target compression test data segment is determined to be incompressible.

[0115] In practical applications, the sliding window is a piece of data with a fixed length, the sliding window compression test data is the data in the sliding window, and the sliding window test data compression result is the data in the sliding window after compression.

[0116] The compression ratio determined based on the data length of the data within the sliding window (sliding window compressed test data) during the sliding window sliding process and the data length of the data after compression within the sliding window (sliding window test data compression result) can be understood as calculating the compression ratio of the data within the sliding window in real time during the sliding window sliding process. When the compression ratio of the data within the sliding window is greater than a preset threshold, it indicates that the data within the sliding window is incompressible. Similarly, when the compression ratio of the data within the sliding window is less than the threshold, it is considered that the data within the sliding window is compressible. Furthermore, when the compression ratio of the data within the sliding window is less than the threshold (i.e., the data within the sliding window is compressible), the sliding window continues to slide to continue compressing the data until it reaches the end of the target compression test data segment, completing the compression of the target compression test data segment and indicating that the target compression test data segment is compressible. Since during the data compression process, some data segments may be difficult to compress, but the data as a whole can be compressed, the target compression test data segment is considered incompressible only when the compression ratio of the data within the preset sliding window is greater than or equal to the threshold for consecutive times, and the sliding of the sliding window is stopped.

[0117] By setting a sliding window on the target compression test data segment, compressing the data in the sliding window during the sliding process of the sliding window and then calculating the compression rate of the data in the sliding window, it can be determined that the target compression test data segment cannot be compressed when the data compression rate in the sliding windows is higher than the threshold for a preset number of consecutive occurrences, thereby further improving the efficiency of judging whether the compressed business data is compressible, and by stopping the sliding operation on the target compression test data segment when the data compression rate in the sliding windows is higher than the threshold for a preset number of consecutive occurrences, the CPU resource consumption is further reduced.

[0118] Step 106: When there is at least one uncompressed business data segment in the uncompressed business data segment set, compression processing is performed on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment.

[0119] In practical applications, the process of performing compression processing on the uncompressed business data segment can be any method of compressing data, which will not be described in detail here. Considering that when determining whether the target compression test data segment is compressible data, the compressible target compression test data segment has already been compressed, the method of performing compression processing on the uncompressed business data segment is preferably the same as the compression method for compressing the target compression test data segment mentioned above, so that the compression result of the business data segment corresponding to the uncompressed business data segment can be obtained by only performing compression once on the uncompressed business data segment excluding the target compression test data segment, thereby further reducing CPU resource consumption.

[0120] Considering that when determining whether the target compression test data segment is compressible data, the compressible target compression test data segment needs to be compressed, the uncompressed service data segment includes the compression test data segment and the reference service data segment to be compressed;

[0121] Furthermore, compression processing is performed on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment, including:

[0122] Determining a target uncompressed service data segment, wherein the target uncompressed service data segment is any one of the uncompressed service data segments;

[0123] Obtaining a test data compression result corresponding to a compressed test data segment in the target uncompressed service data segment;

[0124] performing compression processing on a reference to-be-compressed data segment in the target uncompressed service data segment to obtain a reference to-be-compressed data compression result corresponding to the reference to-be-compressed data segment, wherein the reference to-be-compressed data segment is used to be combined with the compression test data segment to obtain a target uncompressed service data segment;

[0125] The compression result of the service data segment corresponding to the target uncompressed service data segment is determined by combining the test data compression result and the reference to-be-compressed data compression result.

[0126] In actual applications, the reference data segment to be compressed is the data segment remaining after removing the compression test data segment from the data segment to be compressed, the target uncompressed business data segment is any uncompressed business data segment in the uncompressed business data segment set, and the reference data compression result to be compressed is the data after compression processing of the reference data segment to be compressed in the target uncompressed business data segment.

[0127] Specifically, the reference data segment to be compressed can be understood as the data segment to be compressed, after removing the compression test data segment used to determine whether the data segment is an uncompressed data segment, and the remaining byte group data segment. The method of compressing the reference data segment to be compressed can be any data compression method, which will not be described in detail here. Figure 2 As shown in the figure, the first 3K of the 8K-long business data segment to be compressed is a compression test data segment, and the last 5K of data is a reference data segment to be compressed.

[0128] Based on the same reasons as when performing compression on the above-mentioned uncompressed business data segment, the method of performing compression processing on the reference data segment to be compressed is preferably the same as the compression method for compressing the target compressed test data segment mentioned above, so that the reference data segment to be compressed only needs to be compressed once, and then it can be spliced ​​with the compression result of the test data that has been compressed during the previous judgment to obtain the compression result of the business data segment corresponding to the reference data segment to be compressed, thereby further reducing CPU resource consumption.

[0129] Step 108: Determine the service data compression result corresponding to the service data to be compressed according to the compression result of each service data segment and the set of compressed service data segments.

[0130] In actual applications, since the data segments in the compressed business data segment set are already compressed data segments, when determining the business data compression results corresponding to the business data to be compressed, the data segments in the compressed business data segment set need to be compressed again. It is only necessary to sequentially splice the compressed business data segment compression results with the data segments in the compressed business data segment set to complete the compression task of the business data to be compressed.

[0131] By compressing only the uncompressed data segments in the service data to be compressed, the problem of CPU resource waste caused by re-compressing already compressed data can be avoided when compressing the service data to be compressed. In addition, since part of the data in the uncompressed data segment has already been compressed when judging each data segment, when compressing the data after the judgment is completed, only the data in each data segment excluding the data used for judgment can be compressed, further reducing CPU resource consumption during data compression.

[0132] Furthermore, determining the service data compression result corresponding to the service data to be compressed according to the compression result of each service data segment and the set of compressed service data segments includes:

[0133] If there is at least one compressed service data segment in the set of compressed service data segments, determining, based on the service data to be compressed, compression results of each service data segment and sequence information of each compressed service data segment, and concatenating the compression results of each service data segment and each compressed service data segment based on the sequence information to determine a service data compression result corresponding to the service data to be compressed;

[0134] In the case that there is no compressed business data segment in the compressed business data segment set, the sequence information of the compression results of each business data segment is determined based on the business data to be compressed, and the compression results of each business data segment are spliced ​​based on the sequence information to determine the business data compression result corresponding to the business data to be compressed.

[0135] In practical applications, the sequence information represents the order of the individual service data segments to be compressed within the service data to be compressed. Specifically, determining the compression results of the individual service data segments and the sequence information of the compressed service data segments can be understood as determining, based on the pre-segmentation service data to be compressed, the positions of the corresponding service data segments to be compressed in the service data to be compressed, and determining the positions of the compressed service data segments in the service data to be compressed.

[0136] In one embodiment provided herein, the business data to be compressed is data 1, having the structure [ABC]. After segmentation, business data segments A, B, and C are obtained. The position information for each segment is: 0 for business data segment A, 1 for business data segment B, and 2 for business data segment C. Subsequently, it is determined that business data segments A and C are uncompressed data segments and are inserted into a set of uncompressed data segments. Business data segment B is determined to be a compressed data segment and is inserted into a set of compressed data segments. Business data segments A and C in the set of uncompressed data segments are then compressed into business data segment compression results A' and C', respectively. Based on the position information of each data segment obtained when segmenting the business data to be compressed, the position information for business data segment compression result A' is determined to be 0, the position information for compressed data segment B is determined to be 1, and the position information for business data segment compression result C' is determined to be 2. Finally, the business data compression result determined by splicing the position information of each data segment is [A`BC`].

[0137] By compressing only the uncompressed business data segments, compressing the uncompressed data in the business data to be compressed and concatenating it with the compressed data to generate the business data compression result, repeated compression of the compressed data is avoided, thereby reducing CPU resource consumption.

[0138] Furthermore, after determining the uncompressed service data segment set and the compressed service data segment set from each service data segment to be compressed, the method further includes:

[0139] In a case where no uncompressed service data segment exists in the uncompressed service data segment set, it is determined that the service data to be compressed is a service data compression result corresponding to the service data to be compressed.

[0140] Taking into account that the business data to be compressed may have all been compressed, and in the case that the business data to be compressed may have all been compressed, there are no uncompressed business data segments in the business data segments to be compressed divided from the business data to be compressed, so in the above case, the business data to be compressed can be directly regarded as the compression result of the business data that has been compressed.

[0141] By judging the compression degree of each data segment and directly treating the data that does not contain uncompressed data segments as the data compression result, repeated compression of compressed data is avoided, thereby reducing CPU resource consumption.

[0142] By applying the solution of the embodiments of this specification, by segmenting the service data to be compressed into at least one service data segment and determining whether the data segment has been compressed based on the compression test data segment within the data segment, the accuracy of the judgment on whether the data has been compressed can be improved while reducing CPU resource consumption. Subsequently, by compressing only the uncompressed service data segment, repeated compression of already compressed data is avoided, further reducing CPU resource consumption.

[0143] See also Figure 3 , Figure 3 A flowchart of a data compression method applied to a cloud-side device according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0144] Step 302: Receive service data to be compressed from a terminal-side device, and determine at least one service data segment to be compressed based on the service data to be compressed.

[0145] Step 304: Determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed.

[0146] Step 306: When there is at least one uncompressed business data segment in the uncompressed business data segment set, compression processing is performed on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment.

[0147] Step 308: Determine the service data compression result corresponding to the service data to be compressed based on the compression result of each service data segment and the set of compressed service data segments, and return the service data compression result to the terminal device.

[0148] The above is a schematic diagram of a data compression method for cloud-side devices according to this embodiment. It should be noted that the technical solution of this data compression method for cloud-side devices is based on the same concept as the technical solution of the aforementioned data compression method. For details not described in detail in the technical solution of the data compression method for cloud-side devices, please refer to the description of the technical solution of the aforementioned data compression method.

[0149] By applying the solutions of the embodiments of this specification, the cloud-side device segments the service data to be compressed into at least one service data segment and determines whether the data segment has been compressed based on the compression test data segment within the data segment. This improves the accuracy of determining whether the data sent by the end-side device has been compressed while reducing CPU resource consumption on the cloud-side device. Subsequently, by compressing only the uncompressed service data segments, repeated compression of already compressed data is avoided, further reducing CPU resource consumption on the cloud-side device.

[0150] See also Figure 4 , Figure 4 1 shows an architecture diagram of a data compression system provided by an embodiment of the present specification. The data compression system may include a client 100 and a server 200.

[0151] The client 100 is used to send the service data to be compressed to the server 200;

[0152] The server 200 is configured to receive the service data to be compressed sent by the client 100, and determine at least one service data segment to be compressed based on the service data to be compressed; determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed; if at least one uncompressed service data segment exists in the uncompressed service data segment set, perform compression processing on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment; determine a service data compression result corresponding to the service data to be compressed based on the compression result of each service data segment and the compressed service data segment set; and send the service data compression result to the client 100;

[0153] The client 100 is further configured to receive the service data compression result sent by the server 200 .

[0154] Using the solutions of the embodiments of this specification, the server segmented the service data to be compressed into at least one service data segment and determined whether the data segment was compressed based on the compression test data segment within the data segment. This improved the accuracy of determining whether the data sent by the client was compressed while reducing server CPU resource consumption. Subsequently, by compressing only the uncompressed service data segments, repeated compression of already compressed data was avoided, further reducing server CPU resource consumption.

[0155] The data compression system may include multiple clients 100 and a server 200. The clients 100 may be referred to as end-side devices, and the server 200 may be referred to as cloud-side devices. Multiple clients 100 may establish communication connections through the server 200. In a data compression scenario, the server 200 provides data compression services between the multiple clients 100. The multiple clients 100 may act as either senders or receivers, communicating through the server 200.

[0156] Users can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100. In the data compression scenario, users can publish data streams to the server 200 through the client 100. The server 200 generates business data compression results based on the data stream and pushes the business data compression results to other clients with which communication has been established.

[0157] The client 100 and the server 200 are connected via a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, and other processing before being released to the server 200.

[0158] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be based on the software development kit (SDK) of the corresponding service provided by the server 200, such as developed based on the real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0159] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0160] It is worth noting that the data compression methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server and thus execute the data compression methods provided in the embodiments of this specification. In other embodiments, the data compression methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0161] The following combined Figure 5 , taking the application of the data compression method provided in this specification in storing user uploaded data as an example, the data compression method is further explained. Figure 5A flowchart of a processing process of a method for storing user uploaded data provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0162] Step 502: Receive the data uploaded by the user that needs to be stored: data 1 with a length of 23K.

[0163] Step 504: Split the data 1 with a length of 23K into a data segment 1 with a length of 8K, a data segment 2 with a length of 8K, and a data segment 3 with a length of 7K according to the length of 8K.

[0164] Step 506: Compress the first 1K of data in data segment 1, data segment 2, and data segment 3 respectively.

[0165] Step 508: Divide the compressed data length by the uncompressed data length to calculate: the compression rate of the first 1K data of data segment 1 is 68%, the compression rate of the first 1K data of data segment 2 is 98%, and the compression rate of the first 1K data of data segment 3 is 35%.

[0166] Step 510: Based on the preset data compression rate threshold of 95%, it can be known that data segment 1 and data segment 3 are uncompressed data segments, and data segment 2 is a compressed data segment.

[0167] Step 512: Compress the last 7K of data in data segment 1, and concatenate the result of this compression with the compression result of the first 1K of data that has been compressed before to determine the compression result of data segment 1, and compress the last 6K of data in data segment 3, and concatenate the result of this compression with the compression result of the first 1K of data that has been compressed before to determine the compression result of data segment 3.

[0168] Step 514: Concatenate the compression results of data segment 1, data segment 2, and data segment 3 to obtain the compression result of data 1 uploaded by the user.

[0169] Step 516: Store the compression result of data 1 in a storage device.

[0170] The solution of the embodiments of this specification segments the user-uploaded business data to be compressed into at least one business data segment and determines whether the data segment has been compressed using a compression test data segment within the data segment. This improves the accuracy of determining whether the data has been compressed while reducing CPU resource consumption during data storage. Subsequently, by compressing only the uncompressed business data segments, repeated compression of already compressed data is avoided, further reducing CPU resource consumption during data storage while ensuring that less space is occupied on the storage device.

[0171] Corresponding to the above method embodiment, this specification also provides a data compression device embodiment, Figure 6 FIG1 shows a schematic diagram of the structure of a data compression device provided by an embodiment of this specification. Figure 6 As shown, the device includes:

[0172] The data receiving module 602 is configured to receive the service data to be compressed and determine at least one service data segment to be compressed based on the service data to be compressed;

[0173] A set determining module 604 is configured to determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed;

[0174] The data compression module 606 is configured to, when there is at least one uncompressed business data segment in the uncompressed business data segment set, perform compression processing on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment;

[0175] The result determination module 608 is configured to determine the business data compression result corresponding to the business data to be compressed according to the compression result of each business data segment and the set of compressed business data segments.

[0176] Optionally, the data receiving module 602 is further configured to:

[0177] The service data to be compressed is cut according to a preset byte length to determine at least one service data segment to be compressed.

[0178] Optionally, the service data segment to be compressed includes a compression test data segment;

[0179] The set determination module 604 is further configured to:

[0180] Determining a data segment type corresponding to each of the to-be-compressed service data segments according to a compression test data segment in each of the to-be-compressed service data segments, wherein the compression test data segment is a data segment determined by a portion of bytes in the to-be-compressed service data segment;

[0181] The uncompressed service data segment set and the compressed service data segment set are determined according to the data segment type corresponding to each service data segment to be compressed.

[0182] Optionally, the set determination module 604 is further configured to:

[0183] Determining a target service data segment to be compressed, wherein the target service data segment to be compressed is any one of the service data segments to be compressed;

[0184] Determine a target compression test data segment corresponding to the target service data segment to be compressed;

[0185] The data segment type corresponding to the target to-be-compressed business data segment is determined according to the target compression test data segment, wherein the data segment type includes uncompressed business data and compressed business data.

[0186] Optionally, the target service data segment to be compressed includes at least one service data segment byte data;

[0187] The set determination module 604 is further configured to:

[0188] Determining at least one target service test data segment byte data from the byte data of each service data segment corresponding to the target service data segment to be compressed, wherein the target service test data segment byte data is the byte data contained in the target compression test data segment corresponding to the target service data segment to be compressed;

[0189] The target compression test data segment is determined based on the byte data of each target service test data segment.

[0190] Optionally, the set determination module 604 is further configured to:

[0191] In a case where the data segment type corresponding to the target service data segment to be compressed is uncompressed service data, inserting the target service data segment to be compressed into the uncompressed service data segment set;

[0192] In a case where the data segment type corresponding to the target service data segment to be compressed is compressed service data, the target service data segment to be compressed is inserted into the set of compressed service data segments.

[0193] Optionally, the set determination module 604 is further configured to:

[0194] Based on a preset compression test data judgment rule, determining whether the target compression test data segment is compressible;

[0195] In a case where the target compression test data segment is compressible, determining that the data segment type corresponding to the target to-be-compressed service data segment is uncompressed service data;

[0196] In a case where the target compression test data segment is incompressible, it is determined that the data segment type corresponding to the target to-be-compressed business data segment is compressed business data.

[0197] Optionally, the set determination module 604 is further configured to:

[0198] Acquire the data length of the target compression test data segment, and determine the compression ratio corresponding to the target compression test data segment according to the data length of the target compression test data segment;

[0199] When the compression ratio is less than a preset compression threshold, determining that the target compression test data segment is compressible;

[0200] When the compression ratio is greater than or equal to a preset compression threshold, it is determined that the target compression test data segment is incompressible.

[0201] Optionally, the set determination module 604 is further configured to:

[0202] Performing compression processing on the target compressed test data segment to obtain a test data compression result corresponding to the target compressed test data segment;

[0203] Determining the data length of the target compressed test data segment as the data length before compression, and determining the data length of the test data compression result as the data length after compression;

[0204] A compression ratio is determined according to the pre-compression data length and the post-compression data length.

[0205] Optionally, the set determination module 604 is further configured to:

[0206] Determining a sliding window of a preset size, and determining data within the sliding window as sliding window compression test data;

[0207] Performing compression processing on the sliding window compression test data to obtain a sliding window test data compression result corresponding to the sliding window compression test data;

[0208] Determining the data length of the sliding window compression test data as the data length before compression, and determining the data length of the sliding window test data compression result as the data length after compression;

[0209] Determining a compression ratio corresponding to the sliding window according to the pre-compression data length and the post-compression data length;

[0210] If the compression ratio corresponding to the sliding window is less than a preset compression threshold, continue sliding the sliding window until the sliding window slides to the end of the target compressed test data segment, generate a test data compression result corresponding to the target compressed test data segment, and determine that the target compressed test data segment is compressible;

[0211] When the compression ratio corresponding to a preset number of sliding windows is greater than or equal to a preset compression threshold, the sliding window is stopped and the target compression test data segment is determined to be incompressible.

[0212] Optionally, the uncompressed service data segment includes a compressed test data segment and a reference data segment to be compressed;

[0213] The data compression module 606 is further configured to:

[0214] Determining a target uncompressed service data segment, wherein the target uncompressed service data segment is any one of the uncompressed service data segments;

[0215] Obtaining a test data compression result corresponding to a compressed test data segment in the target uncompressed service data segment;

[0216] performing compression processing on a reference to-be-compressed data segment in the target uncompressed service data segment to obtain a reference to-be-compressed data compression result corresponding to the reference to-be-compressed data segment, wherein the reference to-be-compressed data segment is used to be combined with the compression test data segment to obtain a target uncompressed service data segment;

[0217] The compression result of the service data segment corresponding to the target uncompressed service data segment is determined by combining the test data compression result and the reference to-be-compressed data compression result.

[0218] Optionally, the result determination module 608 is further configured to:

[0219] If there is at least one compressed service data segment in the set of compressed service data segments, determining, based on the service data to be compressed, compression results of each service data segment and sequence information of each compressed service data segment, and concatenating the compression results of each service data segment and each compressed service data segment based on the sequence information to determine a service data compression result corresponding to the service data to be compressed;

[0220] In the case that there is no compressed business data segment in the compressed business data segment set, the sequence information of the compression results of each business data segment is determined based on the business data to be compressed, and the compression results of each business data segment are spliced ​​based on the sequence information to determine the business data compression result corresponding to the business data to be compressed.

[0221] Optionally, the data compression device further includes a data transfer module configured to:

[0222] In a case where no uncompressed service data segment exists in the uncompressed service data segment set, it is determined that the service data to be compressed is a service data compression result corresponding to the service data to be compressed.

[0223] Using the solutions of the embodiments of this specification, a data receiving module in a data compression device segments the service data to be compressed into at least one service data segment, and determines whether the data segment has been compressed based on a compression test data segment within the data segment. This improves the accuracy of determining whether the data has been compressed while reducing CPU resource consumption in the data compression device. Subsequently, by compressing only the uncompressed service data segments, repeated compression of already compressed data is avoided, further reducing CPU resource consumption in the data compression device.

[0224] The above is a schematic diagram of a data compression device according to this embodiment. It should be noted that the technical solution of the data compression device and the technical solution of the above-mentioned data compression method are of the same concept. For details not described in detail in the technical solution of the data compression device, please refer to the description of the technical solution of the above-mentioned data compression method.

[0225] Figure 7 7 shows a block diagram of a computing device 700 according to one embodiment of the present disclosure. Components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0226] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0227] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 7 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0228] Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 700 may also be a mobile or stationary server.

[0229] Among them, the processor 720 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the above-mentioned data compression method and the steps of the data compression method applied to the cloud-side device.

[0230] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device is based on the same concept as the aforementioned data compression method and the technical solution of the data compression method applied to the cloud-side device. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data compression method and the data compression method applied to the cloud-side device.

[0231] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-mentioned data compression method and the steps of the data compression method applied to cloud-side devices.

[0232] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the aforementioned data compression method and the technical solution of the data compression method applied to cloud-side devices. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data compression method and the technical solution of the data compression method applied to cloud-side devices.

[0233] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned data compression method and the steps of the data compression method applied to a cloud-side device.

[0234] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of this computer program and the technical scheme of the data compression method and the data compression method applied to the cloud-side device described above are based on the same concept. For details not described in detail in the technical scheme of the computer program, please refer to the description of the technical scheme of the data compression method and the data compression method applied to the cloud-side device described above.

[0235] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0236] The computer instructions include computer program codes, which may be in source code form, object code form, executable files, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0237] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0238] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0239] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data compression method, comprising: receiving service data to be compressed, and determining at least one service data segment to be compressed based on the service data to be compressed; Determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed; When there is at least one uncompressed service data segment in the uncompressed service data segment set, performing compression processing on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment; The business data compression result corresponding to the business data to be compressed is determined according to the compression result of each business data segment and the set of compressed business data segments.

2. The method according to claim 1, wherein the service data segment to be compressed comprises a compression test data segment; Determining an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed includes: Determining a data segment type corresponding to each of the to-be-compressed service data segments according to a compression test data segment in each of the to-be-compressed service data segments, wherein the compression test data segment is a data segment determined by a portion of bytes in the to-be-compressed service data segment; The uncompressed service data segment set and the compressed service data segment set are determined according to the data segment type corresponding to each service data segment to be compressed.

3. The method according to claim 2, wherein determining the data segment type corresponding to each of the service data segments to be compressed based on the compression test data segment in each of the service data segments to be compressed comprises: Determining a target service data segment to be compressed, wherein the target service data segment to be compressed is any one of the service data segments to be compressed; Determine a target compression test data segment corresponding to the target service data segment to be compressed; The data segment type corresponding to the target to-be-compressed business data segment is determined according to the target compression test data segment, wherein the data segment type includes uncompressed business data and compressed business data.

4. The method according to claim 3, wherein the target service data segment to be compressed comprises at least one service data segment byte data; Determining a target compression test data segment corresponding to the target to-be-compressed service data segment includes: Determining at least one target service test data segment byte data from the byte data of each service data segment corresponding to the target service data segment to be compressed, wherein the target service test data segment byte data is the byte data contained in the target compression test data segment corresponding to the target service data segment to be compressed; The target compression test data segment is determined based on the byte data of each target service test data segment.

5. The method according to claim 3, wherein determining the set of uncompressed service data segments and the set of compressed service data segments according to the data segment type corresponding to each service data segment to be compressed comprises: In a case where the data segment type corresponding to the target service data segment to be compressed is uncompressed service data, inserting the target service data segment to be compressed into the uncompressed service data segment set; In a case where the data segment type corresponding to the target service data segment to be compressed is compressed service data, the target service data segment to be compressed is inserted into the set of compressed service data segments.

6. The method according to claim 3, wherein determining the data segment type corresponding to the target service data segment to be compressed according to the target compression test data segment comprises: Based on a preset compression test data judgment rule, determining whether the target compression test data segment is compressible; In a case where the target compression test data segment is compressible, determining that the data segment type corresponding to the target to-be-compressed service data segment is uncompressed service data; In a case where the target compression test data segment is incompressible, it is determined that the data segment type corresponding to the target to-be-compressed business data segment is compressed business data.

7. The method according to claim 6, wherein determining whether the target compression test data segment is compressible comprises: Acquire the data length of the target compression test data segment, and determine the compression ratio corresponding to the target compression test data segment according to the data length of the target compression test data segment; When the compression ratio is less than a preset compression threshold, determining that the target compression test data segment is compressible; When the compression ratio is greater than or equal to a preset compression threshold, it is determined that the target compression test data segment is incompressible.

8. The method according to claim 7, wherein determining the compression ratio corresponding to the target compression test data segment according to the data length of the target compression test data segment comprises: Performing compression processing on the target compressed test data segment to obtain a test data compression result corresponding to the target compressed test data segment; Determining the data length of the target compressed test data segment as the data length before compression, and determining the data length of the test data compression result as the data length after compression; A compression ratio is determined according to the pre-compression data length and the post-compression data length.

9. The method according to claim 6, wherein determining whether the target compression test data segment is compressible comprises: Determining a sliding window of a preset size, and determining data within the sliding window as sliding window compression test data; Performing compression processing on the sliding window compression test data to obtain a sliding window test data compression result corresponding to the sliding window compression test data; Determining the data length of the sliding window compression test data as the data length before compression, and determining the data length of the sliding window test data compression result as the data length after compression; Determining a compression ratio corresponding to the sliding window according to the pre-compression data length and the post-compression data length; If the compression ratio corresponding to the sliding window is less than a preset compression threshold, continue sliding the sliding window until the sliding window slides to the end of the target compressed test data segment, generate a test data compression result corresponding to the target compressed test data segment, and determine that the target compressed test data segment is compressible; When the compression ratio corresponding to a preset number of sliding windows is greater than or equal to a preset compression threshold, the sliding window is stopped and the target compression test data segment is determined to be incompressible.

10. The method according to claim 1, wherein the uncompressed service data segment comprises a compressed test data segment and a reference data segment to be compressed; Performing compression processing on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment includes: Determining a target uncompressed service data segment, wherein the target uncompressed service data segment is any one of the uncompressed service data segments; Obtaining a test data compression result corresponding to a compressed test data segment in the target uncompressed service data segment; performing compression processing on a reference to-be-compressed data segment in the target uncompressed service data segment to obtain a reference to-be-compressed data compression result corresponding to the reference to-be-compressed data segment, wherein the reference to-be-compressed data segment is used to be combined with the compression test data segment to obtain a target uncompressed service data segment; The compression result of the service data segment corresponding to the target uncompressed service data segment is determined by combining the test data compression result and the reference to-be-compressed data compression result.

11. The method according to claim 1, wherein determining at least one service data segment to be compressed based on the service data to be compressed comprises: The service data to be compressed is cut according to a preset byte length to determine at least one service data segment to be compressed.

12. The method according to claim 1, wherein determining the service data compression result corresponding to the service data to be compressed based on the compression result of each service data segment and the set of compressed service data segments comprises: If there is at least one compressed service data segment in the set of compressed service data segments, determining, based on the service data to be compressed, compression results of each service data segment and sequence information of each compressed service data segment, and concatenating the compression results of each service data segment and each compressed service data segment based on the sequence information to determine a service data compression result corresponding to the service data to be compressed; In the case that there is no compressed business data segment in the compressed business data segment set, the sequence information of the compression results of each business data segment is determined based on the business data to be compressed, and the compression results of each business data segment are spliced ​​based on the sequence information to determine the business data compression result corresponding to the business data to be compressed.

13. The method according to claim 1, after determining the uncompressed service data segment set and the compressed service data segment set from each service data segment to be compressed, the method further comprises: In a case where no uncompressed service data segment exists in the uncompressed service data segment set, it is determined that the service data to be compressed is a service data compression result corresponding to the service data to be compressed.

14. A data compression method, applied to a cloud-side device, comprising: receiving service data to be compressed from a terminal-side device, and determining at least one service data segment to be compressed based on the service data to be compressed; Determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed; When there is at least one uncompressed service data segment in the uncompressed service data segment set, performing compression processing on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment; According to the compression results of each business data segment and the set of compressed business data segments, the business data compression result corresponding to the business data to be compressed is determined, and the business data compression result is returned to the end-side device.

15. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 13 or 14 are implemented.

16. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13 or 14.

17. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13 or 14.

Images