Data compression

Abstract

Provided in the present application are data compression methods, a computing device, a computer-readable storage medium and a computer program product. A data compression method comprises: receiving service data to be compressed and, on the basis of said service data, determining at least one service data segment to be compressed; determining from among each of said service data segment an uncompressed service data segment set and a compressed service data segment set; when at least one uncompressed service data segment is present in the uncompressed service data segment set, compressing each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment; and, on the basis of each service data segment compression result and the compressed service data segment set, determining a service data compression result corresponding to said service data.

Description

[0001] Data compression technology field

[0002]

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to data compression.

[0003]

[0002] With the widespread application of distributed storage systems, in order to save storage resources, the system often reduces storage costs by introducing a compression mechanism.

[0004]

[0003] However, in order to control storage costs, user service layers have also begun to use compression technology in their own service layers, resulting in data being compressed before reaching the underlying system. Although this approach appears to improve efficiency, it also brings new problems. Because the system attempts to re-compress the compressed data, this not only wastes CPU resources but also requires decompression operations, increasing CPU resource consumption. Therefore, there is a need for a data compression method that can effectively identify compressed data and avoid repeated compression.

[0005] In view of this, embodiments of this specification provide a data compression method, a data compression method applied to a Yunjia 1 device. One or more embodiments of this specification also relate to a data compression device, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0006]

[0005] According to a first aspect of an embodiment of this specification, a data compression method is provided, comprising: receiving business data to be compressed, and determining at least one business data segment to be compressed based on the business data to be compressed; determining a set of uncompressed business data segments and a set of compressed business data segments from each business data segment to be compressed; when there is at least one uncompressed business data segment in the set of uncompressed business data segments, performing compression processing on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment; and determining a business data compression result corresponding to the business data to be compressed based on the compression result of each business data segment and the set of compressed business data segments.

[0007]

[0006] According to the second aspect of the embodiments of this specification, a data compression method applied to a Yunjia 1 device is provided, comprising: receiving business data to be compressed from an end-user device, and determining at least one business data segment to be compressed based on the business data to be compressed; determining an uncompressed business data segment set and a compressed business data segment set from each business data segment to be compressed; when there is at least one uncompressed business data segment in the uncompressed business data segment set, performing compression processing on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment; determining a business data compression result corresponding to the business data to be compressed based on the compression result of each business data segment and the compressed business data segment set, and returning the business data compression result to the end-user device.

[0008]

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above-mentioned data compression method are implemented.

[0009]

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, storing computer-executable instructions. When the instructions are executed by a processor, the steps of the above-mentioned data compression method are implemented.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the above-mentioned data compression method are implemented.

[0010]

[0010] An embodiment of the present specification implements receiving business data to be compressed, and determining at least one business data segment to be compressed based on the business data to be compressed; determining an uncompressed business data segment set and a compressed business data segment set from each business data segment to be compressed; when there is at least one uncompressed business data segment in the uncompressed business data segment set, performing compression processing on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment; and determining a business data compression result corresponding to the business data to be compressed based on the compression result of each business data segment and the compressed business data segment set.

[0011]

[0011] By applying the solution of the embodiments of this specification, by dividing the data to be compressed into at least one business data segment and determining a set of uncompressed business data segments in the data segment, it is possible to identify the uncompressed business data when some data in the entire data has been compressed, and subsequently compress the uncompressed data in the data segment without compressing the compressed data segment, thereby reducing the problem of CPU resource consumption caused by repeated compression of the compressed data. In addition, when avoiding CPU resource consumption caused by repeated compression, it is also avoided that when some data in the entire data has been compressed, the entire data segment is judged as compressed and the judgment error rate is high, which is caused by the failure to identify the uncompressed data in the data segment when judging whether the data is compressed. This reduces CPU resource consumption while ensuring data compression accuracy.

[0012] FIG1 is a flow chart of a data compression method provided by one embodiment of this specification;

[0013] FIG2 is a schematic diagram of a data compression process provided by one embodiment of this specification;

[0014]

[0014] FIG3 is a flow chart of a data compression method applied to a Yunjia 1 device provided in one embodiment of this specification;

[0015] FIG4 is an architecture diagram of a data compression system provided in accordance with an embodiment of the present disclosure;

[0016]

[0016] FIG5 is a flowchart of a method for storing user uploaded data provided in one embodiment of this specification;

[0017] FIG6 is a schematic diagram of the structure of a data compression device provided in one embodiment of this specification;

[0018]

[0018] FIG7 is a block diagram of a computing device according to an embodiment of the present specification.

[0019] In the following description, a lot of details are described to fully understand this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can do similar promotions without violating the connotation of this specification, so this specification is not subject to the limitation of the specific implementation disclosed below.

[0020] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments and are not intended to limit one or more embodiments of this specification. The singular forms "a", "the", and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms "first," "second," and the like may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of this specification. Depending on the context, the term "if" as used herein may be interpreted as "at the time," "when," or "in response to determining."

[0022]

[0022] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0023] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, typically including hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a foundation model. Pre-training a large model with large-scale unlabeled corpus produces a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities. Examples include large language models (LLMs) and multi-modal pre-training models.

[0024]

[0024] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in fields such as natural language processing (NLP) and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0025]

[0025] First, the terms involved in one or more embodiments of this specification are explained.

[0026] LZ4: This is a compression algorithm that prioritizes speed, providing very high compression and decompression speeds while maintaining a reasonable compression ratio. It is primarily suitable for scenarios where fast data compression and decompression are required, especially when processing large amounts of data. The algorithm reduces the size of the data by finding repeated strings in the data, and is a lossless compression technique.

[0027] Deflate: This compression method combines the LZ77 algorithm and Huffman coding. It provides a good balance between compression efficiency and speed and is applicable to a variety of file types. Deflate is a widely supported compression format, commonly used to compress data in web transmissions to reduce transmission time and bandwidth consumption.

[0028] Brotli: This is a compression algorithm designed to optimize network transmission, providing a higher compression ratio than other common compression methods. It achieves better compression results by using more complex compression techniques while maintaining a reasonable decompression speed. Brotli is particularly suitable for compressing text data, such as web pages, scripts, and style sheets.

[0029] This specification provides a data compression method and a data compression method applied to a Yunjia 1 device. One or more embodiments of this specification also relate to a data compression apparatus, a computing device, a computer-readable storage medium, and a computer program product, each of which is described in detail in the following embodiments.

[0029]

[0030] To conserve storage resources, computer systems often introduce compression mechanisms to reduce storage costs. As user business layers control storage costs, they also begin using compression within their own business layers. Consequently, data arrives at the underlying storage system already compressed. This results in the underlying storage system being unaware that the user data has been compressed, requiring further compression. This wastes CPU resources, and the compressed data must also be decompressed, further wasting CPU resources without reducing storage costs.

[0030]

[0031] Currently, the following two methods are commonly used to determine whether data has been compressed: After compressing the entire data, determine whether the compression ratio is greater than a threshold. If it is greater than the threshold, the data is considered to have been compressed. However, this judgment method has already compressed the data once, which only reduces the data decompression process, and there is still a large waste of CPU resources; the other method is to compress the data in the sliding window and determine whether the compression ratio of the data in the sliding window is greater than a threshold. If the compression ratio of the data in a consecutive number of sliding windows is greater than the threshold, the data is considered to have been compressed. Although this method reduces CPU resources compared to the previous method, because the data uploaded by the user may contain compressed data and uncompressed data, the error rate of using this method to judge the data is high.

[0031]

[0032] 1, which shows a flow chart of a data compression method provided according to an embodiment of this specification, specifically including the following steps.

[0032]

[0033] Step 102: Receive service data to be compressed, and determine at least one service data segment to be compressed based on the service data to be compressed.

[0033]

[0034] In practical applications, the business data to be compressed is business data that needs to be compressed, and the business data segment to be compressed is a segment of data in the business data to be compressed.

[0034]

[0035] Specifically, the business data to be compressed can be understood as any type of data related to the business, including but not limited to, when the business is a video processing business, the business data to be compressed is video data, text data, gradient data, etc., which are data needed in the video processing business; when the business is an audio processing business, the business data to be compressed is audio data, eigenvalue data, audio processing results, etc., which are data needed to perform the audio processing business.

[0035]

[0036] It should be noted that the service data to be compressed is not only data uploaded by users or data that users need to obtain, but can also be data transmitted between devices in a distributed system, or data generated in the same device, etc., which need to be stored. This specification does not impose any restrictions on this.

[0036]

[0037] It's important to note that the business data to be compressed can be the result of multiple tasks or multiple combinations of data. Therefore, due to differences in data processing before combination, some segments of the business data to be compressed may be compressed while others remain uncompressed. By segmenting the business data to be compressed into segments, operations can be performed on each segment. This improves compression judgment accuracy while also allowing for simultaneous compression of each segment, thereby increasing data compression efficiency.

[0038] The business data segments to be compressed can be understood as data segments that constitute the business data to be compressed, which are obtained after the business data to be compressed is divided. The specific method of determining the business data segments to be compressed from the business to be compressed includes but is not limited to using a preset length to cut the business data to be compressed into multiple data segments of corresponding length data, or it can be each segment composed of bytes randomly selected from the business data to be compressed, etc. This specification does not impose any restrictions on the method of cutting the business data segments to be compressed.

[0037]

[0039] Considering the accuracy of subsequent data compression, preferably, determining at least one service data segment to be compressed based on the service data to be compressed includes: cutting the service data to be compressed according to a preset byte length to determine at least one service data segment to be compressed.

[0038]

[0040] In practical applications, the byte length is the length of a preset service data segment to be compressed, and the byte length can be understood as the way in which the service data to be compressed is cut.

[0039]

[0041] Specifically, the byte length can be a preset length. The method for determining the service data segment to be compressed based on the preset length is to intercept the bytes of the preset length from the service data to be compressed and determine the intercepted data segment as the service data segment to be compressed. As shown in FIG2, FIG2 is a schematic diagram of a data compression process provided by one embodiment of this specification. As shown in the figure, the length of the service data to be compressed is 38KB, and the preset byte length is 8KB. Therefore, 8KB of the service data to be compressed is intercepted as the service data segment to be compressed. Since less than 8KB of data remains after intercepting four 8KB data segments, the length of the final service data segment to be compressed is determined to be the remaining byte length, that is, 6KB. The first 3K bytes of the 8K-long business data segment to be compressed are used as the compression test data segment, and the remaining 5K bytes of the business data segment to be compressed are used as the reference data segment to be compressed, so that the compression test data segment can be compressed later. By dividing the business data to be compressed into the preset byte length as the preset length, various types of business data to be compressed can be covered, thereby improving the accuracy of the judgment of the business data to be compressed while reducing the implementation cost of the data compression method.

[0040]

[0042] The byte length can also be a sequence of varying lengths. The method for determining the service data to be compressed based on the preset length is to truncate data segments of corresponding lengths based on the sequence as the service data segments to be compressed. For example, if the length of the service data to be compressed is 38KB and the preset byte lengths are 1, 2, 3, 5, 8, 13, and 21, seven service data segments to be compressed are determined, with lengths of 1KB, 2KB, 3KB, 5KB, 8KB, 13KB, and 6KB, respectively. Similarly, since the remaining bytes of the last service data to be compressed are less than the preset byte length, the length of the last service data segment to be compressed is set to the remaining byte length, i.e., 6KB. By segmenting the service data to be compressed based on the preset byte lengths used as sequences of varying lengths, the segmentation method for the service data to be compressed can be customized based on the characteristics of each type of service data to be compressed, thereby improving the accuracy of the determination of the service data to be compressed.

[0041]

[0043] By intercepting the business data to be compressed according to the preset byte length to determine each business data segment to be compressed, each segment can be judged when judging and compressing the business data to be compressed, thereby further improving the accuracy of the judgment and improving the efficiency of data compression when compressing the data.

[0042]

[0044] Step 104: Determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed.

[0043]

[0045] In practical applications, the uncompressed business data segment set is a set of compressible data segments, and the compressed business data segment set is a set of incompressible data segments.

[0044]

[0046] Specifically, the uncompressed business data segment set can be understood as a set containing compressible data segments, the compressible data segments can be understood as data segments that have not been compressed, the uncompressed business data segment set can be understood as a set containing non-compressible data segments, and the compressible data segments can be understood as data segments that have been compressed.

[0045]

[0047] By determining the data segments that can be compressed and the data segments that cannot be compressed in the business data segments to be compressed, the uncompressed business data segment set and the compressed business data segment set can be determined, thereby distinguishing between the compressed data segments and the uncompressed data segments, and reducing CPU resource consumption when compressing the data subsequently.

[0046]

[0048] Taking into account that judging the entirety of the compressed data increases the consumption of CPU resources, the business data segments to be compressed include compressed test data segments; further, determining an uncompressed business data segment set and a compressed business data segment set from each business data segment to be compressed, including: determining a data segment type corresponding to each business data segment to be compressed based on the compressed test data segment in each business data segment to be compressed, wherein the compressed test data segment is a data segment determined by part of the bytes in the business data segment to be compressed; determining an uncompressed business data segment set and a compressed business data segment set based on the data segment type corresponding to each business data segment to be compressed.

[0047]

[0049] In actual applications, the compression test data segment is a data segment in the business data segment to be compressed, and the data segment type is the type of the business data segment to be compressed.

[0048]

[0050] Specifically, the compression test data segment can be understood as a data segment composed of some bytes obtained from the business data segment to be compressed, which is used to determine the type of the business data segment to be compressed. Since operating the entire business data segment to be compressed to determine the type of the business data segment to be compressed will cause a large consumption of CPU resources, obtaining a data segment from the business data segment to be compressed as the business data segment to be compressed as a feature for determining the type of the business data segment to be compressed can effectively reduce CPU resource consumption when determining whether the business data segment to be compressed is compressible data.

[0049]

[0051] The data segment type can be understood as information indicating whether the service data segment to be compressed is compressible data. Since service data segments to be compressed include compressible data segments and incompressible data segments, the types corresponding to each compressed data segment include uncompressed service data and compressed service data. Uncompressed service data is a type of service data segment that can be compressed, while compressed service data is a type of service data segment that cannot be compressed. Data segments of the uncompressed service data type can be understood as data segments that have not been compressed prior to the current data compression. Data segments of the compressed service data type can be understood as data segments that have already been compressed prior to the current data compression.

[0050]

[0052] By determining the data segment type of each service data segment to be compressed using the compression test data segment in the service data segment to be compressed, CPU resource consumption when determining whether the service data to be compressed is compressible data can be further reduced.

[0051]

[0053] Furthermore, determining the data segment type corresponding to each to-be-compressed business data segment based on the compression test data segment in each to-be-compressed business data segment includes: determining a target to-be-compressed business data segment, wherein the target to-be-compressed business data segment is any one of the to-be-compressed business data segments; determining a target compression test data segment corresponding to the target to-be-compressed business data segment; and determining the data segment type corresponding to the target to-be-compressed business data segment based on the target compression test data segment, wherein the data segment type includes uncompressed business data and compressed business data.

[0054] In practical applications, the target service data segment to be compressed is any service data segment to be compressed segmented from the service data to be compressed, and the target compression test data segment is a portion of bytes obtained from the target service data segment to be compressed.

[0052]

[0055] Specifically, by performing target compression test on each target service data segment to be compressed in the service data segments to be compressed, the data segment type of each service data segment to be compressed can be determined, thereby further reducing CPU resource consumption when determining whether the service data to be compressed is compressible data.

[0053]

[0056] Considering that the business data to be compressed and the segments of the business data to be compressed are both composed of bytes, the target business data segment to be compressed includes at least one business data segment byte data; further, determining the target compression test data segment corresponding to the target business data segment to be compressed includes: determining at least one target business test data segment byte data from the byte data of each business data segment corresponding to the target business data segment to be compressed, wherein the target business test data segment byte data is the byte data contained in the target compression test data segment corresponding to the target business data segment to be compressed; and determining the target compression test data segment based on the byte data of each target business test data segment.

[0054]

[0057] In actual applications, the service data segment byte data is the byte data constituting the service data segment, and the target service test data segment byte data is the byte data corresponding to the target service test segment in the target service data to be compressed.

[0055]

[0058] Specifically, determining at least one target service test data segment byte data from the byte data of each service data segment corresponding to the target service data segment to be compressed can be understood as determining the byte corresponding to the target compression test data segment from each byte in the target service data segment to be compressed. In other words, the bytes corresponding to the determined target compression test data segments can be combined to form the target compression test data segment.

[0056]

[0059] The method for determining the bytes corresponding to the target compression test data segment from the bytes in the target business data segment to be compressed can be to determine the first preset number of bytes of the target business data segment to be compressed as the bytes corresponding to the target compression test data segment (target business test data segment byte data). The method for combining the target business test data segment byte data determined in this way to form the target compression test data segment is sequential combination. As shown in Figure 2, the first 3K bytes of the business data segment to be compressed with a length of 8K are used as the compression test data segment in Figure 2 so that the compression test data segment can be compressed subsequently.

[0057]

[0060] The method for determining the bytes corresponding to the target compression test data segment from the bytes in the target service data segment to be compressed may also be to randomly select bytes corresponding to the target compression test data segment from the target service data segment to be compressed. The method for combining the bytes of the target service test data segment determined in this manner to form the target compression test data segment may be a random combination or a sequential combination. This specification does not impose any limitation on the method for determining the bytes corresponding to the target compression test data segment from the bytes in the target service data segment to be compressed.

[0058]

[0061] By determining the bytes corresponding to the target compression test data segment from each byte in the target business data segment to be compressed, the target compression test data segment can be achieved for each target business data segment to be compressed in the business data segment to be compressed, so that the data segment type of each business data segment to be compressed can be determined, further reducing the CPU resource consumption when determining whether the business data to be compressed is compressible data.

[0059]

[0062] Furthermore, determining an uncompressed business data segment set and a compressed business data segment set according to the data segment type corresponding to each business data segment to be compressed includes: when the data segment type corresponding to the target business data segment to be compressed is uncompressed business data, inserting the target business data segment to be compressed into the uncompressed business data segment set; when the data segment type corresponding to the target business data segment to be compressed is compressed business data, inserting the target business data segment to be compressed into the compressed business data segment set.

[0060]

[0063] In practical applications, inserting data segments of the uncompressed business data type into the uncompressed business data segment set, and inserting data segments of the compressed business data type into the compressed business data segment set, can effectively enable the computer to distinguish between compressible business data segments to be compressed and incompressible business data segments to be compressed, thereby improving the efficiency of data compression.

[0061]

[0064] Furthermore, determining the data segment type corresponding to the target business data segment to be compressed based on the target compression test data segment includes: judging whether the target compression test data segment is compressible based on a preset compression test data judgment rule; if the target compression test data segment is compressible, determining that the data segment type corresponding to the target business data segment to be compressed is uncompressed business data; if the target compression test data segment is incompressible, determining that the data segment type corresponding to the target business data segment to be compressed is compressed business data.

[0062]

[0065] In practical applications, the compression test data judgment rule is a rule for determining whether a compression test data segment is compressible. The compression test data judgment rule is understood to include both an overall judgment rule and a sliding window judgment rule. The overall judgment rule compresses the entire compression test data segment and determines whether the compression result is normal, thereby determining whether the compression test data segment is compressible. The sliding window judgment rule compresses the data within the sliding window and determines whether the compression result is normal, thereby determining whether the compression test data segment is compressible.

[0063]

[0066] Specifically, whether the compression test data segment in the to-be-compressed business data segment can be compressed represents whether the entire to-be-compressed business data segment can be compressed, which can improve data processing efficiency while ensuring the accuracy of data compression judgment, thereby further improving data compression efficiency.

[0064]

[0067] Furthermore, determining whether the target compression test data segment is compressible includes: obtaining the data length of the target compression test data segment, and determining the compression ratio corresponding to the target compression test data segment based on the data length of the target compression test data segment; when the compression ratio is less than a preset compression threshold, determining that the target compression test data segment is compressible; when the compression ratio is greater than or equal to the preset compression threshold, determining that the target compression test data segment is incompressible.

[0065]

[0068] In practical applications, the compression rate is a value that indicates the degree of data compression, and the compression threshold is a value used to determine whether the data is compressible.

[0066]

[0069] Specifically, a method for determining the compression rate corresponding to the target compression test data segment based on the data length of the target compression test data segment can be to compress the target compression test data segment once and obtain the data length after compression, and then determine the data length before compression based on the data length of the target compression test data segment before compression. Thereafter, the compression rate corresponding to the target compression test data segment can be determined based on the change in the obtained data length before and after compression.

[0067]

[0070] The compression ratio determined based on the data length of the target compressed test data segment and the data length of the test data compression result can be understood as determining the degree of compression of the target compressed test data segment after compression. When the compression ratio between the two is greater than a preset threshold, it indicates that the target compressed test data segment is incompressible, which can be understood as the target compressed test data segment being a compressed data segment. Similarly, when the compression ratio between the two is less than the threshold, it is considered that the target compressed test data segment is compressible, which can be understood as the target compressed test data segment being an uncompressed data segment.

[0068]

[0071] Through the relationship between the compression rate and the preset threshold, it is determined whether the target compression test data segment can be compressed, and then it can be realized that whether a certain data segment in the corresponding business data to be compressed can be compressed. Therefore, the business data to be compressed can be compressed only by the target compression test data segment determined in a data segment of the entire data to be compressed, thereby improving the CPU utilization rate when compressing the business data to be compressed.

[0069]

[0072] Furthermore, determining the compression ratio corresponding to the target compression test data segment based on the data length of the target compression test data segment includes: performing compression processing on the target compression test data segment to obtain a test data compression result corresponding to the target compression test data segment; determining the data length of the target compression test data segment as the data length before compression, and determining the data length of the test data compression result as the data length after compression; and determining the compression ratio based on the data length before compression and the data length after compression.

[0070]

[0073] In practical applications, the test data compression result is the data after the compressed test data is compressed, the data length before compression is the length of the original data before compression, and the data length after compression is the length of the data obtained after compression of the original data.

[0071]

[0074] Specifically, there are many ways to compress data, for example, using LZ4 to compress data, using Deflate to compress data, using Brotli to compress data, etc. This specification does not impose any restrictions on this.

[0072]

[0075] The compression ratio is calculated by dividing the compressed data length by the compressed length. Therefore, the compression ratio of compressed data can be further understood as the percentage of storage resources occupied by the compressed data compared to the original data. Therefore, a higher compression ratio indicates a smaller difference between the compressed and uncompressed data. Therefore, a higher compression ratio indicates a lower degree of compression. When the compression ratio exceeds a preset threshold, the difference between the compressed and original data is too small, and the original data can be considered incompressible.

[0073]

[0076] By compressing the target compression test data segment and then calculating the compression rate of the target compression test data segment, it is possible to determine whether the target compression test data segment is compressible based on the compression rate obtained by calculation, and then it is possible to determine whether the corresponding business data segment to be compressed is compressible based on whether the target compression test data segment is compressible, thereby improving the efficiency of judging whether the business data to be compressed is compressible.

[0074]

[0077] Furthermore, determining whether the target compression test data segment is compressible includes: determining a sliding window of a preset size, determining that data within the sliding window is sliding window compression test data; performing compression processing on the sliding window compression test data to obtain a sliding window test data compression result corresponding to the sliding window compression test data; determining that a data length of the sliding window compression test data is a pre-compression data length, and determining that a data length of the sliding window test data compression result is a post-compression data length; determining a compression ratio corresponding to the sliding window based on the pre-compression data length and the post-compression data length; if the compression ratio corresponding to the sliding window is less than a preset compression threshold, continuing to slide the sliding window until the sliding window slides to the end of the target compression test data segment, generating a test data compression result corresponding to the target compression test data segment, and determining that the target compression test data segment is compressible; if the compression ratios corresponding to a preset number of consecutive sliding windows are greater than or equal to the preset compression threshold, stopping sliding the sliding window, and determining that the target compression test data segment is incompressible.

[0075]

[0078] In practical applications, the sliding window is a fixed-length data segment, the sliding window compression test data is the data within the sliding window, and the sliding window test data compression result is the data within the sliding window after compression.

[0076]

[0079] The compression ratio determined based on the data length of the data within the sliding window (sliding window compressed test data) during the sliding window's sliding process and the data length of the data after compression within the sliding window (sliding window test data compression result) can be understood as calculating the compression ratio of the data within the sliding window in real time during the sliding window's sliding process. When the compression ratio of the data within the sliding window is greater than a preset threshold, it indicates that the data within the sliding window is incompressible. Similarly, when the compression ratio of the data within the sliding window is less than the threshold, it is considered that the data within the sliding window is compressible. Furthermore, when the compression ratio of the data within the sliding window is less than the threshold (i.e., the data within the sliding window is compressible), the sliding window continues to compress the data until it reaches the end of the target compression test data segment, completing the compression of the target compression test data segment and indicating that the target compression test data segment is compressible. During the data compression process, a certain segment of data may be difficult to compress, but the data as a whole can be compressed. Therefore, only when the compression ratio of the data in the preset sliding window is greater than or equal to the threshold for a consecutive number of times is the target compression test data segment considered to be uncompressible, and the sliding of the sliding window is stopped.

[0077]

[0080] By setting a sliding window on the target compression test data segment, compressing the data in the sliding window during the sliding process of the sliding window and then calculating the compression rate of the data in the sliding window, it can be determined that the target compression test data segment cannot be compressed when the data compression rate in the sliding windows is higher than the threshold for a preset number of consecutive occurrences, thereby further improving the efficiency of judging whether the compressed business data is compressible. In addition, by stopping the sliding operation on the target compression test data segment when the data compression rate in the sliding windows is higher than the threshold for a preset number of consecutive occurrences, the CPU resource consumption is further reduced.

[0078]

[0081] Step 106: When there is at least one uncompressed service data segment in the uncompressed service data segment set, compression processing is performed on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment.

[0079]

[0082] In practical applications, the process of performing compression processing on the uncompressed service data segment can be any data compression method, and will not be further described here. Considering that the compressible target compression test data segment has already been compressed when determining whether the target compression test data segment is compressible data, the compression processing method for the uncompressed service data segment is preferably the same as the compression method for compressing the target compression test data segment described above. Consequently, subsequent compression of the uncompressed service data segment, excluding the target compression test data segment, can obtain the compression result of the service data segment corresponding to the uncompressed service data segment, thereby further reducing CPU resource consumption.

[0080]

[0083] Taking into account that when judging whether the target compression test data segment is compressible data, the compressible target compression test data segment needs to be compressed, so the uncompressed business data segment includes a compressed test data segment and a reference business data segment to be compressed; further, compression processing is performed on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment, including: determining a target uncompressed business data segment, wherein the target uncompressed business data segment is any one of the uncompressed business data segments; obtaining a test data compression result corresponding to the compressed test data segment in the target uncompressed business data segment; performing compression processing on the reference to-be-compressed data segment in the target uncompressed business data segment to obtain a reference to-be-compressed data compression result corresponding to the reference to-be-compressed data segment, wherein the reference to-be-compressed data segment is used to obtain a target uncompressed business data segment in combination with the compressed test data segment; and determining a business data segment compression result corresponding to the target uncompressed business data segment in combination with the test data compression result and the reference to-be-compressed data compression result.

[0081]

[0084] In practical applications, the reference data segment to be compressed is the data segment remaining after the compression test data segment is removed from the data segment to be compressed, the target uncompressed business data segment is any uncompressed business data segment in the set of uncompressed business data segments, and the compression result of the reference data to be compressed is the data after the reference data segment to be compressed in the target uncompressed business data segment is compressed.

[0082]

[0085] Specifically, the reference data segment to be compressed can be understood as the data segment consisting of the remaining byte groups in the data segment to be compressed, after excluding the compression test data segment used to determine whether the data segment is an uncompressed data segment. The reference data segment to be compressed can be compressed using any data compression method, which will not be described in detail here. The reference data segment to be compressed is shown in FIG2 . In the example of the 8KB service data segment to be compressed in FIG2 , the first 3KB are the compression test data segment, and the last 5KB are the reference data segment to be compressed.

[0083]

[0086] Based on the same reasons as when performing compression on the above-mentioned uncompressed business data segment, the method of performing compression processing on the reference data segment to be compressed is preferably the same as the compression method for compressing the target compressed test data segment mentioned above, so that the reference data segment to be compressed is compressed only once, and then it is spliced ​​with the compression result of the test data that has been compressed in the previous judgment to obtain the compression result of the business data segment corresponding to the reference data segment to be compressed, thereby further reducing the CPU resource consumption.

[0084]

[0087] Step 108: Determine the service data compression result corresponding to the service data to be compressed according to the compression result of each service data segment and the set of compressed service data segments.

[0085]

[0088] In actual applications, since the data segments in the compressed business data segment set are already compressed data segments, when determining the business data compression results corresponding to the business data to be compressed, the data segments in the compressed business data segment set need to be compressed again. The compression task of the business data to be compressed can be completed by simply sequentially splicing the compressed business data segment compression results with the data segments in the compressed business data segment set.

[0086]

[0089] By compressing only the uncompressed data segments in the service data to be compressed, the problem of CPU resource waste caused by re-compressing already compressed data can be avoided when compressing the service data to be compressed. Furthermore, since part of the data in the uncompressed data segment has already been compressed when determining each data segment, when compressing the data after the determination is completed, only the data in each data segment excluding the data used for determination can be compressed, further reducing CPU resource consumption during data compression.

[90] Further, the business data compression result corresponding to the business data to be compressed is determined based on the compression result of each business data segment and the set of compressed business data segments, including: when there is at least one compressed business data segment in the set of compressed business data segments, based on the business data to be compressed, determining the compression result of each business data segment and the sequence information of each compressed business data segment, and splicing the compression result of each business data segment and each compressed business data segment based on the sequence information to determine the business data compression result corresponding to the business data to be compressed; when there is no compressed business data segment in the set of compressed business data segments, determining the sequence information of the compression result of each business data segment based on the business data to be compressed, and splicing the compression result of each business data segment based on the sequence information to determine the business data compression result corresponding to the business data to be compressed.

[0087]

[91] In practical applications, the sequence information indicates the sequence of each to-be-compressed service data segment in the to-be-compressed service data. Specifically, determining the compression results of each service data segment and the sequence information of each compressed service data segment can be understood as determining the position of the to-be-compressed service data segment corresponding to the compression result of each service data segment in the to-be-compressed service data before compression based on the to-be-compressed service data before segmentation, and determining the position of each compressed service data segment in the to-be-compressed service data.

[0088]

[92] In one embodiment provided in this specification, the service data to be compressed is data 1, whose structure is [ABC]. After segmentation, the service data segment A to be compressed, the service data segment B to be compressed, and the service data segment C to be compressed are obtained. The position information of each of the service data segment A to be compressed is 0, the position information of the service data segment B to be compressed is 1, and the position information of the service data segment C to be compressed is 2. Subsequently, it is determined that the service data segment A to be compressed and the service data segment C to be compressed are uncompressed data segments, and the two data segments are inserted into the uncompressed data segment set. The service data segment B to be compressed is determined to be a compressed data segment, and the data segment is inserted into the compressed data segment set. Then, the business data segment A to be compressed and the business data segment C to be compressed in the uncompressed data segment set are compressed into business data segment compression results A' and business data segment compression results C', respectively. Then, based on the position information of each data segment obtained when the business data to be compressed is segmented, the position information of the business data segment compression result A' is determined to be 0, the position information of the compressed data segment B is determined to be 1, and the position information of the business data segment compression result C' is determined to be 2. Finally, the business data compression result determined by splicing the position information of each data segment is [A'BC'L

[0089]

[93] By compressing only the uncompressed business data segments, compressing the uncompressed data in the business data to be compressed and splicing it with the compressed data to generate the business data compression result, repeated compression of the compressed data is avoided, thereby reducing CPU resource consumption.

[0090]

[94] Furthermore, after determining the uncompressed business data segment set and the compressed business data segment set from each business data segment to be compressed, the method further includes: in the case that there is no uncompressed business data segment in the uncompressed business data segment set, determining that the business data to be compressed is the business data compression result corresponding to the business data to be compressed.

[0091]

[95] Considering that the business data to be compressed may have all been compressed, and in the case that the business data to be compressed may have all been compressed, there is no uncompressed business data segment in the business data segments to be compressed, so in the above case, the business data to be compressed can be directly regarded as the compressed result of the business data that has been compressed.

[0096] By judging the compression degree of each data segment and directly regarding the data that does not contain uncompressed data segments as the compressed result of the data, repeated compression of the compressed data is avoided, thereby reducing CPU resource consumption.

[0092] By applying the solution of the embodiments of this specification, the service data to be compressed is segmented into at least one service data segment, and whether the data segment has been compressed is determined by compressing a test data segment within the data segment. This improves the accuracy of determining whether the data has been compressed while reducing CPU resource consumption. Subsequently, by compressing only the uncompressed service data segment, repeated compression of already compressed data is avoided, further reducing CPU resource consumption.

[0093]

[0098] Referring to FIG. 3 , FIG. 3 shows a flow chart of a data compression method applied to a Yunjia I device according to an embodiment of this specification, specifically including the following steps.

[0094]

[0099] Step 302: The receiving end device receives service data to be compressed, and determines at least one service data segment to be compressed based on the service data to be compressed.

[0095]

[0100] Step 304: Determine an uncompressed business data segment set and a compressed business data segment set from each business data segment to be compressed.

[0096]

[0101] Step 306: When there is at least one uncompressed business data segment in the uncompressed business data segment set, compression processing is performed on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment.

[0097]

[0102] Step 308: Determine the service data compression result corresponding to the service data to be compressed according to the compression result of each service data segment and the set of compressed service data segments, and return the service data compression result to the terminal device.

[0098]

[0103] The above is a schematic scheme of a data compression method applied to a Yunjia 1 device in this embodiment. It should be noted that the technical scheme of the data compression method applied to the Yunli device and the technical scheme of the above-mentioned data compression method are of the same concept. For details not described in detail in the technical scheme of the data compression method applied to the Yunli device, please refer to the description of the technical scheme of the above-mentioned data compression method.

[0099]

[0104] By applying the solutions of the embodiments of this specification, the Yunjia 1 device segments the service data to be compressed into at least one service data segment and determines whether the data segment has been compressed by compressing a test data segment within the data segment. This improves the accuracy of determining whether the data sent by the peer Yunjia 1 device has been compressed, while reducing CPU resource consumption of the Yunjia 1 device. Subsequently, by compressing only the uncompressed service data segments, repeated compression of already compressed data is avoided, further reducing CPU resource consumption of the Yunjia 1 device.

[0100]

[0105] Referring to FIG. 4 , FIG. 4 shows an architecture diagram of a data compression system provided in one embodiment of this specification. The data compression system may include a client 100 and a server 200.

[0101]

[0106] The client 100 is used to send the service data to be compressed to the server 200.

[0102]

[0107] The server 200 is used to receive the business data to be compressed sent by the client 100, and determine at least one business data segment to be compressed based on the business data to be compressed; determine an uncompressed business data segment set and a compressed business data segment set from each business data segment to be compressed; when there is at least one uncompressed business data segment in the uncompressed business data segment set, perform compression processing on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment; determine a business data compression result corresponding to the business data to be compressed based on the compression result of each business data segment and the compressed business data segment set; and send the business data compression result to the client 100.

[0103]

[0108] The client 100 is also used to receive the business data compression result sent by the server 200.

[0104]

[0109] By applying the solutions of the embodiments of this specification, the server segmented the service data to be compressed into at least one service data segment and determined whether the data segment had been compressed by compressing a test data segment within the data segment. This improved the accuracy of determining whether the data sent by the client had been compressed while reducing the server's CPU resource consumption. Subsequently, by compressing only the uncompressed service data segment, repeated compression of already compressed data was avoided, further reducing the server's CPU resource consumption.

[0105]

[0110] The data compression system may include multiple clients 100 and a server 200, wherein the clients 100 may be referred to as a client device and the server 200 may be referred to as a cloud device. Multiple clients 100 may establish a communication connection through the server 200. In a data compression scenario, the server 200 is used to provide data compression services between the multiple clients 100. The multiple clients 100 may act as a sender or a receiver, respectively, and communicate through the server 200.

[0106]

[0111] A user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100. In a data compression scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates a business data compression result based on the data stream and pushes the business data compression result to other clients with which communication is established.

[0107]

[0112] The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or optical fiber cables. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

[0108]

[0113] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Languages, version 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be developed based on a software development kit (SDK) of a corresponding service provided by the server 200, such as a real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and rely on the device or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, or a personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0109]

[0114] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that support backend training for models used on clients, and servers that process data sent by clients. It should be noted that the server 200 may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server may also be a server in a distributed system, or a server integrated with a blockchain. The server may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0110] It is worth noting that the data compression method provided in the embodiments of this specification is generally performed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby performing the data compression method provided in the embodiments of this specification. In other embodiments, the data compression method provided in the embodiments of this specification may also be performed jointly by the client and the server.

[0111]

[0116] The following further describes the data compression method provided in this specification using the application of the data compression method in storing user uploaded data as an example, in conjunction with FIG5. FIG5 shows a flowchart of a processing process of a method for storing user uploaded data provided in one embodiment of this specification, which specifically includes the following steps.

[0112]

[0117] Step 502: Receive the data that needs to be stored uploaded by the user: Data 1 with a length of 23K.

[0113]

[0118] Step 504: Divide the data 1 with a length of 23K into a data segment 1 with a length of 8K, a data segment 2 with a length of 8K, and a data segment 3 with a length of 7K according to the length of 8K.

[0114]

[0119] Step 506: Compress the first 1K of data in data segment 1, data segment 2, and data segment 3 respectively.

[0115]

[0120] Step 508: Divide the compressed data length by the uncompressed data length to calculate: the compression rate of the first 1K data of data segment 1 is 68%, the compression rate of the first 1K data of data segment 2 is 98%, and the compression rate of the first 1K data of data segment 3 is 35%.

[0116]

[0121] Step 510: Based on the preset data compression rate threshold of 95%, it can be known that data segment 1 and data segment 3 are uncompressed data segments, and data segment 2 is a compressed data segment.

[0117]

[0122] Step 512: Compress the last 7K of data in data segment 1, and concatenate the result of this compression with the compression result of the first 1K of data that has been compressed previously to determine the compression result of data segment 1. Compress the last 6K of data in data segment 3, and concatenate the result of this compression with the compression result of the first 1K of data that has been compressed previously to determine the compression result of data segment 3.

[0118]

[0123] Step 514: Concatenate the compression results of data segment 1, data segment 2, and data segment 3 to obtain the compression result of data 1 uploaded by the user.

[0119]

[0124] Step 516: Store the compression result of data 1 to a storage device.

[0120]

[0125] By applying the solution of the embodiments of this specification, by segmenting the service data to be compressed uploaded by the user into at least one service data segment, and determining whether the data segment has been compressed by compressing the test data segment in the data segment, the accuracy of determining whether the data has been compressed can be improved while reducing the CPU resource consumption when storing the data. Subsequently, by only compressing the uncompressed service data segment, repeated compression of the compressed data is avoided, thereby further reducing the CPU resource consumption when storing the data while ensuring that less space is occupied on the storage device.

[0126] Corresponding to the above-mentioned method embodiments, this specification also provides an embodiment of a data compression device. FIG6 shows a schematic structural diagram of a data compression device provided in one embodiment of this specification. As shown in Figure 6, the device includes: a data receiving module 602, configured to receive business data to be compressed, and determine at least one business data segment to be compressed based on the business data to be compressed; a set determination module 604, configured to determine an uncompressed business data segment set and a compressed business data segment set from each business data segment to be compressed; a data compression module 606, configured to perform compression processing on each uncompressed business data segment when there is at least one uncompressed business data segment in the uncompressed business data segment set, and obtain a business data segment compression result corresponding to each uncompressed business data segment; a result determination module 608, configured to determine the business data compression result corresponding to the business data to be compressed based on the compression result of each business data segment and the compressed business data segment set.

[0121]

[0127] Optionally, the data receiving module 602 is further configured to: cut the service data to be compressed according to a preset byte length to determine at least one service data segment to be compressed.

[0122]

[0128] Optionally, the business data segments to be compressed include compression test data segments; the set determination module 604 is further configured to: determine the data segment type corresponding to each business data segment to be compressed based on the compression test data segments in each business data segment to be compressed, wherein the compression test data segment is a data segment determined by some bytes in the business data segment to be compressed; and determine the uncompressed business data segment set and the compressed business data segment set based on the data segment type corresponding to each business data segment to be compressed.

[0123]

[0129] Optionally, the set determination module 604 is further configured to: determine a target business data segment to be compressed, wherein the target business data segment to be compressed is any one of the business data segments to be compressed; determine a target compression test data segment corresponding to the target business data segment to be compressed; and determine a data segment type corresponding to the target business data segment to be compressed based on the target compression test data segment, wherein the data segment type includes uncompressed business data and compressed business data.

[0124]

[0130] Optionally, the target business data segment to be compressed includes at least one business data segment byte data; the set determination module 604 is further configured to: determine at least one target business test data segment byte data from the byte data of each business data segment corresponding to the target business data segment to be compressed, wherein the target business test data segment byte data is the byte data contained in the target compression test data segment corresponding to the target business data segment to be compressed; and determine the target compression test data segment based on the byte data of each target business test data segment.

[0125]

[0131] Optionally, the set determination module 604 is further configured to: when the data segment type corresponding to the target business data segment to be compressed is uncompressed business data, insert the target business data segment to be compressed into the uncompressed business data segment set; when the data segment type corresponding to the target business data segment to be compressed is compressed business data, insert the target business data segment to be compressed into the compressed business data segment set.

[0126]

[0132] Optionally, the set determination module 604 is further configured to: determine whether the target compression test data segment is compressible based on a preset compression test data judgment rule; if the target compression test data segment is compressible, determine that the data segment type corresponding to the target to-be-compressed business data segment is uncompressed business data; if the target compression test data segment is incompressible, determine that the data segment type corresponding to the target to-be-compressed business data segment is compressed business data.

[0133] Optionally, the set determination module 604 is further configured to: obtain the data length of the target compression test data segment, and determine the compression ratio corresponding to the target compression test data segment based on the data length of the target compression test data segment; if the compression ratio is less than a preset compression threshold, determine that the target compression test data segment is compressible; if the compression ratio is greater than or equal to the preset compression threshold, determine that the target compression test data segment is incompressible.

[0127]

[0134] Optionally, the set determination module 604 is further configured to: perform compression processing on the target compressed test data segment to obtain a test data compression result corresponding to the target compressed test data segment; determine the data length of the target compressed test data segment as the data length before compression, and determine the data length of the test data compression result as the data length after compression; and determine the compression ratio based on the data length before compression and the data length after compression.

[0128]

[0135] Optionally, the set determination module 604 is further configured to: determine a sliding window of a preset size, determine that data within the sliding window is sliding window compression test data; perform compression processing on the sliding window compression test data, and obtain a sliding window test data compression result corresponding to the sliding window compression test data; determine that the data length of the sliding window compression test data is the data length before compression, and determine that the data length of the sliding window test data compression result is the data length after compression; determine a compression ratio corresponding to the sliding window based on the data length before compression and the data length after compression; if the compression ratio corresponding to the sliding window is less than a preset compression threshold, continue sliding the sliding window until the sliding window slides to the end of the target compression test data segment, generate a test data compression result corresponding to the target compression test data segment, and determine that the target compression test data segment is compressible; if the compression ratios corresponding to a preset number of sliding windows are greater than or equal to the preset compression threshold, stop sliding the sliding window, and determine that the target compression test data segment is incompressible.

[0129]

[0136] Optionally, the uncompressed business data segment includes a compressed test data segment and a reference data segment to be compressed; the data compression module 606 is further configured to: determine a target uncompressed business data segment, wherein the target uncompressed business data segment is any one of the uncompressed business data segments; obtain a test data compression result corresponding to the compressed test data segment in the target uncompressed business data segment; perform compression processing on the reference data segment to be compressed in the target uncompressed business data segment to obtain a reference data compression result corresponding to the reference data segment to be compressed, wherein the reference data segment to be compressed is used to obtain a target uncompressed business data segment in combination with the compressed test data segment; and determine a business data segment compression result corresponding to the target uncompressed business data segment based on the test data compression result and the reference data compression result.

[0130]

[0137] Optionally, the result determination module 608 is further configured to: determine the compression result of each business data segment and the sequence information of each compressed business data segment based on the business data to be compressed when there is at least one compressed business data segment in the compressed business data segment set, and splice the compression results of each business data segment and each compressed business data segment based on the sequence information to determine the business data compression result corresponding to the business data to be compressed; and determine the sequence information of the compression result of each business data segment based on the business data to be compressed when there is no compressed business data segment in the compressed business data segment set, and splice the compression results of each business data segment based on the sequence information to determine the business data compression result corresponding to the business data to be compressed.

[0131]

[0138] Optionally, the data compression device also includes a data transfer module, which is configured to: when there is no uncompressed business data segment in the uncompressed business data segment set, determine that the business data to be compressed is the business data compression result corresponding to the business data to be compressed.

[0132]

[0139] By applying the solution of the embodiments of this specification, the data receiving module in the data compression device segments the service data to be compressed into at least one service data segment, and determines whether the data segment has been compressed based on the compression test data segment in the data segment. This improves the accuracy of determining whether the data has been compressed while reducing CPU resource consumption of the data compression device. Subsequently, by compressing only the uncompressed service data segment, repeated compression of already compressed data is avoided, further reducing CPU resource consumption of the data compression device.

[0133] The above is a schematic diagram of a data compression device according to this embodiment. It should be noted that the technical solution of the data compression device and the technical solution of the above-mentioned data compression method are based on the same concept. For details not described in detail in the technical solution of the data compression device, please refer to the description of the technical solution of the above-mentioned data compression method.

[0134] FIG7 shows a block diagram of a computing device 700 according to one embodiment of the present disclosure. Components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0135]

[0142] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0136] In one embodiment of the present specification, the aforementioned components of the computing device 700 and other components not shown in FIG. 7 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG. 7 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0137] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0138]

[0145] The processor 720 is used to execute the following computer executable instructions, which, when executed by the processor, implement the above-mentioned data compression method and the steps of the data compression method applied to the cloud example device.

[0139]

[0146] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device is based on the same concept as the above-mentioned data compression method and the technical solution of the data compression method applied to the Yunjia 1 device. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-mentioned data compression method and the data compression method applied to the Yunjia 1 device.

[0140]

[0147] An embodiment of this specification also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the above-mentioned data compression method and the data compression method applied to the cloud example device are implemented.

[0141]

[0148] The above is a schematic diagram of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of the storage medium is the same as the technical solution of the above-mentioned data compression method and the data compression method applied to the cloud device. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-mentioned data compression method and the data compression method applied to the cloud device.

[0142]

[0149] An embodiment of this specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned data compression method and the steps of the data compression method applied to the Yunjia 1 device.

[0143]

[0150] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the data compression method and the data compression method applied to the Yunjia 1 device are based on the same concept. For details not described in detail in the technical scheme of the computer program, please refer to the description of the technical scheme of the data compression method and the data compression method applied to the Yunjia 1 device.

[0144]

[0151] The above description of specific embodiments of this specification is provided. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0145]

[0152] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0146]

[0153] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of action combinations. However, those skilled in the art should be aware that the embodiments of this specification are not limited to the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0147]

[0154] In the above embodiments, the description of each embodiment is given with emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0155] The preferred embodiments disclosed above are only used to help illustrate this specification. The optional embodiments do not describe all details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

Claims 1. A data compression method, comprising: receiving service data to be compressed, and determining at least one service data segment to be compressed based on the service data to be compressed; determining an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed; When there is at least one uncompressed service data segment in the uncompressed service data segment set, performing compression processing on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment; The business data compression result corresponding to the business data to be compressed is determined according to the compression result of each business data segment and the set of compressed business data segments.

2. The method according to claim 1, wherein the service data segment to be compressed comprises a compression test data segment; Determining an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed includes: determining a data segment type corresponding to each service data segment to be compressed according to a compression test data segment in each service data segment to be compressed, wherein: The compression test data segment is a data segment determined by some bytes in the service data segment to be compressed; The uncompressed service data segment set and the compressed service data segment set are determined according to the data segment type corresponding to each service data segment to be compressed.

3. The method according to claim 2, wherein determining the data segment type corresponding to each of the service data segments to be compressed according to the compression test data segment in each of the service data segments to be compressed comprises: Determine a target business data segment to be compressed, wherein the target business data segment to be compressed is any one of the business data segments to be compressed; determine a target compression test data segment corresponding to the target business data segment to be compressed; and determine a data segment type corresponding to the target business data segment to be compressed based on the target compression test data segment, wherein the data segment type includes uncompressed business data and compressed business data.

4. The method according to claim 3, wherein the target service data segment to be compressed comprises at least one service data segment byte data; and determining a target compression test data segment corresponding to the target service data segment to be compressed comprises: Determine at least one target business test data segment byte data from the byte data of each business data segment corresponding to the target business data segment to be compressed, wherein the target business test data segment byte data is the byte data contained in the target compression test data segment corresponding to the target business data segment to be compressed; determine the target compression test data segment based on the byte data of each target business test data segment.

5. The method according to claim 3, wherein determining the uncompressed service data segment set and the compressed service data segment set according to the data segment type corresponding to each service data segment to be compressed comprises: In a case where the data segment type corresponding to the target service data segment to be compressed is uncompressed service data, inserting the target service data segment to be compressed into the uncompressed service data segment set; In a case where the data segment type corresponding to the target service data segment to be compressed is compressed service data, the target service data segment to be compressed is inserted into the set of compressed service data segments.

6. The method according to claim 3, wherein determining the data segment type corresponding to the target service data segment to be compressed according to the target compression test data segment comprises: Based on a preset compression test data judgment rule, determining whether the target compression test data segment is compressible; When the target compression test data segment is compressible, the data segment type corresponding to the target business data segment to be compressed is determined to be uncompressed business data; when the target compression test data segment is incompressible, the data segment type corresponding to the target business data segment to be compressed is determined to be compressed business data.

7. The method according to claim 6, wherein determining whether the target compression test data segment is compressible comprises: Acquiring the data length of the target compression test data segment, and determining the compression ratio corresponding to the target compression test data segment according to the data length of the target compression test data segment; When the compression ratio is less than a preset compression threshold, the target compression test data segment is determined to be compressible; when the compression ratio is greater than or equal to the preset compression threshold, the target compression test data segment is determined to be incompressible.

8. The method according to claim 7, wherein determining the compression ratio corresponding to the target compression test data segment according to the data length of the target compression test data segment comprises: Performing compression processing on the target compressed test data segment to obtain a test data compression result corresponding to the target compressed test data segment; Determine the data length of the target compression test data segment as the data length before compression, and determine the data length of the test data compression result as the data length after compression; and determine the compression ratio according to the data length before compression and the data length after compression.

9. The method according to claim 6, wherein determining whether the target compression test data segment is compressible comprises: Determining a sliding window of a preset size, and determining data within the sliding window as sliding window compression test data; Performing compression processing on the sliding window compression test data to obtain a sliding window test data compression result corresponding to the sliding window compression test data; Determining a data length of the sliding window compression test data as a pre-compression data length, and determining a data length of a compression result of the sliding window test data as a post-compression data length; determining a compression ratio corresponding to the sliding window according to the pre-compression data length and the post-compression data length; If the compression ratio corresponding to the sliding window is less than a preset compression threshold, continue sliding the sliding window until the sliding window slides to the end of the target compression test data segment, generate a test data compression result corresponding to the target compression test data segment, and determine that the target compression test data segment is compressible; When the compression ratio corresponding to a preset number of sliding windows is greater than or equal to a preset compression threshold, the sliding window is stopped, and the target compression test data segment is determined to be incompressible.

10. The method according to claim 1, wherein the uncompressed service data segment comprises a compressed test data segment and a reference data segment to be compressed; Performing compression processing on each uncompressed business data segment to obtain a business data segment compression result corresponding to each uncompressed business data segment includes: determining a target uncompressed business data segment, wherein the target uncompressed business data segment is any one of the uncompressed business data segments; obtaining a test data compression result corresponding to a compressed test data segment in the target uncompressed business data segment; Perform compression processing on the reference to-be-compressed data segment in the target uncompressed business data segment to obtain a reference to-be-compressed data compression result corresponding to the reference to-be-compressed data segment, wherein the reference to-be-compressed data segment is used to be combined with the compressed test data segment to obtain the target uncompressed business data segment; and determine the business data segment compression result corresponding to the target uncompressed business data segment by combining the test data compression result and the reference to-be-compressed data compression result.

11. The method according to claim 1, wherein determining at least one service data segment to be compressed based on the service data to be compressed comprises: The service data to be compressed is cut according to a preset byte length to determine at least one service data segment to be compressed.

12. The method according to claim 1, wherein determining the service data compression result corresponding to the service data to be compressed according to the compression result of each service data segment and the set of compressed service data segments comprises: If there is at least one compressed service data segment in the set of compressed service data segments, determining, based on the service data to be compressed, compression results of each service data segment and sequence information of each compressed service data segment, and concatenating the compression results of each service data segment and each compressed service data segment based on the sequence information to determine a service data compression result corresponding to the service data to be compressed; When no compressed business data segment exists in the compressed business data segment set, sequence information of the compression results of each business data segment is determined based on the business data to be compressed, and the compression results of each business data segment are spliced ​​based on the sequence information to determine the business data compression result corresponding to the business data to be compressed.

13. The method according to claim 1, after determining the uncompressed service data segment set and the compressed service data segment set from each service data segment to be compressed, the method further comprises: In a case where no uncompressed service data segment exists in the uncompressed service data segment set, it is determined that the service data to be compressed is a service data compression result corresponding to the service data to be compressed.

14. A data compression method, applied to a Yunjia 1 device, comprising: receiving service data to be compressed from a terminal device, and determining at least one service data segment to be compressed based on the service data to be compressed; Determine an uncompressed service data segment set and a compressed service data segment set from each service data segment to be compressed; When there is at least one uncompressed service data segment in the uncompressed service data segment set, performing compression processing on each uncompressed service data segment to obtain a service data segment compression result corresponding to each uncompressed service data segment; The service data compression result corresponding to the service data to be compressed is determined according to the compression result of each service data segment and the set of compressed service data segments, and the service data compression result is returned to the terminal device.

15. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 13 or 14 are implemented.

16. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13 or 14.

17. A computer program product comprising a computer program / instructions, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13 or 14.

Images