Data compression method and device, equipment and storage medium

Abstract

The invention relates to a data compression method and device, equipment and a storage medium. The method comprises the steps of receiving a target file uploaded by a user, judging whether the target file is matched with a pre-configured compression rule or not, judging whether the target file meets a compression condition or not under the condition that the target file is matched with the pre-configured compression rule, and performing compression on the target file under the condition that the target file meets the compression condition. And compressing the target file through an upper-layer service to obtain a compressed file of the target file, and calling a bottom-layer service to store the compressed file of the target file. Compared with the prior art, the embodiment of the invention has the advantages that the data can be compressed through the upper-layer service, the CPU resources of the upper-layer service can be reasonably utilized, the compression capability can be dynamically controlled through the pre-configured compression rule and the judgment of the compression condition, the compression can be flexibly carried out, the bottom-layer service does not need to be compressed, the pressure of the CPU of the bottom-layer service is reduced, and the compression efficiency is improved. And the requirement on the transmission bandwidth is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data compression method, apparatus, device, and storage medium. Background Art

[0002] With the widespread adoption of the internet and the ongoing advancement of digital transformation, the amount of data that can be captured, stored, and transmitted has expanded to unprecedented scale. This explosive growth in data creates both enormous opportunities and significant challenges. Compression technology is a key tool for addressing the challenges of big data. By employing compression algorithms, the size of data files can be significantly reduced, thereby improving storage efficiency and reducing transmission bandwidth requirements.

[0003] In related technologies, when uploading data to a storage service for storage, the upper-layer service of the storage service does not compress the data, but completely relies on the underlying service of the storage service to compress the data, and then stores the data in the underlying service after compression.

[0004] However, the above approach relies entirely on compression in the underlying service, which increases the pressure on the underlying service's CPU, fails to properly utilize the CPU resources of the upper-layer service, and the compression strategy is not flexible. Moreover, because data is compressed only after it is transmitted to the underlying service, the bandwidth required for data transmission from the upper layer to the bottom layer is relatively high. Summary of the Invention

[0005] In order to solve the above technical problems, the present disclosure provides a data compression method, device, equipment and storage medium, which can compress data through upper-layer services, reasonably utilize the upper-layer service CPU resources, and the compression capability can be dynamically controlled, and compression can be performed flexibly, thereby reducing the pressure on the underlying service CPU and reducing the demand for transmission bandwidth.

[0006] In a first aspect, an embodiment of the present disclosure provides a data compression method, the method comprising:

[0007] Receive target files uploaded by users;

[0008] Determining whether the target file matches a preconfigured compression rule;

[0009] If the target file matches the pre-configured compression rule, determining whether the target file meets the compression condition;

[0010] When the target file meets the compression condition, compressing the target file through the upper layer service to obtain a compressed file of the target file;

[0011] The underlying service is called to store the compressed file of the target file.

[0012] In some embodiments, determining whether the target file matches a preconfigured compression rule includes:

[0013] Obtaining relevant information of the target file, wherein the relevant information of the target file includes user identification, storage location, file prefix information, file suffix information, and file type;

[0014] It is determined whether the target file matches a preconfigured compression rule according to the relevant information of the target file or the identifier of the global compression switch.

[0015] In some embodiments, the preconfigured compression rule includes at least one of the following:

[0016] Target files matching a preset user identifier are allowed to be compressed, target files matching a preset storage location are allowed to be compressed, target files in a file prefix whitelist are allowed to be compressed, target files in a file suffix whitelist are allowed to be compressed, and target files in a file type whitelist are allowed to be compressed;

[0017] Determining whether the target file matches a preconfigured compression rule according to relevant information of the target file includes:

[0018] If the user identifier of the target file matches the preset user identifier, determining that the target file matches the preconfigured compression rule;

[0019] If the storage location of the target file matches the preset storage location, determining that the target file matches the preconfigured compression rule;

[0020] If the file prefix information of the target file is in the file prefix whitelist, determining that the target file matches the preconfigured compression rule;

[0021] If the file suffix information of the target file is in the file suffix whitelist, determining that the target file matches the preconfigured compression rule;

[0022] If the file type of the target file is in the file type whitelist, it is determined that the target file matches the preconfigured compression rule.

[0023] In some embodiments, determining whether the target file matches a preconfigured compression rule based on the relevant information of the target file includes:

[0024] If the file prefix information of the target file is not in the file prefix whitelist, or the file prefix information of the target file is in the file prefix blacklist, it is determined that the target file does not match the preconfigured compression rule;

[0025] If the file suffix information of the target file is not in the file suffix whitelist, or the file suffix information of the target file is in the file suffix blacklist, it is determined that the target file does not match the preconfigured compression rule;

[0026] If the file type of the target file is not in the file type whitelist, or the file type of the target file is in the file type blacklist, it is determined that the target file does not match the preconfigured compression rule.

[0027] In some embodiments, determining whether the target file meets the compression condition includes:

[0028] If the file size of the target file is greater than or equal to a preset threshold, it is determined that the target file meets the compression condition;

[0029] If the compression ratio of the target file is greater than or equal to a preset compression ratio threshold, it is determined that the target file meets the compression condition;

[0030] If the target file is within the range of the preset sampling ratio, it is determined that the target file meets the compression condition.

[0031] In some embodiments, the method further comprises:

[0032] Get the current load status of the upper-layer service;

[0033] The preset sampling ratio is adjusted according to the current load of the upper layer service.

[0034] In some embodiments, compressing the target file through an upper layer service to obtain a compressed file of the target file includes:

[0035] Determining a compression type of the target file;

[0036] Determining a compression algorithm for the target file according to a compression category of the target file;

[0037] The target file is compressed by an upper layer service based on a compression algorithm of the target file to obtain a compressed file of the target file.

[0038] In a second aspect, an embodiment of the present disclosure provides a data compression device, the device comprising:

[0039] Receiving module, used to receive target files uploaded by users;

[0040] A first judgment module is used to judge whether the target file matches a preconfigured compression rule;

[0041] A second judgment module is used to judge whether the target file meets the compression condition when the target file matches the pre-configured compression rule;

[0042] A compression module, configured to compress the target file through an upper layer service to obtain a compressed file of the target file when the target file meets the compression condition;

[0043] The storage module is used to call the underlying service to store the compressed file of the target file.

[0044] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0045] Memory;

[0046] processor; and

[0047] computer programs;

[0048] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0049] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method as described in the first aspect.

[0050] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, implements the method described in the first aspect.

[0051] The data compression method, apparatus, device and storage medium provided by the embodiments of the present disclosure receive a target file uploaded by a user, determine whether the target file matches a pre-configured compression rule, and if the target file matches the pre-configured compression rule, determine whether the target file meets the compression condition. If the target file meets the compression condition, the upper-layer service compresses the target file to obtain a compressed file of the target file, and calls the underlying service to store the compressed file of the target file. Compared with the existing technology, the embodiments of the present disclosure can realize data compression through the upper-layer service, reasonably utilize the CPU resources of the upper-layer service, realize dynamic control of compression capability through pre-configured compression rules and judgment of compression conditions, can flexibly perform compression, and the underlying service does not need to perform compression, thereby reducing the pressure on the underlying service CPU and reducing the demand for transmission bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0053] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 A flow chart of a data compression method provided in an embodiment of the present disclosure;

[0055] Figure 2 A schematic diagram of the overall data compression process provided by an embodiment of the present disclosure;

[0056] Figure 3 A flow chart of a data compression method provided by another embodiment of the present disclosure;

[0057] Figure 4 A flow chart of a data compression method provided by another embodiment of the present disclosure;

[0058] Figure 5 A schematic structural diagram of a data compression device provided in an embodiment of the present disclosure;

[0059] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0060] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0062] With the widespread adoption of the internet and the ongoing advancement of digital transformation, the amount of data that can be captured, stored, and transmitted has expanded to unprecedented scale. This explosive growth in data creates both enormous opportunities and significant challenges. Compression technology is a key tool for addressing the challenges of big data. By employing compression algorithms, the size of data files can be significantly reduced, thereby improving storage efficiency and reducing transmission bandwidth requirements.

[0063] In related technologies, when uploading data to a storage service for storage, the upper-layer service of the storage service does not compress the data, but completely relies on the underlying service of the storage service to compress the data, and then stores the data in the underlying service after compression.

[0064] However, the above approach relies entirely on compression in the underlying service, which increases the pressure on the underlying service's CPU, fails to properly utilize the CPU resources of the upper-layer service, and the compression strategy is not flexible. Moreover, because data is compressed only after it is transmitted to the underlying service, the bandwidth required for data transmission from the upper layer to the bottom layer is relatively high.

[0065] To address this problem, an embodiment of the present disclosure provides a data compression method, which is described below in conjunction with specific embodiments.

[0066] Figure 1 This is a flow chart of a data compression method provided by an embodiment of the present disclosure. The method is performed by an electronic device, which may be an object storage service device. The object storage service device may be a fixed device such as a computer or a server. The server may be a single server or a server cluster, which may be a distributed cluster or a centralized cluster. This method can be applied to scenarios where data is compressed and stored, or where compression is performed through upper-layer services.

[0067] It is understandable that the data compression method provided by the embodiments of the present disclosure can also be applied in other scenarios.

[0068] Below Figure 1 The data compression method shown in FIG. 1 is introduced, which can be applied to electronic devices. The method includes the following specific steps:

[0069] S101: Receive a target file uploaded by a user.

[0070] In this step, if Figure 2 As shown, the user uploads a target file, and the electronic device receives the target file uploaded by the user. The target file can be a data file such as a picture, video, audio, etc., without specific limitation.

[0071] S102: Determine whether the target file matches a preconfigured compression rule.

[0072] In this step, after receiving the target file uploaded by the user, the electronic device determines whether the target file matches the preconfigured compression rules. If the target file matches the preconfigured compression rules, S103 and subsequent steps are executed. The preconfigured compression rules enable dynamic control of compression capabilities, allowing for flexible compression.

[0073] In some embodiments, before determining whether the target file matches the preconfigured compression rule, the method further includes: obtaining the preconfigured compression rule in response to a user's configuration of the compression rule.

[0074] In some embodiments, as Figure 2 As shown, if the target file does not match the pre-configured compression rule, it means that the target file is not compressible, and the underlying service is directly called to store the target file.

[0075] S103: When the target file matches a preconfigured compression rule, determine whether the target file meets a compression condition.

[0076] In this step, if the target file matches the preconfigured compression rule, the electronic device determines whether the target file meets the compression conditions. If the target file meets the compression conditions, S104 and subsequent steps are executed. By configuring the compression conditions, the compression capacity can be dynamically controlled, allowing for flexible compression.

[0077] In some embodiments, as Figure 2 As shown, if the target file does not meet the compression condition, it means that the target file cannot be compressed, and the underlying service is directly called to store the target file.

[0078] S104: When the target file meets the compression condition, compress the target file through an upper layer service to obtain a compressed file of the target file.

[0079] In this step, if it is determined that the target file meets the compression conditions, the electronic device will compress the target file through the upper-layer service to obtain a compressed file of the target file, which can reasonably utilize the upper-layer service CPU resources, reduce the pressure on the underlying service CPU, and reduce the demand for transmission bandwidth.

[0080] S105: Call the underlying service to store the compressed file of the target file.

[0081] In this step, if Figure 2 As shown, after obtaining the compressed file of the target file, the electronic device will call the underlying service to store the compressed file of the target file. The underlying service does not need to perform compression, which releases the CPU pressure of the underlying service.

[0082] The embodiment of the present disclosure receives a target file uploaded by a user, determines whether the target file matches a pre-configured compression rule, and if the target file matches the pre-configured compression rule, determines whether the target file satisfies a compression condition. If the target file satisfies the compression condition, the target file is compressed by the upper-layer service to obtain a compressed file of the target file, and the underlying service is called to store the compressed file of the target file. Compared with the prior art, the embodiment of the present disclosure can realize data compression through the upper-layer service, reasonably utilize the CPU resources of the upper-layer service, realize dynamic control of compression capability through pre-configured compression rules and judgment of compression conditions, can flexibly perform compression, and does not require the underlying service to perform compression, thereby reducing the pressure on the underlying service CPU and reducing the demand for transmission bandwidth.

[0083] Figure 3 A flow chart of a data compression method provided by another embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the method includes the following steps:

[0084] S201: Receive a target file uploaded by a user.

[0085] Specifically, the implementation process and principle of S201 and S101 are the same and will not be repeated here.

[0086] S202: Obtain relevant information of the target file, where the relevant information of the target file includes a user identifier, a storage location, file prefix information, file suffix information, and a file type.

[0087] In this step, the electronic device obtains relevant information about the target file. The relevant information about the target file may include user identification, storage location, file prefix information, file suffix information, and file type, and may also include file size, without specific limitation. Optionally, the storage location is used to represent the directory or bucket where the target file is stored.

[0088] S203: Determine whether the target file matches a preconfigured compression rule based on the relevant information of the target file or the identifier of the global compression switch.

[0089] In this step, the electronic device can determine whether the target file matches the preconfigured compression rules based on the relevant information of the target file or the flag of the global compression switch. If the target file matches the preconfigured compression rules, S204 and the steps following S204 are executed. Optionally, if a global compression switch is configured, when the flag of the global compression switch is on, it indicates that all files can participate in compression, that is, the target file matches the preconfigured compression rules.

[0090] In some embodiments, the preconfigured compression rules include at least one of the following: target files matching a preset user identifier are allowed to be compressed, target files matching a preset storage location are allowed to be compressed, target files in a file prefix whitelist are allowed to be compressed, target files in a file suffix whitelist are allowed to be compressed, and target files in a file type whitelist are allowed to be compressed.

[0091] Optionally, the preconfigured compression rules may include one or a combination of the above rules. The preconfigured compression rules may also include target files in a file prefix blacklist, a file suffix blacklist, and a file type blacklist that are not allowed to be compressed, which is not specifically limited here.

[0092] In some embodiments, determining whether the target file matches the preconfigured compression rule based on the relevant information of the target file in S203 may include but is not limited to S2031, S2032, S2033, S2034, and S2035:

[0093] S2031: If the user identifier of the target file matches a preset user identifier, determining that the target file matches a preconfigured compression rule;

[0094] In this embodiment, by matching with a preset user identifier, it is possible to control at the user dimension whether the data of certain users can be compressed.

[0095] S2032: If the storage location of the target file matches the preset storage location, determining that the target file matches a preconfigured compression rule;

[0096] In this embodiment, by matching with preset storage locations, data in certain storage locations can be controlled to be compressed.

[0097] S2033: If the file prefix information of the target file is in the file prefix whitelist, determining that the target file matches the preconfigured compression rule;

[0098] In this embodiment, the file prefix whitelist is used to configure certain prefixes to participate in compression.

[0099] S2034: If the file suffix information of the target file is in the file suffix whitelist, determining that the target file matches the preconfigured compression rule;

[0100] In this embodiment, the file suffix whitelist is used to configure certain suffixes to participate in compression.

[0101] S2035: If the file type of the target file is in the file type whitelist, determine that the target file matches a preconfigured compression rule.

[0102] In this embodiment, the file type whitelist is used to configure certain types of files to participate in compression. Optionally, the file types are, for example, JPG, MP3, etc., without specific limitation.

[0103] In some embodiments, determining whether the target file matches the preconfigured compression rule according to the relevant information of the target file in S203 may further include S2036, S2037, and S2038:

[0104] S2036: If the file prefix information of the target file is not in the file prefix whitelist, or the file prefix information of the target file is in the file prefix blacklist, it is determined that the target file does not match the preconfigured compression rule;

[0105] In this embodiment, the file prefix blacklist is used to configure certain prefixes to not participate in compression.

[0106] S2037: If the file suffix information of the target file is not in the file suffix whitelist, or the file suffix information of the target file is in the file suffix blacklist, it is determined that the target file does not match the preconfigured compression rule;

[0107] In this embodiment, the file suffix blacklist is used to configure certain suffixes to not participate in compression.

[0108] S2038: If the file type of the target file is not in the file type whitelist, or the file type of the target file is in the file type blacklist, it is determined that the target file does not match the preconfigured compression rule.

[0109] In this embodiment, the file type blacklist is used to configure certain types of files to not participate in compression.

[0110] S204: When the target file matches a preconfigured compression rule, determine whether the target file meets a compression condition.

[0111] Specifically, the implementation process and principle of S204 and S103 are the same and will not be described in detail here.

[0112] S205: If the target file meets the compression condition, determine the compression type of the target file.

[0113] In this step, if it is determined that the target file meets the compression conditions, the electronic device can determine the compression category of the target file. In some optional embodiments, the compression category of the target file can be determined based on a user identifier or storage location. In other optional embodiments, the compression category of the target file can also be determined based on the file type. Optionally, the compression category can include a high compression ratio category, an extremely fast compression category, a fast compression category, a regular compression category, etc., without limitation.

[0114] S206: Determine a compression algorithm for the target file according to the compression type of the target file.

[0115] In this step, after determining the compression category of the target file, the electronic device will determine the compression algorithm of the target file according to the compression category of the target file. Each compression category has a corresponding compression algorithm. For example, if the compression category is a high compression ratio category, the zstd compression algorithm can be used; if the compression category is an ultra-fast compression category, the LZ4 compression algorithm can be used. zstd: has excellent compression ratio and fast compression speed. It can complete the compression of a large amount of data in a short time and can meet the application scenarios of real-time data processing such as video streaming and log recording. If most of the files of a certain customer or a certain bucket are video and log files, the compression algorithm can be configured as zstd. Its excellent compression ratio can reduce the cost of underlying storage capacity. LZ4: has a very fast compression speed and is famous for its ultra-high throughput. If a certain customer or certain buckets are very sensitive to request delays, the compression algorithm can be configured as LZ4. Its excellent compression speed can be used to compress files without affecting the customer's upload and download experience.

[0116] S207: compress the target file based on the compression algorithm of the target file through the upper layer service to obtain a compressed file of the target file.

[0117] In this step, after determining the compression algorithm of the target file, the electronic device will compress the target file based on the compression algorithm of the target file through the upper-layer service to obtain a compressed file of the target file. This can reasonably utilize the upper-layer service CPU resources, reduce the pressure on the underlying service CPU, and reduce the demand for transmission bandwidth.

[0118] S208: Call the underlying service to store the compressed file of the target file.

[0119] Specifically, the implementation process and principle of S208 and S105 are the same and will not be repeated here.

[0120] The disclosed embodiment receives a target file uploaded by a user and obtains relevant information about the target file, including user identification, storage location, file prefix information, file suffix information, and file type. Based on the relevant information about the target file or the identifier of a global compression switch, the method determines whether the target file matches a preconfigured compression rule. Furthermore, if the target file matches the preconfigured compression rule, the method determines whether the target file meets compression conditions. If the target file meets the compression conditions, the method determines the compression category of the target file. Next, based on the compression category of the target file, the method determines a compression algorithm for the target file. The upper-layer service compresses the target file based on the compression algorithm, obtaining a compressed file of the target file. The lower-layer service is then called to store the compressed file of the target file. This method enables data compression through the upper-layer service, effectively utilizing the upper-layer service's CPU resources. By determining the preconfigured compression rules and compression conditions, the method further determines whether the target file matches the preconfigured compression rules. This method enables dynamic control of compression capabilities, allows for flexible compression, and eliminates the need for the lower-layer service to perform compression, thereby reducing pressure on the lower-layer service's CPU and lowering transmission bandwidth requirements.

[0121] Figure 4 A flow chart of a data compression method provided by another embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the method includes the following steps:

[0122] S301: Receive a target file uploaded by a user.

[0123] Specifically, the implementation process and principle of S301 and S101 are the same and will not be described in detail here.

[0124] S302: Obtain the current load status of the upper layer service.

[0125] In this step, the electronic device obtains the current load of the upper layer service. The current load can be the usage of the CPU or disk, without limitation.

[0126] S303: Adjust the preset sampling ratio according to the current load of the upper layer service.

[0127] In this step, after obtaining the current load condition of the upper layer service, the electronic device may adjust the preset sampling ratio according to the current load condition of the upper layer service.

[0128] In some embodiments, when the current load condition of the upper-layer service is less than a preset first threshold, the value of the preset sampling ratio is increased; when the current load condition of the upper-layer service is greater than a preset second threshold, the value of the preset sampling ratio is decreased, wherein the preset first threshold is less than the preset second threshold.

[0129] In S302-S303, the current load of the upper-layer service can be determined based on the monitoring of the upper-layer service's CPU, disk, and other functions. The preset sampling ratio can be dynamically adjusted based on the current load of the upper-layer service. For example, if the current load is less than a preset first threshold, i.e., when the CPU is relatively idle, the preset sampling ratio can be increased, such as to 90%. If the current load is greater than a preset second threshold, such as during peak business hours, the sampling ratio can be reduced, such as to 60%.

[0130] S304: Determine whether the target file matches a preconfigured compression rule.

[0131] Specifically, the implementation process and principle of S304 and S102 are the same and will not be described in detail here.

[0132] S305: When the target file matches the preconfigured compression rule, if the file size of the target file is greater than or equal to a preset threshold, it is determined that the target file meets the compression condition.

[0133] In this step, if the file size of the target file is greater than or equal to a preset threshold, it is determined that the target file meets the compression condition. Because compression itself consumes CPU, if the file is too small, the cost savings are limited, and resources are occupied instead, which is not worth the loss.

[0134] S306: If the compression ratio of the target file is greater than or equal to a preset compression ratio threshold, it is determined that the target file meets the compression condition.

[0135] In this step, if the compression ratio of the target file is greater than or equal to a preset compression ratio threshold, the target file is determined to meet the compression condition. After a file is compressed and stored, decompression is triggered when it is read. Decompression itself consumes resources, so if the compression ratio is not very high, or if some files become larger after compression, there is no need to compress them.

[0136] S307: If the target file is within the range of the preset sampling ratio, it is determined that the target file meets the compression condition.

[0137] In this step, if the target file is within the range of the preset sampling ratio, it is determined that the target file meets the compression condition. By setting the preset sampling ratio, random sampling can be performed, and the current compression ratio can be dynamically controlled. For example, 60% means that there is a 60% chance that the file that meets the conditions will be compressed. In some embodiments, the value of the preset sampling ratio can be gradually increased according to the system resources of the upper-layer service, such as from 30% to 50% to 80%, or even to 100%, or the value of the preset sampling ratio can be maintained at a certain value due to system resource constraints.

[0138] In some embodiments, if the file size of the target file is greater than or equal to a preset threshold, the compression ratio of the target file is greater than or equal to a preset compression ratio threshold, and the target file is within a preset sampling ratio range, and all of the above conditions are met, then the target file is determined to meet the compression condition. If the file size of the target file is less than the preset threshold, or the compression ratio of the target file is less than the preset compression ratio threshold, or the target file is not within the preset sampling ratio range, then the target file is determined to not meet the compression condition.

[0139] S308: When the target file meets the compression condition, compress the target file through an upper layer service to obtain a compressed file of the target file.

[0140] Specifically, the implementation process and principle of S308 and S104 are the same and will not be repeated here.

[0141] S309: Call the underlying service to store the compressed file of the target file.

[0142] Specifically, the implementation process and principle of S309 and S105 are the same and will not be repeated here.

[0143] The disclosed embodiment receives a target file uploaded by a user, obtains the current load of the upper-layer service, and adjusts a preset sampling ratio based on the current load of the upper-layer service. Furthermore, it determines whether the target file matches a preconfigured compression rule. If the target file matches the preconfigured compression rule, if the file size of the target file is greater than or equal to a preset threshold, it is determined that the target file meets the compression condition. If the compression ratio of the target file is greater than or equal to a preset compression ratio threshold, it is determined that the target file meets the compression condition. If the target file is within the range of the preset sampling ratio, it is determined that the target file meets the compression condition. Furthermore, if the target file meets the compression condition, the upper-layer service compresses the target file to obtain a compressed file of the target file, and calls the lower-layer service to store the compressed file of the target file. This method enables data compression through the upper-layer service, rationally utilizing the upper-layer service CPU resources. By using preconfigured compression rules and compression condition determination, this embodiment can further determine whether the target file meets the compression condition, enabling dynamic control of compression capability and flexible compression. The lower-layer service does not need to perform compression, reducing pressure on the lower-layer service CPU and lowering the demand for transmission bandwidth.

[0144] Figure 5 This is a schematic diagram of the structure of the data compression device provided in the embodiment of the present disclosure. The data compression device can be the electronic device described in the above embodiment, or the data compression device can be a component or assembly in the electronic device. The data compression device provided in the embodiment of the present disclosure can execute the processing flow provided in the data compression method embodiment, such as Figure 5 As shown, the data compression device 50 includes: a receiving module 51, a first judgment module 52, a second judgment module 53, a compression module 54, and a storage module 55; wherein the receiving module 51 is used to receive the target file uploaded by the user; the first judgment module 52 is used to judge whether the target file matches the preconfigured compression rule; the second judgment module 53 is used to judge whether the target file meets the compression condition when the target file matches the preconfigured compression rule; the compression module 54 is used to compress the target file through the upper-layer service to obtain a compressed file of the target file when the target file meets the compression condition; the storage module 55 is used to call the underlying service to store the compressed file of the target file.

[0145] Optionally, when the first judgment module 52 judges whether the target file matches the preconfigured compression rules, it is specifically used to: obtain relevant information of the target file, the relevant information of the target file including user identification, storage location, file prefix information, file suffix information and file type; and judge whether the target file matches the preconfigured compression rules based on the relevant information of the target file or the identifier of the global compression switch.

[0146] Optionally, the preconfigured compression rules include at least one of the following: target files matching a preset user identifier are allowed to be compressed, target files matching a preset storage location are allowed to be compressed, target files in a file prefix whitelist are allowed to be compressed, target files in a file suffix whitelist are allowed to be compressed, and target files in a file type whitelist are allowed to be compressed;

[0147] When the first judgment module 52 judges whether the target file matches the preconfigured compression rule based on the relevant information of the target file, it is specifically used to: if the user identifier of the target file matches the preset user identifier, it is determined that the target file matches the preconfigured compression rule; if the storage location of the target file matches the preset storage location, it is determined that the target file matches the preconfigured compression rule; if the file prefix information of the target file is in the file prefix whitelist, it is determined that the target file matches the preconfigured compression rule; if the file suffix information of the target file is in the file suffix whitelist, it is determined that the target file matches the preconfigured compression rule; if the file type of the target file is in the file type whitelist, it is determined that the target file matches the preconfigured compression rule.

[0148] Optionally, when the first judgment module 52 judges whether the target file matches the preconfigured compression rule based on the relevant information of the target file, it is specifically used to: if the file prefix information of the target file is not in the file prefix whitelist, or the file prefix information of the target file is in the file prefix blacklist, then it is determined that the target file does not match the preconfigured compression rule; if the file suffix information of the target file is not in the file suffix whitelist, or the file suffix information of the target file is in the file suffix blacklist, then it is determined that the target file does not match the preconfigured compression rule; if the file type of the target file is not in the file type whitelist, or the file type of the target file is in the file type blacklist, then it is determined that the target file does not match the preconfigured compression rule.

[0149] Optionally, when the second judgment module 53 judges whether the target file meets the compression condition, it is specifically used to: if the file size of the target file is greater than or equal to a preset threshold, then it is judged that the target file meets the compression condition; if the compression ratio of the target file is greater than or equal to a preset compression ratio threshold, then it is judged that the target file meets the compression condition; if the target file is within the range of a preset sampling ratio, then it is judged that the target file meets the compression condition.

[0150] Optionally, the data compression device 50 further includes: an adjustment module 56; the adjustment module 56 is configured to obtain the current load condition of the upper layer service; and adjust the preset sampling ratio according to the current load condition of the upper layer service.

[0151] Optionally, when the compression module 54 compresses the target file through the upper-layer service to obtain a compressed file of the target file, it is specifically used to: determine the compression category of the target file; determine the compression algorithm of the target file according to the compression category of the target file; and compress the target file based on the compression algorithm of the target file through the upper-layer service to obtain a compressed file of the target file.

[0152] Figure 5 The data compression device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0153] Figure 6 This is a schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 6 , which shows a structural diagram of an electronic device 600 suitable for implementing the embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0154] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes to implement the data compression method of the embodiment described in the present disclosure according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0155] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0156] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart, thereby implementing the data compression method described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0157] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0158] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0159] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0160] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0161] Receive target files uploaded by users;

[0162] Determining whether the target file matches a preconfigured compression rule;

[0163] If the target file matches the pre-configured compression rule, determining whether the target file meets the compression condition;

[0164] When the target file meets the compression condition, compressing the target file through the upper layer service to obtain a compressed file of the target file;

[0165] The underlying service is called to store the compressed file of the target file.

[0166] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0167] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0170] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0171] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0172] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0173] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0174] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A data compression method, characterized in that: The method comprises: Receive target files uploaded by users; Determining whether the target file matches a preconfigured compression rule; If the target file matches the pre-configured compression rule, determining whether the target file meets the compression condition; When the target file meets the compression condition, compressing the target file through the upper layer service to obtain a compressed file of the target file; The underlying service is called to store the compressed file of the target file.

2. The method according to claim 1, characterized in that Determining whether the target file matches a preconfigured compression rule includes: Obtaining relevant information of the target file, wherein the relevant information of the target file includes user identification, storage location, file prefix information, file suffix information, and file type; It is determined whether the target file matches a preconfigured compression rule according to the relevant information of the target file or the identifier of the global compression switch.

3. The method according to claim 2, characterized in that The preconfigured compression rules include at least one of the following: Target files matching a preset user identifier are allowed to be compressed, target files matching a preset storage location are allowed to be compressed, target files in a file prefix whitelist are allowed to be compressed, target files in a file suffix whitelist are allowed to be compressed, and target files in a file type whitelist are allowed to be compressed; Determining whether the target file matches a preconfigured compression rule according to relevant information of the target file includes: If the user identifier of the target file matches the preset user identifier, determining that the target file matches the preconfigured compression rule; If the storage location of the target file matches the preset storage location, determining that the target file matches the preconfigured compression rule; If the file prefix information of the target file is in the file prefix whitelist, determining that the target file matches the preconfigured compression rule; If the file suffix information of the target file is in the file suffix whitelist, determining that the target file matches the preconfigured compression rule; If the file type of the target file is in the file type whitelist, it is determined that the target file matches the preconfigured compression rule.

4. The method according to claim 2, characterized in that Determining whether the target file matches a preconfigured compression rule according to relevant information of the target file includes: If the file prefix information of the target file is not in the file prefix whitelist, or the file prefix information of the target file is in the file prefix blacklist, it is determined that the target file does not match the preconfigured compression rule; If the file suffix information of the target file is not in the file suffix whitelist, or the file suffix information of the target file is in the file suffix blacklist, it is determined that the target file does not match the preconfigured compression rule; If the file type of the target file is not in the file type whitelist, or the file type of the target file is in the file type blacklist, it is determined that the target file does not match the preconfigured compression rule.

5. The method according to claim 1, wherein The determining whether the target file meets the compression condition includes: If the file size of the target file is greater than or equal to a preset threshold, it is determined that the target file meets the compression condition; If the compression ratio of the target file is greater than or equal to a preset compression ratio threshold, it is determined that the target file meets the compression condition; If the target file is within the range of the preset sampling ratio, it is determined that the target file meets the compression condition.

6. The method according to claim 5, characterized in that The method further comprises: Get the current load status of the upper-layer service; The preset sampling ratio is adjusted according to the current load of the upper layer service.

7. The method according to claim 1, characterized in that The method of compressing the target file by using an upper layer service to obtain a compressed file of the target file includes: Determining a compression type of the target file; Determining a compression algorithm for the target file according to a compression category of the target file; The target file is compressed by an upper layer service based on a compression algorithm of the target file to obtain a compressed file of the target file.

8. A data compression device, characterized in that: include: Receiving module, used to receive target files uploaded by users; A first judgment module is used to judge whether the target file matches a preconfigured compression rule; A second judgment module is used to judge whether the target file meets the compression condition when the target file matches the pre-configured compression rule; A compression module, configured to compress the target file through an upper layer service to obtain a compressed file of the target file when the target file meets the compression condition; The storage module is used to call the underlying service to store the compressed file of the target file.

9. An electronic device, characterized in that: include: Memory; processor; as well as computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images