Method, apparatus, and system for compressing data, and related device

Abstract

A method, apparatus, and system for compressing data, and a related device, relating to the technical field of data processing. The method comprises: acquiring raw data and running states of a plurality of compression operators on acceleration hardware, each compression operator running by means of an operator unit on the acceleration hardware; and further, on the basis of the running states of the plurality of compression operators, determining an available first portion of compression operators on the acceleration hardware and second portion of compression operators on a general-purpose processor participating in compressing the raw data, so as to use the first portion of compression operators and the second portion of compression operators to compress the raw data to obtain a data compression result. In this way, when the overall load on the acceleration hardware is high, compression operators available on the acceleration hardware can still be used to participate in compressing the raw data, so that resources on the acceleration hardware can be fully utilized, the problem of resource waste of the acceleration hardware is alleviated, and the resource utilization rate of the acceleration hardware is improved, thereby ensuring that the compression effect for the raw data reaches a high level.

Description

Data compression method, device, system and related equipment

[0001] The present application claims priority to the Chinese patent application No. 202410882099.6, filed on July 2, 2024, and entitled “Data compression method, device, system and related equipment”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of data processing, and in particular, to a data compression method, device, system and related equipment. BACKGROUND

[0003] In the era of big data, data is usually compressed and the compressed data is persistently stored to reduce the storage resources required for data storage. Data compression can be generally divided into software compression and hardware compression. Software compression refers to executing program codes corresponding to compression operators by a central processing unit (CPU) to compress data. Hardware compression refers to offloading compression operators to acceleration hardware such as field programmable gate array (FPGA) to compress data by using the acceleration hardware.

[0004] In actual application, software compression or hardware compression of data is usually flexibly selected according to the load conditions of the CPU and the acceleration hardware. Specifically, when the load of the acceleration hardware is small, the hardware compression process for data is performed by the acceleration hardware; when the load of the acceleration hardware is large and the load of the CPU is small, the CPU runs a compression algorithm to perform software compression on the data. When the loads of the acceleration hardware and the CPU are both large, the data is in a state of waiting to be compressed until the load of the CPU is reduced to perform software compression on the data by the CPU, or until the load of the acceleration hardware is reduced to perform hardware compression on the data by the acceleration hardware.

[0005] However, this way of compressing data often has the problem of resource waste, such as low resource utilization rate on the acceleration hardware. SUMMARY

[0006] The present application provides a data compression method to achieve high resource utilization rate and alleviate resource waste on the acceleration hardware while ensuring that the data compression effect can reach a high level. In addition, the present application also provides a data compression device, a data processing method, a data processing device, a storage system, a computing device, a computer readable storage medium and a computer program product.

[0007] In a first aspect, the present application provides a data compression method, which is applied to a storage system including a general processor (such as a CPU, etc.) and a first acceleration hardware including a plurality of operator units, each of which is configured to run a compression operator. The method can be executed by the general processor or a data compression device separately configured in the storage system. The method includes: obtaining original data to be compressed, which can be structured data or unstructured data, and obtaining running states of a plurality of compression operators on the first acceleration hardware, which can indicate whether the compression operators can be used to participate in compression of the original data; then, according to the running states of the plurality of compression operators, determining a first part of compression operators and a second part of compression operators participating in compression of the original data, the first part of compression operators being available compression operators on the first acceleration hardware, and the second part of compression operators being compression operators on the general processor, so as to compress the original data by using the first part of compression operators on the first acceleration hardware and the second part of compression operators on the general processor, and obtain a corresponding data compression result.

[0008] Thus, in the process of compressing the original data, the available compression operators on the first acceleration hardware are used to compress the original data according to the running states of the compression operators on the first acceleration hardware, without considering the load on the first acceleration hardware. This makes it possible to use the available (small load) compression operators on the first acceleration hardware to participate in compression of the original data even when the overall load on the first acceleration hardware is high, so as to fully utilize the hardware resources on the first acceleration hardware, alleviate the problem of resource waste of the first acceleration hardware, and improve the resource utilization rate of the first acceleration hardware. Meanwhile, the compression operators on the general processor are run and participate in compression of the original data, which makes it possible to fully compress the original data based on the first part of compression operators and the second part of compression operators, so as to ensure that the compression effect on the original data reaches a high level. Furthermore, when the loads on the first acceleration hardware and the general processor are both high, the plurality of compression operators required to be executed on the original data are dispersed to different hardware, so that the remaining computing power on the plurality of hardware can meet the computing power requirement of compression of the original data. This makes it possible for the original data to be compressed without being queued, so as to improve the response rate of the storage system to compression of the original data and improve the overall performance of the storage system.

[0009] In a possible implementation, before compressing the original data, the second part of the compression operators is started on the general processor, such as loading and executing program code of the second part of the compression operators on the general processor. In this way, for the second part of the compression operators that need to participate in compressing the original data but are unavailable on the first acceleration hardware, the running can be started on the general processor, so as to realize sufficient compression of the original data and ensure the compression effect.

[0010] In a possible implementation, the storage system is configured with a decision model, which can be a neural network model or other types of models. When determining the first part of the compression operators and the second part of the compression operators that participate in compressing the original data according to the running states of the plurality of compression operators, the decision model can be used to infer according to the original data and the running states of the plurality of compression operators on the first acceleration hardware, to obtain the first part of the compression operators and the second part of the compression operators output by the decision model. In this way, the decision model can be used to determine the first part of the compression operators running on the first acceleration hardware and the second part of the compression operators running on the general processor for the original data, to obtain the combination of the software compression and the hardware compression, so as to ensure the compression effect for the original data while fully utilizing the hardware resources on the first acceleration hardware.

[0011] In a possible implementation, when determining the first part of the compression operators and the second part of the compression operators that participate in compressing the original data according to the running states of the plurality of compression operators, the data features of the original data can be extracted first, and the operator set is determined according to the data features, wherein the plurality of compression operators in the operator set participate in compressing the original data. Then, the first part of the compression operators in the operator set is determined according to the running states of the plurality of compression operators on the first acceleration hardware, and the second part of the compression operators in the operator set is determined according to the first part of the compression operators. In this way, the combination of the software compression and the hardware compression can be determined according to the compression operators in the operator set that participate in compressing the original data and the running states of the compression operators in the operator set on the first acceleration hardware, so as to ensure the compression effect for the original data while fully utilizing the hardware resources on the first acceleration hardware.

[0012] In a possible implementation, before determining the second part of compression operators, the running state of the general processor can also be obtained, which can be the load of the general processor, for example, so that when the first part of compression operators and the second part of compression operators participating in compressing the original data are determined according to the running state of the plurality of compression operators on the first acceleration hardware, the first part of compression operators and the second part of compression operators participating in compressing the original data can be specifically determined according to the running state of the plurality of compression operators and the running state of the general processor. In this way, the combination of operators running on different hardware is determined in combination with the running state of the general processor, which can enable the general processor to support running all the determined second part of compression operators, so as to achieve successful compression of the original data subsequently.

[0013] In a possible implementation, when the first part of compression operators and the second part of compression operators participating in compressing the original data are determined according to the running state of the plurality of compression operators and the running state of the general processor, the plurality of candidate operator sets can be first determined according to the original data and the running state of the plurality of compression operators, and then the target operator set including the first part of compression operators and the second part of compression operators can be determined from the plurality of candidate operator sets according to the running state of the general processor. In this way, the suitable candidate operator set is selected from the plurality of candidate operator sets according to the running state of the general processor, which enables the general processor to have sufficient computing power to support the running of the second part of compression operators after the second part of compression operators in the candidate operator set are deployed to the general processor, so as to achieve successful compression of the original data subsequently.

[0014] In a possible implementation, during the compression of the original data, the data compression result can also be compressed by using at least one available compression operator running on the first acceleration hardware. In this way, the compression rate for the original data can be further reduced, and the compression effect for the original data can be improved.

[0015] In a possible implementation, the first part of compression operators includes a plurality of same first compression operators, and the plurality of same first compression operators are used to perform a parallel compression process on the original data; or the second part of compression operators includes a plurality of same second compression operators, and the plurality of same second compression operators are used to perform a parallel compression process on the original data. In this way, by using the plurality of compression operators on the first acceleration hardware to perform parallel compression on the original data, or using the plurality of compression operators on the general processor to perform parallel compression on the original data, the overall compression efficiency for the original data can be improved, and the overall performance of the storage system can be improved.

[0016] In a possible implementation, the storage system further includes second acceleration hardware, the second acceleration hardware including a plurality of operator units, each of the plurality of operator units on the second acceleration hardware being configured to run one compression operator; and when determining the first part of compression operators and the second part of compression operators participating in compressing the original data according to the running states of the plurality of compression operators, the determination can be specifically according to the running states of the plurality of compression operators on the first acceleration hardware and the running states of the plurality of compression operators on the second acceleration hardware, to determine the first part of compression operators, the second part of compression operators, and a third part of compression operators participating in compressing the original data, the third part of compression operators being available compression operators on the second acceleration hardware; and correspondingly, when compressing the original data by using the first part of compression operators running on the first acceleration hardware and the second part of compression operators running on the general processor, the compression can be specifically by using the first part of compression operators running on the first acceleration hardware, the second part of compression operators running on the general processor, and the third part of compression operators running on the second acceleration hardware. In this way, the compression operators on the plurality of acceleration hardware are used to participate in the hardware compression process of the original data, which can make full use of the hardware resources on the plurality of acceleration hardware, alleviate the problem of resource waste of the plurality of acceleration hardware, and improve the resource utilization of the plurality of acceleration hardware.

[0017] In a possible implementation, the running state of a compression operator includes a throughput of the compression operator, a number of tasks processed in parallel by the compression operator, or a health state of the compression operator. Alternatively, the running state of a compression operator can also be whether the compression operator is occupied, and the like, which is not limited in this regard.

[0018] In a second aspect, the present application provides a data processing method, which is applied to a data processing system including a general processor and first acceleration hardware, the first acceleration hardware including a plurality of operator units, each of the plurality of operator units being configured to run one calculation operator; the method including: obtaining original data to be processed and running states of a plurality of calculation operators on the first acceleration hardware; determining a first part of calculation operators and a second part of calculation operators participating in calculating the original data according to the running states of the plurality of calculation operators, the first part of calculation operators being available calculation operators on the first acceleration hardware, and the second part of calculation operators being calculation operators on the general processor; and processing the original data by using the first part of calculation operators on the first acceleration hardware and the second part of calculation operators on the general processor to obtain a data processing result.

[0019] Thus, in the process of computing the original data, the original data is computed by using available computing operators on the first acceleration hardware according to the running state of the computing operators on the first acceleration hardware, without considering the load on the first acceleration hardware, which enables the available (smaller load) computing operators on the first acceleration hardware to participate in computing the original data even when the overall load on the first acceleration hardware is high, thereby fully utilizing the hardware resources on the first acceleration hardware, alleviating the problem of resource waste of the first acceleration hardware, and improving the resource utilization of the first acceleration hardware. Meanwhile, the computing operators are run on the general processor and participate in computing the original data, which enables the original data to be fully computed based on the first part of the computing operators and the second part of the computing operators, thereby ensuring that the computing effect for the original data reaches a high level. Moreover, when the loads on the first acceleration hardware and the general processor are both high, by dispersing the multiple computing operators required to be executed for the original data to different hardware, the remaining computing power on the multiple hardware can meet the demand for computing power for computing the original data, which enables the original data to be computed without being in a state of queuing and waiting, thereby improving the response rate of the data processing system for computing the original data and improving the overall performance of the data processing system.

[0020] In a third aspect, the present application provides a data compression device, which is applied to a storage system, the storage system comprising a general processor and a first acceleration hardware, the first acceleration hardware comprising a plurality of operator units, each of the plurality of operator units being configured to run a compression operator, the device comprising: an acquisition module configured to acquire original data to be compressed and running states of a plurality of compression operators on the first acceleration hardware; an operator determination module configured to determine, according to the running states of the plurality of compression operators, a first part of compression operators and a second part of compression operators participating in compressing the original data, the first part of compression operators being available compression operators on the first acceleration hardware, and the second part of compression operators being compression operators on the general processor; and a compression module configured to compress the original data by using the first part of compression operators on the first acceleration hardware and the second part of compression operators on the general processor, to obtain a data compression result.

[0021] In a possible implementation, the data compression device further comprises an operator starting module configured to start the second part of compression operators on the general processor before compressing the original data.

[0022] In a possible implementation, the storage system is configured with a decision model, and the operator determination module is specifically configured to perform reasoning according to the original data and the running states of the compression operators on the first acceleration hardware by using the decision model, to obtain the first part of compression operators and the second part of compression operators output by the decision model.

[0023] In a possible implementation, the operator determination module is configured to: extract data features of the original data; determine a set of operators according to the data features, a plurality of compression operators in the set of operators participating in compression of the original data; determine the first part of compression operators in the set of operators according to the running states of the plurality of compression operators on the first acceleration hardware; and determine the second part of compression operators in the set of operators according to the first part of compression operators.

[0024] In a possible implementation, the obtaining module is further configured to obtain a running state of the general processor, and the operator determination module is specifically configured to determine the first part of compression operators and the second part of compression operators participating in compression of the original data according to the running states of the plurality of compression operators and the running state of the general processor.

[0025] In a possible implementation, the operator determination module is specifically configured to: determine a plurality of candidate sets of operators according to the original data and the running states of the plurality of compression operators; and determine a target set of operators from the plurality of candidate sets of operators according to the running state of the general processor, the target set of operators including the first part of compression operators and the second part of compression operators.

[0026] In a possible implementation, the compression module is further configured to compress the data compression result by using at least one available compression operator running on the first acceleration hardware.

[0027] In a possible implementation, the first part of compression operators includes a plurality of same first compression operators, and the plurality of same first compression operators are used to perform a parallel compression process on the original data; or the second part of compression operators includes a plurality of same second compression operators, and the plurality of same second compression operators are used to perform a parallel compression process on the original data.

[0028] In a possible implementation, the storage system further includes second acceleration hardware, the second acceleration hardware including a plurality of operator units, each of the plurality of operator units on the second acceleration hardware being configured to run one compression operator; the operator determining module is specifically configured to determine, according to the running states of the plurality of compression operators on the first acceleration hardware and the running states of the plurality of compression operators on the second acceleration hardware, the first part of compression operators, the second part of compression operators, and a third part of compression operators participating in compressing the original data, the third part of compression operators being available compression operators on the second acceleration hardware; and the compression module is specifically configured to compress the original data by using the first part of compression operators running on the first acceleration hardware, the second part of compression operators running on the general processor, and the third part of compression operators running on the second acceleration hardware.

[0029] In a possible implementation, the running state of the compression operator includes a throughput of the compression operator, a number of tasks processed in parallel by the compression operator, or a health state of the compression operator.

[0030] The data compression apparatus provided in the third aspect corresponds to the data compression method provided in the first aspect, and any implementation manner in the third aspect has the technical effects described with reference to the corresponding implementation manner in the first aspect, which will not be described herein.

[0031] In the fourth aspect, the present application provides a data processing apparatus, which is applied to a data processing system, the data processing system including a general processor and first acceleration hardware, the first acceleration hardware including a plurality of operator units, each of the plurality of operator units being configured to run one calculation operator; the apparatus includes: an acquisition module configured to acquire original data to be processed and running states of a plurality of calculation operators on the first acceleration hardware; an operator determining module configured to determine, according to the running states of the plurality of calculation operators, a first part of calculation operators and a second part of calculation operators participating in calculating the original data, the first part of calculation operators being available calculation operators on the first acceleration hardware, and the second part of calculation operators being calculation operators on the general processor; and a processing module configured to process the original data by using the first part of calculation operators on the first acceleration hardware and the second part of calculation operators on the general processor to obtain a data processing result.

[0032] The data processing apparatus provided in the fourth aspect corresponds to the data processing method provided in the second aspect, and the implementation manner in the fourth aspect has the technical effects described with reference to the implementation manner in the second aspect, which will not be described herein.

[0033] In a fifth aspect, the present application provides a computing device, comprising a processor, a memory. The processor and the memory are in communication with each other. The processor is configured to execute instructions stored in the memory, so as to cause the computing device to perform the operation steps of the data compression method according to the first aspect or any implementation manner of the first aspect, or so as to cause the computing device to perform the operation steps of the data processing method according to the second aspect or any implementation manner of the second aspect. It should be noted that the memory can be integrated into the processor, or can be independent of the processor. The computing device can further comprise a bus. The processor is connected to the memory through the bus. The memory can comprise a readable memory and a random access memory.

[0034] In a sixth aspect, the present application provides a storage system, comprising a general processor, acceleration hardware, at least one storage medium. The general processor is configured to perform the operation steps of the data compression method according to the first aspect or any implementation manner of the first aspect. The acceleration hardware comprises a plurality of operator units, each of the plurality of operator units is configured to run one compression operator. The at least one storage device is configured to persistently store the data compression result.

[0035] In a seventh aspect, the present application provides a storage system, comprising a general processor, acceleration hardware, at least one storage device and a data compression device. The data compression device is configured to perform the operation steps of the data compression method according to the first aspect or any implementation manner of the first aspect. The at least one storage medium is configured to persistently store the data compression result.

[0036] In an eighth aspect, the present application provides a computer readable storage medium, which stores instructions. When the instructions are executed on a computing device, the computing device is caused to perform the operation steps of the data compression method according to the first aspect or any implementation manner of the first aspect, or the computing device is caused to perform the operation steps of the data processing method according to the second aspect or any implementation manner of the second aspect.

[0037] In a ninth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, cause the computing device to perform the operation steps of the data compression method according to the first aspect or any implementation manner of the first aspect, or cause the computing device to perform the operation steps of the data processing method according to the second aspect or any implementation manner of the second aspect.

[0038] On the basis of the implementation manners of the above aspects, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0039] Fig. 1a is a structural schematic diagram of an exemplary storage system;

[0040] Fig. 1b is a structural schematic diagram of another exemplary storage system;

[0041] Fig. 2 is a schematic diagram of resource waste on acceleration hardware;

[0042] Fig. 3 is a flow schematic diagram of a data compression method provided by the present application;

[0043] Fig. 4 is a flow schematic diagram of compression on original data;

[0044] Fig. 5 is a schematic diagram of utilizing compression operators on acceleration hardware for multiple compression tasks;

[0045] Fig. 6 is a schematic diagram of scheduling compression operators on multiple acceleration hardware to compress original data;

[0046] Fig. 7 is a flow schematic diagram of a data processing method provided by the present application;

[0047] Fig. 8 is a structural schematic diagram of a data compression apparatus provided by the present application;

[0048] Fig. 9 is a structural schematic diagram of a data processing apparatus provided by the present application;

[0049] Fig. 10 is a hardware structural schematic diagram of a computing device provided by the present application. DETAILED DESCRIPTION

[0050] In order to make the above objectives, features and advantages of the present application more apparent and comprehensible, various non-limiting embodiments in the present application will be exemplarily described below in conjunction with the accompanying drawings. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained based on the above content are within the scope of protection of the present application.

[0051] Referring to Fig. 1a, it is a structural schematic diagram of an exemplary storage system. As shown in Fig. 1a, the storage system 10 includes a general-purpose processor 100, at least one acceleration hardware, and at least one storage medium. For ease of illustration, Fig. 1a illustrates an example including M acceleration hardware and N storage medium, wherein the M acceleration hardware are acceleration hardware 1 to acceleration hardware M respectively, the N storage medium are storage medium 1 to storage medium N respectively, and M and N are positive integers.

[0052] The general-purpose processor 100 may, for example, be a central processing unit (CPU) or other types of processors with general-purpose computing power.

[0053] Each acceleration hardware component can be, for example, an FPGA or an application-specific integrated circuit (ASIC). Typically, acceleration hardware can have multiple operator units, each capable of running a compression operator. Taking an FPGA as an example, an FPGA can include multiple programmable logic units (PLUs), and multiple operator units can be constructed based on these PLUs. Each operator unit includes at least one PLU for running a compression operator. A compression operator is an operator used to compress data. Typically, after effective compression using a compression operator, the amount of compressed data is less than the amount of data before compression. A compression operator can be an operator in a compression algorithm that produces a data compression effect, and a single compression algorithm can include one or more compression operators.

[0054] Each storage medium, such as a solid-state drive (SSD), hard disk drive (HDD), or hard disk, can be used for persistent data storage. Furthermore, the storage system 10 can include N storage media of the same type, or it can include N storage media of different types.

[0055] As shown in Figure 1a, the storage system 10 can establish communication connections with one or more applications, and each application can send data to the storage system 10 for persistent storage. For ease of description and distinction, this data will be referred to as raw data below. For example, the application shown in Figure 1a may be a client on the user side, or a process running on a server.

[0056] The general processor 100 can receive the raw data sent by the application through a communication interface or a network card, etc. Generally, the amount of raw data sent by the application is large, therefore, the general processor 100 can first compress the raw data by using the general processor 100 and / or at least one acceleration hardware to reduce the storage space required for data storage. For example, when the raw data is "ABCADABABC", the general processor 100 can schedule a dictionary matching type operator (a compression operator) running on the acceleration hardware to compress the raw data. The dictionary matching type operator can compress the string by using a pre-created dictionary, and the repeated characters in the string can be indicated by the specified encoding in the dictionary. Assuming that the dictionary includes the strings AB, ABC, ABCD, and the specified encoding of each string is 1, 2, and 3 respectively, the dictionary matching type operator on the acceleration hardware can be used to compress the raw data "ABCADABABC" to "2AD12". Alternatively, the general processor 100 can run the dictionary matching type operator to compress the raw data, etc. In actual application, the compression of the raw data can be completed by using multiple compression operators to compress the data to obtain a data compression result with the smallest data amount as possible.

[0057] In this process, if the general processor 100 selects to compress the raw data by using the acceleration hardware when the overall load of the acceleration hardware is low, and selects to compress the raw data by using the general processor 100 when the overall load of the acceleration hardware is high according to the overall load of the acceleration hardware, this will cause that when the overall load of the acceleration hardware is high, some operator units corresponding to the compression operators are still in idle state, which will cause the waste of hardware resources for executing the compression operators. That is to say, when the overall load of the acceleration hardware is high, even if there are idle computing units on the acceleration hardware that can run the compression operators to compress the raw data, the general processor 100 will not schedule the computing units to execute the compression process for the raw data.

[0058] For example, as shown in FIG. 2, in the process of hardware compression of data in the compression task 1 by using the Lempel-Ziv (LZ) compression operator, the huffman compression operator, and the Burrows-Wheeler transform (BWT) compression operator on the acceleration hardware 1, the overall load of the acceleration hardware 1 is generally high, but the litDelta compression operator on the acceleration hardware 1 is still in an idle state at this time. When the storage system 10 needs to compress data in the compression task 2 by using the litDelta compression operator, the storage system 10 will select to use the general processor 100 to run the litDelta compression operator (and other compression operators) and perform a software compression process on the data, without deciding whether to schedule the acceleration hardware 1 to compress the original data according to whether the operator on the acceleration hardware 1 is idle. Thus, the hardware resources corresponding to the litDelta compression operator on the acceleration hardware 1 are in an idle state for a long time, resulting in waste of hardware resources.

[0059] Therefore, in the storage system 10 provided in the present application, the general processor 100 improves the resource utilization rate of the acceleration hardware by combining software compression and hardware compression.

[0060] In a specific implementation, the general processor 100 can obtain the running states of a plurality of compression operators on the acceleration hardware 1 (assuming that the storage system 1 includes only one acceleration hardware), wherein the running state of a compression operator can be used to indicate whether the compression operator is currently available (i.e., whether the compression operator can be used to participate in compression of the original data), such as when the load of the compression operator is large (the running state), the compression operator is not available, and when the load of the compression operator is small, the compression operator is available. Then, the general processor 100 determines, according to the original data and the running states of the plurality of compression operators on the acceleration hardware 1, a first part of compression operators (such as the load of the first part of compression operators is small) that are available for compression of the original data and run on the acceleration hardware 1, and determines a second part of compression operators (such as the load of the second part of compression operators is large) that need to participate in compression of the original data but are in an unavailable state on the acceleration hardware 1, and then the general processor 100 can execute the second part of compression operators to participate in compression of the original data. For example, the general processor 100 can execute the program code corresponding to the second part of compression operators. Thus, the general processor 100 can compress the original data by using the first part of compression operators running on the acceleration hardware 1 and the second part of compression operators running on the general processor 100, and obtain a corresponding data compression result. Further, the general processor 100 can send the generated data compression result to one or more storage media for storage.

[0061] Thus, in the process of compressing the original data, the general processor 100 selects the available compression operators (i.e., the first part of the compression operators) on the acceleration hardware 1 to compress the original data according to the running state of the compression operators on the acceleration hardware 1, without considering the load on the acceleration hardware 1, which enables the idle (small load) compression operators on the acceleration hardware 1 to participate in the compression of the original data in the case of high load on the acceleration hardware 1, so as to fully utilize the hardware resources on the acceleration hardware 1, alleviate the problem of waste of resources of the acceleration hardware 1, and improve the resource utilization rate of the acceleration hardware 1. Meanwhile, the second part of the compression operators run on the general processor 100 and participate in the compression of the original data, which enables the original data to be fully compressed based on the first part of the compression operators and the second part of the compression operators, so as to ensure that the compression effect on the original data reaches a high level.

[0062] Moreover, in the case of high load on both the acceleration hardware 1 and the general processor 100, by dispersing the multiple compression operators required to be executed on the original data to different hardware (i.e., to the acceleration hardware 1 and the general processor 100), the total remaining computing power of the multiple hardware can meet the demand for computing power for compressing the original data, which enables the original data to be compressed without being in a state of queuing, so as to improve the response rate of the storage system 10 to the compression of the original data and improve the overall performance of the storage system 10.

[0063] Similarly, when the storage system 10 includes multiple acceleration hardware, the general processor 100 can utilize the available compression operators on the multiple acceleration hardware and the compression operators running on the general processor to compress the original data according to the running state of the multiple compression operators on the multiple acceleration hardware, and send the generated data compression result to the corresponding storage medium for storage.

[0064] It is worth noting that in the storage system 10 shown in FIG. 1a, the general processor 100 schedules the acceleration hardware and the general processor 100 runs multiple compression operators to compress the original data, and in other implementations, a data compression device separately configured in the storage system 10 can also schedule the acceleration hardware and the general processor 100 to run multiple compression operators to compress the original data. As shown in FIG. 1b, the storage system 10 shown in FIG. 1b can further include a data compression device 200 on the basis of the storage system 10 shown in FIG. 1a, and the data compression device 200 can interact with the application to obtain the original data to be compressed, and schedule part of the compression operators on the acceleration hardware 1 to participate in the compression of the original data and schedule another part of the compression operators on the general processor 100 to participate in the compression of the original data according to the running state of the multiple compression operators on the acceleration hardware 1, so as to obtain the final compression result.

[0065] When the data compression apparatus 200 is integrated with the acceleration hardware 1 in one device, the data compression apparatus 200 can be connected with the acceleration hardware 1 through a bus, and the data compression apparatus 200 can access the acceleration hardware 1 (and other acceleration hardware) based on the bus to obtain the running states of the plurality of compression operators on the acceleration hardware 1; or the running states of the plurality of compression operators on the acceleration hardware 1 can be collected by the general processor 100, so that when the data compression apparatus 200 requests the running states of the compression operators, the general processor 100 can send the collected running states of the plurality of compression operators on the acceleration hardware 1 to the data compression apparatus 200, so that the data compression apparatus 200 subsequently schedules the compression operators on the acceleration hardware 1 to participate in compressing the original data.

[0066] When the data compression apparatus 200 and the acceleration hardware 1 are deployed in different devices, the running states of the plurality of compression operators on the acceleration hardware 1 (and other acceleration hardware) can be collected by the general processor 100, so that the data compression apparatus 200 can request the running states of the compression operators from the general processor 100, and the general processor 100 sends the running states of the plurality of compression operators on the acceleration hardware 1 to the data compression apparatus 200. The specific implementation of the data compression apparatus 200 obtaining the running states of the plurality of compression operators on the acceleration hardware 1 (and other acceleration hardware) is not limited in the present application. For example, the data compression apparatus 200 can be a management node in the storage system 10, and can be responsible for scheduling the general processors and the acceleration hardware in each device to perform data compression tasks.

[0067] Exemplarily, the data compression apparatus 200 in FIG. 1b can be implemented by a processor or at least one physical device including the processor. The processor can be any one of a central processing unit (CPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, a data processing unit (DPU), or any combination thereof. The number of processors included in the data compression apparatus 200 can be one or more, and the types of processors included can be one or more. The number and types of processors can be set according to actual application requirements, and the present embodiment does not limit the number and types of processors.

[0068] In actual applications, the N storage media in the storage system 10 shown in FIG. 1a or FIG. 1b can be deployed in a centralized manner or in a distributed manner.

[0069] When deployed in a centralized manner, the storage system 10 can be deployed in a storage server configured with N storage media. For example, a plurality of SSDs (each SSD being one of the storage media shown in FIG. 1a or FIG. 1b) can be inserted into the storage server. The general processor 100 / data compression apparatus 200 in the storage system 10 can compress data and store the compression result in the configured storage media.

[0070] When distributed deployment is adopted, the N storage media in the storage system 10 can be distributed in multiple different storage servers, each of which can be configured with one or more storage media, for example, multiple SSDs (each of which is one storage medium shown in FIG. 1a or FIG. 1b) can be inserted on each storage server. And each storage server can be configured with a general processor. In the storage system 10 shown in FIG. 1b, the data compression device 200 can run in one of the storage servers, or can run in a separate device, or can be implemented through hardware in the storage server or hardware in a separately deployed device. Thus, the general processor 100 or the data compression device 200 can store the compression result to the storage medium in at least one storage server after compressing the data. In the distributed storage system, different storage servers can communicate based on the compute express link (CXL) protocol, the unified bus (UB or Ubus) protocol, or the remote direct memory access (RDMA) protocol, or can communicate based on other protocols.

[0071] It is worth noting that the storage system 10 shown in FIG. 1a or FIG. 1b is only an exemplary illustration and is not intended to be limiting. For example, in other possible storage systems, the storage system can include an application for generating data to be compressed, or can include more other hardware devices for supporting the storage system to have more other functions.

[0072] For ease of understanding, the embodiments of the data compression method provided by the present application are described below with reference to the accompanying drawings.

[0073] Referring to FIG. 3, FIG. 3 is a flow diagram of a data compression method provided by an embodiment of the present application. The method can be applied to the storage system 10 described in FIG. 1a or FIG. 1b, or can be applied to other applicable storage systems. For ease of illustration, the general processor 100 in the storage system 10 shown in FIG. 1a is taken as an example for exemplary illustration in this embodiment.

[0074] Specifically, the data compression method shown in FIG. 3 can include the following steps.

[0075] S301: The application sends the original data to be compressed to the storage system 10.

[0076] The original data to be compressed can be structured data or unstructured data. The structured data refers to data stored in a set format or rule, such as data stored in a table. The unstructured data refers to data with incomplete or irregular data structure, such as image data, audio / video data, etc. Different unstructured data usually has different data characteristics, and thus is suitable for compression by different compression operators.

[0077] In a possible implementation, the application can generate original data during running and send the original data to the storage system 10. For example, when the application is a client on the user side, the client can generate a data storage request according to a storage operation instruction of the user for the original data, and send the data storage request to the storage system 10, where the data storage request includes the original data. Accordingly, the storage system 10 can receive the data storage request through the communication interface and parse the original data designated to be saved by the user from the data storage request. In actual application, the storage system 10 can also obtain the original data through other manners, which is not limited.

[0078] Accordingly, the general processor 100 can receive the original data through the corresponding communication interface.

[0079] S302: The general processor 100 obtains the running states of the plurality of compression operators on the acceleration hardware 1.

[0080] The number of acceleration hardware included in the storage system 10 can be one or more. For ease of understanding and description, the storage system 10 includes one acceleration hardware (i.e., the acceleration hardware 1) in this embodiment.

[0081] For example, the general processor 100 can periodically collect the running states of the plurality of compression operators on the acceleration hardware 1 to determine whether the plurality of compression operators on the acceleration hardware 1 can be used to compress data. Alternatively, the general processor 100 can collect the running states of the plurality of compression operators on the acceleration hardware 1 after determining that there is data to be compressed.

[0082] As an implementation example, the running state of the compression operator can include one or more of a throughput of the compression operator, a number of tasks processed in parallel by the compression operator, and a health state of the compression operator, or can be other state information possessed by the compression operator during operation. The throughput of the compression operator refers to the amount of data compressed by the compression operator per unit time. The number of tasks processed in parallel by the compression operator refers to the number of tasks in the process of compressing data in parallel by the compression operator. The health state of the compression operator can be used to indicate whether the compression operator has an abnormality, for example, whether the compression operator has an abnormality can be determined according to the temperature, voltage and other working parameters of the operator unit running the compression operator.

[0083] Taking the running state including the throughput, the number of tasks processed in parallel, and the health state as an example, when the throughput of the compression operator is less than a threshold 1, the number of tasks processed in parallel by the compression operator is less than a threshold 2, and the health state of the compression operator indicates that there is no abnormality, the general processor 100 can determine that the compression operator is in an available state (this time can be used to participate in compressing the original data). Conversely, when the throughput of the compression operator is greater than the threshold 1, it indicates that the current load of the compression operator is large and it is difficult to participate in compressing the original data, at this time, the general processor 100 can determine that the compression operator is in an unavailable state (this time cannot be used to participate in compressing the original data). Or, when the number of tasks processed in parallel by the compression operator is greater than the threshold 2, it indicates that the current number of tasks processed in parallel by the compression operator is too large and it is difficult to process new tasks, at this time, the general processor 100 can determine that the compression operator is in an unavailable state. Or, when the health state of the compression operator indicates that there is an abnormality, it indicates that the compression operator cannot operate normally, that is, it is difficult to execute the process of compressing the original data, at this time, the general processor 100 can determine that the compression operator is in an unavailable state.

[0084] As another implementation example, the running state of the compression operator can be used to indicate whether the compression operator is occupied by other data compression tasks. That is, when the compression operator is currently executing other data compression tasks (executing a data compression process), the general processor 100 can determine that the compression operator is in an unavailable state, and when the compression operator is not currently executing a data compression task, the general processor 100 can determine that the compression operator is in an available state.

[0085] The compression operators running on the acceleration hardware 1 can be, for example, Lempel-Ziv (LZ) compression operators, huffman compression operators, Burrows-Wheeler transform (BWT) compression operators, hzdbPre compression operators, hzbcPer compression operators, litDelta compression operators, asymmetric numeral systems (ANS) compression operators, or finite state entropy (FSE) compression operators, or other types of compression operators, without any limitation.

[0086] When obtaining the running states of the compression operators on the acceleration hardware 1, the general processor 100 can obtain the running states of all the compression operators on the acceleration hardware 1 by accessing the acceleration hardware 1.

[0087] Alternatively, the general processor 100 can only collect the running states of the compression operators on the acceleration hardware 1 that are suitable for participating in compressing the original data, and the general processor 100 can not need to collect the running states of the compression operators that are not suitable for compressing the data. The implementation of the general processor 100 determining the compression operators suitable for participating in compressing the original data is introduced below.

[0088] First, when the amount of the original data is large, the general processor 100 can sample the original data, for example, by using a random sampling algorithm to sample the original data, to obtain a small amount of sample data. Then, the general processor 100 can extract the features of the sample data, and determine one or more data types to which the sample data belongs according to the features of the sample data. For example, the general processor 100 can calculate the proportion of numerical values or the proportion of strings in the sample data as a feature of the sample data. Generally, since the sample data is obtained by sampling the original data, the features of the sample data are similar to the features of the original data, so the general processor 100 can use the features extracted from the sample data as the features of the original data. In actual applications, when the amount of the original data is small, the general processor 100 can directly extract the features of the original data.

[0089] Then, the general processor 100 can determine the data type to which the original data belongs according to the characteristics of the original data. For example, when the proportion of the number of numerical values in the original data to the total number of data is large (e.g., greater than 0.4, etc.), the general processor 100 can determine that the original data belongs to the numerical value type; at the same time, when the proportion of the number of strings in the original data to the total number of data is large (e.g., greater than 0.5, etc.), the general processor 100 can determine that the original data also belongs to the string type. At this time, the original data belongs to both the numerical value type and the string type. When the proportion of the number of numerical values in the original data is small and the proportion of the number of strings is large, the general processor 100 can also determine that the data type to which the original data belongs is only the string type.

[0090] Finally, the general processor 100 can determine a plurality of compression operators that match the data type to which the original data belongs according to the correspondence between the data type and the compression operator that is pre-saved. For example, when the data type to which the original data belongs includes the numerical value type and the string type, the general processor 100 can determine that the compression operator corresponding to the numerical value type is the LZ compression operator, and determine that the compression operator corresponding to the string type is the huffman compression operator. The correspondence between the data type and the compression operator can be pre-configured by the technician according to experience in the general processor 100, so that the general processor 100 can determine the appropriate compression operator for the original data according to the correspondence. In actual application, the general processor 100 can also determine the plurality of compression operators used to compress the original data in other ways, which is not limited.

[0091] In this way, after determining the plurality of compression operators used to compress the original data, the general processor 100 can collect the running states of these compression operators running on the acceleration hardware 1, so as to subsequently determine which compression operators can be used to compress the original data from these compression operators.

[0092] S303: The general processor 100 determines the first part of the compression operators and the second part of the compression operators participating in compressing the original data according to the running states of the plurality of compression operators, the first part of the compression operators being the available compression operators running on the acceleration hardware 1, and the second part of the compression operators being the compression operators on the general processor 100.

[0093] In actual application, part of the compression operators on the acceleration hardware 1 can be compressing the data of other tasks, so that the part of the compression operators cannot currently participate in compressing the original data. Therefore, the general processor 100 can determine the compression operators that can participate in compressing the original data according to the running states of the plurality of compression operators on the acceleration hardware 1. For the convenience of distinguishing and describing, the compression operators are referred to as the first part of the compression operators in this embodiment.

[0094] In this embodiment, the following implementation examples of determining the first part of compression operators and the second part of compression operators for the original data are provided.

[0095] In the first implementation example, the general processor 100 can be configured with a decision model, which can be a neural network model or a random forest model, etc., and can be used to infer the multiple compression operators participating in data compression and the first part of compression operators running on the acceleration hardware 1 from the original data.

[0096] Specifically, the general processor 100 can first extract data features from the original data, and input the data features and the running states of each compression operator on the acceleration hardware 1 into the decision model, execute the corresponding inference process by the decision model, and output the inference result, which includes the identification of the multiple compression operators (such as the names of the multiple compression operators, etc.), and the identification of the first part of compression operators in the multiple compression operators, which can be a mark indicating that the compression operator is running on the acceleration hardware 1, etc.

[0097] Alternatively, as shown in FIG. 4, the general processor 100 can first filter at least one compression operator available for participating in compressing the original data on the acceleration hardware 1 according to the running states of each compression operator, and the general processor 100 can also collect the data features of the original data (such as by analyzing the original data to extract the data features therefrom), and input the data features and the at least one compression operator in the available state (specifically, the identification of the compression operator) into the decision model, infer and output the identification of the multiple compression operators and the identification of the first part of compression operators by the decision model, as shown in FIG. 4, wherein the identification of the other compression operators in the identification of the multiple compression operators except for the identification of the first part of compression operators is the identification of the second part of compression operators.

[0098] Thus, the general processor 100 can determine the first part of compression operators used for compressing the original data on the acceleration hardware 1 according to the inference result output by the decision model, and determine the second part of compression operators which are not run on the acceleration hardware 1 according to the identification of the plurality of compression operators included in the inference result and the identification of the first part of compression operators. As shown in FIG. 4, the general processor 100 can determine that the first part of compression operators run on the acceleration hardware 1 is the compression operator 1, and determine that the second part of compression operators which need to be run on the general processor 100 later is the compression operator b. In actual application scenarios, the storage system 10 can perform a plurality of data compression tasks in parallel, so that the general processor 100 can not only start to run the compression operator b and compress the original data by using the compression operator b and the compression operator 1 on the acceleration hardware 1, but also run the compression operator a and perform a data compression process on other data by using the compression operator a and the compression operator 2 on the acceleration hardware 1.

[0099] In the embodiment, the second part of compression operators are run on the general processor 100, and the running state of the general processor 100 also affects the selection of the compression operators participating in the compression of the original data by the general processor 100. For example, when the number of compression operators running on the general processor 100 is too large, the remaining computing power of the general processor 100 cannot support the running of the compression operators in the large number.

[0100] Therefore, in a further possible implementation, the general processor 100 can also determine the compression operators participating in the compression of the original data in combination with the running state of the general processor 100. Specifically, before the inference by the decision model, the data processing apparatus 100 can obtain the running state of the general processor 100, as shown in FIG. 4. The running state of the general processor 100 may, for example, include the utilization rate and the memory usage rate of the general processor 100, and then the general processor 100 can input the data features of the original data, the running state of each compression operator on the acceleration hardware 1, and the running state of the general processor 100 into the decision model to obtain the identification of the plurality of compression operators and the identification of the first part of compression operators output by the decision model, as shown in FIG. 4. Thus, when the second part of compression operators in the plurality of compression operators are run on the general processor 100 later, the computing power of the general processor 100 can support the running of the second part of compression operators.

[0101] In the second implementation example, the general processor 100 can extract a data feature of the original data, and analyze a set of operators participating in compressing the original data according to the data feature, the set of operators including a plurality of compression operators. Wherein, the general processor 100 determines the specific implementation manner of the plurality of compression operators in the set of operators according to the data feature, which can be referred to the foregoing description, and will not be described here. Then, the general processor 100 can collect the running states of the plurality of compression operators belonging to the set of operators on the acceleration hardware 1 according to the set of operators, so as to determine a first part of compression operators in the set of operators running on the acceleration hardware 1 and in the available state according to the running states of the compression operators, and determine a second part of compression operators belonging to the set of operators on the acceleration hardware 1 but in the unavailable state.

[0102] Further, the general processor 100 can also obtain the running state of the general processor 100 before determining the second part of compression operators. In this way, the general processor 100 can determine m compression operators which need to participate in compressing the original data but are in the unavailable state on the acceleration hardware 1 according to the first part of compression operators and the plurality of compression operators which have been determined, m being a positive integer. Then, the general processor 100 can determine the second part of compression operators supported by the general processor 100 to run from the m compression operators according to the running state of the general processor 100 and the demand of each compression operator in the m compression operators for computing power.

[0103] Wherein, when the running state of the general processor 100 indicates that the load of the general processor is small (can support running the m compression operators), the general processor 100 can take all the m compression operators as the second part of compression operators. When the running state of the general processor 100 indicates that the load of the general processor is large (difficult to support running the m compression operators), the general processor 100 can take part of the m compression operators as the second part of compression operators. For example, the general processor 100 can determine the matching degree between each compression operator in the m compression operators and the original data according to the data feature of the original data, wherein the higher the matching degree corresponding to the compression operator, the higher the compression effect of the compression operator on the original data, so that the general processor 100 can determine the first k compression operators with the highest matching degree from the m compression operators as the second part of compression operators according to the remaining computing power of the general processor 100, k being a positive integer less than m, and the total computing power required by the k compression operators does not exceed the remaining computing power of the general processor 100.

[0104] In a third implementation example, the general processor 100 can first determine a plurality of candidate operator sets according to the original data and the running states of the respective compression operators on the acceleration hardware 1. For example, the general processor 100 can determine the plurality of candidate operator sets based on the manners described in the first implementation example or the second implementation example. The compression operators included in different candidate operator sets can be different, such as different candidate operator sets can include different numbers of compression operators, or different candidate operator sets can include different types of compression operators, and the like. Each candidate operator set includes two parts of compression operators, one part of the compression operators can be used to compress the original data and is in an available state on the acceleration hardware 1, and the other part of the compression operators can be used to compress the original data but is in an unavailable state on the acceleration hardware 1. The compression operators included in different candidate operator sets and in the available state on the acceleration hardware 1 can be different.

[0105] Then, the general processor 100 can traverse the plurality of candidate operator sets according to the running state of the general processor 100 to determine a target operator set (that is, one of the candidate operator sets) from the plurality of candidate operator sets, and the total amount of computing power required by at least one compression operator in the target operator set required to run on the general processor 100 does not exceed the remaining computing power of the general processor 100. Thus, the general processor 100 can determine the first part of the compression operators and the second part of the compression operators in the target operator set. In this way, after the second part of the compression operators is deployed to the general processor 100, the general processor 100 can have sufficient computing power to support the running of the second part of the compression operators.

[0106] In a fourth implementation example, the general processor 100 can first determine an operator set for the original data, and the operator set includes a plurality of compression operators. At the same time, the general processor 100 can also obtain the running states of the respective compression operators in the operator set on the acceleration hardware 1 to determine whether the respective compression operators in the operator set are available on the acceleration hardware 1. For example, when the running state of a compression operator indicates that the compression operator is currently performing a data compression task, such as the throughput of the compression operator is not 0, or the number of tasks performed by the compression operator is not 0, and the like, the general processor 100 can determine that the compression operator is in an unavailable state. When the running state of a compression operator indicates that the compression operator is not currently performing a data compression task, the general processor 100 can determine that the compression operator is in an available state. Thus, the general processor 100 can determine which compression operators in the plurality of compression operators included in the operator set are in an available state on the acceleration hardware 1 (that is, the first part of the compression operators), and which compression operators are in an unavailable state on the acceleration hardware 1 (that is, the second part of the compression operators).

[0107] It should be noted that the above various implementations are only as some exemplary descriptions, and in actual application, the general processor 100 can also determine the first part of the compression operator and the second part of the compression operator for the original data in other manners, such as combination or transformation of the above various implementations, and the like, which is not limited.

[0108] S304: Start the second part of the compression operator on the general processor 100.

[0109] It can be understood that since the second part of the compression operator on the acceleration hardware 1 is unavailable, that is, the second part of the compression operator on the acceleration hardware 1 cannot be used to participate in the hardware compression of the original data. Therefore, the general processor 100 can start the second part of the compression operator on the general processor 100, so as to perform a software compression process on the original data based on the second part of the compression operator. As shown in FIG. 4, the general processor 100 can start to run the compression operator b (that is, the second part of the compression operator) participating in the compression of the original data.

[0110] In a specific implementation, before determining the second part of the compression operator, the general processor 100 does not run the second part of the compression operator, and when it is determined that the second part of the compression operator needs to be run, the general processor 100 can load and execute the program code corresponding to the second part of the compression operator to start the running of the second part of the compression operator. For example, the general processor 100 can access an operator library according to the identifier of the second part of the compression operator to obtain the program code of the second part of the compression operator from the operator library, and the operator library includes the program code corresponding to a plurality of compression operators respectively. Then, the general processor 100 can execute the program code to start to run the second part of the compression operator on the general processor 100. In this way, the general processor 100 starts to run the compression operator dynamically, which not only meets the demand for running the compression operator in the process of compressing the original data, but also avoids running unnecessary compression operators in the process of not performing data compression (that is, before compressing the original data), thereby avoiding wasting the computing power of the general processor 100.

[0111] It should be noted that in the embodiment, the general processor 100 does not run the second part of the compression operator before obtaining the original data, and in other embodiments, the general processor 100 can also run the second part of the compression operator in advance, such as the general processor 100 running the second part of the compression operator in the process of executing the last compression task, and the general processor 100 does not unload the second part of the compression operator after the execution of the compression task is completed. Therefore, in the process of compressing the original data this time, the general processor 100 can continue to perform the process of compressing the original data by using the unloaded second part of the compression operator (and the first part of the compression operator on the acceleration hardware 1).

[0112] S305: The general processor 100 compresses the original data by using the first part of compression operators on the acceleration hardware 1 and the second part of compression operators on the general processor 100 to obtain a data compression result.

[0113] In a specific implementation, the general processor 100 can schedule the acceleration hardware 1 and the general processor 100 to perform the data compression process in sequence by using the running first part of compression operators and the second part of compression operators respectively according to a predetermined compression strategy, which can be used to indicate the compression order of each compression operator. For example, the general processor 100 can schedule the acceleration hardware 1 to compress the original data by using the compression operator 1 first to obtain intermediate data, and schedule the compression operator b on the general processor 100 to compress the intermediate data.

[0114] The compression strategy can be provided by the decision model. For example, in the process in which the general processor 100 determines the first part of compression operators and the second part of compression operators for compressing the original data by using the decision model, the decision model can also output the compression strategy together, so that the general processor 100 can control each compression operator to start performing the data compression process in sequence according to the compression strategy.

[0115] Alternatively, in the process in which the general processor 100 determines the multiple compression operators participating in compressing the original data according to the data characteristics of the original data, the general processor 100 can determine the order of compressing the original data according to the logical relationship between different compression operators, so as to generate a compression strategy for indicating the order.

[0116] In this way, in the process of compressing the original data, the general processor 100 executes the hardware compression process for the original data by using the available compression operators on the acceleration hardware 1 according to the running state of the compression operators on the acceleration hardware 1, and executes the software compression process for the original data by deploying the remaining compression operators that need to be run on the general processor 100, so as to effectively compress the original data by flexibly combining the software compression mode and the hardware compression mode, and ensure the compression rate of the original data. Moreover, the remaining computing power on the acceleration hardware 1 and the remaining computing power on the general processor 100 are comprehensively used to compress the original data, which can effectively avoid the insufficient remaining computing power of a single hardware affecting the compression effect of the original data, such as a lower compression rate or a too large compression delay, and improve the overall performance of the storage system 10.

[0117] In an actual application scenario, for data under other compression tasks, the general processor 100 can also schedule the compression operators on the acceleration hardware 1 to participate in hardware compression of the data under the compression tasks based on the similar manner, and deploy the compression operators on the general processor 100 to participate in software compression of the data under the compression tasks. In this way, the hardware resources on the acceleration hardware 1 can be fully utilized, so as to alleviate the problem of resource waste of the acceleration hardware 1 and improve the resource utilization rate of the acceleration hardware 1. As shown in FIG. 5, assuming that the acceleration hardware 1 includes a litDelta compression operator, a hzdbpre compression operator, an FSE compression operator, and an LZ compression operator, for data 1 under a compression task 1, the general processor 100 can schedule the litDelta compression operator and the FSE compression operator to perform a hardware compression process for the data 1. For data 2 under a compression task 2, the hzdbpre compression operator, the FSE compression operator, and the LZ compression operator need to be used for compression, and the general processor 100 can schedule the hzdbpre compression operator and the LZ compression operator to perform a hardware compression process for the data 2. Since the FSE compression operator is currently compressing the data 1 under the compression task 1, the FSE compression operator is unavailable, and therefore the general processor 100 can deploy and run the FSE compression operator on the general processor 100 to perform a software compression process for the data 2. In this way, all the compression operators on the acceleration hardware 1 can be utilized, and the resource utilization rate of the acceleration hardware 1 is improved.

[0118] In an actual application, the second part of the compression operators running on the general processor 100 can include the same compression operators and the like.

[0119] It can be understood that the compression operators running on the acceleration hardware 1 and the compression operators running on the general processor 100 can have different data compression rates. For example, the data compression rate of the compression operators on the acceleration hardware 1 can be greater than the data compression rate of the compression operators on the general processor 100. Therefore, the general processor 100 can instruct the general processor 100 to deploy the same plurality of compression operators, so as to utilize the same plurality of compression operators to perform parallel compression on the compression results output by the compression operators on the acceleration hardware 1, so as to ensure the overall compression efficiency for the original data. The parallelization number of the compression operators on the general processor 100 can be determined according to the data compression rate. For example, the data compression rate of a compression operator A on the acceleration hardware 1 can be n times the data compression rate of a compression operator B on the general processor 100, where n is a positive integer greater than 1. ​is a floor function. Then, the general processor 100 can instruct the general processor 100 to deploy n identical compression operators B, so that the general processor 100 can perform parallel compression process on the compressed results output by the compression operator A using the n compression operators B.

[0120] Alternatively, the first part of the compression operators running on the acceleration hardware 1 can include identical compression operators, such as the data compression rate of the compression operator x on the general processor 100 is greater than the data compression rate of the compression operator y on the acceleration hardware 1, etc. At this time, the data processing apparatus 100 can instruct the acceleration hardware 1 to perform parallel compression on the compressed results output by the compression operator x on the general processor 100 using multiple compression operators y, so as to ensure the overall compression efficiency for the original data.

[0121] S306: The general processor 100 sends the data compression result to the corresponding storage medium for persistent storage.

[0122] After completing the compression of the original data and obtaining the data compression result, the general processor 100 can send the data compression result to one or more storage media, so as to perform persistent storage of the data compression result using the storage media. In this way, the storage space required for storing the data compression result in the storage medium is usually smaller than the storage space required for storing the original data, so as to reduce the occupation of the storage resource by the data storage.

[0123] In addition, after completing the compression of the data and outputting it to the storage medium for persistent storage, the general processor 100 can also feed back a notification message to the application through the communication interface, so as to notify the application of the completion of the data storage, or notify the application of the compressed storage of the data, etc.

[0124] Further, the general processor 100 can also instruct the acceleration hardware 1 or the general processor 100 to run additional compression operators to further reduce the compression rate (i.e. the ratio between the amount of data after compression and the amount of data before compression) of the original data, so as to improve the compression effect for the original data.

[0125] As a first implementation example, since the load of the acceleration hardware 1 and the general processor 100 can dynamically change in the actual application scenario, for example, the load of the acceleration hardware 1 and the general processor 100 is high when the general processor 100 determines the first part of the compression operators and the second part of the compression operators, and the acceleration hardware 1 and the general processor 100 cannot support more compression operators to participate in compressing the original data. However, during the process of compressing the original data based on the first part of the compression operators and the second part of the compression operators, the load of the acceleration hardware 1 can be reduced, or the load of the general processor 100 can be reduced, so that the acceleration hardware 1 or the general processor 100 can support running other compression operators that can be used to participate in compressing the original data.

[0126] Therefore, before compressing the original data, the general processor 100 can also determine whether the first part of the compression operators and the second part of the compression operators are all compression operators that can be used to participate in compressing the original data. If not, the general processor 100 can determine other compression operators that can be used to participate in compressing the original data, which are referred to as the third part of the compression operators. In this way, during the process of compressing the original data by using the acceleration hardware 1 and the general processor 100, the general processor 100 can also monitor whether the third part of the compression operators on the acceleration hardware 1 is in an available state. If the third part of the compression operators is in the available state, the general processor 100 can also schedule the third part of the compression operators on the acceleration hardware 1 that changes from the unavailable state to the available state to the original data, so as to use the third part of the compression operators to perform the compression process for the original data, and improve the compression efficiency for the original data. Alternatively, the general processor 100 can monitor whether the remaining computing power on the general processor 100 can support the third part of the compression operators to run, and if the remaining computing power can support the third part of the compression operators to run (for example, the tasks on the general processor 100 are completed and the computing power is released), the general processor 100 can instruct the general processor 100 to additionally deploy the third part of the compression operators, and use the third part of the compression operators to participate in performing the compression process for the original data.

[0127] As a second implementation example, after the general processor 100 completes the compression on the original data based on the first partial compression operator on the acceleration hardware 1 and the second partial compression operator on the general processor 100 and obtains the data compression result, the general processor 100 can determine whether the third partial compression operator on the acceleration hardware 1 is available according to the running state of the third partial compression operator. If yes, the general processor 100 can continue to compress the data compression result by using the third partial compression operator on the acceleration hardware 1 to further reduce the compression rate of the original data. Alternatively, after obtaining the data compression result, the general processor 100 can monitor whether the remaining computing power on the general processor 100 can support the running of the third partial compression operator. If yes, the general processor 100 can instruct the general processor 100 to run the third partial compression operator and compress the data compression result by using the third partial compression operator to further reduce the compression rate of the original data.

[0128] As a third implementation example, after obtaining the data compression result and sending it to the storage medium for storage, the general processor 100 can monitor whether the load of the acceleration hardware 1 or the load of the general processor 100 is less than a threshold. If yes, the general processor 100 can read the data compression result in the storage medium and further compress the data compression result by using the third partial compression operator running on the acceleration hardware 1 or the third partial compression operator running on the general processor 100, and save the data obtained after compression to the storage medium, so as to further reduce the storage space required by the original data in the storage system 10.

[0129] It is worth noting that the execution order of each step shown in FIG. 3 is only an example, and in other embodiments, the execution order of each step can also be in other orders, such as executing step S301 in parallel with step S302, or executing step S302 before step S301.

[0130] In addition, in the embodiment shown in FIG. 3, the general processor 100 uses the compression operator on one acceleration hardware to perform hardware compression on the original data, and in other embodiments, the general processor 100 can also schedule the compression operators on multiple acceleration hardware to perform hardware compression on the original data.

[0131] With the general processor 100 scheduling the compression operators on the acceleration hardware 1 and the acceleration hardware 2 to compress the original data as an example. The general processor 100 can acquire the running states of the plurality of compression operators on the acceleration hardware 1 and the running states of the plurality of compression operators on the acceleration hardware 2, and determine the first part of compression operators, the second part of compression operators and the third part of compression operators for the original data according to the running states of the plurality of compression operators on each acceleration hardware, wherein the third part of compression operators are the available compression operators running on the acceleration hardware 2. Wherein the specific implementation manner of the general processor 100 to determine the first part of compression operators and the second part of compression operators can be referred to the description of the related places in the above-mentioned embodiment shown in FIG. 3, and will not be described here. In addition, the specific implementation manner of the general processor 100 to determine the third part of compression operators can be referred to the description of the related places of determining the first part of compression operators, and will not be described here.

[0132] In this way, the general processor 100 can compress the original data by using the first part of compression operators running on the acceleration hardware 1, the third part of compression operators running on the acceleration hardware 2, and the second part of compression operators deployed and running on the general processor 100, to obtain the corresponding data compression result.

[0133] For example, it is assumed that the compression operators participating in compressing the original data include FSE compression operators, LZ compression operators, BWT compression operators and huffman compression operators, and these compression operators are also deployed on the acceleration hardware 1 and the acceleration hardware 2 respectively, as shown in FIG. 6. Among them, the FSE compression operators, the BWT compression operators and the huffman compression operators on the acceleration hardware 1 are in an unavailable state, and the FSE compression operators and the BWT compression operators on the acceleration hardware 2 are in an unavailable state. Then, the general processor 100 can execute the hardware compression process for the original data by using the LZ compression operators on the acceleration hardware 1 and the FSE compression operators and the BWT compression operators on the acceleration hardware 2 according to the running states of the plurality of compression operators on each acceleration hardware, and deploy and run the huffman compression operators on the general processor 100, so as to execute the software compression process for the original data by using the huffman compression operators on the general processor 100, as shown in FIG. 6.

[0134] In addition, FIG. 3 is used to illustrate the case that the general processor 100 determines the first partial compression operator and the second partial compression operator, in other embodiments, when applied to the storage system 10 shown in FIG. 1b, the above steps S301 to S304 can be performed by the data compression device 200 in the storage system 10, and the acceleration hardware 1 is scheduled to participate in the compression of the original data using the first partial compression operator, and the general processor 100 is scheduled to participate in the compression of the original data using the second partial compression operator, and the data compression device 200 sends the obtained compression result to the corresponding storage medium for storage. For specific implementation, please refer to the description of the related part in the above embodiment shown in FIG. 3, which will not be repeated here.

[0135] In this application, in addition to improving the resource utilization rate of the acceleration hardware in the data compression scene, the resource utilization rate of the acceleration hardware in the data calculation scene can also be improved, that is, the calculation operator running on the acceleration hardware and the calculation operator running on the general processor can be used to perform the corresponding data calculation process on the data.

[0136] Referring to FIG. 7, a flowchart of a data calculation method provided by the present application is shown. The method can be applied to a data processing system including a general processor and an acceleration hardware, such as FPGA or ASIC, which has a plurality of operator units, each of which is used to run a calculation operator. As shown in FIG. 7, the method can specifically include the following steps.

[0137] S701: The application sends the original data to be processed to the general processor.

[0138] The application can be a user-side client, or a process running in a server, etc., which is not limited. During the running process, the application can generate data that needs to be calculated and processed, which is referred to as original data below, and the application can send the original data to the general processor for corresponding calculation.

[0139] S702: The general processor obtains the running state of the plurality of calculation operators on the acceleration hardware 1.

[0140] The calculation operator refers to an operator used for calculating data. For example, in the field of face detection, the calculation operator can be an operator for detecting the face position in an image, an operator for detecting the face key points in an image, an operator for positioning the key parts of the face in an image, an operator for extracting the face features in an image, an operator for adjusting the light and angle in an image, etc. In actual application, the calculation operator can be an operator in a calculation algorithm that can be used for calculating data, and a single calculation algorithm can include one or more calculation operators.

[0141] Correspondingly, the running state of the computing operator can be used to indicate whether the computing operator can be used to compute the original data this time. Illustratively, the running state of the computing operator can include one or more of the throughput of the computing operator, the number of tasks processed in parallel by the computing operator, and the health status of the computing operator, or can be other state information possessed by the computing operator during operation.

[0142] S703: The general processor determines, according to the running states of the plurality of computing operators, a first part of computing operators and a second part of computing operators to participate in computing the original data, the first part of computing operators being the computing operators available on the acceleration hardware 1, and the second part of computing operators being the computing operators on the general processor.

[0143] The general processor can determine the first part of computing operators and the second part of computing operators according to the running states of the plurality of computing operators using an AI model, or can determine the first part of computing operators and the second part of computing operators according to the running states of the plurality of computing operators and the running state of the general processor using an AI model. Alternatively, the general processor can also determine the computing operators running on different hardware based on other manners, and the specific implementation manner can be referred to the description of the related part in the embodiment shown in FIG. 3 about determining the first part of compression operators and the second part of compression operators, which will not be described here. The AI model can be a neural network model, or can be a random forest model, etc., which is not limited.

[0144] S704: The general processor starts the second part of computing operators on the general processor.

[0145] Illustratively, the general processor can instruct the general processor to run the program code corresponding to the second part of computing operators. For example, the general processor can send a command to the general processor, the command including the identifier of the second part of computing operators, so that the general processor can acquire the program code of the second part of computing operators according to the identifier carried in the command, and run the program code to realize running the second part of computing operators on the general processor.

[0146] S705: The general processor processes the original data using the first part of computing operators running on the acceleration hardware 1 and the second part of computing operators running on the general processor to obtain a data processing result.

[0147] In a specific implementation, the general processor can schedule the acceleration hardware and the general processor to respectively execute corresponding data calculation processes on the original data by using the running first part of calculation operators and the second part of calculation operators, to obtain corresponding data processing results. The execution order of each calculation operator can be specified by the general processor, for example, the general processor can determine the execution order according to the calculation logic between different calculation operators.

[0148] In this way, in the process of calculating the original data, the general processor executes the hardware calculation process on the original data by using the available calculation operators on the acceleration hardware according to the running state of the calculation operators on the acceleration hardware, and executes the software calculation process on the original data by deploying the remaining calculation operators that need to be run on the general processor, so as to effectively calculate the original data by flexibly combining the software calculation method and the hardware calculation method. Moreover, the remaining calculation power on the acceleration hardware and the remaining calculation power on the general processor are comprehensively used to calculate the original data, which can effectively avoid the insufficient remaining calculation power of a single hardware affecting the calculation of the original data.

[0149] Further, after completing the calculation of the original data and obtaining the corresponding data processing results, the general processor 100 can also feed back a notification message to the application through the communication interface to notify the application that the processing process of the original data is completed.

[0150] In addition, when the acceleration hardware all includes calculation operators, the general processor can also schedule the calculation operators on multiple acceleration hardware to calculate and process the original data.

[0151] Taking the general processor scheduling the calculation operators on the acceleration hardware 1 and the acceleration hardware 2 to calculate the original data as an example. The general processor can obtain the running states of the multiple calculation operators on the acceleration hardware 1 and the running states of the multiple calculation operators on the acceleration hardware 2, and determine the first part of calculation operators, the second part of calculation operators, and the third part of calculation operators for the original data according to the running states of the calculation operators on each acceleration hardware, wherein the third part of calculation operators are the available calculation operators running on the acceleration hardware 2. The specific implementation of the general processor determining the first part of calculation operators and the second part of calculation operators can be referred to the description in the above related places, and will not be described here. In addition, the specific implementation of the general processor determining the third part of calculation operators can be referred to the description of determining the first part of calculation operators, and will not be described here. In this way, the general processor can process the original data by using the first part of calculation operators running on the acceleration hardware 1, the third part of calculation operators running on the acceleration hardware 2, and the second part of calculation operators deployed and running on the general processor, to obtain corresponding data processing results.

[0152] It should be noted that FIG. 7 is used as an example to determine the first part of the calculation operator and the second part of the calculation operator by the general processor, and in other embodiments, the data processing system can also include a data processing device, and the data processing device can perform the above steps S701 to S704, and schedule the first part of the calculation operator to participate in the calculation of the original data by the acceleration hardware 1, and schedule the second part of the compression operator to participate in the calculation of the original data by the general processor. For specific implementation, please refer to the description of the related parts in the above embodiment shown in FIG. 7, which will not be repeated here.

[0153] It is worth noting that other reasonable combinations of steps that can be thought of by those skilled in the art based on the above description are also within the scope of protection of the present application. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily required by the present application.

[0154] The above describes the storage system, data compression method and data processing method provided by the embodiments of the present application in combination with FIGS. 1a to 7. Next, the structure of the data compression device, data processing device and computing device provided by the embodiments of the present application will be introduced in combination with the accompanying drawings.

[0155] Referring to FIG. 8, a structural schematic diagram of a data compression device is shown, wherein the data compression device 800 shown in FIG. 8 is applied to a storage system, and the storage system includes a general processor and a first acceleration hardware, and the first acceleration hardware includes a plurality of operator units, and each operator unit in the plurality of operator units is used to run a compression operator. As shown in FIG. 8, the data compression device 800 includes:

[0156] The acquisition module 801 is configured to acquire original data to be compressed and running states of a plurality of compression operators on the first acceleration hardware.

[0157] The operator determination module 802 is configured to determine, according to the running states of the plurality of compression operators, a first part of compression operators and a second part of compression operators participating in compression of the original data, the first part of compression operators being available compression operators on the first acceleration hardware, and the second part of compression operators being compression operators on the general processor.

[0158] The compression module 803 is configured to compress the original data by using the first part of compression operators on the first acceleration hardware and the second part of compression operators on the general processor to obtain a data compression result.

[0159] In a possible implementation, the data compression device 800 further includes:

[0160] The operator starting module 804 is configured to start the second part of compression operators on the general processor before compressing the original data.

[0161] In a possible implementation, the storage system is configured with a decision model.

[0162] The operator determining module 802 is specifically configured to utilize the decision model to infer according to the original data and the running states of the plurality of compression operators on the first acceleration hardware, to obtain a first part of compression operators and a second part of compression operators in the decision model output.

[0163] In a possible implementation, the operator determining module 802 is specifically configured to:

[0164] extract data features of the original data;

[0165] determine an operator set according to the data features, and a plurality of compression operators in the operator set participate in compressing the original data;

[0166] determine a first part of compression operators in the operator set according to the running states of the plurality of compression operators on the first acceleration hardware;

[0167] determine a second part of compression operators in the operator set according to the first part of compression operators.

[0168] In a possible implementation, the obtaining module 801 is further configured to obtain a running state of the general processor;

[0169] The operator determining module 802 is specifically configured to determine a first part of compression operators and a second part of compression operators participating in compressing the original data according to the running states of the plurality of compression operators and the running state of the general processor.

[0170] In a possible implementation, the operator determining module 802 is specifically configured to:

[0171] determine a plurality of candidate operator sets according to the original data and the running states of the plurality of compression operators;

[0172] determine a target operator set from the plurality of candidate operator sets according to the running state of the general processor, and the target operator set includes the first part of compression operators and the second part of compression operators.

[0173] In a possible implementation, the compression module 803 is further configured to utilize at least one available compression operator running on the first acceleration hardware to compress the data compression result.

[0174] In a possible implementation, the first part of compression operators includes a plurality of same first compression operators, and the plurality of same first compression operators are used to perform a parallel compression process on the original data;

[0175] Alternatively, the second part of compression operators comprises a plurality of identical second compression operators, and the plurality of identical second compression operators are used to perform a parallel compression process on the original data.

[0176] In a possible implementation, the storage system further comprises second acceleration hardware, and the second acceleration hardware comprises a plurality of operator units, and each operator unit in the plurality of operator units on the second acceleration hardware is used to run one compression operator.

[0177] The operator determining module 802 is specifically configured to determine, according to the running states of the plurality of compression operators on the first acceleration hardware and the running states of the plurality of compression operators on the second acceleration hardware, a first part of compression operators, a second part of compression operators, and a third part of compression operators participating in compressing the original data, the third part of compression operators being available compression operators on the second acceleration hardware.

[0178] The compression module 803 is specifically configured to compress the original data by using the first part of compression operators running on the first acceleration hardware, the second part of compression operators running on the general processor, and the third part of compression operators running on the second acceleration hardware.

[0179] In a possible implementation, the running state of a compression operator comprises a throughput of the compression operator, a number of tasks processed in parallel by the compression operator, or a health state of the compression operator.

[0180] Since the data compression apparatus 800 shown in FIG. 8 corresponds to the general processor 100 in the embodiment shown in FIG. 3 or the data processing apparatus 200, for details of the implementation mode of the data compression apparatus 800 shown in FIG. 8 and the technical effects thereof, refer to the description of the related parts in the embodiment shown in FIG. 3, and no further description is given here.

[0181] Referring to FIG. 9, a structural schematic diagram of a data processing apparatus is shown, wherein the data processing apparatus 900 shown in FIG. 9 is applied to a data processing system, and the data processing system comprises a general processor and first acceleration hardware, and the first acceleration hardware comprises a plurality of operator units, and each operator unit in the plurality of operator units is used to run one calculation operator. As shown in FIG. 9, the data processing apparatus 900 comprises:

[0182] The obtaining module 901 is configured to obtain original data to be processed and running states of a plurality of calculation operators on the first acceleration hardware.

[0183] The operator determining module 902 is configured to determine, according to the running states of the plurality of calculation operators, a first part of calculation operators and a second part of calculation operators participating in calculating the original data, the first part of calculation operators being available calculation operators on the first acceleration hardware, and the second part of calculation operators being calculation operators on the general processor.

[0184] The processing module 903 is configured to process the original data by using the first part of the calculation operators on the first acceleration hardware and the second part of the calculation operators on the general processor, to obtain a data processing result.

[0185] The data processing apparatus 900 shown in FIG. 9 corresponds to the general processor in the embodiment shown in FIG. 7 or the data processing apparatus described above, and therefore, the specific implementation mode of the data processing apparatus 900 shown in FIG. 9 and the technical effects thereof are described in the related parts of the embodiment shown in FIG. 7, which will not be repeated here.

[0186] FIG. 10 is a schematic diagram of a hardware structure of a computing device 1000 provided by the present application, which can implement the general processor 100 or the data compression apparatus 200 in the embodiment shown in FIG. 3, and the like.

[0187] As shown in FIG. 10, the computing device 1000 includes a processor 1001, a memory 1002, and a communication interface 1003. The processor 1001, the memory 1002, and the communication interface 1003 communicate through a bus 1004, and can also communicate through wireless transmission or other means. The memory 1002 is configured to store instructions, and the processor 1001 is configured to execute the instructions stored in the memory 1002. Further, the computing device 1000 can also include a memory unit 1005, and the memory unit 1005 can be connected to the processor 1001, the storage medium 1002, and the communication interface 1003 through the bus 1004. The memory 1002 stores program codes, and the processor 1001 can invoke the program codes stored in the memory 1002 to perform the following operations:

[0188] obtaining original data to be compressed and running states of a plurality of compression operators on first acceleration hardware, wherein the first acceleration hardware belongs to a storage system, the storage system further includes a general processor, the first acceleration hardware includes a plurality of operator units, and each operator unit is configured to run a compression operator;

[0189] determining, according to the running states of the plurality of compression operators, a first part of compression operators and a second part of compression operators participating in compression of the original data, the first part of compression operators being available compression operators on the first acceleration hardware, and the second part of compression operators being compression operators on the general processor;

[0190] compressing the original data by using the first part of compression operators on the first acceleration hardware and the second part of compression operators on the general processor, to obtain a data compression result.

[0191] Alternatively, the processor 1001 can invoke program codes stored in the memory 1002 to perform the following operations:

[0192] obtain raw data to be processed, and running states of a plurality of computing operators on a first acceleration hardware, wherein the first acceleration hardware belongs to a data processing system, the data processing system further includes a general-purpose processor, and the first acceleration hardware includes a plurality of operator units, each of which is configured to run a computing operator;

[0193] determine, according to the running states of the plurality of computing operators, a first part of computing operators and a second part of computing operators participating in computing the raw data, the first part of computing operators being available computing operators on the first acceleration hardware, and the second part of computing operators being computing operators on the general-purpose processor;

[0194] process the raw data by using the first part of computing operators on the first acceleration hardware and the second part of computing operators on the general-purpose processor, to obtain a data processing result.

[0195] It should be understood that, in this embodiment, the processor 1001 can be a CPU, and can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, or the like. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0196] The memory 1002 can include read-only memory and random access memory, and provide instructions and data to the processor 1001. The memory 1002 can also include non-volatile random access memory.

[0197] The memory 1002 can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory, among others. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, and not limitation, many forms of RAM are available, for example, static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). The memory 1002 can be used to store data and / or instructions that can be used by the processing circuit 1001. For example, the memory 1002 can store software or firmware instructions that are executed by the processing circuit 1001. In some embodiments, the memory 1002 can store data that is used by the processing circuit 1001. In some embodiments, the memory 1002 can store data that is received from the input device 1005 or the communication interface 1003. In some embodiments, the memory 1002 can store data that is transmitted from the output device 1006 or the communication interface 1003.

[0198] The communication interface 1003 is used to communicate with other devices connected to the computing device 1000. The bus 1004 can include a data bus, a power bus, a control bus, and a state signal bus, among others. However, for clarity, the various buses are shown as a bus 1004 in the figure.

[0199] It should be understood that the computing device 1000 according to the embodiments of the present application can correspond to the general processor 100 or the data compression apparatus 200 in the embodiments shown in FIG. 3 described above, and can correspond to the method performed by the general processor 100 or the data compression apparatus 200 in the embodiments shown in FIG. 3, or can correspond to the general processor or the data processing apparatus in the embodiments shown in FIG. 7 described above, and can correspond to the method performed by the general processor or the data processing apparatus in the embodiments shown in FIG. 7. The above and other operations and / or functions implemented by the computing device 1000 are respectively used to implement the flow of the corresponding method in FIG. 3 or FIG. 7, and are not described herein for brevity.

[0200] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be used to store the instructions that can be executed by a computing device. The computer readable storage medium can be a magnetic-based medium (e.g., a floppy diskette, a hard disk drive, a magnetic tape), an optical-based medium (e.g., a compact disc (CD), a DVD), or a semiconductor-based medium (e.g., a solid state hard drive), etc. The computer readable storage medium includes instructions that are executable by a computing device to implement the data compression method or the data processing method.

[0201] The embodiments of the present application also provide a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the computer instructions generate, in whole or in part, the processes or functions described in the embodiments of the present application.

[0202] The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, e.g., the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0203] The computer program product can be a software package, which can be downloaded and executed on a computing device in the case that any of the data compression method or the data processing method needs to be used.

[0204] The above-described embodiments can be implemented in part or in whole through software, hardware, firmware or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions produce the above-described processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, from a website site, a computer, a server, or a data center to another website site, a computer, a server, or a data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing a set of one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0205] The terms used in the above-described embodiments are only for the purpose of describing particular embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the application, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "one or more" as used in the application embodiments refers to one, two, or more than two. The character " / " generally indicates that the associated objects before and after the " / " are in an "or" relationship. In the embodiments of the present application, "simultaneously" means in the same time period, including the case of being at the same time. The terms "first", "second", and the like in the specification and claims of the present application and the accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a particular order or sequence. It should be understood that the terms used in this way can be interchanged as appropriate, and this is merely a way of distinguishing the objects of the same attribute used in describing the embodiments of the present application.

[0206] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment, but can refer to one or more but not all embodiments. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. The terms "including", "containing", and "having" are intended to be

[0207] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data compression method characterized by, The method is applied to a storage system, the storage system comprising a general processor and a first acceleration hardware, the first acceleration hardware comprising a plurality of operator units, each of the plurality of operator units being configured to run one compression operator; the method comprising: obtaining original data to be compressed, and running states of a plurality of compression operators on the first acceleration hardware; determining, according to the running states of the plurality of compression operators, a first part of compression operators and a second part of compression operators to participate in compressing the original data, the first part of compression operators being available compression operators on the first acceleration hardware, and the second part of compression operators being compression operators on the general processor; compressing the original data by using the first part of compression operators on the first acceleration hardware and the second part of compression operators on the general processor to obtain a data compression result.

2. The method of claim 1, wherein, The method further comprises: starting the second part of compression operators on the general processor before compressing the original data.

3. The method according to claim 1 or 2, characterized in that, The storage system is configured with a decision model, and the determining, according to the running states of the plurality of compression operators, the first part of compression operators and the second part of compression operators to participate in compressing the original data comprises: using the decision model to infer according to the original data and the running states of each compression operator on the first acceleration hardware to obtain the first part of compression operators and the second part of compression operators output by the decision model.

4. The method according to claim 1 or 2, characterized in that, The determining, according to the running states of the plurality of compression operators, the first part of compression operators and the second part of compression operators to participate in compressing the original data comprises: extracting data features of the original data; determining an operator set according to the data features, a plurality of compression operators in the operator set participating in compressing the original data; determining the first part of compression operators in the operator set according to the running states of a plurality of compression operators on the first acceleration hardware; determining the second part of compression operators in the operator set according to the first part of compression operators.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: obtaining a running state of the general processor; The determining, according to the running states of the plurality of compression operators, the first part of compression operators and the second part of compression operators to participate in compressing the original data comprises: determining, according to the running states of the plurality of compression operators and the running state of the general processor, the first part of compression operators and the second part of compression operators to participate in compressing the original data.

6. The method of claim 5, wherein, The determining, according to the running states of the plurality of compression operators and the running state of the general processor, the first part of compression operators and the second part of compression operators to participate in compressing the original data comprises: determining a plurality of candidate operator sets according to the original data and the running states of the plurality of compression operators; determining a target operator set from the plurality of candidate operator sets according to the running state of the general processor, the target operator set comprising the first part of compression operators and the second part of compression operators.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: compressing the data compression result using at least one available compression operator running on the first acceleration hardware.

8. The method according to any one of claims 1 to 7, characterized in that, The first part of compression operators comprises a plurality of identical first compression operators for performing a parallel compression process on the original data. Alternatively, the second part of compression operators comprises a plurality of identical second compression operators for performing a parallel compression process on the original data.

9. The method according to any one of claims 1 to 8, characterized in that, The storage system further comprises a second acceleration hardware comprising a plurality of operator units, each of the plurality of operator units on the second acceleration hardware for running one compression operator. The determining of the first part of compression operators and the second part of compression operators participating in compressing the original data according to the running states of the plurality of compression operators comprises: The determining of the first part of compression operators, the second part of compression operators, and a third part of compression operators participating in compressing the original data according to the running states of the plurality of compression operators on the first acceleration hardware and the running states of the plurality of compression operators on the second acceleration hardware, the third part of compression operators being available compression operators on the second acceleration hardware. The compressing of the original data using the first part of compression operators running on the first acceleration hardware and the second part of compression operators running on the general processor comprises: The compressing of the original data using the first part of compression operators running on the first acceleration hardware, the second part of compression operators running on the general processor, and the third part of compression operators running on the second acceleration hardware.

10. The method according to any one of claims 1 to 9, characterized in that, The running state of the compression operator comprises a throughput of the compression operator, a number of tasks processed in parallel by the compression operator, or a health state of the compression operator.

11. A data processing method, characterized by, The method is applied to a data processing system comprising a general processor and a first acceleration hardware comprising a plurality of operator units, each of the plurality of operator units for running one computing operator; the method comprises: obtaining original data to be processed and running states of a plurality of computing operators on the first acceleration hardware; determining a first part of computing operators and a second part of computing operators participating in computing the original data according to the running states of the plurality of computing operators, the first part of computing operators being available computing operators on the first acceleration hardware, and the second part of computing operators being computing operators on the general processor; processing the original data using the first part of computing operators on the first acceleration hardware and the second part of computing operators on the general processor to obtain a data processing result.

12. A data compression device, characterized by The apparatus is applied to a storage system comprising a general processor and a first acceleration hardware comprising a plurality of operator units, each of the plurality of operator units for running one compression operator; the apparatus comprises: obtain a raw data to be compressed, running states of a plurality of compression operators on the first acceleration hardware; determine, according to the running states of the plurality of compression operators, a first part of compression operators and a second part of compression operators participating in compressing the raw data, the first part of compression operators being available compression operators on the first acceleration hardware, and the second part of compression operators being compression operators on the general processor; compress the raw data by using the first part of compression operators on the first acceleration hardware and the second part of compression operators on the general processor to obtain a data compression result.

13. The apparatus of claim 12, wherein, The apparatus further comprises: start the second part of compression operators on the general processor before compressing the raw data.

14. The apparatus of claim 12 or 13, wherein, The storage system is configured with a decision model, and the operator determining module is specifically configured to use the decision model to perform reasoning according to the raw data and the running states of each compression operator on the first acceleration hardware to obtain the first part of compression operators and the second part of compression operators output by the decision model.

15. The apparatus of claim 12 or 13, wherein, The operator determining module is configured to: extract data features of the raw data; determine an operator set according to the data features, a plurality of compression operators in the operator set participating in compressing the raw data; determine the first part of compression operators in the operator set according to the running states of a plurality of compression operators on the first acceleration hardware; determine the second part of compression operators in the operator set according to the first part of compression operators.

16. The apparatus according to any one of claims 12 to 15, wherein the obtaining module is further configured to obtain a running state of the general processor; the operator determining module is specifically configured to determine the first part of compression operators and the second part of compression operators participating in compressing the raw data according to the running states of the plurality of compression operators and the running state of the general processor.

17. The apparatus of claim 16, wherein, The operator determining module is specifically configured to: determine a plurality of candidate operator sets according to the raw data and the running states of the plurality of compression operators; determine a target operator set from the plurality of candidate operator sets according to the running state of the general processor, the target operator set including the first part of compression operators and the second part of compression operators.

18. The apparatus of any one of claims 12 to 17, wherein, The compression module is further configured to compress the data compression result by using at least one available compression operator running on the first acceleration hardware.

19. The apparatus of any one of claims 12 to 18, wherein, The first part of compression operators includes a plurality of same first compression operators, and the plurality of same first compression operators are used to perform a parallel compression process on the raw data. Alternatively, the second part of compression operators includes a plurality of same second compression operators, and the plurality of same second compression operators are used to perform a parallel compression process on the raw data.

20. The apparatus of any one of claims 12 to 19, wherein, The storage system further includes second acceleration hardware, the second acceleration hardware including a plurality of operator units, each of the plurality of operator units on the second acceleration hardware being configured to run one compression operator; The operator determining module is specifically configured to determine, according to the running states of the plurality of compression operators on the first acceleration hardware and the running states of the plurality of compression operators on the second acceleration hardware, the first part of compression operators, the second part of compression operators, and a third part of compression operators participating in compressing the original data, the third part of compression operators being available compression operators on the second acceleration hardware; The compression module is specifically configured to compress the original data by using the first part of compression operators running on the first acceleration hardware, the second part of compression operators running on the general processor, and the third part of compression operators running on the second acceleration hardware.

21. The apparatus of any one of claims 12 to 20, wherein, The running state of the compression operator includes a throughput of the compression operator, a number of tasks processed in parallel by the compression operator, or a health state of the compression operator.

22. A data processing apparatus, characterized in that, The device is applied to a data processing system including a general processor and first acceleration hardware, the first acceleration hardware including a plurality of operator units, each of the plurality of operator units being configured to run one calculation operator; the device includes: An obtaining module configured to obtain original data to be processed and running states of a plurality of calculation operators on the first acceleration hardware; An operator determining module configured to determine, according to the running states of the plurality of calculation operators, a first part of calculation operators and a second part of calculation operators participating in calculating the original data, the first part of calculation operators being available calculation operators on the first acceleration hardware, and the second part of calculation operators being calculation operators on the general processor; A processing module configured to process the original data by using the first part of calculation operators on the first acceleration hardware and the second part of calculation operators on the general processor to obtain a data processing result.

23. A computing device, comprising: The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the computing device performs steps of the method according to any one of claims 1 to 11.

24. A storage system, comprising: The storage system includes a general processor, acceleration hardware, at least one storage medium, the general processor being configured to execute the method according to any one of claims 1 to 10, the acceleration hardware including a plurality of operator units, each of the plurality of operator units being configured to run one compression operator, and the at least one storage device being configured to persistently store a data compression result.

25. A storage system, comprising: The storage system includes a general processor, acceleration hardware, at least one storage device, and a data compression device, the data compression device being configured to execute the method according to any one of claims 1 to 10, the general processor being configured to run a compression operator, the acceleration hardware including a plurality of operator units, each of the plurality of operator units being configured to run one compression operator, and the at least one storage device being configured to persistently store a data compression result.

26. A computer-readable storage medium, characterized in that, comprising instructions that, when executed on a computing device, cause the computing device to perform the steps of the method of any one of claims 1 to 10.

27. A computer program product comprising instructions, wherein: causing at least one computing device to perform the steps of the method of any one of claims 1 to 10 when executed on the at least one computing device.

Images