Data compression and storage method and apparatus, and system, medium and product

Abstract

The present application relates to the technical field of storage. Disclosed are a data compression and storage method and apparatus, and a system, a medium and a product. The method is applied to a first data processor network card on a storage control side. The first data processor network card is connected to a central processing unit on the storage control side, and is connected to a second data processor network card on a storage side by means of a network, and the second data processor network card is connected to hard disks on the storage side. In the method, the first data processor network card performs data compression, and the central processing unit is not responsible for data compression any more, so that the usage rate of the central processing unit can be reduced, I / O processing programs of the central processing unit can be conserved, and the performance can be improved. Moreover, a first data processor compresses data and then transmits same to the second data processor network card, and the second data processor network card writes the compressed data into a hard disk, such that the volume of data transmitted in the network can be reduced, the usage rate of the bandwidth of the network can be increased, and performance bottlenecks brought about by the congestion of the network can be avoided.

Description

Data compression storage method, device, system, medium and product

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. CN202410704392.3, filed on June 3, 2024, and entitled "Data compression storage method, device, system, medium and computer program product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of storage, in particular to a data compression storage method, a data compression storage device, a storage system, a medium and a computer readable instruction product. BACKGROUND

[0004] Data centers usually use distributed storage systems containing hundreds of nodes. In such a large-scale distributed storage system, a large amount of data transmission between nodes becomes the norm, and the data center needs to provide higher performance networks. However, the growth rate of network bandwidth hardware is much slower than that of data volume. Therefore, the common practice is to reduce the amount of data transmission through data reduction techniques such as compression and deduplication. Various compression and decoding techniques in distributed storage systems involve a large amount of computation and cross-node data transmission, which can cause a large computational and transmission overhead to the system. The current mainstream solution is to use a CPU (Central Processing Unit) to run software to perform compression and decoding calculations. When the I / O bandwidth of the system is large, compression and decompression can occupy a large amount of CPU resources, and the CPU also needs to process the business of other software modules. The CPU and memory resources occupied by compression and decompression calculations can affect the overall performance. SUMMARY

[0005] In a first aspect, an embodiment of the present application provides a data compression storage method applied to a first data processor network card of a storage control side. The first data processor network card is connected with a central processor of the storage control side, and the first data processor network card is connected with a second data processor network card of a storage side through a network. The second data processor network card is connected with each hard disk of the storage side.

[0006] The data compression storage method comprises the following steps:

[0007] receiving notification information sent by the central processor;

[0008] acquiring, according to the notification information, to-be-compressed data from a memory of the storage control side;

[0009] The compressed data is obtained by compressing the to-be-compressed data, and the compressed data is cached.

[0010] The second data processor network card is sent a request so that the second data processor network card reads the compressed data from the first data processor network card according to the request and writes the compressed data into the hard disk.

[0011] The notification information is used for notifying the first data processor network card to compress data, and includes the position of the to-be-compressed data and the data length.

[0012] In some embodiments, the notification information further includes the data type of the to-be-compressed data.

[0013] In some embodiments, the to-be-compressed data is obtained from the memory of the storage control side according to the notification information, including: reading the to-be-compressed data from the memory of the storage control side according to the position and the data length of the to-be-compressed data.

[0014] In some embodiments, the compressed data is obtained by compressing the to-be-compressed data, including the following steps:

[0015] The compression algorithm and the compression rate are determined according to the data type of the to-be-compressed data; and

[0016] The to-be-compressed data is compressed by using the compression algorithm and the compression rate to obtain the compressed data.

[0017] In some embodiments, the data compression storage method further includes: after the to-be-compressed data is compressed to obtain the compressed data and the compressed data is cached, notifying the central processing unit that the data compression is completed.

[0018] In some embodiments, the data compression storage method further includes: receiving a completed data writing result sent by the second data processor network card after the compressed data is written into the hard disk, and sending the completed data writing result to the central processing unit.

[0019] In some embodiments, the data compression storage method further includes the following steps:

[0020] The reading data command sent by the central processing unit is received;

[0021] The reading data command is sent to the second data processor network card so that the second data processor network card reads the target compressed data from the hard disk according to the reading data command;

[0022] The target compressed data sent by the second data processor network card is received;

[0023] The target data is obtained by decompressing the target compressed data, and the target data is written into the memory of the storage control side; and

[0024] The central processing unit is informed that the data has been decompressed, so that the central processing unit reads the target data from the memory of the storage control side.

[0025] In some embodiments, the data compression storage method further comprises: receiving a notification of completing data reading sent by the central processing unit after reading the target data from the memory of the storage control side.

[0026] In some embodiments, the first data processor network card comprises: a network card central processing unit, a field programmable gate array, and a network card memory. The network card central processing unit internally integrates a compressor, and the network card central processing unit is used for data compression and decompression. The field programmable gate array is used for receiving and sending data.

[0027] In some embodiments, the network card central processing unit and the field programmable gate array are connected through a peripheral component interconnect express interface, the field programmable gate array is connected to a remote direct memory access network port, and the field programmable gate array is connected to the central processing unit of the storage control side through the peripheral component interconnect express interface.

[0028] In the second aspect, the embodiments of the present application further provide a data compression storage method applied to a second data processor network card of a storage side. The second data processor network card is connected to each hard disk, and the second data processor network card is connected to a first data processor network card of a storage control side through a network. The first data processor network card is connected to a central processing unit of the storage control side. The data compression storage method comprises the following steps:

[0029] receiving a request sent by the first data processor network card;

[0030] reading compressed data from the first data processor network card according to the request; and

[0031] writing the compressed data into the hard disk according to the request.

[0032] The first data processor network card is used for receiving notification information sent by the central processing unit, obtaining to-be-compressed data from the memory of the storage control side according to the notification information, performing compression processing on the to-be-compressed data to obtain compressed data, buffering the compressed data, and sending a request to the second data processor network card. The notification information is used for notifying the first data processor network card to perform data compression, and the notification information comprises a position and a data length of the to-be-compressed data.

[0033] In some embodiments, the data compression storage method further comprises: after writing the compressed data into the hard disk, sending a data writing completion result to the first data processor network card.

[0034] In some embodiments, the data compression storage method further comprises: receiving a data reading command sent by the first data processor network card;

[0035] read target compressed data from the hard disk according to a read data command; and

[0036] send the target compressed data to the first data processor network card, so that the first data processor network card decompresses the target compressed data to obtain target data, and writes the target data into the memory of the storage control side.

[0037] In some embodiments, the second data processor network card comprises a network card central processor, a field programmable gate array and a network card memory; the network card central processor internally integrates a compressor; the network card central processor is configured to perform data compression and decompression, and the field programmable gate array is configured to receive and send data.

[0038] In some embodiments, the network card central processor and the field programmable gate array are connected through a peripheral component interconnect express interface, the field programmable gate array is connected to a remote direct memory access network port, and the field programmable gate array is connected to the hard disk through the peripheral component interconnect express interface.

[0039] In a third aspect, the embodiments of the present application further provide a data compression storage device, applied to a first data processor network card of a storage control side, the first data processor network card being connected to a central processor of the storage control side, and the first data processor network card being connected to a second data processor network card through a network, the second data processor network card being connected to each hard disk.

[0040] The data compression storage device comprises the following modules:

[0041] The receiving module is configured to receive notification information sent by the central processor;

[0042] The obtaining module is configured to obtain, according to the notification information, to-be-compressed data from the memory of the storage control side;

[0043] The compression module is configured to compress the to-be-compressed data to obtain compressed data, and buffer the compressed data; and

[0044] The sending module is configured to send a request to the second data processor network card, so that the second data processor network card reads the compressed data from the first data processor network card according to the request and writes the compressed data into the hard disk.

[0045] The notification information is used to notify the first data processor network card to perform data compression; the notification information comprises a position of the to-be-compressed data and a data length.

[0046] In a fourth aspect, the embodiments of the present application further provide a data compression storage device, which is applied to a second data processor network card of a storage side, the second data processor network card is connected with each hard disk, and the second data processor network card is connected with a first data processor network card of a storage control side through a network, and the first data processor network card is connected with a central processing unit of the storage control side.

[0047] The data compression storage device comprises the following units:

[0048] A request receiving unit is configured to receive a request sent by the first data processor network card.

[0049] A data reading unit is configured to read compressed data from the first data processor network card according to the request; and

[0050] A data writing unit is configured to write the compressed data into the hard disk according to the request.

[0051] The first data processor network card is configured to receive notification information sent by the central processing unit, acquire the to-be-compressed data from the memory of the storage control side according to the notification information, perform compression processing on the to-be-compressed data to obtain compressed data, cache the compressed data, and send a request to the second data processor network card, wherein the notification information is used to notify the first data processor network card to perform data compression, and the notification information comprises the position and the data length of the to-be-compressed data.

[0052] In a fifth aspect, the embodiments of the present application further provide a storage system, which comprises a storage control side and a storage side. The storage control side comprises a central processing unit, a first data processor network card and a memory, and the storage side comprises a second data processor network card and a plurality of hard disks. The first data processor network card is connected with the central processing unit, and the first data processor network card is connected with the second data processor network card through a network, and the second data processor network card is connected with the hard disks.

[0053] The first data processor network card is configured to receive notification information sent by the central processing unit, the notification information is used to notify the first data processor network card to perform data compression, the notification information comprises the position and the data length of the to-be-compressed data, acquire the to-be-compressed data from the memory according to the notification information, perform compression processing on the to-be-compressed data to obtain compressed data, cache the compressed data, and send a request to the second data processor network card.

[0054] The second data processor network card is configured to read compressed data from the first data processor network card according to the request, and write the compressed data into the hard disk according to the request.

[0055] In a sixth aspect, the embodiments of the present application further provide a non-volatile readable storage medium, and the non-volatile readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor to implement the steps of the data compression storage method applied to the first data processor network card or the steps of the data compression storage method applied to the second data processor network card.

[0056] In a seventh aspect, the embodiments of the present application further provide a computer readable instruction product, and the computer readable instruction product comprises computer readable instructions, and the computer readable instructions are executed by a processor to implement the steps of the data compression storage method applied to the first data processor network card or the steps of the data compression storage method applied to the second data processor network card.

[0057] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features and advantages of the present application will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the prior art and the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0059] FIG. 1 is a flowchart of a data compression storage method provided by an embodiment of the present application;

[0060] FIG. 2 is a schematic diagram of a hardware architecture of a storage system provided by an embodiment of the present application;

[0061] FIG. 3 is a schematic diagram of a data write processing flow provided by an embodiment of the present application;

[0062] FIG. 4 is a schematic diagram of a read data processing flow provided by an embodiment of the present application;

[0063] FIG. 5 is a schematic diagram of a data processor network card provided by an embodiment of the present application;

[0064] FIG. 6 is a schematic diagram of an internal data processing flow of a data processor network card provided by an embodiment of the present application;

[0065] FIG. 7 is a flowchart of another data compression storage method provided by an embodiment of the present application;

[0066] FIG. 8 is a schematic diagram of a data compression storage device provided by an embodiment of the present application;

[0067] FIG. 9 is a schematic diagram of another data compression storage device according to an embodiment of the present application;

[0068] FIG. 10 is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] The present application provides a data compression storage method, which can reduce the usage of a central processing unit, save I / O (Input / Output) processing programs of the central processing unit, improve the performance of the central processing unit, reduce the transmission amount of data in a network, and improve the efficiency of network bandwidth usage.

[0070] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0071] In a first aspect, referring to FIG. 1, FIG. 1 is a flowchart of a data compression storage method according to an embodiment of the present application. The method is applied to a first data processor network card on a storage control side. The first data processor network card is connected to a central processing unit on the storage control side, and the first data processor network card is connected to a second data processor network card on a storage side through a network. The second data processor network card is connected to each hard disk on the storage side. As shown in FIG. 1, the data compression storage method comprises the following steps.

[0072] Step S101: receiving notification information sent by the central processing unit. The notification information is used to notify the first data processor network card to perform data compression. The notification information comprises the position of data to be compressed and the length of the data.

[0073] The storage control side refers to a storage controller. The storage side refers to a JBOF, which is an English full name of Just a Bunch of Flash drives. The data compression storage method provided in the embodiment is mainly implemented based on a DPU (Data Processing Unit) network card. In order to distinguish the DPU network card on the storage control side from the DPU network card on the storage side, the DPU network card on the storage control side is referred to as the first data processor network card, and the DPU network card on the storage side is referred to as the second data processor network card.

[0074] Referring to FIG. 2, the storage control side 210 mainly includes a central processor 211, a first data processor network card 212, and a memory. The memory can be a dynamic random access memory (DRAM) 213. The central processor 211 of the storage control side is not responsible for data compression, which is implemented by the first data processor network card 212. When writing data, the central processor 211 of the storage control side fills in the original data to be compressed, i.e., the compressed data, and places the compressed data in the data request buffer in the memory. The central processor 211 of the storage control side sends notification information to the first data processor network card 212 of the storage control side. The notification information is used to notify the first data processor network card 212 to perform data compression. The notification information includes the location of the compressed data and the data length. The storage side 220 includes a second data processor network card 222 and a plurality of solid state disks 223.

[0075] In some embodiments, the notification information further includes the data type of the compressed data.

[0076] After the central processor of the storage control side places the compressed data in the data request buffer in the memory, the first data processor network card is notified that the compressed data has been placed in the memory, and the location, data length, and data type of the compressed data are notified to the first data processor network card.

[0077] Step S102: obtaining the compressed data from the memory of the storage control side according to the notification information.

[0078] The first data processor network card obtains the compressed data from the memory of the storage control side according to the notification information.

[0079] In some embodiments, obtaining the compressed data from the memory of the storage control side according to the notification information includes:

[0080] According to the location and data length of the compressed data, the compressed data is read from the memory of the storage control side.

[0081] The central processor of the storage control side notifies the first data processor network card of the location and data length of the compressed data, and the first data processor network card reads the compressed data of the corresponding data length from the corresponding location in the memory according to the location and data length of the compressed data.

[0082] Step S103: performing compression processing on the compressed data to obtain compressed data, and buffering the compressed data.

[0083] The first data processor network card reads the to-be-compressed data from the memory, compresses the to-be-compressed data to obtain compressed data, and buffers the compressed data in the first data processor network card without copying the compressed data to the memory on the storage control side. The compressed data can be directly transmitted from the first data processor network card to the second data processor network card, which can significantly reduce the data delay.

[0084] In some embodiments, the compression processing of the to-be-compressed data to obtain the compressed data includes:

[0085] The compression algorithm and the compression rate are determined according to the data type of the to-be-compressed data, and the compression algorithm and the compression rate are used to compress the to-be-compressed data to obtain the compressed data.

[0086] For different types of data, different compression algorithms and / or compression rates can be used. The first data processor network card determines the compression algorithm and the compression rate according to the data type of the to-be-compressed data, and then uses the determined compression algorithm to compress the to-be-compressed data according to the determined compression rate to obtain the compressed data.

[0087] As for the correspondence between the data type, the compression algorithm, and the compression rate, it can be set differently according to the actual situation, which is not limited in the present application.

[0088] In some embodiments, the data compression storage method further includes: after the compression processing of the to-be-compressed data to obtain the compressed data and buffering the compressed data, notifying the central processor that the data compression is completed.

[0089] After the first data processor network card compresses the to-be-compressed data and buffers the compressed data, the first data processor network card further notifies the central processor that the data compression is completed and can be sent.

[0090] Step S104: sending a request to the second data processor network card to make the second data processor network card read the compressed data from the first data processor network card according to the request and write the compressed data to the hard disk.

[0091] The first data processor network card sends a request to the second data processor network card, and the second data processor network card parses the compressed data from the request to obtain the address information of the hard disk. The second data processor network card reads the compressed data from the first data processor network card and writes the read compressed data to the hard disk. The hard disk can be an SSD (Solid State Drive).

[0092] In some embodiments, the data compression storage method further includes: receiving a complete data write result sent by the second data processor network card after writing the compressed data to the hard disk, and sending the complete data write result to the central processor. In some embodiments, the data compression storage method further includes: receiving a complete data write result sent by the second data processor network card after writing the compressed data to the hard disk, and sending the complete data write result to the central processor.

[0093] The second data processor network card sends the data write completion result to the first data processor network card after writing the compressed data to the hard disk, and the first data processor network card sends the data write completion result to the central processor.

[0094] In some embodiments, the data write processing flow can refer to that shown in FIG. 3.

[0095] The central processor 211 on the storage control side fills in the original data to be compressed, i.e., the compressed data, and places the compressed data to be placed in the data request buffer in the DRAM 213.

[0096] The central processor 211 on the storage control side informs the first data processor network card 212 that the compressed data to be compressed has been placed in the DRAM 213, and informs the first data processor network card 212 of the location, data length and data type of the compressed data to be compressed.

[0097] The first data processor network card 212 on the storage control side obtains the compressed data to be compressed from the DRAM 213 according to the location and data length of the compressed data to be compressed.

[0098] The first data processor network card 212 on the storage control side obtains the data type information of the compressed data to be compressed.

[0099] The first data processor network card 212 on the storage control side configures the compression function of the first data processor network card 212 according to the data type information of the compressed data to be compressed, including the compression algorithm and the compression rate.

[0100] The first data processor network card 212 on the storage control side compresses the compressed data to be compressed using the configured compression algorithm and compression rate.

[0101] The first data processor network card 212 on the storage control side packs the compressed compressed data and caches it to the first data processor network card 212, waiting for data transmission.

[0102] The first data processor network card 212 on the storage control side informs the central processor 211 that the data has been compressed and can be transmitted after buffering the compressed data to the first data processor network card 212.

[0103] The central processor sends a data read request to the second data processor network card 222 through the first data processor network card 212.

[0104] The second data processor network card 222 obtains the address information of the compressed data written to the SSD from the data read request.

[0105] The second data processor network card 222 reads the compressed data from the data cache area of the first data processor network card 212.

[0106] The second data processor network card writes the read compressed data into the SSD 223.

[0107] The second data processor network card 222 sends the completed write data result to the central processor 211 through the first data processor network card 212.

[0108] In some embodiments, the data compression storage method further comprises the following steps:

[0109] Receiving the read data command sent by the central processor;

[0110] Sending the read data command to the second data processor network card to make the second data processor network card read the target compressed data from the hard disk according to the read data command;

[0111] Receiving the target compressed data sent by the second data processor network card;

[0112] Decompressing the target compressed data to obtain target data and writing the target data into the memory of the storage control side; and

[0113] Notifying the central processor that the data has been decompressed to make the central processor read the target data from the memory of the storage control side.

[0114] When reading data, the central processor of the storage control side sends a read data command to the first data processor network card, and the first data processor network card further sends the read data command to the second data processor network card. The second data processor network card reads the target compressed data according to the read data command and sends the target compressed data to the first data processor network card. The first data processor network card receives the target compressed data sent by the second data processor network card and decompresses the target compressed data to obtain target data, which is written into the memory of the storage control side. After the first data processor network card writes the target data into the memory, it notifies the central processor that the data has been decompressed, so that the central processor reads the target data from the memory.

[0115] In some embodiments, the data compression storage method further comprises the following steps:

[0116] Receiving the notification of completing data reading sent by the central processor after reading the target data from the memory of the storage control side.

[0117] After the first data processor network card writes the target data into the memory, it notifies the central processor that the data has been decompressed. The central processor reads the target data from the memory and sends a notification of completing data reading to the first data processor network card.

[0118] In some embodiments, the data read processing flow can refer to that shown in FIG. 4.

[0119] The storage control side central processor 211 sends a read data command to the first data processor network card 212.

[0120] The storage control side first data processor network card 212 sends a read data command to the second data processor network card 222.

[0121] The second data processor network card 222 reads the target compressed data.

[0122] The second data processor network card 222 sends the target compressed data to the first data processor network card 212.

[0123] The storage control side first data processor network card 212 decompresses the target compressed data and writes the decompressed data to the memory 213.

[0124] The first data processor network card 212 notifies the central processor 211 that the data has been decompressed.

[0125] The central processor 211 reads the decompressed data and sends a read operation completion to the first data processor network card 212.

[0126] In some embodiments, the first data processor network card includes a network card central processor, a field programmable gate array, and a network card memory; the network card central processor internally integrates a compressor; the network card central processor is used for data compression and decompression, and the field programmable gate array is used for data reception and transmission.

[0127] The first data processor network card has data compression function and data transmission and reception function, and supports data buffering. Referring to FIG. 5, the first data processor network card 510 in the embodiment mainly includes a network card central processor 511, an FPGA 512 (Field Programmable Gate Array), and a network card memory 513. The network card central processor 511 is a central processor inside the data processor network card. The network card memory 513 is a memory inside the data processor network card. The network card central processor 511 internally integrates a compressor 514. The network card central processor 511 is used for data compression and decompression, and the field programmable gate array 512 is used for data reception and transmission, supporting TCP (Transmission Control Protocol) and RDMA (Remote Direct Memory Access) (RoCEv2) protocols.

[0128] In some embodiments, the network card central processor 511 is connected with the field programmable gate array 512 through the peripheral component interconnect express (PCIe) interface 520, the field programmable gate array 512 is connected with the remote direct memory access (RDMA) network interface 521, and the field programmable gate array 512 is connected with the storage control side central processor through the peripheral component interconnect express interface 522.

[0129] Referring to FIG. 5, the network card central processor 511 is connected with the field programmable gate array 512 through the peripheral component interconnect express (PCIe) interface 520, the field programmable gate array 512 is connected with the remote direct memory access (RDMA) network interface 521, and the field programmable gate array 512 is connected with the storage control side central processor through the peripheral component interconnect express interface 522. The network card central processor 511 can support 6 cores and 12 threads. The interface between the field programmable gate array 512 and the network card central processor 511 is a GEN4 X16 PCIE interface. The interface between the data processor network card 510 and the server is a GEN4 X16 PCIE interface.

[0130] Based on the above structure of the first data processor network card, referring to FIG. 6 (the central processor 611 in FIG. 6 is the central processor inside the network card, and the DRAM 613 is the DRAM inside the network card), when writing data, the data processing flow inside the first data processor network card can be as follows:

[0131] The field programmable gate array 612 inside the first data processor network card notifies the central processor 611 inside the first data processor network card to write data, and the central processor 611 inside the first data processor network card uses the direct memory access (DMA) to carry the data to be compressed from the storage control side DRAM. The data to be compressed is carried to the DRAM 613 inside the first data processor network card by the DMA. The central processor 611 inside the first data processor network card performs data compression and packaging on the data to be compressed. The central processor 611 inside the first data processor network card notifies the field programmable gate array 612 that the data compression is completed and returns the data address, etc. In addition, the central processor 611 inside the first data processor network card notifies the field programmable gate array 612 to read the compressed data. The field programmable gate array 612 reads the compressed data from the DRAM 613 inside the first data processor network card. After the second data processor network card writes the compressed data to the hard disk, the central processor 611 inside the first data processor network card notifies the field programmable gate array 612 that the data writing is completed, and then the field programmable gate array 612 notifies the storage control side central processor that the data writing is completed.

[0132] In summary, the data compression storage method provided by the application, the data compression is realized by the first data processor network card, and the central processor is no longer responsible for data compression, so that the use rate of the central processor can be effectively reduced, the I / O processing program of the central processor is saved, and the performance of the central processor is improved. Meanwhile, the first data processor transmits the compressed data to the second data processor network card on the storage side, the second data processor network card writes the compressed data into the hard disk, the transmission amount of data in the network can be reduced, the network bandwidth use rate is improved, and the performance bottleneck caused by network congestion is avoided. Moreover, the method has the independent expansion capability of compression performance, and the data transmission amount between storage controllers can be greatly reduced.

[0133] In the second aspect, referring to FIG. 7, FIG. 7 is a flow diagram of another data compression storage method provided by an embodiment of the application, the method is applied to the second data processor network card on the storage side, the second data processor network card is connected with each hard disk, and the second data processor network card is connected with the first data processor network card on the storage control side through a network, and the first data processor network card is connected with the central processor on the storage control side; referring to FIG. 7, the data compression storage method comprises the following steps:

[0134] Step S701: receiving a request sent by the first data processor network card;

[0135] Step S702: reading compressed data from the first data processor network card according to the request; the first data processor network card is used for receiving notification information sent by the central processor, obtaining to-be-compressed data from the memory on the storage control side according to the notification information, performing compression processing on the to-be-compressed data to obtain compressed data, buffering the compressed data, sending a request to the second data processor network card, wherein the notification information is used for notifying the first data processor network card to perform data compression, and the notification information comprises the position and the data length of the to-be-compressed data; and

[0136] Step S703: writing the compressed data into the hard disk according to the request.

[0137] When writing data, the central processor of the storage control side fills in the original data to be compressed, i.e., the compressed data, and places the compressed data into a data request buffer in the memory of the storage control side. The central processor of the storage control side sends notification information to the first data processor network card of the storage control side. The notification information is used to notify the first data processor network card to perform data compression. The notification information includes the location of the compressed data and the data length. The first data processor network card acquires the compressed data from the memory of the storage control side according to the notification information. After the first data processor network card reads the compressed data from the memory, the first data processor network card performs compression processing on the compressed data to obtain compressed data, and buffers the compressed data in the first data processor network card. The first data processor network card sends a request to the second data processor network card. After the second data processor network card receives the request sent by the first data processor network card, the second data processor network card parses the compressed data from the request and writes the compressed data to the address information of the hard disk. The second data processor network card reads the compressed data from the first data processor network card, and writes the read compressed data to the hard disk.

[0138] In some embodiments, the data compression storage method further comprises: after writing the compressed data into the hard disk, sending a data write completion result to the first data processor network card.

[0139] After the second data processor network card writes the compressed data into the hard disk, the second data processor network card sends a data write completion result to the first data processor network card, and then the first data processor network card sends the data write completion result to the central processor.

[0140] In some embodiments, the data compression storage method further comprises: receiving a read data command sent by the first data processor network card; reading target compressed data from the hard disk according to the read data command; and sending the target compressed data to the first data processor network card, so that the first data processor network card decompresses the target compressed data to obtain target data, and writes the target data into the memory of the storage control side.

[0141] When reading data, the central processor of the storage control side sends a read data command to the first data processor network card, and the first data processor network card further sends the read data command to the second data processor network card. After receiving the read data command, the second data processor network card reads target compressed data according to the read data command, and sends the target compressed data to the first data processor network card. The first data processor network card receives the target compressed data sent by the second data processor network card, and decompresses the target compressed data to obtain target data, and writes the target data into the memory of the storage control side. After the first data processor network card writes the target data into the memory, the first data processor network card notifies the central processor that the data has been decompressed, so that the central processor reads the target data from the memory.

[0142] In some embodiments, the second data processor network card comprises: a network card central processor, a field programmable gate array, and a network card memory; the network card central processor internally integrates a compressor; the network card central processor is configured to perform data compression and decompression, and the field programmable gate array is configured to receive and send data.

[0143] In some embodiments, the second data processor network card can adopt the same structure as the first data processor network card, for example, the second data processor network card can adopt the structure of FIG. 5.

[0144] In some embodiments, the second data processor network card mainly comprises a network card central processor, an FPGA, and a network card memory. The network card central processor is a central processor inside the data processor network card. The network card memory is a memory inside the data processor network card. The network card central processor internally integrates a compressor. The network card central processor is configured to perform data compression and decompression, and the field programmable gate array is configured to receive and send data, supporting TCP and RDMA (RoCEv2) protocols.

[0145] In some embodiments, the network card central processor and the field programmable gate array are connected through a peripheral component interconnect express interface, the field programmable gate array is connected to a remote direct memory access network port, and the field programmable gate array is connected to a hard disk through a peripheral component interconnect express interface.

[0146] The network card central processor and the FPGA are connected through a PCIe interface, the FPGA is connected to an RDMA network port, and the FPGA is connected to a hard disk through a PCIe interface. The network card central processor can support 6 cores and 12 threads. The interface from the FPGA to the network card central processor is a GEN4 X16 PCIE interface. The interface from the data processor network card to the server is a PCIe GEN4 X16 PCIE interface.

[0147] In summary, the data compression storage method provided by the present application, the data compression is realized by the first data processor network card, and the central processor is no longer responsible for data compression, which can effectively reduce the utilization rate of the central processor, save the I / O processing program of the central processor, and improve the performance of the central processor. At the same time, the first data processor transmits the compressed data to the second data processor network card on the storage side, and the second data processor network card writes the compressed data into the hard disk, which can reduce the transmission amount of data in the network, improve the network bandwidth utilization rate, and avoid the performance bottleneck caused by network congestion. Moreover, the method has independent expansion capability of compression performance, which can greatly reduce the data transmission amount between storage controllers.

[0148] In a third aspect, the present application also provides a data compression storage device, which can be correspondingly referred to the method described above. Please refer to FIG. 8, which is a schematic diagram of a data compression storage device provided by an embodiment of the present application. The device is applied to a first data processor network card of a storage control side, the first data processor network card is connected with a central processing unit of the storage control side, and the first data processor network card is connected with a second data processor network card through a network, and the second data processor network card is connected with hard disks. As shown in FIG. 8, the data compression storage device comprises the following modules:

[0149] The receiving module 10 is configured to receive the notification information sent by the central processing unit.

[0150] The obtaining module 11 is configured to obtain the to-be-compressed data from the memory of the storage control side according to the notification information.

[0151] The compression module 12 is configured to compress the to-be-compressed data to obtain compressed data, and buffer the compressed data.

[0152] The sending module 13 is configured to send a request to the second data processor network card, so that the second data processor network card reads the compressed data from the first data processor network card according to the request and writes the compressed data into the hard disks.

[0153] The notification information is used to notify the first data processor network card to compress data, and the notification information comprises the position of the to-be-compressed data and the data length.

[0154] In some embodiments, the notification information further comprises the data type of the to-be-compressed data.

[0155] In some embodiments, the obtaining module 11 is specifically configured to read the to-be-compressed data from the memory of the storage control side according to the position and the data length of the to-be-compressed data.

[0156] In some embodiments, the compression module 12 comprises a determining unit configured to determine a compression algorithm and a compression rate according to the data type of the to-be-compressed data, and a compression unit configured to compress the to-be-compressed data by using the compression algorithm and the compression rate to obtain the compressed data.

[0157] In some embodiments, the data compression storage device further comprises a notification module configured to notify the central processing unit that the data compression is completed after the to-be-compressed data is compressed to obtain the compressed data and the compressed data is buffered.

[0158] In some embodiments, the data compression storage device further comprises a data write result receiving and sending module configured to receive a complete data write result sent by the second data processor network card after the compressed data is written into the hard disks, and send the complete data write result to the central processing unit.

[0159] In some embodiments, the data compression storage device further comprises: a command receiving module configured to receive a read data command sent by the central processor; a command sending module configured to send the read data command to the second data processor network card to cause the second data processor network card to read target compressed data from the hard disk according to the read data command; a compressed data receiving module configured to receive the target compressed data sent by the second data processor network card; a decompression module configured to decompress the target compressed data to obtain target data and write the target data into the memory on the storage control side; and a decompression completion notification module configured to notify the central processor that the data has been decompressed to cause the central processor to read the target data from the memory on the storage control side.

[0160] In some embodiments, the data compression storage device further comprises: a data read result receiving module configured to receive a notification of completion of data reading sent by the central processor after reading the target data from the memory on the storage control side.

[0161] In some embodiments, the first data processor network card comprises: a network card central processor, a field programmable gate array and a network card memory, the network card central processor internally integrating a compressor, the network card central processor being configured to perform data compression and decompression, and the field programmable gate array being configured to receive and send data.

[0162] In some embodiments, the network card central processor and the field programmable gate array are connected through a peripheral component interconnect express interface, the field programmable gate array is connected to a remote direct memory access network port, and the field programmable gate array is connected to the central processor on the storage control side through the peripheral component interconnect express interface.

[0163] The data compression storage device provided in the present application implements data compression by the first data processor network card, and the central processor is no longer responsible for data compression, which can effectively reduce the usage rate of the central processor, save the I / O processing program of the central processor, and improve the performance of the central processor. At the same time, the first data processor transmits the compressed data to the second data processor network card on the storage side, and the second data processor network card writes the compressed data into the hard disk, which can reduce the transmission amount of data in the network, improve the network bandwidth usage rate, and avoid the performance bottleneck caused by network congestion. Moreover, the device has independent expandability of compression performance, which can greatly reduce the data transmission amount between storage controllers.

[0164] In a fourth aspect, the present application also provides another data compression storage device, which can be correspondingly referred to the method described above. Please refer to FIG. 9, which is a schematic diagram of another data compression storage device provided by the embodiments of the present application. The device is applied to a second data processor network card of a storage side, the second data processor network card is connected with each hard disk, and the second data processor network card is connected with a first data processor network card of a storage control side through a network, and the first data processor network card is connected with a central processing unit of the storage control side. In combination with the device shown in FIG. 9, the device comprises:

[0165] a request receiving unit 20, configured to receive a request sent by the first data processor network card;

[0166] a data reading unit 21, configured to read compressed data from the first data processor network card according to the request; and

[0167] a data writing unit 22, configured to write the compressed data into the hard disk according to the request.

[0168] In the device, the first data processor network card is configured to receive notification information sent by the central processing unit, acquire the to-be-compressed data from the memory of the storage control side according to the notification information, perform compression processing on the to-be-compressed data to obtain the compressed data, cache the compressed data, and send a request to the second data processor network card. The notification information is used to notify the first data processor network card to perform data compression, and the notification information comprises the position and the data length of the to-be-compressed data.

[0169] In some embodiments, the data compression storage further comprises a data writing result sending unit, configured to send a data writing completion result to the first data processor network card after the compressed data is written into the hard disk.

[0170] In some embodiments, the data compression storage further comprises a command receiving unit, configured to receive a read data command sent by the first data processor network card; a data reading unit, configured to read target compressed data from the hard disk according to the read data command; and a data sending unit, configured to send the target compressed data to the first data processor network card, so that the first data processor network card performs decompression processing on the target compressed data to obtain target data, and writes the target data into the memory of the storage control side.

[0171] In some embodiments, the second data processor network card comprises a network card central processing unit, a field programmable gate array, and a network card memory. The network card central processing unit internally integrates a compressor. The network card central processing unit is configured to perform data compression and decompression. The field programmable gate array is configured to receive data and send data.

[0172] In some embodiments, the network card central processor is connected with the field programmable gate array through a peripheral component interconnect express interface, the field programmable gate array is connected with the remote direct memory access network port, and the field programmable gate array is connected with the hard disk through the peripheral component interconnect express interface.

[0173] The data compression storage device provided by the application is realized by the first data processor network card, and the central processor is no longer responsible for data compression, so that the use rate of the central processor can be effectively reduced, the I / O processing program of the central processor is saved, and the performance of the central processor is improved. Meanwhile, the first data processor transmits compressed data to the second data processor network card on the storage side, and the second data processor network card writes the compressed data into the hard disk, so that the data transmission amount in the network can be reduced, the network bandwidth use rate can be improved, and the performance bottleneck caused by network congestion can be avoided. Moreover, the device has independent expansion capability of compression performance, and the data transmission amount between storage controllers can be greatly reduced.

[0174] In a fifth aspect, the application further provides a storage system, comprising: a storage control side and a storage side. The storage control side comprises a central processor, a first data processor network card and a memory, and the storage side comprises a second data processor network card and a plurality of hard disks. The first data processor network card is connected with the central processor, and the first data processor network card is connected with the second data processor network card through a network, and the second data processor network card is connected with the hard disks.

[0175] The first data processor network card is used for receiving notification information sent by the central processor. The notification information is used for notifying the first data processor network card to perform data compression, and the notification information comprises the position and the data length of to-be-compressed data. The to-be-compressed data is obtained from the memory according to the notification information, the to-be-compressed data is compressed to obtain compressed data, the compressed data is cached, and a request is sent to the second data processor network card.

[0176] The second data processor network card is used for reading the compressed data from the first data processor network card according to the request, and writing the compressed data into the hard disk according to the request.

[0177] The storage system provided by the application is realized by the first data processor network card, and the central processor is no longer responsible for data compression, so that the use rate of the central processor can be effectively reduced, the I / O processing program of the central processor is saved, and the performance of the central processor is improved. Meanwhile, the first data processor transmits compressed data to the second data processor network card on the storage side, and the second data processor network card writes the compressed data into the hard disk, so that the data transmission amount in the network can be reduced, the network bandwidth use rate can be improved, and the performance bottleneck caused by network congestion can be avoided. Moreover, the storage system has independent expansion capability of compression performance, and the data transmission amount between storage controllers can be greatly reduced.

[0178] For the storage system provided in the present application, refer to the above method embodiments, which will not be repeated here.

[0179] The present application also provides a computer device, in some embodiments, the computer device is shown in FIG. 10, and the computer device includes a non-volatile computer readable storage medium.

[0180] In a sixth aspect, the present application also provides a non-volatile readable storage medium, as shown in FIG. 10, the medium can be a non-volatile computer readable storage medium in a computer device, and the medium stores computer readable instructions, and the computer readable instructions are executed by a processor to implement the steps of the above data compression storage method applied to the first data processor network card or implement the steps of the above data compression storage method applied to the second data processor network card.

[0181] The medium can include a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store program codes.

[0182] For the non-volatile readable storage medium provided in the present application, refer to the above method embodiments, which will not be repeated here.

[0183] In a seventh aspect, the present application also provides a computer readable instruction product, which can be a computer device as shown in FIG. 10, and the computer device includes computer readable instructions / instructions, and the computer readable instructions are executed by a processor to implement the steps of the above data compression storage method applied to the first data processor network card or implement the steps of the above data compression storage method applied to the second data processor network card.

[0184] The embodiments in the description are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. For the apparatus, device and medium disclosed in the embodiments, since they correspond to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0185] Those skilled in the art will further realize that the mere conception of the examples described herein is not inducing the patentable subject matter recited in each of the claims. Accordingly, those skilled in the art will recognize that the examples described herein can be implemented with a variety of computer systems or other hardware, software, firmware, or a combination thereof, applying the teachings of the examples described herein to the general principles of computer science and technology. Accordingly, although the examples have been described with reference to specific exemplifications, it will be recognized that the examples are not limited thereto and that the examples can be practiced with reference to any number of software and hardware configurations. For a software implementation, the techniques described herein can be implemented with software at a machine level. The software can be stored on a program storage memory or system memory, for example, as an application program interface (API) or a program, and can be implemented in a standardized or proprietary language. The software can be executed with a general- purpose computer or a dedicated computer. For an hardware implementation, the techniques described herein can be implemented with at least one application specific integrated circuit (ASIC) adapted at least in part to perform the steps described herein. As will be realized, the examples described herein are capable of outputs more than the specific implementations described herein. Therefore, many modifications and other implementations are possible without departing from the spirit and the scope of the examples described herein.

[0186] The steps of a method or algorithm described in connection with the examples disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The exemplary storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0187] The data compression storage method, device, system, medium and computer readable instruction product provided by the present application are described in detail above. The principles and implementation modes of the present application are described by applying specific examples in the present document. The above description of the examples is only to help understand the method and its core idea of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A data compression storage method characterized by, The application discloses a first data processor network card applied to a storage control side, the first data processor network card is connected with a central processor of the storage control side, and the first data processor network card is connected with a second data processor network card of a storage side through a network, and the second data processor network card is connected with hard disks of the storage side. Receiving notification information sent by the central processor; the notification information is used for notifying the first data processor network card to perform data compression; the notification information comprises the position of to-be-compressed data and the data length; Obtaining to-be-compressed data from the memory of the storage control side according to the notification information; Compressing the to-be-compressed data to obtain compressed data, and buffering the compressed data; and Sending a request to the second data processor network card, so that the second data processor network card reads the compressed data from the first data processor network card according to the request and writes the compressed data into the hard disks.

2. The data compression storage method according to claim 1, wherein, The notification information further comprises the data type of the to-be-compressed data.

3. The data compression storage method of claim 1, wherein, Obtaining to-be-compressed data from the memory of the storage control side according to the notification information comprises: Reading the to-be-compressed data from the memory of the storage control side according to the position and the data length of the to-be-compressed data.

4. The data compression storage method of claim 2, wherein, Compressing the to-be-compressed data to obtain compressed data comprises: Determining a compression algorithm and a compression rate according to the data type of the to-be-compressed data; and Compressing the to-be-compressed data by using the compression algorithm and the compression rate to obtain the compressed data.

5. The data compression storage method of claim 1, wherein, Further comprising: After the to-be-compressed data is compressed to obtain the compressed data and the compressed data is buffered, notifying the central processor that the data compression is completed.

6. The data compression storage method of claim 1, wherein, Further comprising: Receiving a complete data writing result sent by the second data processor network card after the second data processor network card writes the compressed data into the hard disks, and sending the complete data writing result to the central processor.

7. The data compression storage method of claim 1, wherein, Further comprising: Receiving a read data command sent by the central processor; Sending the read data command to the second data processor network card, so that the second data processor network card reads target compressed data from the hard disks according to the read data command; Receiving the target compressed data sent by the second data processor network card; Decompressing the target compressed data to obtain target data, and writing the target data into the memory of the storage control side; And Notifying the central processor that the data has been decompressed, so that the central processor reads the target data from the memory of the storage control side.

8. The data compression storage method according to claim 7, wherein, Further comprising: Receiving a notification of completing data reading sent by the central processor after the central processor reads the target data from the memory of the storage control side.

9. The data compression storage method according to any one of claims 1 to 8, characterized by, The first data processor network card comprises a network card central processor, a field programmable gate array and a network card memory; the network card central processor is internally integrated with a compressor; the network card central processor is used for performing data compression and decompression, and the field programmable gate array is used for receiving data and sending data.

10. The data compression storage method of claim 9, wherein, The network card central processor is connected with the field programmable gate array through a peripheral component interconnect express interface, the field programmable gate array is connected with a remote direct memory access network port, and the field programmable gate array is connected with the central processor of the storage control side through the peripheral component interconnect express interface.

11. A data compression storage method characterized by comprising: The second data processor network card applied to the storage side is connected with each hard disk, and the second data processor network card is connected with the first data processor network card of the storage control side through a network, and the first data processor network card is connected with the central processor of the storage control side. Receiving the request sent by the first data processor network card; According to the request, the compressed data is read from the first data processor network card; the first data processor network card is used for receiving the notification information sent by the central processor, obtaining the to-be-compressed data from the memory of the storage control side according to the notification information, performing compression processing on the to-be-compressed data to obtain compressed data, and caching the compressed data, and sending a request to the second data processor network card, wherein the notification information is used for notifying the first data processor network card to perform data compression, and the notification information includes the position and data length of the to-be-compressed data; and According to the request, the compressed data is written into the hard disk.

12. The data compression storage method of claim 11, wherein, Further comprising: After the compressed data is written into the hard disk, a data writing completion result is sent to the first data processor network card.

13. The data compression storage method of claim 11, wherein, Further comprising: Receiving a read data command sent by the first data processor network card; According to the read data command, target compressed data is read from the hard disk; And The target compressed data is sent to the first data processor network card, so that the first data processor network card performs decompression processing on the target compressed data to obtain target data, and the target data is written into the memory of the storage control side.

14. The data compression storage method of claim 11, wherein, The second data processor network card comprises a network card central processor, a field programmable gate array and a network card memory; the network card central processor is internally integrated with a compressor; the network card central processor is used for data compression and decompression, and the field programmable gate array is used for receiving and sending data.

15. The data compression storage method of claim 14, wherein, The network card central processor is connected with the field programmable gate array through a peripheral component interconnect express interface, the field programmable gate array is connected with a remote direct memory access network port, and the field programmable gate array is connected with the central processor of the storage control side through the peripheral component interconnect express interface.

16. A data compression storage device, characterized by, The first data processor network card applied to the storage control side is connected with the central processor of the storage control side, and the first data processor network card is connected with the second data processor network card through a network, and the second data processor network card is connected with each hard disk; comprising: A receiving module is used for receiving notification information sent by the central processor; the notification information is used for notifying the first data processor network card to perform data compression; the notification information includes the position and data length of the to-be-compressed data; An obtaining module is used for obtaining to-be-compressed data from the memory of the storage control side according to the notification information; a compression module, configured to compress the to-be-compressed data to obtain compressed data and buffer the compressed data; and a sending module, configured to send a request to the second data processor network card, so that the second data processor network card reads the compressed data from the first data processor network card and writes the compressed data into the hard disk according to the request.

17. A data compression storage device, characterized by, The second data processor network card applied to the storage side is connected with each hard disk, and is connected with the first data processor network card of the storage control side through a network. The first data processor network card is connected with a central processing unit of the storage control side. The second data processor network card comprises: a request receiving unit configured to receive a request sent by the first data processor network card; a data reading unit configured to read compressed data from the first data processor network card according to the request; the first data processor network card is configured to receive notification information sent by the central processing unit, acquire to-be-compressed data from a memory of the storage control side according to the notification information, compress the to-be-compressed data to obtain compressed data, buffer the compressed data, and send a request to the second data processor network card, wherein the notification information is used to notify the first data processor network card to compress data, and the notification information comprises a position and a data length of the to-be-compressed data; and a data writing unit configured to write the compressed data into the hard disk according to the request.

18. A storage system, comprising: The storage control side and the storage side are connected. The storage control side comprises a central processing unit, a first data processor network card, and a memory. The storage side comprises a second data processor network card and a plurality of hard disks. The first data processor network card is connected with the central processing unit, and is connected with the second data processor network card through a network. The second data processor network card is connected with the hard disks. The first data processor network card is configured to receive notification information sent by the central processing unit. The notification information is used to notify the first data processor network card to compress data. The notification information comprises a position and a data length of to-be-compressed data. The to-be-compressed data is acquired from the memory according to the notification information. The to-be-compressed data is compressed to obtain compressed data, and the compressed data is buffered. A request is sent to the second data processor network card. The second data processor network card is configured to read the compressed data from the first data processor network card according to the request. The compressed data is written into the hard disk according to the request. The non-volatile readable storage medium stores computer readable instructions. When the computer readable instructions are executed by a processor, the steps of the data compression storage method according to any one of claims 1 to 10 or the steps of the data compression storage method according to any one of claims 11 to 15 are implemented.

19. A non-volatile readable storage medium, characterized by The computer readable instructions are executed by a processor to implement the steps of the data compression storage method according to any one of claims 1 to 10 or the steps of the data compression storage method according to any one of claims 11 to 15.

20. A computer readable instruction product comprising computer readable instructions, characterized in that, ​

Images