Method, apparatus and computer program product for transmitting data blocks

Abstract

The invention relates to a method, a device and a computer program product for transmitting data blocks. The method includes receiving, from a client, a request to read a data block. The method also includes determining, at a data block layer, whether the data block needs to be compressed at a protocol layer based on a flag indicating whether the data block has been compressed and a storage size of the data block. The method also includes communicating, from the data block layer to the protocol layer, an indication that the data block does not need to be compressed at the protocol layer in response to determining that the data block does not need to be compressed at the protocol layer. In addition, the method also includes sending the data block to the client over the protocol layer without compressing the data block at the protocol layer. In this way, processor and memory resources can be saved, and overall response time can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of data storage, and more particularly, to methods, devices, and computer program products for transmitting data blocks. Background Art

[0002] Protocol layer compression is to compress files at the protocol layer of the server or client before sending them, then send the compressed files over the network, and then decompress the compressed files at the server or client that receives them. Many file sharing protocols support protocol layer compression, which can reduce the amount of data transmitted over the network. When data transmission is limited by network bandwidth, it helps to reduce data transmission time and improve actual data transmission throughput. Summary of the invention

[0003] Embodiments of the present disclosure provide a method, device and computer program product for transmitting data blocks. In an embodiment of the present disclosure, a server receives a request to read a data block from a client. Then, the server can obtain compression information related to the data block at the data block layer. The compression information may include a flag indicating whether the data block is stored after being compressed, and the size of the disk space actually occupied by the data block in the storage system. Based on the compression information, the server can determine whether it is necessary to compress the data block at the protocol layer. If the data block does not need to be compressed at the protocol layer, the indication that compression is not required is passed from the data block layer to the protocol layer, and then the uncompressed data block is directly sent to the client at the protocol layer without further compression. In this way, compression of data blocks with poor compressibility can be reduced at the protocol layer, and compression of these data blocks with poor compressibility will result in a reduction in the amount of transmitted data and a reduction in transmission time while increasing the consumption of processor and memory resources. Therefore, this method can enable the protocol layer to perform appropriate operation processes for different data blocks by passing the compressibility information of the data block from the data block layer to the protocol layer, thereby reducing the compression and decompression operations on the data blocks with poor compressibility, saving processor and memory resources. In addition, since this method reduces the compression and decompression operations, the time consumed by performing the compression and decompression operations is saved, and the overall response time can be reduced.

[0004] In a first aspect of an embodiment of the present disclosure, a method for transmitting a data block is provided. The method includes receiving a request for reading a data block from a client. The method also includes determining, at the data block layer, whether the data block needs to be compressed at the protocol layer based on a flag indicating whether the data block has been compressed and the storage size of the data block. The method also includes, in response to determining that the data block does not need to be compressed at the protocol layer, passing an indication that the data block does not need to be compressed at the protocol layer from the data block layer to the protocol layer. In addition, the method also includes sending the data block to the client through the protocol layer without compressing the data block at the protocol layer.

[0005] In a second aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a method for transmitting a data block. The method includes receiving a request for reading a data block from a client. The method also includes determining, at the data block layer, whether the data block needs to be compressed at the protocol layer based on a flag indicating whether the data block has been compressed and the storage size of the data block. The method also includes, in response to determining that the data block does not need to be compressed at the protocol layer, passing an indication that the data block does not need to be compressed at the protocol layer from the data block layer to the protocol layer. In addition, the method also includes sending the data block to the client through the protocol layer without compressing the data block at the protocol layer.

[0006] In a third aspect of an embodiment of the present disclosure, a computer program product is provided, which is tangibly stored on a non-volatile computer-readable medium and includes machine executable instructions, which, when executed, cause a machine to implement a method for transmitting a data block. The method includes receiving a request to read a data block from a client. The method also includes determining, at the data block layer, whether the data block needs to be compressed at the protocol layer based on a flag indicating whether the data block has been compressed and the storage size of the data block. The method also includes, in response to determining that the data block does not need to be compressed at the protocol layer, passing an indication that the data block does not need to be compressed at the protocol layer from the data block layer to the protocol layer. In addition, the method also includes sending the data block to the client through the protocol layer without compressing the data block at the protocol layer.

[0007] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure may be implemented;

[0010] Figure 2 A flow chart showing a method for transmitting a data block according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A schematic diagram showing a process of generating data block compression information according to some embodiments of the present disclosure;

[0012] Figure 4 A schematic diagram showing an example of storing data block compression information according to some embodiments of the present disclosure;

[0013] Figure 5 A schematic diagram showing an example process from receiving a request to read a data block to sending the data block according to some embodiments of the present disclosure;

[0014] Figure 6 A schematic diagram illustrating an example process of passing a compression indication from a data block layer to a protocol layer according to some embodiments of the present disclosure;

[0015] Figure 7 A schematic diagram showing an example header structure used at the protocol layer according to some embodiments of the present disclosure;

[0016] Figure 8 A schematic diagram showing an example of including multiple payload headers in a single network packet according to some embodiments of the present disclosure;

[0017] Fig. 9 Another schematic diagram illustrating an example process of passing a compression indication from a data block layer to a protocol layer according to some embodiments of the present disclosure; and

[0018] Fig.10 A block diagram of a device in which various embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0021] As mentioned above, protocol layer compression is to compress the file at the protocol layer of the server or client before sending the file, then send the compressed file through the network, and then decompress the compressed file at the server or client that receives the compressed file. In this way, the amount of data transmitted through the network can be reduced, which helps to reduce the data transmission time and improve the actual data transmission throughput when the data transmission is limited by the network bandwidth. However, since this method requires the compression and decompression operations to be performed on the client and server respectively, it will consume processor resources. How to reduce unnecessary processor resource consumption while improving data transmission efficiency is a problem that needs to be solved.

[0022] The File Transfer Protocol (FTP), Secure File Transfer Protocol (SFTP), and Server Message Block (SMB) protocols all support protocol layer compression with connection-level compression, which automatically negotiates the compression / decompression protocol between the server and the client, and then reduces the size of the data transferred over the connection. FTP uses the "COMpress" subcommand to enable compressed data transfer mode. SFTP uses the "compression(-C)" switch to enable connection-level compression. SMB compression can be enabled with protocol layer compression through the configuration of the client and server.

[0023] Taking the SMB protocol as an example, SMB compression allows clients to request compression of files before sending them over the network. When the storage server is capable of SMB compression, it can accept the request and transmit the compressed data. If SMB compression is not enabled or the client chooses not to use SMB compression, normal I / O requests will continue without compression. Compressing files can save the amount of data transmitted over the network, and the higher the compression ratio, the more data is saved, the less bandwidth is consumed, and the less transmission time is consumed.

[0024] However, for high entropy data streams (i.e., data streams with a large amount of information in the data and no obvious regularity or pattern), compression can provide little help because only a small amount of space can be saved through compression calculations. In some traditional schemes, SMB compression sampling is introduced to minimize the impact. SMB compression sampling attempts to compress the first 524,288,000 bytes (500 MiB) of the file during transmission. If the space that can be saved by compression in the 500 MiB is less than 104,857,600 bytes (100 MiB), SMB compression will stop trying to compress the rest of the file. If at least 100 MiB is saved in the first 500 MiB, SMB compression will continue to compress the rest of the file. However, the results of this compression sampling are not accurate because the amount of data that can be compressed in the first 500 MiB may be small, but the rest of the data that can be compressed is large.

[0025] In other conventional schemes, when a client or server requests to read a file, SMB always attempts to compress the entire file, that is, when a client or server requests to read a file, compression is always performed on all data blocks. This means that for data blocks with poor compressibility (i.e., data blocks with a small difference between the size before and after compression), processor and memory resources are wasted on unnecessary compression attempts. In addition, the time spent on these compression attempts also results in longer response times.

[0026] To this end, an embodiment of the present disclosure proposes a scheme for concurrently transmitting data. In an embodiment of the present disclosure, a server receives a request to read a data block from a client. Then, the server can obtain compression information related to the data block at the data block layer. The compression information may include a flag indicating whether the data block is stored after being compressed, and the size of the disk space actually occupied by the data block in the storage system. Based on this compression information, the server can determine whether it is necessary to compress the data block at the protocol layer. If the data block does not need to be compressed at the protocol layer, an indication that compression is not required is passed from the data block layer to the protocol layer, and then the uncompressed data block is sent directly to the client at the protocol layer without compressing it.

[0027] In this way, the compressibility information of the data block can be passed from the data block layer to the protocol layer, so that the protocol layer can perform appropriate operation processes for different data blocks, thereby reducing the compression and decompression operations on data blocks with poor compressibility, saving processor and memory resources. In addition, since this method reduces the compression and decompression operations, thereby saving the time consumed in performing the compression and decompression operations, the overall response time can be reduced.

[0028] Figure 1 1 is a schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. Figure 1 As shown, the environment 100 includes a client 102 and a server 104, wherein the client 102 includes a protocol layer 106, and the server 104 includes a protocol layer 108, a file system layer 110, a data block layer 112, and data blocks 114-1, 114-2, ..., 114-N (collectively referred to as data blocks 114) are stored. At the protocol layer 106 of the client 102 and the protocol layer 108 of the server 104, the transmission of data in the network, the compression of the data blocks before transmission, and the decompression of the compressed data blocks after receiving them can be realized. At the file system layer 110 of the server 104, the organization, storage, and access of files and directories can be managed. At the data block layer 112 of the server 104, actual storage and read-write support of data can be provided for the file system layer 110.

[0029] exist Figure 1 In the environment 100 shown, the data block 114-1 stored in the server 104 may be a compressed data block or an uncompressed original data block, and a compression flag 116 indicating whether the data block 114-1 has been compressed is stored in the server 104. In some embodiments, a value of "1" of the compression flag 116 may indicate that the stored data block 114-1 is a compressed data block, and a value of "0" of the compression flag 116 may indicate that the stored data block 114-1 is an uncompressed original data block. In addition, a storage size 118 of the data block 114-1 is also stored in the server 104. Depending on the settings of the storage system, the data block 114 has a set size when it is not compressed, for example, 8KB. If the stored data block 114 is an original data block, the storage size 118 is 8KB; if the stored data block 114 is a compressed data block, the storage size 118 may be, for example, 4KB. It should be understood that for the sake of simplicity, in Figure 1 In the illustrated environment 10 , only the compression flag 116 and the storage size 118 corresponding to the data block 114 - 1 are shown, but the server 104 may obtain the compression flag and the storage size corresponding to each data block 114 .

[0030] like Figure 1As shown, the client 102 may send a read request 120 to the server 104 to read a data block. For example, the read request 120 may request to read the data block 114-1. Then, the server 104 may generate a compression indication 122 at the data block layer 112, and the compression indication 122 indicates whether the data block 114-1 needs to be compressed at the protocol layer 108 before being transmitted to the client 102. At the protocol layer 108, if the compression indication 122 indicates that the data block 114-1 does not need to be compressed, the server 104 directly transmits the uncompressed data block 126 (i.e., the uncompressed data block 114-1) to the client 102; if the compression indication 122 indicates that the data block 114-1 needs to be compressed, the server 104 compresses the data block 114-1 before transmitting it to generate a compressed data block 124, and then transmits it to the protocol layer 106. For the data block compressed at the protocol layer 108 of the server 104 , the client 102 decompresses the data block at the protocol layer 106 after receiving the data block, thereby obtaining the requested data block.

[0031] Figure 2 FIG. 2 is a flow chart of a method 200 for transmitting a data block according to some embodiments of the present disclosure. Figure 2 As shown, at block 202, method 200 receives a request from a client to read a data block. Figure 1 In the illustrated environment 100 , the server 104 may receive a read request 120 for reading a data block from the client 102 . The data block to be read may be one or more data blocks in the data blocks 114 .

[0032] At block 204, method 200 may determine at the data block layer whether the data block needs to be compressed at the protocol layer based on a flag indicating whether the data block has been compressed and a storage size of the data block. Figure 1 In the illustrated environment 100, the server 104 includes a protocol layer 108, a file system layer 110, and a data block layer 112, and compression information of a data block 114 can be obtained at the data block layer 112. The compression information may include a compression flag indicating whether the data block stored in the disk has been compressed (e.g., a compression flag 116 indicating whether the data block 114-1 has been compressed and stored) and a storage size indicating the space actually occupied by the data block in the disk (e.g., a storage size 118 indicating the space actually occupied by the data block 114-1 in the disk). When a read request 120 requests to read the data block 114-1, the server 104 may determine whether the data block 114-1 needs to be compressed at the protocol layer based on the compression flag 116 and the storage size 118.

[0033] At block 206, in response to determining that the data block does not need to be compressed at the protocol layer, the method 200 communicates an indication from the data block layer to the protocol layer that the data block does not need to be compressed at the protocol layer. Figure 1 In the illustrated environment 100, if it is determined that the data block 114-1 does not need to be compressed at the protocol layer, a compression indication 122 indicating that the data block 114-1 does not need to be compressed can be passed from the data block layer to the protocol layer 108 via the file system layer 110. In some embodiments, the compression indication 122 can be a flag in a data structure corresponding to the data block 114-1 that is passed to the protocol layer along with the data block 114-1.

[0034] At block 208, method 200 sends the data block to the client via the protocol layer without compressing the data block at the protocol layer. Figure 1 In the illustrated environment 100, at the protocol layer 108, the data block 114-1 (i.e., the uncompressed data block 126) may be sent to the client 102 without being compressed at the protocol layer according to the compression indication 122 indicating that the data block 114-1 does not need to be compressed. In some embodiments, if the compression indication 122 indicates that the data block 114-1 needs to be compressed at the protocol layer, the protocol layer 108 will compress the data block 114-1 and then send the compressed data block 124 to the client 102, and the client 102 will then decompress the compressed data block 124 at the protocol layer 106 to obtain the data block 114-1.

[0035] In this way, the compressibility information of the data block can be passed from the data block layer to the protocol layer, so that the protocol layer can perform appropriate operation processes for different data blocks, thereby reducing the compression and decompression operations on data blocks with poor compressibility, saving processor and memory resources. In addition, since this method reduces the compression and decompression operations, it saves the time consumed by performing compression and decompression operations, and thus can reduce the overall response time.

[0036] In the conventional solution, since the compression information of the data block cannot be known at the protocol layer, it is impossible to perform appropriate operations for different data blocks. In the embodiment provided by the present disclosure, the compression information for the data block can be generated at the data block layer, and the data block layer can evaluate whether the data block needs to be compressed at the protocol layer based on the compression information of the data block requested to be read by the client, and then the indication indicating whether compression is needed is passed from the data block layer to the protocol layer.

[0037] In order to generate and store compression information of a data block, in some embodiments, before the data block is written to the disk, upon receiving an original data block to be written (also referred to as a first data block), a compressed data block (also referred to as a second data block) may be generated by compressing the original data block. Then, the size of the original data block and the size of the compressed data block may be determined, and if the ratio of the size of the original data block to the size of the compressed data block is greater than or equal to a predetermined threshold, the compressed data block is stored. Then, a flag corresponding to the original data block indicating whether the original data block has been compressed may be set to a value indicating that the original data block has been compressed, and the size of the compressed data block may be recorded. In some embodiments, if the ratio of the size of the original data block to the size of the compressed data block is less than a predetermined threshold, the original data block is stored. Then, a flag corresponding to the original data block indicating whether the original data block has been compressed may be set to a value indicating that the original data block has not been compressed.

[0038] Figure 3 FIG. 3 is a schematic diagram showing an example process 300 for generating data block compression information according to some embodiments of the present disclosure. Figure 3 In the example shown, process 300 receives original data blocks 302 and 312. In a storage server (e.g., Figure 1 In the example of the server 104 shown in FIG. 1 , the file can be stored in the form of data blocks on the disk, and the size of the data blocks can be specified in a configurable manner, for example, each data block can be 4KB, 8KB, 16KB, 1MB, etc. In this example, the size of the data block is configured to be 8KB, so the size of the original data blocks 302 and 312 is 8KB. Before writing the original data blocks 302 and 312 to the disk, the process 300 can compress the original data block 302 to generate a compressed data block 304, and compress the original data block 312 to generate a compressed data block 314 using a compression algorithm with stronger performance than the compression algorithm used by the protocol layer. Figure 3 In the example shown, the size of compressed data block 304 is 2 KB, and the size of compressed data block 314 is 6 KB.

[0039] Then, process 300 can determine the compression ratio of original data block 302 by calculating the ratio of the size of original data block 302 to the size of compressed data block 304, and determine the compression ratio of original data block 312 by calculating the ratio of the size of original data block 312 to the size of compressed data block 314. Figure 3 In the example shown, the compression ratio of the original data block 302 is 4 (ie, 8KB / 2KB), and the compression ratio of the original data block 312 is approximately 1.3 (ie, 8KB / 6KB).

[0040] When writing a data block, process 300 may compare the compression ratio to a predetermined threshold ratio. Figure 3 In the example shown, the threshold ratio is set to 2, and the compression ratio of the original data block 302 (i.e., 4) is greater than the threshold ratio (i.e., 2), which means that the original data block 302 has good compressibility, and storing the compressed data block 304 can save more disk space, although it is worthwhile to consume processor, memory resources and time costs to decompress the data block when reading it in the future. Therefore, the process 300 can write the compressed data block 304 to the disk, and set the corresponding compression flag to indicate that the written compressed data block 304 is a compressed data block (e.g., "compressed" or "1"), and record the size of the compressed data block 304 (i.e., 2KB).

[0041] On the contrary, if the compression ratio (i.e., 1.3) of the original data block 312 is less than the threshold ratio (i.e., 2), it means that the compressibility of the original data block 312 is poor. If storing the compressed data block 314 can only save a small amount of disk space, but it will take up processor, memory resources and time cost to decompress it in the future, it is not worth it. Therefore, in this case, the process 300 can write the original data block 312 to the disk, and set the corresponding compression flag to a value indicating that the written original data block 312 is an uncompressed data block (e.g., "uncompressed" or "0"). Since the size of the original data block 312 is set to 8KB, there is no need to record the size of the disk space actually occupied.

[0042] In this way, some data blocks with poor compressibility can be marked when they are written to the disk. If the more powerful compression algorithm used by the storage server cannot effectively compress the data block when writing it, then the lighter and less powerful compression algorithm used at the protocol layer cannot effectively compress it. Therefore, when the client needs to read the data block, it can be directly determined that it does not need to be compressed at the protocol layer, thereby saving computing time and improving computing efficiency. In addition, for data blocks with good compressibility, since process 300 records the storage size of the compressed data block when writing the compressed data block to the disk, when determining whether it needs to be compressed at the protocol layer, the storage size can be directly used to calculate the compression ratio without performing the compression operation, thereby saving computing time, improving computing efficiency, and reducing the consumption of processor resources and memory resources.

[0043] In order to store the compression information of the data block, in some embodiments, an indirect block may be used to store the compression flag, and a data block mapping table may be used to store the size of the compressed data block. Figure 4FIG. 4 is a schematic diagram showing an example 400 of storing data block compression information according to some embodiments of the present disclosure. Figure 4 As shown, example 400 includes indirect block 402, indirect block 404, data block mapping table 406, and compressed data storage area 408. In some embodiments, the storage structure of the disk may be a tree structure (e.g., a B-tree and a variant of a B-tree, an LSM tree, etc.), and indirect blocks 402 and 404 may be leaf indirect blocks in the tree structure. Figure 4 As shown, indirect blocks 402 and 404 include a compression flag field and a pointer field, and each entry in indirect blocks 402 and 404 corresponds to a data block. The compression flag field indicates whether the data block written to the disk has been compressed. In example 400, the value of the compression flag field of entries 411, 412, 413, and 414 is "1", indicating that the data blocks corresponding to these entries have been compressed. The corresponding entries in the data block mapping table can be found according to the pointer fields of these entries. For example, in example 100, the pointers of entries 411, 412, 413, and 414 all point to the data block mapping table 406.

[0044] The data block mapping table is used to track the mapping relationship between various data in the storage server, and it includes information associated with the compressed data block. In example 400, the data block mapping table 406 includes an entry 421 corresponding to entry 411, an entry 422 corresponding to entry 412, an entry 423 corresponding to entry 413, and an entry 414 corresponding to entry 414. Entry 421 includes a compression length field, which indicates the address length of the compressed data block in the storage space, and may also indicate the actual storage size of the compressed data block. For example, the compression flag of entry 411 in the indirect block 402 is "1", and the compression length of entry 421 in the data block mapping table 406 corresponding to entry 411 is 2048 bytes, which may indicate that the data blocks corresponding to entries 411 and 421 are stored in the disk after being compressed, and the storage size of the compressed data block is 2048 bytes (i.e., 2KB).

[0045] The data block mapping table 406 may be associated with the compressed data storage area 408, and the compressed data blocks are stored in the compressed data storage area 408. The compressed data storage area 408 includes compressed data 431, 432, 433, and 434, and each compressed data has corresponding header data, wherein the header data stores the address length of the storage space occupied by the compressed data, and the compressed length of the data block mapping table 406 includes the address length of the space occupied by the compressed data in the compressed data storage area and the address length of the space occupied by the corresponding header data. In some embodiments, the compressed length in the data block mapping table 406 may be used as the storage size of the compressed data block, and in other embodiments, the address length of the space occupied by the compressed data (e.g., compressed data 431) in the compressed data storage area 408 may be used as the storage size of the compressed data block.

[0046] In this way, compression information such as the compression flag and storage size of the data block can be effectively stored, preparing for the subsequent determination of whether the data block needs to be compressed at the protocol layer, saving computing time and improving computing efficiency.

[0047] Figure 5 FIG. 5 is a schematic diagram showing an example process 500 from receiving a request to read a data block to sending a data block according to some embodiments of the present disclosure. Figure 5 As shown, at block 502, process 500 receives a request from a client to read a data block. Figure 1 In the illustrated environment 100 , the server 104 may receive a request 120 to read a data block from the client 102 . The data block to be read may be one or more data blocks in the data blocks 114 .

[0048] At block 504, process 500 obtains compression information associated with the data block requested to be read at the data block level, wherein the compression information includes a compression flag indicating whether the data block has been compressed and a storage size of the data block. Figure 1 In the illustrated environment 100, the read request 120 may be a read request for the data block 114-1, and the process 500 may obtain the compression flag 116 and storage size 118 of the data block 114-1 at the data block layer 112. The compression flag 116 may be obtained from, for example, Figure 4 4. If the value of the compression flag 116 is "1", it means that the data block 114-1 has been compressed when it was written to the disk. In this case, the storage size 118 can be obtained from, for example, Figure 4The storage size 118 can be obtained from the compression length field of the data block mapping table 406 shown in the figure, and can indicate the size of the compressed data block 114-1 (for example, 2KB). If the value of the compression flag 116 is "0", it means that the data block 114-1 is written as an uncompressed original data block when it is written to the disk. In this case, the configuration of the server for the original data block size (for example, 8KB) can be obtained as the storage size 118.

[0049] At block 506, the process 500 determines at the data block layer whether the data block has been compressed. If the compression flag of the data block indicates that the data block is not compressed (e.g., the value of the compression flag is "0"), indicating that the compressibility of the data block is poor, the process 500 can pass an indication that the data block does not need to be compressed from the data block layer to the protocol layer, and then the process 500 proceeds to block 508. At block 508, the process 500 receives an indication at the protocol layer that the data block does not need to be compressed, so the original data block is packaged into a network packet. For example, at Figure 1 In the illustrated environment 100, the process 500 receives a compression indication 122 at the protocol layer 108 that the data block 114-1 does not need to be compressed, and then directly packages the uncompressed data block 114-1 into a network packet without compressing it at the protocol layer 108. At block 510, the process 500 sends the network packet to the client requesting to read the data block.

[0050] Returning to block 506, if the process 500 determines that the compression flag of the data indicates that the data block has been compressed (e.g., the value of the compression flag is "1"), indicating that the compressibility of the data block may be good, but further determination of its compressibility is still required, the process 500 proceeds to block 512. At block 512, the process 500 calculates the compression ratio of the data block at the data block layer based on the storage size of the data block and the size of the original data block configured by the storage server. For example, Figure 4 In the example shown, the original data block size of the data block corresponding to entry 421 in the data block mapping table 406 is 8KB, and the compressed length (i.e., storage size) is 2KB. The compression ratio can be determined to be 4 (i.e., 8KB / 2KB) by calculating the ratio of the original data block size to the compressed length.

[0051] At block 514, process 500 may compare the compression ratio of the data block with a predetermined threshold compression ratio, and if the compression ratio of the data block is less than the threshold ratio, process 500 proceeds to block 508. For example, if the predetermined threshold ratio is 3 and the compression ratio of the data block is 2, indicating that the compressibility of the data block still does not meet the requirements, process 500 may pass an indication that the data block does not need to be compressed from the data block layer to the protocol layer, and then process 500 proceeds to block 508. If the compression ratio of the data block is greater than or equal to the threshold ratio, the process proceeds to block 516. For example, if the predetermined threshold ratio is 3 and the compression ratio of the data block is 4, indicating that the compressibility of the data block is good, process 500 proceeds to block 516.

[0052] At block 516, process 500 compresses the original data block at the protocol layer and packages the compressed data block into a network packet. Figure 1 In the illustrated environment 100, the process 500 may compress the uncompressed data block 114-1 at the protocol layer 108 to generate the compressed data block 124, and then package the compressed data block 124 into a network packet. The process 500 then proceeds to block 510, where the network packet including the compressed data block is sent to the client.

[0053] In this way, the compressibility information of the data block can be passed from the data block layer to the protocol layer, so that the protocol layer can perform appropriate operation processes for different data blocks, thereby reducing the compression and decompression operations on data blocks with poor compressibility, saving processor and memory resources. In addition, since this method reduces the compression and decompression operations, it saves the time consumed by performing compression and decompression operations, and thus can reduce the overall response time.

[0054] In order to pass the indication that the data block does not need to be compressed from the data block layer to the protocol layer, in some embodiments, the indication can be passed from the data block layer to the file system layer, and then from the file system layer to the data block layer. In some embodiments, the compression indication can be set in the requested data block and passed from the data block layer to the protocol layer via the input / output (I / O) stack along with the data block. In some embodiments, since the compression indication has only two states (i.e., compression is required and compression is not required), only the indication that compression is not required can be passed. If the indication that the data block does not need to be compressed is not received at the protocol layer, the data block is compressed at the protocol layer. Figure 6 6 is a schematic diagram showing an example process 600 for passing a compression indication from a data block layer to a protocol layer according to some embodiments of the present disclosure. Figure 6As shown, process 600 requires passing data blocks 602, 604 and 606 from the data block layer along the I / O stack to the protocol layer, and then sending them to the client at the protocol layer, where data blocks 602 and 604 are compressed data blocks (i.e., the compression flag is, for example, "1"), and data block 606 is an uncompressed data block (i.e., the compression flag is, for example, "0").

[0055] like Figure 6 As shown, process 600 utilizes data block compressibility assessment module 608 at the data block level to assess whether data blocks 602, 604, and 606 need to be compressed. Figure 6 In the example shown, the data block compressibility evaluation module 608 determines that the compression ratio of the data block 602 is greater than or equal to the predetermined threshold ratio (for example, the original data block size of the data block 602 is 8KB, the compressed storage size is 2KB, the compression ratio is 4, and the predetermined threshold ratio is 3), indicating that its compressibility is good, so the data block 602 can be marked as requiring compression (or the data block is not marked, if the data block is not marked as not requiring compression, it means that compression is required), and generate the data block 612. Then, the process 600 can pass the data block 612 to the file system layer, and convert the data block 612 into a data block 622 that can be used by the protocol layer at the file system layer. Then, the process 600 can pass the data block 622 to the protocol layer. At the protocol layer, the process 600 determines that the data block 622 needs to be compressed, so the compression engine 630 can be used to compress the data block 622, and then the compressed data block is sent to the client.

[0056] In addition, the process 600 can also use the data block compressibility evaluation module 608 to determine that the compression ratio of the data block 604 is less than a predetermined threshold (for example, the original data block size of the data block 604 is 8KB, the compressed storage size is 4KB, the compression ratio is 2, and the predetermined threshold ratio is 3), indicating that its compressibility is poor, so the data block 604 can be marked as not requiring compression, and a data block 614 is generated. Then, the process 600 can pass the data block 614 to the file system layer, and convert the data block 614 into a data block 624 that can be used by the protocol layer at the file system layer. Then, the process 600 can pass the data block 624 to the protocol layer. At the protocol layer, the process 600 determines that the data block 624 does not need to be compressed, so the data block 624 can be directly sent to the client without using the compression engine 630 to compress it.

[0057] In addition, the process 600 can also use the data block compressibility assessment module 608 to determine that the data block 606 is an uncompressed data block, indicating that it is determined to be poorly compressible when it is written to the disk, so the data block 606 can be marked as not requiring compression, and a data block 616 can be generated. Then, the process 600 can pass the data block 616 to the file system layer, and at the file system layer, the data block 616 is converted into a data block 626 that can be used by the protocol layer. Then, the process 600 can pass the data block 626 to the protocol layer. At the protocol layer, the process 600 determines that the data block 626 does not need to be compressed, so the data block 626 can be directly sent to the client without using the compression engine 630 to compress it.

[0058] It should be noted that although Figure 6 In the example shown, only the data block layer, the file system layer, and the protocol layer are shown, but it is not intended to limit the specific implementation to only three layers from the data block layer to the protocol layer. On the contrary, in the specific implementation, the data block layer, the file system layer, and the protocol layer can often be divided into multiple layers. However, no matter how the specific implementation is, the requested data block is passed from the bottom layer of the data block layer to the protocol layer layer by layer, so the compression indication can also be passed from the bottom layer of the data block layer to the protocol layer layer by layer along with the data block, and in each layer, the data block and the compression indication are converted into a format that can be used by the next layer.

[0059] In this way, it is possible to evaluate at the data block layer whether the requested data needs to be compressed at the protocol layer based on compression information that cannot be known at the protocol layer, and the indication that it does not need to be compressed can be passed from the data block layer to the protocol layer layer by layer, so that the protocol layer can perform appropriate operations for data blocks with different compressibility, thereby reducing the compression and decompression operations on data blocks with poor compressibility, saving processor and memory resources. In addition, since this method reduces the compression and decompression operations, thereby saving the time consumed in performing the compression and decompression operations, the overall response time can be reduced.

[0060] In some embodiments, the protocol layer processes the data into a data stream and rearranges the data blocks into different packets. For example, in the SMB protocol, a client or server can use a COMPRESSION_PAYLOAD_HEADER header to send a compressed payload. Figure 7 FIG. 7 is a schematic diagram showing an example header structure 700 used at the protocol layer according to some embodiments of the present disclosure. Figure 7As shown, the header structure 700 includes a compression algorithm field 702, which is 16 bits long, and different values ​​indicate that different compression algorithms are used to perform protocol layer compression on the data block. For example, SMB compression supports XPRESS (LZ77), XPRESS Huffman (LZ77+Huffman), (LZNT1), and PATTERN_V1* compression algorithms, and the SMB protocol automatically uses the XPRESS compression algorithm for protocol layer compression. When the value of the compression algorithm field 702 is "0x0000" (i.e., indicating "None", or no compression algorithm is specified), the protocol layer will send the original data block without compressing it. Therefore, for data blocks that do not need to perform protocol layer compression, the compression algorithm field 702 in the header structure 700 can be set to "0x0000".

[0061] Use Figure 7 The header structure 700 shown, in some embodiments, can apply multiple payload headers when splitting or merging data blocks into SMB data packets. Figure 8 8 is a schematic diagram showing an example 800 of including multiple payload headers in a single network packet according to some embodiments of the present disclosure. Figure 8 As shown, example 800 includes headers 802 and 804. In header 802, the value of the CompressionAlgorithm field is "0x0000" (i.e., indicating "None", or no compression algorithm is specified), indicating that the data block does not need to be compressed at the protocol layer, but the original data block is directly sent to the client. In header 804, the value of the CompressionAlgorithm field is "0x0002" (i.e., indicating the LZ77 compression algorithm), indicating that the LZ77 compression algorithm needs to be used at the protocol layer to compress the data block before sending it to the client.

[0062] In this way, compression information for the requested data block can be collected at the data block layer and passed to the protocol layer. The protocol layer compression will use this knowledge to choose whether the requested data block needs to be compressed.

[0063] Fig. 9 Another schematic diagram of an example process 900 for passing a compression indication from a data block layer to a protocol layer according to some embodiments of the present disclosure is shown. Fig. 9 As shown, process 900 needs to send data blocks 901, 902, 903, 904 and 905 to the client. Process 900 can utilize data block compressibility evaluation module 910 (e.g., Figure 6 The data block compressibility evaluation module 608 in the example is used to evaluate the compressibility of these data blocks. Fig. 9 In the example shown, data blocks 902 and 904 are uncompressed original data blocks stored in the disk, indicating that their compressibility is poor, so process 900 can mark them as not requiring compression at the data block layer, and generate data blocks 912 and 914. Data blocks 901, 903, and 905 are compressed data blocks stored in the disk. After being evaluated by data block compressibility evaluation module 910, the compression ratio of data block 905 is less than a predetermined threshold, indicating that the compressibility of the data block also does not meet the requirements, so process 900 can decompress it at the data block layer and mark it as not requiring compression, thereby generating decompressed original data block 915. Fig. 9 In the example shown, for data blocks 901 and 903, after being evaluated by the data block compressibility evaluation module 910, the compression ratio of data block 905 is greater than or equal to the predetermined threshold, indicating that the compressibility of the data block is good. Therefore, process 900 can decompress it at the data block layer and mark it as requiring compression (or not mark it and compress it by default at the protocol layer), thereby generating decompressed original data blocks 911 and 913.

[0064] After obtaining data blocks 911, 912, 913, 914, and 915, process 900 performs compression at the protocol layer according to the compression flag and utilizes compression engine 920 (e.g., Figure 6 The compression engine 630 in the process 900 compresses the data blocks 911 and 913 that need protocol layer compression to generate compressed data blocks 921 and 923, and sends them to the client. In addition, since data blocks 912, 914, and 915 are marked as not needing protocol layer compression, the process 900 can bypass the compression engine 920 and directly send them to the client without performing protocol layer compression on them.

[0065] In this way, the compression and decompression operations on the data blocks with poor compressibility can be reduced, saving processor and memory resources. In addition, since this method reduces the compression and decompression operations, the time consumed by performing the compression and decompression operations is saved, so the overall response time can be reduced.

[0066] Fig.10A schematic block diagram of an example device 1000 that can be used to implement an embodiment of the present disclosure is shown. As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1002 or loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0067] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0068] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the method 200 in any other appropriate manner (eg, by means of firmware).

[0069] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0070] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0071] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, although each operation is depicted in a specific order, this should be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0072] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for transmitting a data block, comprising: receiving a request from a client to read the data block; At the data block layer, based on a flag indicating whether the data block has been compressed and a storage size of the data block, determining whether the data block needs to be compressed at the protocol layer; In response to determining that the data block does not need to be compressed at the protocol layer, communicating an indication that the data block does not need to be compressed at the protocol layer from the data block layer to the protocol layer; as well as The data block is sent to the client through the protocol layer without compressing the data block at the protocol layer.

2. The method according to claim 1, further comprising: In response to receiving the first data block, generating a second data block by compressing the first data block; Determining a size of the first data block and a size of the second data block; as well as In response to a ratio of the size of the first data block to the size of the second data block being greater than or equal to a predetermined threshold, the second data block is stored.

3. The method according to claim 2, further comprising: setting a flag corresponding to the first data block indicating whether the first data block has been compressed to a first value, the first value indicating that the first data block has been compressed; as well as The size of the second data block is recorded.

4. The method according to claim 2, further comprising: In response to the ratio of the size of the first data block to the size of the second data block being less than the predetermined threshold, the first data block is stored.

5. The method according to claim 4, further comprising: A flag corresponding to the first data block indicating whether the first data block has been compressed is set to a second value, the second value indicating that the first data block is not compressed.

6. The method of claim 1, wherein the flag indicating whether the data block is currently compressed is stored in an indirect block associated with the data block.

7. The method according to claim 1, wherein determining at the data block layer whether the data block needs to be compressed at the protocol layer comprises: At the data block layer, determining that the data block has been compressed based on the flag indicating whether the data block is currently compressed; determining a compression ratio of the data block based on the storage size of the data block and an original size of the data block before being compressed; as well as By determining that the compression ratio of the data block is less than a predetermined threshold, it is determined that the data block does not need to be compressed at the protocol layer.

8. The method according to claim 7, wherein sending the data block to the client through the protocol layer comprises: Decompressing the data block to obtain an original data block; as well as The original data block is sent.

9. The method according to claim 1, wherein determining at the data block layer whether the data block needs to be compressed at the protocol layer further comprises: determining that the data block is not compressed based on the flag indicating whether the data block is currently compressed; as well as By determining that the data block is not compressed, it is determined that the data block does not need to be compressed at the protocol layer.

10. The method of claim 1, wherein conveying an indication from the data block layer to the protocol layer that the data block does not need to be compressed at the protocol layer comprises: passing the indication from the data block layer to the file system layer; as well as The indication is passed from the file system layer to the data block layer.

11. An electronic device, comprising: at least one processor; as well as coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the electronic device to perform actions, the actions comprising: receiving a request from a client to read a block of data; At the data block layer, based on a flag indicating whether the data block has been compressed and a storage size of the data block, determining whether the data block needs to be compressed at the protocol layer; In response to determining that the data block does not need to be compressed at the protocol layer, communicating an indication that the data block does not need to be compressed at the protocol layer from the data block layer to the protocol layer; and The data block is sent to the client through the protocol layer without compressing the data block at the protocol layer.

12. The device according to claim 11, further comprising: In response to receiving the first data block, generating a second data block by compressing the first data block; Determining a size of the first data block and a size of the second data block; as well as In response to a ratio of the size of the first data block to the size of the second data block being greater than or equal to a predetermined threshold, the second data block is stored.

13. The apparatus according to claim 12, further comprising: setting a flag corresponding to the first data block indicating whether the first data block has been compressed to a first value, the first value indicating that the first data block has been compressed; as well as The size of the second data block is recorded.

14. The apparatus according to claim 12, further comprising: In response to the ratio of the size of the first data block to the size of the second data block being less than the predetermined threshold, the first data block is stored.

15. The apparatus according to claim 14, further comprising: A flag corresponding to the first data block indicating whether the first data block has been compressed is set to a second value, the second value indicating that the first data block is not compressed.

16. The apparatus of claim 11, wherein the flag indicating whether the data block is currently compressed is stored in an indirect block associated with the data block.

17. The apparatus of claim 11, wherein determining at the data block layer whether the data block needs to be compressed at the protocol layer comprises: At the data block layer, determining that the data block has been compressed based on the flag indicating whether the data block is currently compressed; determining a compression ratio of the data block based on the storage size of the data block and an original size of the data block before being compressed; as well as By determining that the compression ratio of the data block is less than a predetermined threshold, it is determined that the data block does not need to be compressed at the protocol layer.

18. The apparatus of claim 17, wherein sending the data block to the client through the protocol layer comprises: Decompressing the data block to obtain an original data block; as well as The original data block is sent.

19. The device according to claim 11, wherein determining at the data block layer whether the data block needs to be compressed at the protocol layer further comprises: determining that the data block is not compressed based on the flag indicating whether the data block is currently compressed; as well as By determining that the data block is not compressed, it is determined that the data block does not need to be compressed at the protocol layer.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions that when executed cause a machine to perform actions comprising: receiving a request from a client to read a block of data; At the data block layer, based on a flag indicating whether the data block has been compressed and a storage size of the data block, determining whether the data block needs to be compressed at the protocol layer; In response to determining that the data block does not need to be compressed at the protocol layer, communicating an indication that the data block does not need to be compressed at the protocol layer from the data block layer to the protocol layer; as well as The data block is sent to the client through the protocol layer without compressing the data block at the protocol layer.

Images