A data transmission method, system, apparatus, device, and medium

Abstract

This application discloses a data transmission method, including: a first device sending first information of a first data block to a second device; the second device searching for a second data block similar to the first data block from a first index based on the first information; the second device sending second information of the second data block to the first device; the first device obtaining a reference block identical to the second data block from its local storage based on the second information; the first device compressing the difference between the first data block and the reference block to obtain first compression information; the first device sending the first compression information to the second device; and the second device obtaining the first data block based on the first compression information and the second data block. This application also provides a system, apparatus, device, and medium. For a data block to be transmitted, if there is a similar data block that has already been transmitted, only the difference between the two data blocks is transmitted, improving the transmission efficiency of data synchronization technology.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a data transmission method, system, apparatus, device and medium. Background Technology

[0002] Data synchronization technology is used for remote transmission, sending data from the source (sender) to the target (receiver), thus enabling remote data replication or backup. It is a crucial component of remote disaster recovery solutions. Due to cost and other limitations, such as in intercity travel, the link bandwidth between the source and target is typically limited. Therefore, data synchronization technology must address how to effectively utilize limited link bandwidth to maximize data transmission efficiency.

[0003] In existing technologies, deduplication techniques are mainly used to improve the transmission efficiency of data synchronization. For example, if the data block to be sent is the same as the data block that has already been transmitted, the source end only transmits the description information of the data block to be sent, so that the target end can obtain the data block locally based on the description information, thereby reducing the data link.

[0004] However, in actual work, the duplication rate of data blocks in some database scenarios is not high, which limits the use of deduplication technology and makes it unable to effectively reduce the data in the link.

[0005] Therefore, the aforementioned problems in the existing technology still need to be improved. Summary of the Invention

[0006] This application provides a data transmission method, system, apparatus, device, and medium to solve the problem of link reduction in output transmission between devices.

[0007] In view of the above, a first aspect of this application provides a data transmission method, comprising: a first device sending first information of a first data block to a second device, the first data block being a data block to be sent by the first device to the second device, the first information describing the attributes of the first data block; the second device searching for a second data block similar to the first data block from a first index based on the first information, the first index recording information of data blocks already received by the second device from the first device; the second device sending second information of the second data block to the first device, the second information describing the attributes of the second data block; the first device obtaining a reference block from a local source based on the second information, the reference block being identical to the second data block; the first device compressing the difference portion of the first data block relative to the reference block to obtain first compression information; the first device sending the first compression information to the second device; and the second device obtaining the first data block based on the first compression information and the second data block.

[0008] In this embodiment, before sending the first data block to the second device, the first device sends first information about the first data block to the second device. The second device then searches in a first index for a data block similar to the first data block that it has already received from the first device. If a second data block similar to the first data block exists in the second device, the second device sends second information about the second data block to the first device. This allows the first device to retrieve a reference block locally, which is identical to the second data block. The first device then compresses the differences between the first data block and the reference block to obtain first compressed information, which is then sent to the second device. This allows the second device to reconstruct the first data block locally based on the first compressed information. This achieves similarity deduplication technology. For a data block to be sent, if a similar data block has already been transmitted, only the differences between the two data blocks are transmitted, effectively reducing the data link and improving the transmission efficiency of data synchronization technology.

[0009] In conjunction with the first aspect described above, in a first possible implementation, the first information includes a first similar fingerprint, which is a weak hash value of the first data block. Optionally, a similar fingerprint (SFP) is used to characterize the similarity between data blocks; when two data blocks have the same similar fingerprint, the two data blocks are similar. The first index records a first correspondence between the similar fingerprints of data blocks and a second address of the data block in the second device. The second device obtains a second data block similar to the first data block based on the first information, including: the second device searching for a second similar fingerprint in the first index that is the same as the first similar fingerprint, which is the similar fingerprint of the second data block; the second device obtaining a second address from the first correspondence based on the second similar fingerprint, which is the second address of the second data block in the second device; and the second device obtaining the second data block based on the second address.

[0010] In this embodiment, optionally, the first information further includes a first address and a first strong fingerprint. The first address is used to record the local storage address of the first data block on the first device, and the first strong fingerprint is used to describe the characteristics of the first data block. The similar fingerprint (SFP) is used to characterize the similarity between data blocks. When two data blocks have the same similar fingerprint, the two data blocks are similar. Therefore, through the similar fingerprint, the second device can find a second data block similar to the first data block in the first index.

[0011] In conjunction with the first aspect described above, in a second possible implementation, the second information further includes the storage address of the reference block in the first device; the first device retrieves the reference block locally based on the second information, including: the first device retrieves the reference block locally based on the storage address of the reference block in the first device.

[0012] In this embodiment, the first device obtains the local reference block based on the address information sent by the second device, thereby realizing the subsequent differential compression step. Optionally, as a substitute for the address information, the second information may also include tag information, so that the first device can directly obtain the reference block from the local device based on the tag information, thereby further reducing the data transmission steps.

[0013] In conjunction with the first possible implementation of the first aspect described above, in the third possible implementation, the first correspondence includes the correspondence between strong fingerprints and similar fingerprints and the second address. Optionally, the first index containing the first correspondence is stored in an external hard drive. The strong fingerprint is a strong hash value of a data block, which is used to describe the data attributes of the corresponding data block. When the second device finds M target similar fingerprints that are identical to the first similar fingerprint in the first index, M is a positive integer greater than 1. The method further includes: the second device obtaining M target addresses corresponding to the M target similar fingerprints from the first correspondence based on the M target similar fingerprints; the second device obtaining M target data blocks based on the M target addresses; the second device obtaining the strong fingerprints of the M target data blocks; and the second device obtaining one target data block from the M target data blocks as the second data block, wherein the strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

[0014] In this embodiment, in order to prevent the acquired second data block from being rewritten locally, the second device acquires M target data blocks at once, and then selects the unrewritten data block from the M target data blocks as the second data block by comparing strong fingerprints. This improves the efficiency of the second device in acquiring the second data block and avoids repeated reading and writing of data caused by the rewriting of data blocks.

[0015] In conjunction with the third possible implementation of the first aspect described above, in the fourth possible implementation, when the second device obtains N data blocks similar to the first data block from the M target data blocks, the second information includes strong fingerprints and tagging information of the N candidate data blocks, and the first addresses of the N candidate data blocks in the first device. The N candidate data blocks are each similar to the first data block, and the strong fingerprints of the N candidate data blocks are the strong hash values ​​of the N candidate data blocks. N is a positive integer greater than 1. The first device obtains the base block from its local storage based on the second information, including: the first device obtaining the N candidate data blocks from its local storage based on the first addresses of the N candidate data blocks in the first device. The first device acquires N candidate data blocks; the first device acquires strong fingerprints of the N candidate data blocks; the first device acquires a target candidate data block from the N candidate data blocks as the base block, the strong fingerprint of the target candidate data block being the same as the strong fingerprint recorded in the second information; the first device sends the first compression information to the second device, including: the first device sending the tag information of the target candidate data block to the second device; the second device obtains the first data block based on the first compression information and the candidate data blocks, including: the second device acquiring the target candidate block from the local storage based on the tag information of the target candidate data block; the second device obtaining the first data block based on the first compression information and the target candidate data block.

[0016] In this embodiment, in order to prevent the candidate block that is the same as the second data block from being overwritten in the first device, the second device obtains N data blocks similar to the first data block from M target data blocks at one time as N candidate data blocks, so that the first device can select a data block that has not been overwritten locally as the base block, thereby preventing the failure of similar deduplication due to the overwriting of data blocks.

[0017] In conjunction with the fourth possible implementation of the first aspect described above, in the fifth possible implementation, the second information further includes sorting information, which is used to sort the similarity between the N candidate data blocks and the first data block. Then, the first device obtains the strong fingerprints of the N candidate data blocks by: the first device obtaining the strong fingerprints of the N candidate data blocks one by one according to the sorting information; and the first device selecting the first target candidate data block whose strong fingerprint is the same as the strong fingerprint recorded in the second information as the candidate block.

[0018] In this embodiment, after the second device acquires N candidate data blocks, it further sorts these N candidate data blocks. The sorting principle can be based on the similarity between the candidate data blocks and the first data block, with the candidate blocks having higher similarity ranking higher. This allows the first device to determine whether each of the N candidate data blocks has been rewritten based on the sorting information. The first target candidate data block whose strong fingerprint matches the strong fingerprint recorded in the second information is selected as the candidate block. In this way, the first device can determine a data block that has not been rewritten and is the most suitable as the base block.

[0019] In conjunction with the fourth possible implementation of the first aspect described above, in the fifth possible implementation, the second device is used to synchronize data in the first device, the synchronized data block has the same storage address in both the first and second devices, and the base block has the same storage address in the first device as the second address of the second data block in the second device; or, the second device is used to back up data in the first device, and the backup data block has a different storage address in both the first and second devices, then before the first device obtains the second information from the second device, it further includes: when the first device sends the second data block to the second device, it sends a third address, which is the storage address of the base block that is the same as the second data block in the first device; then the storage address of the second data block included in the second information in the first device is the third address.

[0020] In conjunction with the first aspect and the first to fifth possible implementations of the first aspect, in the sixth possible implementation, the first compression information includes a first compression block and difference information, wherein the first compression block includes the difference portion between the first data block and the reference block, and the difference information is used to describe the difference between the first data block and the reference block; the second device obtains the first data block based on the first compression information and the second data block, including: the second device obtains the first data block based on the first compression block, the difference information and the second data block.

[0021] In this embodiment, the first compressed block records the information on the difference between the first data block and the reference block. Since the reference block and the second data block are completely identical data blocks, the data of the second data block plus the data of the first compressed block can be used to obtain all the data of the first data block. Furthermore, the difference information, as descriptive information, records the differences between the first data block and the second data block. Thus, the second device can obtain the first data block locally based on the first compressed block, the difference information, and the second data block.

[0022] A second aspect of this application provides a data transmission method, comprising: a first device acquiring first information of a first data block, the first data block being a data block to be sent by the first device to a second device, the second device being used to receive the data block sent by the first device, the first information being used to describe the attributes of the first data block; the first device searching for a second data block similar to the first data block in a first index based on the first information, the first index being used to record information of data blocks already sent by the first device to the second device; the first device compressing the difference portion of the first data block relative to the second data block to obtain first compression information; and the first device sending the first compression information to the second device, so that the second device obtains the first data block based on the first compression information.

[0023] In this embodiment, for a first data block to be sent by the first device, the first device searches its local first index for a second data block that is similar to the first data block among the data blocks already sent to the second device. If a second data block is found, the first device compresses the difference between the first data block and the second data block to obtain first compression information. This first compression information is then sent to the second device, allowing the second device to reconstruct the first data block locally based on the first compression information. Since the first compression information only includes the difference between the first and second data blocks, the amount of data transmitted is greatly reduced, thus lowering the data transmission volume.

[0024] In conjunction with the second aspect described above, in a first possible implementation, the first information includes a first similar fingerprint, which is a weak hash value of the first data block. The first index records a first correspondence between similar fingerprints of data blocks and storage addresses of data blocks in the first device. The first device then searches for a second data block similar to the first data block in the first index based on the first information, including: the first device searching for a second similar fingerprint identical to the first similar fingerprint in the first index, where the second similar fingerprint is a weak hash value of the second data block; the first device obtaining a second address from the first correspondence based on the second similar fingerprint, where the second address is the storage address of the second data block in the first device; and the first device obtaining the second data block based on the second address.

[0025] In this embodiment, optionally, the first information further includes a first address and a first strong fingerprint. The first address is used to record the local storage address of the first data block on the first device, and the first strong fingerprint is used to describe the characteristics of the first data block. The similar fingerprint (SFP) is used to characterize the similarity between data blocks. When two data blocks have the same similar fingerprint, the two data blocks are similar. Therefore, through the similar fingerprint, the first device can find a second data block similar to the first data block in the first index.

[0026] In conjunction with the first possible implementation of the second aspect described above, in the second possible implementation, the first correspondence is a correspondence between strong fingerprints and similar fingerprints and the storage address. The strong fingerprint is a strong hash value of the data block, which is used to describe the data attributes of the corresponding data block. When the first device finds M target similar fingerprints that are identical to the first similar fingerprint in the first index, M is a positive integer greater than 1. The method further includes: the first device obtaining M target addresses corresponding to the M target similar fingerprints from the first correspondence based on the M target similar fingerprints; the first device obtaining M target data blocks based on the M target addresses; the first device obtaining the strong fingerprints of the M target data blocks; and the first device obtaining one target data block from the M target data blocks as the second data block, wherein the strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

[0027] In this embodiment, in order to prevent the acquired second data block from being rewritten locally, the first device acquires M target data blocks at once, and then selects the unrewritten data block from the M target data blocks as the second data block by comparing strong fingerprints. This improves the efficiency of the first device in acquiring the second data block and avoids repeated reading and writing of data caused by the rewriting of data blocks.

[0028] In conjunction with the second possible implementation of the second aspect described above, in the third possible implementation, when the data attributes of the (M-1)th target data block are inconsistent with the data attributes described by the corresponding strong fingerprint, the method further includes: the first device updating the entry order of the (M-1)th target data block in the first index, the similar fingerprint of the (M-1)th target data block, and the strong fingerprint in the first index.

[0029] In this embodiment, for the M target data blocks acquired by the first device, when the first device verifies that a data block has been overwritten using a strong fingerprint method, it updates the information of the overwritten data block into the first index, thereby maintaining the first index during operation. This facilitates improving the accuracy of subsequent similar block searches.

[0030] In conjunction with the third possible implementation of the second aspect described above, in the fourth possible implementation, the second device is used to synchronize the data in the first device. The synchronized data block has the same storage address in the first device and the second device. After the first device performs differential compression on the first data block to obtain the first compressed block and differential information, the method further includes: the first device sending the second address of the second data block to the second device so that the second device can obtain the base block locally based on the second address.

[0031] In this embodiment, in a synchronization scenario, if the data block being synchronized has the same storage address in the first device and the second device, then the first device sends the second address of the second data block to the second device, and the second device can obtain the same reference block as the second data block locally based on the second address.

[0032] In conjunction with the third possible implementation of the second aspect described above, in the fifth possible implementation, the second device is used to back up the data in the first device. The data block being backed up has a different storage address in the first device and the second device. Before the first device obtains the first information of the first data block, the method further includes: the first device obtaining a second feedback message from the second device, the second feedback message being used to record the third address where the second data block is stored in the second device; after the first device performs differential compression on the first data block according to the second data block to obtain a first compressed block and differential information, the method further includes: the first device sending the third address to the second device so that the second device can obtain the second data block locally according to the third address.

[0033] In this embodiment, in the backup scenario, the data block being backed up has the same storage address in both the first device and the second device. When the first device sends the second data block to the second device, the first device obtains a second feedback message from the second device. This second feedback message is used to record the third address where the second data block is stored in the second device, thereby enabling the first device to know the storage location of the reference block identical to the second data block in the second device. When the first device performs differential compression on the first data block based on the second data block, the first device sends the third address along with the first compression information to the second device, so that the second device can obtain the reference block identical to the second data block from its local storage via the third address.

[0034] In conjunction with the fifth possible implementation of the second aspect described above, in the sixth possible implementation, after the first device sends the third address to the second device, the method further includes: the first device obtaining a first feedback message from the second device, the first feedback message being used to record the fourth address of the first data block stored in the second device; the first device updating the correspondence between the fourth address and the first strong fingerprint and the first similar fingerprint to the first correspondence, wherein the first strong fingerprint is the strong fingerprint of the first data block.

[0035] In this embodiment, since the storage addresses of the same data block are not the same in the first device and the second device in the backup scenario, after the second device restores the first data block locally, it needs to inform the first device of the current storage address of the first data block in the second device through a feedback message, so that the first device can record the information of the first data block in the first index, thereby realizing the update and maintenance of the first index and facilitating the accuracy of the query of the first index in subsequent work.

[0036] In conjunction with the second aspect and the first to sixth possible implementations of the second aspect, in the seventh possible implementation, the first compression information includes a first compression block and difference information, wherein the first compression block includes the difference portion between the first data block and the second data block, and the difference information is used to describe the difference between the first data block and the second data block; so that the second device obtains the first data block based on the first compression block, the difference information and the reference block, wherein the reference block is the same data block in the second device as the second data block.

[0037] In this embodiment, the first compression block records the difference information between the first data block and the second data block. Therefore, by adding the data of the second data block to the data of the first compression block, all the data of the first data block can be obtained. Furthermore, the difference information, as descriptive information, records the differences between the first data block and the second data block. Thus, the second device can obtain the first data block locally based on the first compression block, the difference information, and the same reference block as the second data block.

[0038] A third aspect of this application provides a data transmission method, comprising: a second device obtaining first compression information from a first device, the first compression information being information obtained by the first device compressing the difference between a first data block and a second data block, the first data block being a data block to be sent by the first device to the second device, and the second data block being a data block in the first device that has already been sent to the second device; the second device obtaining a reference block from a local device, the reference block being identical to the second data block in the first device; and the second device obtaining the first data block based on the first compression information and the reference block.

[0039] In this embodiment, the first device performs differential compression on the first data block to be sent based on the second data block already sent to the second device. This generates first compression information that distinguishes the first data block from the second data block, which is then sent to the second device. Upon receiving this first compression information, the second device decompresses it using a local reference block identical to the second data block, thus obtaining the first data block locally. This method significantly reduces the amount of data transmitted compared to directly transmitting the first data block, as the first compression information only contains the differences between the first and second data blocks, thereby improving data transmission efficiency.

[0040] In conjunction with the third aspect above, in a first possible implementation, the first compression information includes a first compression block and difference information, wherein the first compression block includes a difference portion between the first data block and the second data block, and the difference information is used to describe the difference between the first data block and the second data block; the second device obtains the first data block based on the first compression information and the reference block, including: the second device obtains the first data block based on the first compression block, the difference information, and the reference block.

[0041] In this embodiment, the first compression block records the difference information between the first data block and the second data block. Therefore, by adding the data of the second data block to the data of the first compression block, all the data of the first data block can be obtained. Furthermore, the difference information, as descriptive information, records the differences between the first data block and the second data block. Thus, the second device can obtain the first data block locally based on the first compression block, the difference information, and the same reference block as the second data block.

[0042] In conjunction with the third aspect or the first possible implementation of the third aspect, in the second possible implementation, the first compression information includes the storage address of the reference block in the second device, and the second device obtains the reference block from the local device according to the first compression information, including: the second device obtains the reference block from the local device according to the storage address of the reference block in the second device.

[0043] In this embodiment, the first compression information includes the storage address of the reference block in the second device, so that the second device can obtain the same reference block as the second data block from the local device according to the address.

[0044] This application provides a data transmission system in a fourth aspect. The system includes a first device and a second device, wherein the second device is used to receive data sent by the first device. The first device includes: a first sending unit, used to send first information of a first data block to the second device, the first data block being a data block to be sent by the first device to the second device, the first information describing the attributes of the first data block; the second device includes: a lookup unit, used to search for a second data block similar to the first data block from a first index based on the first information sent by the sending unit, the first index recording information about data blocks already received by the second device from the first device; and a second sending unit... The first device further includes: a second sending unit, configured to send second information of the second data block found by the search unit to the first device, the second information describing the attributes of the second data block; the first device further includes: an acquisition unit, configured to acquire a reference block from a local source based on the second information, the reference block being identical to the second data block; a compression unit, configured to compress the difference between the first data block and the reference block acquired by the acquisition unit to obtain first compression information; the first sending unit is further configured to send the first compression information to the second device; the second device further includes: a decompression unit, configured to obtain the first data block based on the first compression information and the second data block.

[0045] In this embodiment, before sending the first data block to the second device, the first device sends first information about the first data block to the second device. The second device then searches in a first index for a data block similar to the first data block that it has already received from the first device. If a second data block similar to the first data block exists in the second device, the second device sends second information about the second data block to the first device. This allows the first device to retrieve a reference block locally, which is identical to the second data block. The first device then compresses the differences between the first data block and the reference block to obtain first compressed information, which is then sent to the second device. This allows the second device to reconstruct the first data block locally based on the first compressed information. This achieves similarity deduplication technology. For a data block to be sent, if a similar data block has already been transmitted, only the differences between the two data blocks are transmitted, effectively reducing the data link and improving the transmission efficiency of data synchronization technology.

[0046] In conjunction with the fourth aspect above, in a first possible implementation, the first information includes a first similar fingerprint, which is a weak hash value of the first data block; the first index records a first correspondence between similar fingerprints of data blocks and a second address of the data block in the second device; the search unit is further configured to: search in the first index for a second similar fingerprint that is the same as the first similar fingerprint, which is a similar fingerprint of the second data block; obtain a second address from the first correspondence based on the second similar fingerprint, which is a second address of the second data block in the second device; and obtain the second data block based on the second address.

[0047] In this embodiment, optionally, the first information further includes a first address and a first strong fingerprint. The first address is used to record the local storage address of the first data block on the first device, and the first strong fingerprint is used to describe the characteristics of the first data block. The similar fingerprint (SFP) is used to characterize the similarity between data blocks. When two data blocks have the same similar fingerprint, the two data blocks are similar. Therefore, through the similar fingerprint, the second device can find a second data block similar to the first data block in the first index.

[0048] In conjunction with the fourth aspect above, in a second possible implementation, the second information further includes the storage address of the reference block in the first device; the acquisition unit is further configured to: acquire the reference block from the local device according to the storage address of the reference block in the first device.

[0049] In this embodiment, the first device obtains the local reference block based on the address information sent by the second device, thereby realizing the subsequent differential compression step. Optionally, as a substitute for the address information, the second information may also include tag information, so that the first device can directly obtain the reference block from the local device based on the tag information, thereby further reducing the data transmission steps.

[0050] In conjunction with the first possible implementation of the fourth aspect described above, in the second possible implementation, the first correspondence includes the correspondence between strong fingerprints and similar fingerprints and the second address. The strong fingerprint is a strong hash value of the data block, which is used to describe the data attributes of the corresponding data block. When the second device finds M target similar fingerprints that are identical to the first similar fingerprint in the first index, M is a positive integer greater than 1. The second device further includes a filtering unit, which is used to: obtain M target addresses corresponding to the M target similar fingerprints from the first correspondence based on the M target similar fingerprints; obtain M target data blocks based on the M target addresses; obtain the strong fingerprints of the M target data blocks; and obtain one target data block from the M target data blocks as the second data block, wherein the strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

[0051] In this embodiment, in order to prevent the acquired second data block from being rewritten locally, the second device acquires M target data blocks at once, and then selects the unrewritten data block from the M target data blocks as the second data block by comparing strong fingerprints. This improves the efficiency of the second device in acquiring the second data block and avoids repeated reading and writing of data caused by the rewriting of data blocks.

[0052] In conjunction with the second possible implementation of the fourth aspect described above, in the third possible implementation, when the second device obtains N data blocks similar to the first data block from the M target data blocks, the second information includes strong fingerprints of the N candidate data blocks, tag information, and the first addresses of the N candidate data blocks in the first device. The N candidate data blocks are each similar to the first data block, and the strong fingerprints of the N candidate data blocks are the strong hash values ​​of the N candidate data blocks. N is a positive integer greater than 1. The obtaining unit is further configured to: [The text abruptly ends here, so the translation stops as well.] The first device retrieves the N candidate data blocks from local storage using a first address; retrieves the strong fingerprints of the N candidate data blocks; retrieves a target candidate data block from the N candidate data blocks as the base block, the strong fingerprint of the target candidate data block being the same as the strong fingerprint recorded in the second information; the first sending unit is further configured to: send the tagging information of the target candidate data block to the second device; the decompression unit is further configured to: retrieve the target candidate block from local storage based on the tagging information of the target candidate data block; and obtain the first data block based on the first compression information and the target candidate data block.

[0053] In this embodiment, in order to prevent the candidate block that is the same as the second data block from being overwritten in the first device, the second device obtains N data blocks similar to the first data block from M target data blocks at one time as N candidate data blocks, so that the first device can select a data block that has not been overwritten locally as the base block, thereby preventing the failure of similar deduplication due to the overwriting of data blocks.

[0054] In conjunction with the fourth aspect and the first to third possible implementations of the fourth aspect, in the fourth possible implementation, the first compression information includes a first compression block and difference information, wherein the first compression block includes the difference portion between the first data block and the reference block, and the difference information is used to describe the difference between the first data block and the reference block; the decompression unit is further configured to: obtain the first data block based on the first compression block, the difference information and the second data block.

[0055] In this embodiment, the first compressed block records the information on the difference between the first data block and the reference block. Since the reference block and the second data block are completely identical data blocks, the data of the second data block plus the data of the first compressed block can be used to obtain all the data of the first data block. Furthermore, the difference information, as descriptive information, records the differences between the first data block and the second data block. Thus, the second device can obtain the first data block locally based on the first compressed block, the difference information, and the second data block.

[0056] A fifth aspect of this application provides a data transmission device, comprising: an acquisition unit for acquiring first information of a first data block, the first data block being a data block to be sent by the data transmission device to a second device, the second device being used to receive the data block sent by the data transmission device, the first information being used to describe the attributes of the first data block; a search unit for searching for a second data block similar to the first data block in a first index based on the first information acquired by the acquisition unit, the first index being used to record information of data blocks already sent by the data transmission device to the second device; a compression unit for compressing the difference between the first data block and the second data block searched by the search unit to obtain first compression information; and a transmission unit for transmitting the first compression information obtained by the compression unit to the second device, so that the second device obtains the first data block based on the first compression information.

[0057] In this embodiment, for a first data block to be sent by the data sending device, the data sending device searches in a local first index for a second data block that is similar to the first data block among the data blocks already sent to the second device. If a second data block is found, the data sending device compresses the difference between the first data block and the second data block to obtain first compression information. This first compression information is then sent to the second device, allowing the second device to reconstruct the first data block locally based on the first compression information. Since the first compression information only includes the difference between the first and second data blocks, the amount of data transmitted is greatly reduced, thus lowering the data transmission volume.

[0058] In conjunction with the fifth aspect above, in a first possible implementation, the first information includes a first similar fingerprint, which is a weak hash value of the first data block, and the first index records a first correspondence between similar fingerprints of data blocks and storage addresses of data blocks in the data transmitting device; then the search unit is further configured to: search in the first index for a second similar fingerprint that is the same as the first similar fingerprint, where the second similar fingerprint is a weak hash value of the second data block;

[0059] The second address is obtained from the first correspondence based on the second similar fingerprint. The second address is the storage address of the second data block in the data transmitting device. The second data block is obtained based on the second address.

[0060] In this embodiment, optionally, the first information further includes a first address and a first strong fingerprint. The first address is used to record the local storage address of the first data block on the data transmitting device, and the first strong fingerprint is used to describe the characteristics of the first data block. The similar fingerprint (SFP) is used to characterize the similarity between data blocks. When two data blocks have the same similar fingerprint, the two data blocks are similar. Therefore, through the similar fingerprint, the data transmitting device can find a second data block similar to the first data block in the first index.

[0061] In conjunction with the first possible implementation of the fifth aspect described above, in the second possible implementation, the first correspondence is a correspondence between strong fingerprints and similar fingerprints and the storage address. The strong fingerprint is a strong hash value of the data block, which is used to describe the data attributes of the corresponding data block. When the data sending device finds M target similar fingerprints that are the same as the first similar fingerprint in the first index, M is a positive integer greater than 1. The device further includes a filtering unit, which is used to: obtain M target addresses corresponding to the M target similar fingerprints from the first correspondence based on the M target similar fingerprints; obtain M target data blocks based on the M target addresses; obtain the strong fingerprints of the M target data blocks; and obtain one target data block from the M target data blocks as the second data block, wherein the strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

[0062] In this embodiment, in order to prevent the acquired second data block from being rewritten locally, the data sending device acquires M target data blocks at once, and then selects the unrewritten data block from the M target data blocks as the second data block by comparing strong fingerprints. This improves the efficiency of the data sending device in acquiring the second data block and avoids repeated reading and writing of data caused by the rewriting of data blocks.

[0063] In conjunction with the fifth aspect and the first to second possible implementations of the fifth aspect, in the third possible implementation, the first compression information includes a first compression block and difference information, wherein the first compression block includes the difference portion between the first data block and the second data block, and the difference information is used to describe the difference between the first data block and the second data block; so that the second device obtains the first data block based on the first compression block, the difference information and the reference block, wherein the reference block is the same data block in the second device as the second data block.

[0064] In this embodiment, the first compression block records the difference information between the first data block and the second data block. Therefore, by adding the data of the second data block to the data of the first compression block, all the data of the first data block can be obtained. Furthermore, the difference information, as descriptive information, records the differences between the first data block and the second data block. Thus, the second device can obtain the first data block locally based on the first compression block, the difference information, and the same reference block as the second data block.

[0065] A sixth aspect of this application provides a data receiving device, comprising: a receiving unit configured to acquire first compression information from a first device, the first compression information being information obtained by the first device compressing the difference between a first data block and a second data block, the first data block being a data block to be sent by the first device to the data receiving device, and the second data block being a data block in the first device that has already been sent to the data receiving device; an acquisition unit configured to acquire a reference block from a local device based on the first compression information, the reference block being identical to the second data block in the first device; and a decompression unit configured to obtain the first data block based on the first compression information received by the receiving unit and the reference block acquired by the acquisition unit.

[0066] In this embodiment, the first device performs differential compression on the first data block to be transmitted based on the second data block that has already been sent to the data receiving device. This generates first compression information that distinguishes the first data block from the second data block, which is then sent to the data receiving device. After receiving the first compression information, the data receiving device decompresses the first compression information based on a local reference block identical to the second data block, thus obtaining the first data block locally. This method significantly reduces the amount of data transmitted compared to directly transmitting the first data block, as the first compression information only contains the differences between the first and second data blocks, thereby improving data transmission efficiency.

[0067] In conjunction with the sixth aspect above, in a first possible implementation, the first compression information includes a first compression block and difference information, wherein the first compression block includes the difference portion between the first data block and the second data block, and the difference information is used to describe the difference between the first data block and the second data block; the decompression unit is further configured to: obtain the first data block based on the first compression block, the difference information and the reference block.

[0068] In this embodiment, the first compression block records the difference information between the first data block and the second data block. Therefore, by adding the data of the second data block to the data of the first compression block, all the data of the first data block can be obtained. Furthermore, the difference information, as descriptive information, records the differences between the first data block and the second data block. Thus, the data receiving device can obtain the first data block locally based on the first compression block, the difference information, and the same reference block as the second data block.

[0069] In conjunction with the sixth aspect or the first possible implementation of the sixth aspect, in the second possible implementation, the first compressed information includes the storage address of the reference block in the data receiving device, and the acquisition unit is further configured to: acquire the reference block from the local storage according to the storage address of the reference block in the data receiving device.

[0070] In this embodiment, the first compression information includes the storage address of the reference block in the data receiving device, so that the data receiving device can obtain the same reference block as the second data block from the local machine based on the address.

[0071] A seventh aspect of this application provides an electronic device comprising: an interactive device, an input / output (I / O) interface, a processor, and a memory storing program instructions; the interactive device is configured to acquire operation instructions input by a user; and the processor is configured to execute the program instructions stored in the memory to perform the method shown in the first aspect or any possible implementation thereof.

[0072] The eighth aspect of this application provides an electronic device, which includes: an interactive device, an input / output (I / O) interface, a processor, and a memory storing program instructions; the interactive device is used to acquire operation instructions input by a user; the processor is used to execute the program instructions stored in the memory to perform the method shown in the second aspect or any possible implementation of the second aspect.

[0073] A ninth aspect of this application provides an electronic device comprising: an interactive device, an input / output (I / O) interface, a processor, and a memory storing program instructions; the interactive device is configured to acquire operation instructions input by a user; and the processor is configured to execute the program instructions stored in the memory to perform the method shown in the third aspect or any possible implementation thereof.

[0074] The tenth aspect of this application provides a computer-readable storage medium including instructions that, when executed on a computer device, cause the computer device to perform the method shown in the first aspect or any possible implementation thereof.

[0075] The eleventh aspect of this application provides a computer-readable storage medium including instructions that, when executed on a computer device, cause the computer device to perform the method shown in the second aspect or any possible implementation thereof.

[0076] The twelfth aspect of this application provides a computer-readable storage medium including instructions that, when executed on a computer device, cause the computer device to perform the method shown in the third aspect or any possible implementation thereof.

[0077] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0078] The data transmission method provided in this embodiment includes: a first device sending first information of a first data block to a second device, wherein the first data block is a data block to be sent by the first device to the second device, and the first information describes the attributes of the first data block; the second device searching for a second data block similar to the first data block from a first index based on the first information, wherein the first index records information about data blocks already received by the second device from the first device; the second device sending second information of the second data block to the first device, wherein the second information describes the attributes of the second data block; the first device obtaining a reference block locally based on the second information, wherein the reference block is identical to the second data block; the first device compressing the difference between the first data block and the reference block to obtain first compression information; the first device sending the first compression information to the second device; and the second device obtaining the first data block based on the first compression information and the second data block. This achieves similarity deduplication technology. For a data block to be sent, if there is a similar data block that has already been transmitted, only the difference between the two data blocks is transmitted, effectively reducing the data link and improving the transmission efficiency of data synchronization technology.

[0079] The data transmission method provided in this embodiment includes: a first device acquiring first information of a first data block, wherein the first data block is a data block to be sent by the first device to a second device, and the second device receiving the data block sent by the first device, and the first information describing the attributes of the first data block; the first device searching for a second data block similar to the first data block in a first index based on the first information, the first index recording information of data blocks already sent by the first device to the second device; the first device compressing the difference between the first data block and the second data block to obtain first compression information; and the first device sending the first compression information to the second device so that the second device can obtain the first data block based on the first compression information. For the first data block to be sent by the first device, the first device searches in its local first index for a second data block similar to the first data block among the data blocks already sent to the second device. If a second data block exists, the first device compresses the difference between the first data block and the second data block to obtain first compression information, and then sends the first compression information to the second device so that the second device can reconstruct the first data block locally based on the first compression information. Since the first compression information only includes the difference between the first data block and the second data block, the amount of data transmitted is greatly reduced, thus lowering the data transmission volume.

[0080] The data transmission method provided in this embodiment includes: a second device obtaining first compression information from a first device, wherein the first compression information is information obtained by the first device compressing the difference between a first data block and a second data block, the first data block is a data block to be sent by the first device to the second device, and the second data block is a data block already sent to the second device from the first device; the second device obtaining a reference block from its local storage, the reference block being identical to the second data block in the first device; and the second device obtaining the first data block based on the first compression information and the reference block. Through this working method, since the first compression information only contains the difference information between the first data block and the second data block, compared to transmitting the first data block directly, the amount of data transmitted is greatly reduced, thus improving the transmission efficiency of data transmission. Attached Figure Description

[0081] Figure 1 This is a system architecture diagram illustrating an application scenario of the data transmission method provided in this application embodiment;

[0082] Figure 2 This is a schematic diagram illustrating one implementation of the data transmission method provided in this application.

[0083] Figure 3 This is a schematic diagram illustrating another implementation of the data transmission method provided in the embodiments of this application;

[0084] Figure 4 This is a schematic diagram illustrating another implementation of the data transmission method provided in the embodiments of this application;

[0085] Figure 5a A system architecture diagram illustrating another application scenario of the data transmission method provided in this application embodiment;

[0086] Figure 5b A system architecture diagram illustrating another application scenario of the data transmission method provided in this application embodiment;

[0087] Figure 6 This is a schematic diagram illustrating another implementation of the data transmission method provided in the embodiments of this application;

[0088] Figure 7 This is a schematic diagram illustrating another implementation of the data transmission method provided in the embodiments of this application;

[0089] Figure 8 A system architecture diagram illustrating another application scenario of the data transmission method provided in this application embodiment;

[0090] Figure 9 A system device diagram illustrating another application scenario of the data transmission method provided in the embodiments of this application;

[0091] Figure 10 A schematic diagram of data flow in a system device, illustrating one embodiment of the data transmission method provided in this application.

[0092] Figure 11 A schematic diagram of data flow in a system device, illustrating another implementation of the data transmission method provided in the embodiments of this application;

[0093] Figure 12 A schematic diagram of the electronic device provided in the embodiments of this application;

[0094] Figure 13 This is a schematic diagram of the data transmission system provided in an embodiment of this application;

[0095] Figure 14 This is a schematic diagram of the data transmission device provided in the embodiments of this application;

[0096] Figure 15 This is a schematic diagram of a data receiving device provided in an embodiment of this application. Detailed Implementation

[0097] This invention provides a data transmission method, system, apparatus, device, and medium that can solve the link compression problem during data transmission by using similar deduplication technology.

[0098] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0099] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0100] Data synchronization technology is used for remote transmission, sending data from the source (sender) to the target (receiver), thereby enabling remote replication or backup of data. It is a crucial component of remote disaster recovery solutions. For example, the system architecture of the application scenario in this application can be as follows: Figure 1 As shown, Figure 1 In this system, production center 101 is the source end, used to generate data, and disaster recovery center 102 is the target end. To back up the data generated by production center 101, production host 1011 and production storage device 1012 of production center 101 are connected via small computer system interface (SCSI) or internet small computer system interface (iSCSI). Disaster recovery host 1021 of disaster recovery center 102 is connected to disaster recovery storage device 1022 via SCSI or iSCSI. Production storage device 1012 sends data blocks to disaster recovery storage device 1022, thereby realizing data backup of production center 101 by disaster recovery center 102.

[0101] Furthermore, in addition to backup scenarios, the source and target ends can also be used for data replication, which is not limited in this embodiment.

[0102] Optionally, the source and target ends can be any hardware devices with storage functions, such as data nodes, disks, servers, or mobile smart terminals with data storage functions, and this application embodiment does not limit this.

[0103] In practical applications, due to cost and other limitations, such as in intercity travel between different cities, the link bandwidth between the source and destination is usually limited. Therefore, data synchronization technology needs to address how to effectively utilize the limited link bandwidth to maximize data transmission efficiency.

[0104] Currently, deduplication technology is mainly used to improve the transmission efficiency of data synchronization. For example, if the data block A to be sent is the same as the data block B that has already been transmitted, the source end only transmits the description information of the data block A to be sent, so that the target end can obtain the data block B locally based on the description information of the data block A. Since the data block A is the same as the data block B, the target end obtains the data block that the source end needs to send, thereby realizing the reduction of the data link.

[0105] However, in actual work, in some database scenarios, the duplication rate of data blocks is not high; at the same time, small modifications to the data cause small changes in the data blocks, which makes them different from the original blocks, so that deduplication technology cannot effectively reduce the data link.

[0106] To address the aforementioned problems, embodiments of this application provide a data transmission method based on similarity deduplication technology. This method identifies data blocks similar to the data block to be sent and performs differential compression on the data block to be sent based on these similar data blocks. This achieves efficient reduction of the data link while enhancing the adaptability of the solution. For ease of understanding, the method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0107] It should be noted that the data transmission method provided in this application records the information of data blocks that have been sent from the source end to the target end in an index. In subsequent operations, when the source end needs to send a data block to the target end, it will search the index to see if the data block to be sent is similar to a data block that has already been sent. If there is a data block that has already been sent that is similar to the data block to be sent, the subsequent steps of similarity deduplication will be performed. The index can be stored on the target end, and the target end can perform the search for similar blocks. Alternatively, the index can be stored on the source end, and the source end can perform the search for similar blocks. This application will describe these two cases separately.

[0108] First, the index is stored on the target side, and the target side performs the search for similar blocks.

[0109] Please see Figure 2 ,like Figure 2 As shown, an embodiment of the data transmission method provided in this application includes the following steps.

[0110] 201. The first device sends the first information of the first data block to the second device.

[0111] In this embodiment, the first device is the source end, the second device is the target end, the first data block is the data block to be sent, the second device is used to receive the data block sent by the first device, and the first information is used to describe the attributes of the first data block.

[0112] Furthermore, the first information may include a first address, a first strong fingerprint, and a first similar fingerprint of the first data block. The first address is the storage address of the first data block in the first device, the first strong fingerprint is the strong hash value of the first data block, and the first similar fingerprint is the weak hash value of the first data block. The strong fingerprint (FP) describes the attributes of the data block; comparing the strong fingerprints of data blocks can determine whether the data block has been overwritten. The similar fingerprint (SFP) is used to compare the similarity between data blocks; when the similar fingerprints of two data blocks are the same, the two data blocks can be considered similar.

[0113] 202. The second device searches for a second data block similar to the first data block in the first index based on the first information.

[0114] In this embodiment, the first index is used to record information about data blocks that the second device has received from the first device. The first index is updated whenever the second device obtains a new data block from the first device. Optionally, the first index records a first correspondence between similar fingerprints and strong fingerprints of data blocks and the storage address of the data blocks in the second device. The process of finding the second data block may specifically include the following steps.

[0115] 1. The second device searches for a second similar fingerprint that is identical to the first similar fingerprint in the first index.

[0116] In this embodiment, the second similar fingerprint is the weak hash value of the second data block. When two data blocks are similar (i.e., partially the same), there is a high probability that the similar fingerprints are the same. Therefore, if a second similar fingerprint that is the same as the first similar fingerprint is found in the first index, it can be determined that the second data block corresponding to the second similar fingerprint is a similar block to the first data block.

[0117] 2. The second device obtains the second address from the first correspondence based on the second similar fingerprint.

[0118] In this embodiment, the second address is the storage address of the second data block in the second device. Since the first index records the first correspondence between similar fingerprints and storage addresses, the second device can query the storage address of the second data block in the second device from the first index based on a second similar fingerprint.

[0119] 3. The second device obtains the second data block based on the second address.

[0120] In this embodiment, the second device acquires the second data block, thereby realizing the acquisition of similar blocks. The second data block is a data block that is similar to the first data block to be sent from the data blocks that the second device has already received.

[0121] 203. The second device sends the second information of the second data block to the first device.

[0122] In this embodiment, the second information is used to describe the attributes of the second data block. For example, the second information includes the second strong fingerprint and the second similar fingerprint of the second data block, wherein the second strong fingerprint is the descriptive information of the second data block.

[0123] Optionally, the second information also includes the storage address of the reference block in the first device. The reference block is the same data block as the second data block stored in the first device. In practice, after the first device sends the second data block to the second device, it saves the data of the second data block locally. To distinguish it from the second data block stored in the second device, the second data block stored in the first device is designated as the reference block. The method provided in this application can be applied to two scenarios: data synchronization and data backup between devices. In the data synchronization scenario, the synchronized data block has the same storage address in both the first and second devices; in the data backup scenario, the backup data block has different storage addresses in the first and second devices. Therefore, for these two different situations, the following two different methods need to be provided so that the second information includes the storage address of the reference block in the first device.

[0124] 1. Data synchronization scenarios.

[0125] In a data synchronization scenario, the second information can directly record the second address, which is the storage address of the second data block in the second device. Since the storage address of the synchronized data block is the same in both the first and second devices, the second address is also the storage address of the reference block in the first device. Therefore, the second information directly carries the second address as the storage address of the reference block in the first device. Thus, the first device can directly obtain the same reference block as the second data block locally based on the first address.

[0126] 2. Data backup scenarios.

[0127] In a data backup scenario, if the storage addresses of the data blocks to be backed up are inconsistent between the first and second devices, then in the preceding steps, when the first device sends a data block to the second device, it includes the storage address of that data block in its own device. For example, when the first device sends the second data block, it also sends a third address, which is the storage address of the base block in the first device. The base block and the second data block are the same data block. The data block that has already been sent to the second device is denoted as the second data block, and the data block stored locally on the first device is denoted as the base block. This allows the second device to know the storage address of the base block in the first device. At this time, the second information carries the third address as the storage address of the base block in the first device. In the data backup scenario, the second device records this third address in the first index. In this scenario, the first correspondence includes the correspondence between the (strong fingerprint + similar fingerprint) and (local storage address + first device storage address) of the data block. The second device updates the first correspondence with the third address, thereby allowing the second device to know the storage address of each received data block in the first device.

[0128] In the data backup scenario, in step 203, the second information sent by the second device to the first device includes a third address.

[0129] 204. The first device compresses the difference between the first data block and the reference block to obtain first compressed information.

[0130] In this embodiment, the first compression information is used to record the difference data between the first data block and the reference block. Optionally, the first compression information includes a first compressed block and difference information. The first compressed block includes the difference portion between the first data block and the second data block, and the difference information describes the difference between the first data block and the second data block. Optionally, this step can be implemented by the following steps.

[0131] 1. The first device acquires the reference block stored locally.

[0132] In this embodiment, as described above, the second information includes the storage address of the reference block in the first device. The first device can obtain the reference block locally based on this address. In the data synchronization scenario, the first device obtains the reference block based on the second address, and in the data backup scenario, the first device obtains the reference block based on the third address.

[0133] 2. The first device compares the first data block and the second data block to obtain the difference information.

[0134] In this embodiment, the differences between the first data block and the reference block may be that the first part of the data is the same, the second part of the data is the same, the first and second parts of the data are partially the same, or the two data blocks may have interspersed identical data. The first device acquires and records these differences in the difference information.

[0135] 3. The first device performs differential compression on the first data block based on the difference information to obtain the first compressed block.

[0136] In this embodiment, based on the difference information, the first device deletes the part of the first data block that is repeated with the reference block, thereby realizing the similarity deduplication of the first data block. The resulting first compressed block only records the difference between the first data block and the reference block. At the same time, how these differences are different from the reference block is recorded in the difference information. It should be noted that differential compression technology is a known technology. Those skilled in the art can choose different differential compression methods to perform differential compression on the first data block based on the reference block according to actual needs. This embodiment of the application does not limit this.

[0137] 205. The first device sends the first compressed information to the second device.

[0138] In this embodiment, the first compression information may include a first compression block and difference information. Since the first compression block only includes a portion of the content in the first data block and has been compressed, and the difference information is also small-format descriptive data, the amount of data transmitted is greatly reduced compared to transmitting the first data block itself. In transmission scenarios with low bandwidth, the amount of data transmitted can be greatly reduced, thereby reducing the transmission link.

[0139] 206. The second device obtains the first data block based on the first compression information and the second data block.

[0140] In this embodiment, after the second device obtains the first compression information, it processes the first compression information. For example, the first compression information includes a first compression block and difference information. First, the first compression block is decompressed to obtain the first difference data between the first data block and the second data block. Then, according to the difference information, the first difference data and the second data block are assembled into the first data block. For example, according to the difference information, the size of the first data block is 100 bytes. The first data block is different from the first 30 bytes of the reference block, and the last 70 bytes are the same. Since the reference block and the second data block are the same data block, the second device can replace the first 30 bytes of the second data block with the data recorded in the first difference data according to the difference information, thereby obtaining the first data block.

[0141] In this embodiment, a first device sends first information of a first data block to a second device. The first data block is the data block to be sent from the first device to the second device, and the first information describes the attributes of the first data block. The second device searches for a second data block similar to the first data block from a first index based on the first information. The first index records information about data blocks already received by the second device from the first device. The second device sends second information of the second data block to the first device, which describes the attributes of the second data block. The first device obtains a reference block locally based on the second information, and the reference block is identical to the second data block. The first device compresses the difference between the first data block and the reference block to obtain first compression information. The first device sends the first compression information to the second device. The second device obtains the first data block based on the first compression information and the second data block. This achieves similarity deduplication technology. For a data block to be sent, if there is a similar data block that has already been transmitted, only the difference between the two data blocks is transmitted, effectively reducing the data link and improving the transmission efficiency of data synchronization technology.

[0142] It should be noted that in the above process, the first index is used to record information about data blocks that have been transmitted to the second device. Specifically, this includes the correspondence between the strong fingerprint (FP) and similar fingerprint (SFP) of these data blocks and their logical block address (LBA). The strong fingerprint can be implemented using secure hash algorithm 1 (SHA1), and the correspondence can be abbreviated as (SFP + SHA1) → LBA. Since a large amount of data is transmitted between the first and second devices, the first index will occupy a significant amount of memory. In practice, the first index can be stored on an external storage device. For example, the second device can connect to a hard drive via an external interface, and a key-value database can be built on this hard drive using LevelDB technology. This key-value database will then function as the first index. When the second device needs to perform a lookup on the first index, it can access the key-value database on the external hard drive through the external interface.

[0143] Correspondingly, based on the implementation of the first index described above, the data transmission method provided in this application embodiment is implemented through the following steps, which will be described in detail below with reference to the accompanying drawings for ease of understanding.

[0144] Please see Figure 3 ,like Figure 3 As shown, Embodiment 2 of the data transmission method provided in this application includes the following steps.

[0145] 301. The first device sends the first information of the first data block to the second device.

[0146] In this embodiment, this step can be referred to step 201 above, and will not be repeated here.

[0147] 302. The second device searches for a second data block similar to the first data block in the first index based on the first information.

[0148] In this embodiment, this step can refer to step 202 above. Based on step 202, in this embodiment, the first index is stored on an external storage device, such as a hard drive connected to the second device through an external interface. The second device reads the first index running on the hard drive through the external interface, thereby realizing the search for the second similar fingerprint.

[0149] It should be noted that during actual operation, after the second device receives the data block sent by the first device, it may rewrite the data block. This rewriting may not be updated in the first index in a timely manner. For example, the first device sends data block A to the second device. At this time, the second device records the information of data block A in the first index, including the similar fingerprint A and the storage address A. During the operation of the second device, data block A is rewritten as data block A'. At this time, the second device does not update the information of data block A' in the first index. Subsequently, the first device sends a similar fingerprint B, which is used as a similar fingerprint for the data block B to be sent. The second device finds a similar fingerprint A in the first index based on similar fingerprint B, indicating that data block A and data block B are similar data blocks. However, since data block A has been rewritten, the data block obtained by the second device based on the storage address is no longer data block A, but the rewritten data block A'. At this point, data block A' is no longer a similar data block to data block B. Therefore, after finding a similar data block in the first index, the second device needs to perform corresponding verification steps. This is to determine whether the acquired data block has been rewritten as described above.

[0150] Furthermore, since the first index is stored in an external storage device, each read requires going through an external data interface. In order to improve query efficiency and prevent repeated readings, the second device can query the information of multiple data blocks similar to the first data block at once, and then verify them one by one to select the data blocks that have not been modified.

[0151] As described above, the step of the second device searching for a second data block similar to the first data block in the first index based on the first information can be implemented through the following steps.

[0152] 1. The second device finds M target similar fingerprints that are identical to the first similar fingerprint in the first index.

[0153] In this embodiment, M is a positive integer greater than 1. The second device finds multiple target similar fingerprints that are identical to the first similar fingerprint from the first index at once, so that they can be used for verification and filtering in subsequent steps to prevent the second device from repeatedly reading from the external hard drive. It is understood that M is a positive integer greater than 1, which is an ideal working condition. In actual operation, if only one similar fingerprint identical to the first similar fingerprint is found, then M can also be equal to 1. This embodiment of the application does not limit this.

[0154] 2. The second device obtains the M target addresses corresponding to the M target similar fingerprints from the first correspondence based on the M target similar fingerprints.

[0155] In this embodiment, the first correspondence is the first correspondence recorded in the first index. The first correspondence records the correspondence between (strong fingerprint + similar fingerprint) and storage address. The second device can obtain the target address of the M target data blocks corresponding to the M target similar fingerprints in the second device from the first correspondence based on the M target similar fingerprints.

[0156] 3. The second device acquires M target data blocks one by one according to the M target addresses.

[0157] In this embodiment, the data blocks are stored locally on the second device, and the second device can obtain the M target data blocks based on the M target addresses.

[0158] 4. The second device determines whether each of the M target data blocks has been rewritten according to the preset rules.

[0159] In this embodiment, the preset rule is as follows: determine whether the strong fingerprint of the target data block is the same as the strong fingerprint recorded in the first index. If they are different, it means that the data block has been rewritten; otherwise, if they are the same, it means that the data block has not been rewritten. Since the strong fingerprint is used to record the data attributes of the data block, it is possible to determine whether the data block has been rewritten by whether the strong fingerprint has changed.

[0160] In the specific operation process, the first index records the correspondence between (strong fingerprint + similar fingerprint) and storage address. Then, the second device can obtain the M target strong fingerprints corresponding to the M target similar fingerprints based on the M target similar fingerprints. The strong fingerprint can be implemented by SHA1. The strong fingerprint is used to record the data attributes of the data block. Therefore, through the strong fingerprint, the second device can determine whether the obtained target data block has been rewritten.

[0161] In the above manner, the second device determines whether each of the M target data blocks has been overwritten. If the (M-1)th data block among the M target data blocks is overwritten, the (M-1)th data block is discarded and the Mth data block is obtained. This process continues until a data block that has not been overwritten is obtained, and then the data block is identified as a second data block similar to the first data block.

[0162] Furthermore, through the above steps, the second device knows which data blocks among the M target data blocks have been rewritten. At this time, the second device needs to update the information of the rewritten data blocks in the first index to improve the accuracy of the first index.

[0163] Specifically, in the above steps, if the second device determines that the (M-1)th data block has been rewritten, then the second device updates the entry order of the (M-1)th target data block in the first index, the similar fingerprint of the (M-1)th target data block, and the strong fingerprint of the (M-1)th target data block in the first index.

[0164] 303. The second device adds the first tag information to the second data block.

[0165] In this embodiment, the first marking information is used to mark the second data block, and the first marking information uniquely marks the second data block in subsequent steps.

[0166] 304. The second device acquires the second information of the second data block.

[0167] In this embodiment, the second device obtains the second strong fingerprint, the second similar fingerprint, and the storage address of the second data block in the first device as second information. Furthermore, the second information also includes the aforementioned first marking information.

[0168] It should be noted that the method provided in this application can be applied to two scenarios: data synchronization and data backup between devices. In the data synchronization scenario, the synchronized data blocks have the same storage address in both the first and second devices; in the data backup scenario, the backed-up data blocks have different storage addresses in both devices. Therefore, two different methods are needed to address these two different situations, ensuring that the second information includes the storage address of the second data block in the first device. The specific implementation can be found in step 203 above, and will not be repeated here.

[0169] 305. The second device sends a second message to the first device.

[0170] In this embodiment, the second information includes the second strong fingerprint, the second similar fingerprint, and the storage address of the reference block in the first device. Furthermore, it also includes the first tag information of the second data block, which is used to tag the second data block.

[0171] 306. The second device caches the second data block locally.

[0172] In this embodiment, the second device caches the second data block in a local cache for use in the subsequent restoration operation of the first data block. Optionally, it can be cached in a local cache.

[0173] 307. The first device obtains the reference block based on the second information.

[0174] In this embodiment, the first device obtains the reference block stored locally according to the storage address of the reference block recorded in the second information. Since the reference block is the same data block as the second data block stored in the second device, the first device obtains a data block that is the same as the second data block locally.

[0175] 308. The first device performs differential compression on the first data block based on the reference block to obtain the first compressed block and the difference information.

[0176] In this embodiment, the first compression block includes the difference between the first data block and the second data block. The difference information is used to describe the difference between the first data block and the second data block. For the specific implementation method, please refer to the description of step 204 above, which will not be repeated here.

[0177] 309. The first device sends the first compressed block, difference information and first tag information to the second device.

[0178] In this embodiment, the first marking information is used to inform the second device that the first compressed block and the difference information are obtained based on the second data block marked by the first marking information.

[0179] 310. The second device obtains the first data block based on the first compressed block, the difference information, and the first marking information.

[0180] In this embodiment, the specific implementation of obtaining the first data block can be achieved through the following steps.

[0181] 1. The second device retrieves the second data block from the local cache based on the first tag information.

[0182] In this embodiment, the first tag information can be a tag information of only a few bytes. Compared with the embodiment one which obtains the second data block by sending the storage address, sending the first tag information can further reduce the amount of data link transmission.

[0183] 2. The second device restores the first data block locally based on the second data block, the first compressed block, and the difference information.

[0184] In this embodiment, the specific working method can be found in step 206 above, and will not be repeated here.

[0185] In this embodiment, when the second device queries the first index for a similar fingerprint that is the same as the first similar fingerprint, it acquires multiple target similar fingerprints at once. Then, it acquires the corresponding target similar blocks based on the target similar fingerprints and selects the data blocks that have not been overwritten from the multiple target similar blocks as the second data blocks, thereby preventing the failure of similar deduplication due to the data blocks being overwritten in the second device.

[0186] It should be noted that the first device may also have the problem of data blocks being rewritten. Therefore, the second device can query multiple data blocks similar to the first data block according to the method in Embodiment 2 based on the first information, and send them to the first device as candidate data blocks, so that the first device can select a data block that has not been rewritten locally from the multiple candidate data blocks to perform differential compression on the first data block. For ease of understanding, this situation will be described in detail below.

[0187] Please see Figure 4 ,like Figure 4 As shown, Embodiment 3 of the data transmission method provided in this application includes the following steps.

[0188] 401. The first device sends the first information of the first data block to the second device.

[0189] In this embodiment, this step can be referred to step 201 above, and will not be repeated here.

[0190] 402. The second device searches for N alternative data blocks similar to the first data block in the first index based on the first information.

[0191] In this embodiment, the second data block mentioned in Embodiment 2 is one of N candidate data blocks, where N is a positive integer greater than 1; the N candidate data blocks are obtained by the second device from the above M target data blocks, and the acquisition steps of each candidate data block in the N candidate data blocks can refer to the method described in step 302 above, which will not be repeated here.

[0192] Optionally, after the second device acquires N candidate data blocks, it sorts the N candidate data blocks according to their similarity to the first data block to obtain sorting information. The sorting information can be in descending order, arranging the candidate data blocks from the most similar to the first data block to the candidate data blocks with the lowest similarity to the first data block. Specifically, the second device can determine the similarity between the candidate data blocks and the first data block by the similarity between similar fingerprints; the higher the similarity between the similar fingerprints of the candidate data block and the similar fingerprints of the first data block, the more similar the candidate data block is to the first data block.

[0193] 403. The second device obtains the strong fingerprints and similar fingerprints of N candidate data blocks, as well as the storage addresses of the N candidate data blocks in the first device.

[0194] In this embodiment, strong fingerprints and similar fingerprints are data recorded in the first index, which the second device can directly obtain from the first index. Regarding the storage addresses of N candidate data blocks in the first device, the method provided in this embodiment can be applied to both data synchronization and data backup scenarios between devices. In the data synchronization scenario, the synchronized data blocks have the same storage address in both the first and second devices; in the data backup scenario, the backup data blocks have different storage addresses in both devices. Therefore, two different methods are needed to ensure that the second information includes the storage address of the second data block in the first device. The specific implementation method can be found in step 203 above, and will not be repeated here.

[0195] 404. The second device adds tag information to N candidate data blocks.

[0196] In this embodiment, each tag is used to uniquely tag one of the N candidate data blocks, so that the second device and the first device can identify the corresponding candidate data block based on the tag.

[0197] Optionally, the second device adds tagging information to the N candidate data blocks in the following manner: the second device sorts the N candidate data blocks, assigns a sequence number to each candidate data block according to the sorting order, and uses this sequence number as the tagging information of the candidate data block. The sorting method for the N candidate data blocks can be based on the order in which they are found in the first index, or it can be based on any other preset rule; this embodiment does not limit the specific method used.

[0198] 405. The second device sends the second information to the first device.

[0199] In this embodiment, the second information includes strong fingerprints and tagging information of N candidate data blocks, and the storage addresses of the N candidate data blocks in the first device. Optionally, the second information may also include sorting information generated in step 402. The second device sends the second information to the first device so that the first device performs differential compression based on the second information.

[0200] 406. The second device caches N alternative data blocks locally.

[0201] In this embodiment, the second device caches the second data block in a local cache for use in the subsequent restoration operation of the first data block. Optionally, it can be cached in a local cache.

[0202] 407. The first device retrieves N candidate data blocks from local storage based on the storage addresses of the N candidate data blocks in the first device.

[0203] In this embodiment, the first device retrieves N candidate data blocks from local storage based on the storage address recorded in the second information.

[0204] 408. The first device determines whether the N candidate data blocks have been overwritten.

[0205] In this embodiment, the second information includes a strong fingerprint of each candidate data block. The strong fingerprint is used to record the data attributes of the corresponding candidate data block. The first device compares the data attributes of the candidate data block obtained from the local storage with the data attributes recorded by the strong fingerprint to determine whether the candidate data block has been rewritten in the first device.

[0206] For example, candidate data block A is one of the aforementioned N candidate data blocks, and the second information records the strong fingerprint A of candidate data block A. After the first device obtains candidate data block A from local storage, it calculates the strong fingerprint A' of candidate data block A. If the strong fingerprint A' is the same as the strong fingerprint A, it means that candidate data block A has not been overwritten, and the original information is still recorded in candidate data block A. In this case, candidate data block A can constitute a similar block to the first data block. Conversely, if the strong fingerprint A' is different from the strong fingerprint A, it means that candidate data block A has been overwritten in the first device. In this case, candidate data block A is no longer a similar block to the first data block.

[0207] 409. The first device selects one unmodified candidate data block from N candidate data blocks as the base block.

[0208] In this embodiment, according to the above-mentioned preset rules, the first device selects a candidate data block from N candidate data blocks whose strong fingerprint is the same as the strong fingerprint in the second information as a reference block; since the strong fingerprint is used to describe the data block, it can be determined by the strong fingerprint whether the candidate data block found by the second device has been rewritten in the first device.

[0209] In this embodiment, the N candidate data blocks are all data blocks that are similar to the first data block on the local machine of the second device. The second device sends the information of the N candidate data blocks to the first device. The first device determines whether the N candidate data blocks have been rewritten locally according to the above method. For data blocks that have not been rewritten, the first device obtains one of them as the base block.

[0210] Optionally, when the second information includes sorting information, the first device obtains the strong fingerprints of N candidate data blocks one by one according to the sorting information to determine whether the candidate data blocks have been rewritten locally. The first device takes the first candidate data block that has not been rewritten as the reference block and uses the reference block to perform subsequent differential compression steps on the first data block.

[0211] Furthermore, if the first device does not find a candidate data block among the N candidate data blocks whose strong fingerprint is the same as the strong fingerprint recorded in the second information, it means that all N candidate data blocks have been rewritten locally on the first device. In this case, the first device directly compresses the first data block into a second compressed block and sends it to the second device. After the second device obtains the second compressed block, it decompresses the second compressed block into the first data block locally and does not execute subsequent steps 410 to 413.

[0212] 410. The first device performs differential compression on the first data block based on the reference block to obtain the first compressed block and differential information.

[0213] In this embodiment, the first compressed block includes the difference between the first data block and the reference block. The difference information is used to describe the difference between the first data block and the reference block. For the specific implementation method, please refer to step 204 above, which will not be repeated here.

[0214] 411. The first device sends the first compressed block, the difference information, and the first tag information to the second device.

[0215] In this embodiment, the reference block is one of N candidate data blocks. The first device selects one of the N candidate data blocks that has not been modified locally as the reference block to perform differential compression on the first data block. The first tag information is used to tag the second data block, and this first tag information is the tag information added by the second device to the second data block in step 404 above. Based on the first tag information, the second device can know that the reference block selected by the first device from the N candidate data blocks is the same data block as the second data block, and the first device performs differential compression on the first data block to be sent based on the reference block.

[0216] 412. The second device retrieves the second data block from the local cache based on the first tag information.

[0217] In this embodiment, after the second device sends N candidate data blocks to the first device, the second device caches the N candidate data blocks locally. When the second device obtains the first tag information sent by the first device, the second device knows that the first device has selected the second data block from the N candidate data blocks for differential compression. At this time, the second device retrieves the second data block from the local cache according to the first tag information to perform the subsequent data restoration steps.

[0218] 413. The second device obtains the first data block based on the first compressed block, the difference information, and the second data block.

[0219] In this embodiment, the specific implementation steps of the second device obtaining the first data block based on the first compression block, the difference information, and the second data block can be referred to step 206 above, and will not be repeated here.

[0220] In this embodiment, to prevent similar blocks found by the second device from being overwritten at the first device, the second device searches for multiple candidate data blocks similar to the first data block and sends the information of these candidate data blocks to the first device. The first device then selects one unrewritten data block from these candidate data blocks as the reference block to perform differential compression on the first data block to be sent. Afterward, the first device sends the marking information of the second data block to the second device. The second data block marked by this marking information is the same data block as the reference block. This allows the second device to know which candidate data block the first device has selected to perform differential compression, thereby ensuring the smooth progress of the data restoration work of the second device.

[0221] Furthermore, after the second device obtains the first data block, it updates the information of the first data block in the first index, including the similar fingerprint, strong fingerprint, storage address of the first data block in the first device, and the correspondence between (similar fingerprint + strong fingerprint) and storage address.

[0222] It should be noted that during the above process, the second information fed back from the second device to the first device may also be of the following two types: 1. The second device found a second data block in the first index that is completely identical to the first data block; 2. The second device did not find a data block that is the same as or similar to the first data block in the first index. In the first case, after receiving the second information, the first device skips the current processing of the first data block, and the second device retrieves the second data block locally as the first data block. In the second case, after receiving the second information, the first device directly compresses the first data block into a second compressed block and sends it to the second device.

[0223] The above situation 1 is the same deduplication technology in the prior art, and the above situation 2 is the data compression and transmission technology in the prior art. Those skilled in the art can obtain the specific implementation methods in these two situations, so the embodiments of this application will not be described in detail.

[0224] The above describes the first scenario provided by the embodiments of this application: the index is stored on the target end, and the target end performs the search for similar blocks. This method can achieve good working results in more complex system architectures, for example... Figure 5a In the architecture shown, a single target end simultaneously receives data sent by multiple source ends. Figure 5a In the diagram, there are first devices A 501, first devices B 502, and first devices C 503. These first devices A 501 to C 503 are all the same source (i.e., first devices). Assume there are p first devices, where p is a positive integer greater than 1. Second device 504 is used to back up or synchronize the data of these p first devices. Second device 504 has a first index for recording data block information received from first device A 501; a second index for recording data block information received from first device B 502; and so on, with a Pth index for recording data block information received from first device C 503.

[0225] As can be seen, the target end second device is equipped with P indexes corresponding to P first devices, thereby enabling the second device to execute the data transmission method provided in the embodiments of this application on the P first devices. Each of the P first devices and the second device constitute a data transmission link, and the similarity block search of each link can be uniformly performed by the second device, thereby reducing the service pressure on the first devices and improving the working efficiency of the data transmission method provided in this application.

[0226] Furthermore, it can also be used for, for example Figure 5b In the architecture shown, Q second devices acquire data blocks sent by the first device, where Q is a positive integer greater than 1. The number of first devices can be one or more, and this embodiment of the application does not limit this. Figure 5b In this setup, Q second devices can be considered as Q data nodes. Each data node contains a portion of the first index. For example, when Q=2, it includes second device A 506 and second device B 507. Second device A 506 and second device B 507 are the same two second devices. First index A in second device A 506 stores content with odd-numbered tails of the strong fingerprint of the first index; first index B in second device B 507 stores content with even-numbered tails of the strong fingerprint of the first index. When first device 505 sends first information to the second devices, if the tail of the strong fingerprint in the first information is odd, then second device A 506 performs a search: retrieving information about a second data block similar to the first data block from first index A based on the first information; if the tail of the strong fingerprint in the first information is even, then second device B 507 performs a search: retrieving information about a second data block similar to the first data block from first index B based on the first information. This distributes the search pressure on the first index among multiple second devices, achieving load balancing among them through this clustered deployment of second devices.

[0227] The data transmission method provided in this embodiment includes: a first device sending first information of a first data block to a second device, wherein the first data block is a data block to be sent by the first device to the second device, and the first information describes the attributes of the first data block; the second device searching for a second data block similar to the first data block from a first index based on the first information, wherein the first index records information about data blocks already received by the second device from the first device; the second device sending second information of the second data block to the first device, wherein the second information describes the attributes of the second data block; the first device obtaining a reference block locally based on the second information, wherein the reference block is identical to the second data block; the first device compressing the difference between the first data block and the reference block to obtain first compression information; the first device sending the first compression information to the second device; and the second device obtaining the first data block based on the first compression information and the second data block. This achieves similarity deduplication technology. For a data block to be sent, if there is a similar data block that has already been transmitted, only the difference between the two data blocks is transmitted, effectively reducing the data link and improving the transmission efficiency of data synchronization technology.

[0228] It should be noted that in actual operation, indexes can be deployed not only on the target end (the receiving end) but also on the source end (the sending end). For example, in a backup scenario, the target end has relatively low business pressure, so the index can be deployed on the target end to fully utilize its computing resources. In a replication scenario, data synchronization between the source and target ends is required, so the index can be deployed on the source end to minimize interaction overhead. The above embodiments illustrate the working scenario of setting the index on the target end; the following, in conjunction with the accompanying drawings, provides a detailed explanation of the working scenario of deploying the index on the source end.

[0229] Second, the index is stored at the source end, and the search for similar blocks is performed at the source end.

[0230] Please see Figure 6 ,like Figure 6 As shown, Embodiment 4 of the data transmission method provided in this application includes the following steps.

[0231] 601. The first device acquires the first information of the first data block.

[0232] In this embodiment, the first device is the end that sends data, the first data block is the data block to be sent, and the first information is used to describe the attributes of the first data block.

[0233] Furthermore, the first information may include a first address, a first strong fingerprint, and a first similar fingerprint of the first data block. The first address is the storage address of the first data block in the first device, the first strong fingerprint is the strong hash value of the first data block, and the first similar fingerprint is the weak hash value of the first data block. The strong fingerprint (FP) describes the attributes of the data block; comparing the strong fingerprints of data blocks can determine whether the data block has been overwritten. The similar fingerprint (SFP) is used to compare the similarity between data blocks; when the similar fingerprints of two data blocks are the same, the two data blocks can be considered similar.

[0234] 602. The first device searches for a second data block similar to the first data block in the first index based on the first information.

[0235] In this embodiment, the first index is used to record information about data blocks that the first device has sent to the second device. The first index is updated whenever the first device sends a new data block to the second device. The second device is used to receive data blocks sent by the first device.

[0236] Optionally, the first index records a first correspondence between similar fingerprints and strong fingerprints of data blocks and the storage address of the data blocks in the first device. The process of finding the second data block may specifically include the following steps.

[0237] 1. The first device searches for a second similar fingerprint that is identical to the first similar fingerprint in the first index.

[0238] In this embodiment, the second similar fingerprint is the weak hash value of the second data block. When two data blocks are similar (i.e., partially the same), there is a high probability that the similar fingerprints are the same. Therefore, if a second similar fingerprint that is the same as the first similar fingerprint is found in the first index, it can be determined that the second data block corresponding to the second similar fingerprint is a similar block to the first data block.

[0239] 2. The first device obtains the second address from the first correspondence based on the second similar fingerprint.

[0240] In this embodiment, the second address is the storage address of the second data block in the first device. Since the first index records the first correspondence between similar fingerprints and storage addresses, the first device can query the storage address of the second data block in the first device from the first index based on a second similar fingerprint.

[0241] 3. The first device obtains the second data block based on the second address.

[0242] In this embodiment, the first device acquires the second data block, thereby realizing the acquisition of similar blocks. The second data block is a data block that is similar to the first data block to be sent at present, which is among the data blocks that the first device has already sent to the second device.

[0243] 603. The first device compresses the difference between the first data block and the second data block to obtain first compressed information.

[0244] In this embodiment, the first compression information is used to record the difference information between the first data block and the second data block. Optionally, the first compression information includes a first compressed block and difference information, wherein the first compressed block includes the difference portion between the first data block and the second data block, and the difference information is used to describe the difference between the first data block and the second data block. Optionally, this step can be implemented by the following steps.

[0245] 1. The first device compares the first data block and the second data block to obtain the difference information.

[0246] In this embodiment, the differences between the first data block and the second data block may be that the first part of the data is the same, the second part of the data is the same, the first and second parts of the data are partially the same, or the two data blocks may have interspersed identical data. The first device acquires and records these differences in the difference information.

[0247] 2. The first device performs differential compression on the first data block based on the difference information to obtain the first compressed block.

[0248] In this embodiment, based on the difference information, the first device deletes the part of the first data block that is repeated with the second data block, thereby realizing the similarity deduplication of the first data block. The resulting first compressed block only records the difference between the first data block and the second data block. At the same time, how these differences are different from the second data block is recorded in the difference information. It should be noted that differential compression technology is a known technology. Those skilled in the art can choose different differential compression methods to perform differential compression on the first data block based on the second data block according to actual needs. This embodiment of the application does not limit this.

[0249] 604. The first device sends the first compressed information to the second device.

[0250] In this embodiment, optionally, the first device sends a first compressed block and difference information to the second device, so that the second device obtains a first data block based on the first compressed block, the difference information, and a local reference block. The local reference block of the second device is the same data block as the second data block stored in the first device. After sending the second data block to the second device, the first device stores the second data block locally. To easily distinguish the same data block stored in two different devices, the second data block sent by the first device to the second device is designated as the reference block. Since the first compressed block only contains information about the differences between the first and second data blocks, and the difference information is merely descriptive information, the data volume of both is much smaller than that of the first data block, thereby effectively reducing the amount of data transmitted and achieving a reduction in the data transmission link.

[0251] Optionally, the first device further sends the storage address of the reference block in the second device to the second device, so that the second device can retrieve the reference block from its local storage based on the address. The method provided in this application embodiment can be applied to both data synchronization and data backup scenarios between devices. In the data synchronization scenario, the storage address of the synchronized data block is the same in both the first and second devices; in the data backup scenario, the storage address of the backed-up data block is different in both the first and second devices. Therefore, for these two different situations, the following two different methods need to be provided so that the first device can send the storage address of the second data block in the second device to the second device.

[0252] 1. Data synchronization scenarios.

[0253] In the scenario of data synchronization, since the storage address of the synchronized data block is the same in the first device and the second device, the first device can directly send the second address to the second device. The second address of the second data block in the first device is the storage address of the reference block in the second device. The reference block and the second data block are the same data block. For easy distinction, the data block stored in the first device is called the second data block, and the data block stored in the second device is called the reference block.

[0254] 2. Data backup scenarios.

[0255] In a data backup scenario, if the storage addresses of the backed-up data blocks are inconsistent between the first and second devices, then in the preceding steps, after the first device sends the data block to the second device, it obtains a feedback message sent by the second device. This feedback message records the storage address of the data block in the second device. For example, after the first device sends the second data block, the second data block stored locally on the first device is recorded as the second data block, and the second data block sent to the second device is recorded as the base block. The first device obtains a second feedback message from the second device, which records a third address where the base block is stored in the second device. The first device records this third address in a first index. In this scenario, the first correspondence includes the correspondence between the data block's (strong fingerprint + similar fingerprint) and (local storage address + second device storage address). The first device updates the third address in the first correspondence, thereby enabling the first device to know the storage address of each data block sent to the second device in the second device.

[0256] In the data backup scenario, in step 604, the first device sends a third address to the second device.

[0257] 605. The second device obtains the reference block from the local machine.

[0258] In this embodiment, the second device can obtain a base block from the local cache according to the storage address sent by the first device. For example, in the above data synchronization scenario, the second device obtains a base block from the local storage according to the second address; in the above data backup scenario, the second device obtains a base block from the local storage according to the third address.

[0259] Optionally, the second device may also obtain tagging information from the first device. This tagging information is used to tag the reference block and obtain the reference block locally. Since the amount of data in the tagging information is smaller than that in the address information, this method can further reduce the amount of data transmitted.

[0260] 606. The second device obtains the first data block based on the first compression information and the reference block.

[0261] In this embodiment, optionally, the first compression information includes a first compression block and difference information. The first compression block records the data that differs between the first data block and the second data block. The difference information records the specific differences between the first data block and the second data block. Since the reference block and the second data block are the same data block, the second device can reconstruct the second data block based on the first compression block, the difference information, and the reference block. For example, according to the difference information, the first data block is 100 bytes in size, where the first 30 bytes of data in the first data block and the second data block are different, and the last 70 bytes are the same. Since the reference block and the second data block are the same data block, the second device can then replace the first 30 bytes of data in the reference block with the data recorded in the first difference information, thereby obtaining the first data block.

[0262] Optionally, in a data backup scenario, the following steps may be performed further.

[0263] The second device sends a first feedback message to the first device.

[0264] In this embodiment, the first feedback message is used to record the fourth address where the first data block is stored in the second device. In the backup scenario, the second device needs to provide feedback to the first device on the storage address of each received data block in the second device. Therefore, when the second device restores the first data block locally, it needs to provide feedback to the first device on the local storage address of the first data block. At the same time, the first device updates the correspondence between (strong fingerprint + similar fingerprint) and (local storage address + storage address in the second device) of the first data block in the local first index.

[0265] Furthermore, during the operation of the second device, the reference block may be rewritten, to the point that the second device cannot obtain the reference block locally based on the address sent by the first device. In this case, the second device needs to send a failure message to the first device, indicating that the second device did not find the second data block locally. At this point, the first device directly compresses the first data block into a second compressed block and sends the second compressed block to the second device. The second compressed block includes all the data of the first data block. The method of compressing the first data block into the second compressed block can be any compression method in the prior art, and this application is not limited to it.

[0266] In this embodiment, this method prevents data transmission failure due to the inability to perform similar deduplication of the data block to be sent because the data is rewritten in the second device. When the similar block sent by the first device is rewritten in the second device, the second device informs the first device through a failure message to directly send the compressed block of the data block to be sent, thereby overcoming this problem.

[0267] In this embodiment, a first device performs a lookup of a first index. The first device obtains first information about a first data block, which is the data block to be sent by the first device to a second device. The second device receives the data block sent by the first device. The first information describes the attributes of the first data block. The first device searches for a second data block similar to the first data block in the first index based on the first information. The first index records information about data blocks already sent by the first device to the second device. The first device compresses the differences between the first data block and the second data block to obtain first compression information. The first device sends the first compression information to the second device, enabling the second device to obtain the first data block based on the first compression information. This achieves similarity deduplication technology. For a data block to be sent, if there is a similar data block that has already been transmitted, only the differences between the two data blocks are transmitted, effectively reducing the data link and improving the transmission efficiency of data synchronization technology.

[0268] It should be noted that in the above embodiment four, the first index is used to record information about data blocks that have been transmitted to the second device. Specifically, this includes the correspondence between the strong fingerprint (FP) and similar fingerprint (SFP) of these data blocks and their logical block address (LBA). The strong fingerprint can be implemented using secure hash algorithm 1 (SHA1), and this correspondence can be abbreviated as (SFP + SHA1) → LBA. Since the first device needs to send and transmit a large amount of data to the second device, the first index will occupy a significant amount of memory. In practice, the first index can be stored on an external storage device. For example, the first device can connect to a hard drive via an external interface, and a key-value database can be built on this hard drive using LevelDB technology. This key-value database will then function as the first index. When the second device needs to perform a lookup on the first index, it can access the key-value database on the external hard drive through the external interface.

[0269] Correspondingly, based on the implementation of the first index described above, the data transmission method provided in this application embodiment is implemented through the following steps, which will be described in detail below with reference to the accompanying drawings for ease of understanding.

[0270] Please see Figure 7 ,like Figure 7 As shown, Embodiment 5 of the data transmission method provided in this application includes the following steps.

[0271] 701. The first device acquires the first information of the first data block.

[0272] In this embodiment, the specific implementation of this step can be found in step 601 above, and will not be repeated here.

[0273] 702. The first device finds M target similar fingerprints that are identical to the first similar fingerprint in the first index.

[0274] In this embodiment, the first similar fingerprint is the similar fingerprint of the first data block. Based on the first similar fingerprint, the first device finds M target similar fingerprints that are the same as the first similar fingerprint in the first index, where M is a positive integer greater than 1.

[0275] Specifically, the first index is stored on an external storage device, such as a hard drive connected to the first device via an external interface. The first device reads the first index running on the hard drive through the external interface, thereby enabling the search for M similar fingerprints.

[0276] It should be noted that during actual operation, after the first device sends a data block to the second device, it may rewrite the data block. This rewriting may not be updated in the first index in a timely manner. For example, the first device sends data block A to the second device. At this time, the first device records the information of data block A in the first index, including the similar fingerprint A and the storage address A. During the operation of the first device, data block A is rewritten as data block A'. At this time, the first device does not update the information of data block A' in the first index. Subsequently, when the first device prepares to send data block B to the second device, it obtains the similar fingerprint B of data block B and finds that the similar fingerprint B is the same as the similar fingerprint A in the first index, indicating that data block A and data block B are similar data blocks. However, at this time, because data block A has been rewritten, the data block obtained by the first device based on the storage address is no longer data block A, but the rewritten data block A'. At this point, data block A' is no longer a similar data block to data block B. Therefore, after the first device finds a similar data block in the first index, it needs to perform corresponding verification steps. This is to determine whether the acquired data block has been rewritten as described above.

[0277] Furthermore, since the first index is stored in an external storage device, each read requires access to an external data interface. To improve query efficiency and prevent duplicate reads, the first device can query information from multiple data blocks similar to the first data block at once, and then verify them one by one, selecting the data blocks that have not been modified. Therefore, in step 702, the first device needs to find M target similar fingerprints that are identical to the first similar fingerprint in the first index.

[0278] 703. The first device obtains the M target addresses corresponding to the M target similar fingerprints from the first correspondence relationship based on the M target similar fingerprints.

[0279] In this embodiment, the first index records the correspondence between similar fingerprints of each data block and storage addresses. The storage address includes the local storage address of the data block in the first device. According to the correspondence, the first device can find the storage address of the data block corresponding to the M target similar fingerprints in the first index based on the M target similar fingerprints.

[0280] 704. The first device obtains M target data blocks based on M target addresses.

[0281] In this implementation, the first device can obtain M target data blocks from local storage based on the obtained M target addresses.

[0282] 705. The first device determines whether each of the M target data blocks has been rewritten according to the preset rules.

[0283] In this embodiment, the preset rule is as follows: determine whether the strong fingerprint of the target data block is the same as the strong fingerprint recorded in the first index. If they are different, it means that the data block has been rewritten; otherwise, if they are the same, it means that the data block has not been rewritten. Since the strong fingerprint is used to record the data attributes of the data block, it is possible to determine whether the data block has been rewritten by whether the strong fingerprint has changed.

[0284] In the specific operation process, the first index records the correspondence between (strong fingerprint + similar fingerprint) and storage address. Then, the first device can obtain the M target strong fingerprints corresponding to the M target similar fingerprints based on the M target similar fingerprints. The strong fingerprint can be implemented by SHA1. The strong fingerprint is used to record the data attributes of the data block. Therefore, through the strong fingerprint, the first device can determine whether the obtained target data block has been rewritten.

[0285] 706. The first device obtains the second data block that has not been modified from M target data blocks.

[0286] In this embodiment, the first device determines whether each of the M target data blocks has been overwritten in the manner described above. If the (M-1)th data block among the M target data blocks is overwritten, the (M-1)th data block is discarded and the Mth data block is obtained. This process continues until a data block that has not been overwritten is obtained, and then the data block is identified as a second data block that is similar to the first data block.

[0287] It should be noted that the first device may obtain multiple unmodified target data blocks from M target data blocks. In this case, the first device may obtain any one of them as the second data block, or it may select the first unmodified target data block obtained as the second data block according to the determination order, and no longer perform subsequent determinations on other target data blocks.

[0288] Preferably, the first device can obtain N candidate data blocks similar to the first data block from M target data blocks, then sort the similarity between the N candidate data blocks and the first data block, and finally select the candidate data block with the highest similarity to the first data block as the second data block. Specifically, the first device can determine the similarity between the candidate data block and the first data block by the similarity between similar fingerprints; the higher the similarity between the similar fingerprints of the candidate data block and the similar fingerprints of the first data block, the more similar the candidate data block is to the first data block.

[0289] Furthermore, through the above steps, the first device knows which data blocks among the M target data blocks have been rewritten. At this time, the first device needs to update the information of the rewritten data blocks in the first index to improve the accuracy of the first index.

[0290] Specifically, in the above steps, if the first device determines that the (M-1)th data block has been rewritten, then the first device updates the entry order of the (M-1)th target data block in the first index, the similar fingerprint of the (M-1)th target data block, and the strong fingerprint of the (M-1)th target data block in the first index.

[0291] Subsequent steps 707 to 710 can be found in steps 603 to 606 above, and will not be repeated here.

[0292] Furthermore, when the first device sends the storage address of the reference block in the second device, it can also send a strong fingerprint of the reference block to the second device. This allows the second device to determine whether the reference block in its local storage has been overwritten based on the strong fingerprint. If it has been overwritten, the second device sends a failure message to the first device. This failure message indicates that the second data block in the second device's local storage has been overwritten. At this point, the first device directly compresses the first data block into a second compressed block and then sends the second compressed block to the second device. The second compressed block includes all the data of the first data block. The method of compressing the first data block into the second compressed block can be any compression method in the prior art, and this application is not limited to any of them.

[0293] In this embodiment, when the first device queries a similar fingerprint that is the same as the first similar fingerprint in the first index, it obtains multiple target similar fingerprints at once, and then obtains the corresponding target similar blocks according to the target similar fingerprints. It selects data blocks that have not been overwritten from the multiple target similar blocks as the second data blocks, thereby preventing the situation where similar deduplication fails due to the data blocks being overwritten in the first device.

[0294] It should be noted that during the above process, after the first device searches in the first index based on the first information, the following two results may occur: 1. The first device finds a second data block in the first index that is completely identical to the first data block; 2. The first device does not find a data block that is identical or similar to the first data block in the first index. In the first case, the first device can directly send the description information of the second data block to the second device, so that the second device can directly retrieve the second data block from its local storage as the first data block based on the description information. In the second case, the first device directly compresses the first data block into a second compressed block and sends it to the second device.

[0295] The above situation 1 is the same deduplication technology in the prior art, and the above situation 2 is the data compression and transmission technology in the prior art. Those skilled in the art can obtain the specific implementation methods in these two situations, so the embodiments of this application will not be described in detail.

[0296] The above describes the second scenario provided by the embodiments of this application: the index is stored at the source end, and the source end performs the search for similar blocks. This method can achieve good results in more complex system architectures, for example... Figure 8 In the architecture shown, the source sends data to multiple target ends simultaneously. Figure 8 In this process, the first device 801 is connected to the second devices A 802, B 803 and C 804. The second devices A 802 to C 804 are all the same second devices. The first device 801 performs the lookup of the first index in a unified manner, thereby reducing data link transmission and improving the efficiency of the lookup.

[0297] Furthermore, in Figure 8In this scenario, the second devices A 802 to B 804 may be located in the same computer room or be close to each other. In this case, the N second devices have a large transmission bandwidth and are not limited by link transmission, allowing them to transmit data blocks to each other. For example, if the first device 801 needs to send a first data block to the second device B 803, and the first device 801 finds a second data block already sent to the second device A 802 that is similar to the first data block in the first index, then the first device 801 sends the first compressed block and difference information to the second device B 803 in the manner described above. Simultaneously, it also sends an instruction to the second device B 803, instructing it to obtain the second data block from the second device A 802, so that the second device B 803 can reconstruct the first data block locally based on the second data block, the first compressed block, and the difference information. In this scenario, the transmission bandwidth between the second device B 803 and the second device A 802 is relatively large, enabling the second devices to transmit data blocks that have been received from the first device 801 to each other, thereby further improving the data transmission efficiency.

[0298] The above describes in detail the two methods provided in the embodiments of this application. The first method is that the index is stored on the target end (second device) and the target end performs the similar block search. The second method is that the index is stored on the source end (first device) and the source end performs the similar block search. Both methods can be implemented by the following hardware structure.

[0299] Please see Figure 9 ,like Figure 9 As shown, the source and target ends can each be implemented using n nodes (vnodes). The source and target ends can each contain only one node or multiple nodes. Figure 9 The example uses multiple nodes to illustrate this, thus achieving a node cluster deployment, enabling features such as... Figure 5bThe load balancing query described herein. A node 91 of the first device (source end) includes a first replication module 901 and a first interface module 902. The first replication module 901 is used to perform data replication (which can be data synchronization or data backup) with the second device, and the first interface module 902 is used to communicate with other nodes. A communication interface is provided between the first replication module 901 and the first storage module Block 903, enabling the first replication module 901 to obtain data blocks to be sent from the first storage module Block 903. Further, the first replication module 901 includes a first data movement engine (DME) 9011 and a first transceiver module NGW IoRouter 9012, wherein the DME includes a first link reduction module Dedup 90111.

[0300] A node 92 of the second device (target end) includes a second replication module 904 and a second interface module 905. The second replication module 904 is used to perform data replication with the first device (which can be data synchronization or data backup), and the second interface module 905 is used to communicate with other nodes. A communication interface is provided between the second replication module 904 and the second storage module Block 906, enabling the second replication module 904 to read or write data blocks from the second storage module Block 906. Further, the second replication module 904 includes a data movement engine (DME) 9041 and a second transceiver module NGW IoRouter 9042, wherein the DME includes a second link reduction module Dedup 90411.

[0301] The following is based on Figure 9 The architecture shown provides a detailed explanation of the data flow and specific data processing methods for the two schemes provided in this application.

[0302] First, the index is stored on the target side, and the target side performs the search for similar blocks.

[0303] Please see Figure 10 ,like Figure 10 As shown, based on Figure 9 The system architecture shown in this application embodiment includes the following steps in the data processing method of the first case.

[0304] 1001. The first DME obtains the first data snapshot from the first storage module at the first moment.

[0305] In this embodiment, the first data snapshot is used to record the data storage status in the first storage module at a first moment. This first data snapshot can be identified as snapid1 in DME.

[0306] 1002. The first DME obtains a second data snapshot from the first storage module at the second moment.

[0307] In this embodiment, the second time point is the time point after the first time point, and the second data snapshot is used to record the data storage status in the first storage module at the second time point. This second data snapshot can be identified as snapid2 in DME.

[0308] 1003. The first device compares the first data snapshot and the second data snapshot to obtain the first data block.

[0309] In this embodiment, by comparing the first data snapshot and the second data snapshot, the first device knows what changes have occurred in the locally stored data from the first moment to the second moment. These changed data are the data that needs to be sent to the second device for backup. Therefore, the data that differs from the data in the second data snapshot is obtained as the first data block that needs to be sent to the second device.

[0310] Furthermore, after the first device acquires the first data block, it acquires the similar fingerprint and strong fingerprint of the first data block; then the first device generates first information, which includes the similar fingerprint and the strong fingerprint.

[0311] 1004. The first DME sends the first information to the second transceiver module through the first transceiver module, so that the second transceiver module forwards the first information to the second DME.

[0312] In this embodiment, the first information is the information of the first data block. The specific content of the first information can be found in Embodiments 1 to 3 above, and will not be repeated here.

[0313] 1005. The second Dedup in the second DME searches for data block information similar to the first data block based on the first information.

[0314] In this embodiment, the second Dedup performs the lookup of the first index. The specific implementation of the second Dedup using the first index to find information about data blocks similar to the first data block can be found in Embodiments 1 to 3 above, and will not be repeated here.

[0315] 1006. The second Dedup obtains the information of the target data block from the second storage module based on the data block information similar to the first data block.

[0316] In this embodiment, the information of the data block similar to the first data block is the information of the target data block. This information may include the strong fingerprint information of the target data block, storage address information, and similar fingerprint information, etc. For specific implementation methods, please refer to Embodiments 1 to 3 above, which will not be repeated here.

[0317] 1007. When the second Dedup determines that the target data block has not been overwritten, it sends the target data block information to the first Dedup.

[0318] In this embodiment, the number of target data blocks can be one or more, as can be found in the descriptions of embodiments one to three above, which will not be repeated here.

[0319] 1008. The first Dedup obtains the target data block from the first storage module through the first interface module based on the information of the target data block.

[0320] In this embodiment, the first Dedup retrieves the target data block from the local storage: the first storage module, based on the storage address recorded in the target data block information. If there are multiple target data blocks, the first Dedup also needs to select one unmodified data block as the base block.

[0321] 1009. The first Dedup performs differential compression on the first data block based on the target data block to obtain the first compressed block and the difference information.

[0322] In this embodiment, the specific method of differential compression can be found in the descriptions of Embodiments 1 to 3 above, and will not be repeated here.

[0323] 1010. The first Dedup sends the first compressed block and difference information to the second Dedup.

[0324] In this embodiment, the first compressed block and the difference information pass through the first transceiver module and the second transceiver module to reach the second DME, which then sends them to the second Dedup. Furthermore, the first Dedup can also send the storage address or identification information of the target data block in the second device. For details, please refer to the descriptions of Embodiments 1 to 3 above, which will not be repeated here.

[0325] 1011. The second Dedup restores the first data block based on the first compressed block, the difference information, and the target data block.

[0326] In this embodiment, the specific implementation of the second Dedup restoring the first data block based on the first compressed block, the difference information, and the target data block can be found in the descriptions of Embodiments 1 to 3 above, and will not be repeated here.

[0327] 1012. The second Dedup writes the first data block into the second storage module.

[0328] In this embodiment, the second Dedup writes the first data block to the second storage module through the second interface module, thereby writing the newly acquired first data block into the local storage.

[0329] 1013. The second storage module sends the disk write result to the second Dedup.

[0330] In this embodiment, the second storage module sends the disk writing result to the second Dedup through the second interface module, so that the second Dedup knows the disk writing status of the first data block.

[0331] 1014. The second Dedup sends the copy result to the first Dedup.

[0332] In this embodiment, the second Dedup sends the copy result to the first Dedup so that the first device knows whether the data block was successfully sent. If it is unsuccessful, the first device can also resend the complete compressed block of the first data block to the second device.

[0333] 1015. The second Dedup updates the information of the first data block into the first index.

[0334] In this embodiment, the first index can reside on the same node as the second Dedup or on another target node, which is also a second device. Different portions of the first index are stored in multiple second device nodes, thereby achieving load-balanced lookup of the first index. When the second Dedup needs to update the information of the first data block to the Dedup on another second device node, the second Dedup sends the information of the first data block to that Dedup, enabling that Dedup to update the first index.

[0335] In this embodiment, the data transmission method shown in Embodiments 1 to 3 is realized through the collaborative work of various modules in the system architecture.

[0336] Second, the index is stored at the source end, and the search for similar blocks is performed at the source end.

[0337] Please see Figure 11 ,like Figure 11 As shown, based on Figure 9 The system architecture shown in this application embodiment includes the following steps in the data processing method of the second case.

[0338] Steps 1101 to 1103 can be found in steps 1001 to 1003 above, and will not be repeated here.

[0339] 1104. The first Dedup in the first DME searches for data block information similar to the first data block based on the first information.

[0340] In this embodiment, the first information is obtained by the first Dedup from the first data block, and the first index is set on the first device. The first Dedup performs the lookup of the first index. The specific implementation of the first Dedup using the first index to find data block information similar to the first data block can be found in Embodiments 4 or 5 above, and will not be repeated here.

[0341] 1105. The first Dedup obtains the target data block from the first storage module through the first interface module.

[0342] In this embodiment, a Dedup finds a target data block that is similar to the first data block based on first information, including the storage address of the target data block in the first storage module. Therefore, the first Dedup can obtain the target data block based on this address. Furthermore, the first Dedup can further verify whether the target data block has been rewritten, as detailed in Embodiments 4 and 5 above, which will not be repeated here.

[0343] 1106. The first Dedup performs differential compression on the first data block based on the target data block to obtain the first compressed block and the difference information.

[0344] In this embodiment, the target data block is the data block that the first Dedup finds based on the first information that is similar to the first data block. The first Dedup performs differential compression on the first data block based on the target data block to obtain the first compressed block and the difference information. For the specific working method, please refer to the above embodiment four or embodiment five, which will not be repeated here.

[0345] 1107. The first Dedup sends the first compressed block, the difference information, and the target data block information to the second Dedup.

[0346] In this embodiment, the specific working method of the first Dedup sending the first compressed block, difference information and target data block information to the second Dedup can be referred to in Embodiment 4 or Embodiment 5 above, and will not be repeated here.

[0347] 1108. The second Dedup retrieves the target data block from the second storage module based on the information of the target data block.

[0348] In this embodiment, the information of the target data block can be the storage address of the target data block in the second storage module, or it can be the identification information of the target data block. For the specific working method, please refer to the above embodiment four or embodiment five, which will not be repeated here.

[0349] Steps 1109 to 1112 can be referred to steps 1011 to 1014 above, and will not be repeated here.

[0350] 1113. The first Dedup updates the information of the first data block into the first index.

[0351] In this embodiment, based on the copying result sent by the second Dedup, the first Dedup can know that the first data block has been successfully written to the disk in the second device. At this time, the first Dedup updates the information of the first data block to the first index.

[0352] In this embodiment, the data transmission method shown in Embodiments 4 and 5 is realized through the collaborative work of various modules in the system architecture.

[0353] The data transmission method provided in this embodiment includes: a first device sending first information of a first data block to a second device, wherein the first data block is a data block to be sent by the first device to the second device, and the first information describes the attributes of the first data block; the second device searching for a second data block similar to the first data block from a first index based on the first information, wherein the first index records information about data blocks already received by the second device from the first device; the second device sending second information of the second data block to the first device, wherein the second information describes the attributes of the second data block; the first device obtaining a reference block locally based on the second information, wherein the reference block is identical to the second data block; the first device compressing the difference between the first data block and the reference block to obtain first compression information; the first device sending the first compression information to the second device; and the second device obtaining the first data block based on the first compression information and the second data block. This achieves similarity deduplication technology. For a data block to be sent, if there is a similar data block that has already been transmitted, only the difference between the two data blocks is transmitted, effectively reducing the data link and improving the transmission efficiency of data synchronization technology.

[0354] From a hardware structure perspective, the above method can be implemented by a single physical device, multiple physical devices working together, or a logical functional module within a single physical device. This application does not specifically limit this aspect.

[0355] For example, the above method can be used Figure 12 This is achieved through electronic devices within the system. Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application; the electronic device can be the electronic device in the embodiment of this invention, or it can be a terminal device. The electronic device includes at least one processor 1201, a communication line 1202, a memory 1203, and at least one communication interface 1204.

[0356] The processor 1201 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (server IC), or one or more integrated circuits used to control the execution of the program of the present application.

[0357] Communication line 1202 may include a path for transmitting information between the aforementioned components.

[0358] Communication interface 1204 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0359] The memory 1203 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via communication line 1202. The memory may also be integrated with the processor.

[0360] The memory 1203 stores computer execution instructions for implementing the scheme of this application, and the processor 1201 controls the execution. The processor 1201 executes the computer execution instructions stored in the memory 1203, thereby implementing the billing management method provided in the following embodiments of this application.

[0361] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0362] In a specific implementation, as one embodiment, the processor 1201 may include one or more CPUs, for example... Figure 12 CPU0 and CPU1 in the CPU.

[0363] In a specific implementation, as one example, an electronic device may include multiple processors, for example... Figure 12 Processors 1201 and 1207 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0364] In a specific implementation, as one embodiment, the electronic device may further include an output device 1205 and an input device 1206. The output device 1205 communicates with the processor 1201 and can display information in various ways. For example, the output device 1205 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 1206 communicates with the processor 1201 and can receive user input in various ways. For example, the input device 1206 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0365] The aforementioned electronic device can be a general-purpose device or a special-purpose device. In specific implementations, the electronic device can be a server, a wireless terminal device, an embedded device, or something else. Figure 12 Devices with similar structures. The embodiments of this application do not limit the type of electronic device.

[0366] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0367] For example, when dividing the functional units using an integrated approach. Figure 13 A schematic diagram of the structure of a data transmission system provided in an embodiment of this application is shown.

[0368] like Figure 13As shown in the embodiment of this application, a data transmission system includes: a first device and a second device, wherein the second device is used to receive data sent by the first device, and the first device includes:

[0369] The first sending unit 1301 is used to send first information of a first data block to a second device. The first data block is a data block to be sent by the first device to the second device. The first information is used to describe the attributes of the first data block.

[0370] The second device includes:

[0371] The lookup unit 1302 is used to search for a second data block similar to the first data block from a first index based on the first information sent by the sending unit. The first index is used to record information about data blocks that the second device has already received from the first device.

[0372] The second sending unit 1303 is used to send the second information of the second data block found by the lookup unit 1302 to the first device. The second information is used to describe the attributes of the second data block.

[0373] The first device also includes:

[0374] Acquisition unit 1304 is used to acquire a reference block from the local source based on the second information, the reference block being the same as the second data block;

[0375] Compression unit 1305 is used to compress the difference between the first data block and the reference block acquired by the acquisition unit 1304 to obtain first compression information;

[0376] The first sending unit 1301 is also used to send the first compressed information to the second device;

[0377] The second device also includes:

[0378] The decompression unit 1306 is used to obtain the first data block based on the first compression information and the second data block.

[0379] Optionally, the first information includes a first similar fingerprint, which is a weak hash value of the first data block; the first index records a first correspondence between the similar fingerprints of the data blocks and the second address of the data blocks in the second device; the lookup unit 1302 is further configured to:

[0380] Search the first index for a second similar fingerprint that is the same as the first similar fingerprint. The second similar fingerprint is the similar fingerprint of the second data block.

[0381] The second address is obtained from the first correspondence based on the second similar fingerprint, and the second address is the second address of the second data block in the second device;

[0382] The second data block is obtained based on the second address.

[0383] Optionally, the second information may also include the storage address of the reference block in the first device; the acquisition unit 1304 is further configured to:

[0384] The reference block is retrieved locally based on its storage address in the first device.

[0385] Optionally, the first correspondence includes the correspondence between strong fingerprints and similar fingerprints and the second address, where the strong fingerprint is a strong hash value of the data block, and the strong hash value is used to describe the data attributes of the corresponding data block; when the second device finds M target similar fingerprints that are the same as the first similar fingerprint in the first index, where M is a positive integer greater than 1, the second device further includes a filtering unit 1307, which is used for:

[0386] Based on the M target similar fingerprints, obtain the M target addresses corresponding to the M target similar fingerprints from the first correspondence;

[0387] Obtain M target data blocks based on the M target addresses:

[0388] Obtain the strong fingerprints of the M target data blocks;

[0389] One target data block is obtained from the M target data blocks as the second data block, and the strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

[0390] Optionally, when the second device obtains N data blocks similar to the first data block from the M target data blocks, the second information includes strong fingerprints of the N candidate data blocks, tag information, and the first addresses of the N candidate data blocks in the first device, wherein the N candidate data blocks are each similar to the first data block, and the strong fingerprints of the N candidate data blocks are the strong hash values ​​of the N candidate data blocks; N is a positive integer greater than 1; the obtaining unit 1304 is further configured to:

[0391] The N candidate data blocks are retrieved from the local storage based on their first addresses in the first device.

[0392] Obtain the strong fingerprints of the N candidate data blocks;

[0393] One target candidate data block is obtained from the N candidate data blocks as the base block, and the strong fingerprint of the target candidate data block is the same as the strong fingerprint recorded in the second information.

[0394] The first transmitting unit 1301 is also used for:

[0395] The tagging information of the target candidate data block is sent to the second device;

[0396] The decompression unit 1306 is also used for:

[0397] The target candidate data block is retrieved from the local machine based on its tagging information.

[0398] The first data block is obtained based on the first compression information and the target candidate data block.

[0399] Optionally, the first compression information includes a first compression block and difference information, wherein the first compression block includes the difference portion between the first data block and the reference block, and the difference information is used to describe the difference between the first data block and the reference block; the decompression unit 1306 is further configured to:

[0400] The first data block is obtained based on the first compressed block, the difference information, and the second data block.

[0401] Furthermore, such as Figure 14 As shown in the embodiment of this application, a data transmission device includes:

[0402] The acquisition unit 1401 is used to acquire first information of a first data block, the first data block being a data block to be sent by the data transmitting device to a second device, the second device being used to receive the data block sent by the data transmitting device, and the first information being used to describe the attributes of the first data block.

[0403] The lookup unit 1402 is used to look up a second data block similar to the first data block in a first index based on the first information obtained by the acquisition unit 1401. The first index is used to record information about data blocks that the data sending device has sent to the second device.

[0404] Compression unit 1403 is used to compress the difference between the first data block and the second data block searched by the lookup unit 1402 to obtain first compressed information.

[0405] The sending unit 1404 is used to send the first compression information obtained by the compression unit 1403 to the second device, so that the second device can obtain the first data block according to the first compression information.

[0406] Optionally, the first information includes a first similar fingerprint, which is a weak hash value of the first data block, and the first index records a first correspondence between the similar fingerprints of the data blocks and the storage addresses of the data blocks in the first device; then the lookup unit 1402 is further configured to:

[0407] Search the first index for a second similar fingerprint that is identical to the first similar fingerprint. The second similar fingerprint is the weak hash value of the second data block.

[0408] The second address is obtained from the first correspondence based on the second similar fingerprint, and the second address is the storage address of the second data block in the first device;

[0409] The second data block is obtained based on the second address.

[0410] Optionally, the first correspondence is the correspondence between strong fingerprints and similar fingerprints and the storage address, where the strong fingerprint is a strong hash value of the data block, and the strong hash value is used to describe the data attributes of the corresponding data block; when the first device finds M target similar fingerprints that are the same as the first similar fingerprint in the first index, where M is a positive integer greater than 1, the device further includes a filtering unit 1405, which is used for:

[0411] Based on the M target similar fingerprints, obtain the M target addresses corresponding to the M target similar fingerprints from the first correspondence;

[0412] Obtain M target data blocks based on the M target addresses:

[0413] Obtain the strong fingerprints of the M target data blocks;

[0414] One target data block is obtained from the M target data blocks as the second data block, and the strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

[0415] Optionally, the first compression information includes a first compression block and difference information, wherein the first compression block includes the difference portion between the first data block and the second data block, and the difference information is used to describe the difference between the first data block and the second data block; so that the second device obtains the first data block based on the first compression block, the difference information and the reference block, wherein the reference block is the same data block in the second device as the second data block.

[0416] Furthermore, such as Figure 15 As shown in the figure, an embodiment of this application provides a data receiving device, including:

[0417] The receiving unit 1501 is used to obtain first compression information from the first device. The first compression information is information obtained by the first device compressing the difference between the first data block and the second data block. The first data block is the data block to be sent by the first device to the data receiving device, and the second data block is the data block that has been sent by the first device to the data receiving device.

[0418] Acquisition unit 1502 is used to acquire a reference block from the local device according to the first compression information, the reference block being the same as the second data block in the first device;

[0419] The decompression unit 1503 is used to obtain the first data block based on the first compression information received by the receiving unit 1501 and the reference block obtained by the acquiring unit 1502.

[0420] Optionally, the first compression information includes a first compression block and difference information, wherein the first compression block includes the difference portion between the first data block and the second data block, and the difference information is used to describe the differences between the first data block and the second data block; the decompression unit 1503 is further configured to:

[0421] The first data block is obtained based on the first compressed block, the difference information, and the reference block.

[0422] Optionally, the first compression information includes the storage address of the reference block in the second device, and the acquisition unit 1502 is further configured to:

[0423] The reference block is retrieved locally based on its storage address in the second device.

[0424] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0425] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0426] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0427] In the several embodiments provided in this application, it should be understood that the disclosed communication methods, relay devices, host base stations, and computer storage media can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0428] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0429] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0430] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0431] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data transmission method, characterized by, The method comprises the following steps: A first device sends first information of a first data block to a second device, the first data block being a data block to be sent by the first device to the second device, the first information being used to describe attributes of the first data block; The second device finds second information of a second data block similar to the first data block from a first index according to the first information, the first index being used to record information of data blocks that have been received by the second device from the first device; The second device sends the second information of the second data block to the first device, the second information being used to describe attributes of the second data block; The first device acquires a reference block from the local device according to the second information, the reference block being identical to the second data block; The first device compresses a difference part of the first data block relative to the reference block to obtain first compression information; The first device sends the first compression information to the second device; The second device obtains the first data block according to the first compression information and the second data block; The first information comprises a first similar fingerprint, the first similar fingerprint being a weak hash value of the first data block; the first index records a first correspondence relationship between similar fingerprints of data blocks and second addresses of the data blocks in the second device; the second address is a second address of the second data block in the second device; the first correspondence relationship comprises a correspondence relationship between strong fingerprints and similar fingerprints and the second addresses, the strong fingerprint being a strong hash value of a data block, the strong hash value being used to describe data attributes of the corresponding data block; when the second device finds M target similar fingerprints identical to the first similar fingerprint in the first index, the M being a positive integer greater than 1, the method further comprises the following steps: The second device acquires M target addresses corresponding to the M target similar fingerprints from the first correspondence relationship according to the M target similar fingerprints; The second device acquires M target data blocks according to the M target addresses; The second device acquires strong fingerprints of the M target data blocks; The second device acquires a target data block from the M target data blocks as the second data block, the strong fingerprint of the second data block being identical to the strong fingerprint recorded in the first index.

2. The method of claim 1, wherein, The second device acquires the second data block similar to the first data block according to the first information, comprising the following steps: The second device finds a second similar fingerprint identical to the first similar fingerprint in the first index, the second similar fingerprint being a similar fingerprint of the second data block; The second device acquires a second address from the first correspondence relationship according to the second similar fingerprint; The second device acquires the second data block according to the second address.

3. The method of claim 1, wherein, The second information further comprises a storage address of the reference block in the first device; The first device acquires the reference block from the local device according to the second information, comprising the following step: The first device acquires the reference block from the local device according to the storage address of the reference block in the first device.

4. The method of claim 1, wherein, When the second device obtains N data blocks similar to the first data block from the M target data blocks, the second information comprises strong fingerprints of the N candidate data blocks, tag information of the N candidate data blocks and first addresses of the N candidate data blocks in the first device, wherein the N candidate data blocks are similar to the first data block respectively, the strong fingerprints of the N candidate data blocks are strong hash values of the N candidate data blocks respectively, the N is a positive integer greater than 1; the first device obtains a reference block from the local according to the second information, comprising: The first device obtains the N candidate data blocks from the local storage according to the first addresses of the N candidate data blocks in the first device; The first device obtains the strong fingerprints of the N candidate data blocks; The first device obtains a target candidate data block from the N candidate data blocks as the reference block, the strong fingerprint of the target candidate data block is the same as the strong fingerprint recorded in the second information; The first device sends the first compressed information to the second device, comprising: The first device sends the tag information of the target candidate data block to the second device; The second device obtains the first data block according to the first compressed information and the candidate data block, comprising: The second device obtains the target candidate data block from the local according to the tag information of the target candidate data block; The second device obtains the first data block according to the first compressed information and the target candidate data block.

5. The method according to any one of claims 1 to 4, characterized in that, The first compressed information comprises a first compressed block and difference information, wherein the first compressed block comprises a difference part of the first data block and the reference block, and the difference information is used to describe the difference between the first data block and the reference block; the second device obtains the first data block according to the first compressed information and the second data block, comprising: The second device obtains the first data block according to the first compressed block, the difference information and the second data block.

6. A data transmission method, characterized by, Comprising: The first device obtains first information of a first data block, the first data block is a data block to be sent by the first device to a second device, the second device is used to receive the data block sent by the first device, and the first information is used to describe the attribute of the first data block; The first device finds a second data block similar to the first data block according to the first information and a first index, the first index is used to record information of the data block sent by the first device to the second device; The first device compresses a difference part of the first data block relative to the second data block to obtain first compressed information; The first device sends the first compressed information to the second device, so that the second device obtains the first data block according to the first compressed information; The first device sends the first compressed information to the second device, so that the second device obtains the first data block according to the first compressed information; The first information includes a first similar fingerprint, the first similar fingerprint is a weak hash value of the first data block, and the first index records a first correspondence relationship between a similar fingerprint of a data block and a storage address of the data block in the first device; the first correspondence relationship is a correspondence relationship between a strong fingerprint and the storage address, the strong fingerprint is a strong hash value of the data block, and the strong hash value is used to describe a data attribute of the corresponding data block; when the first device finds M target similar fingerprints same as the first similar fingerprint in the first index, the M is a positive integer greater than 1, and the method further includes: The first device obtains M target addresses corresponding to the M target similar fingerprints from the first correspondence relationship according to the M target similar fingerprints; The first device obtains M target data blocks according to the M target addresses: The first device obtains a strong fingerprint of the M target data blocks; The first device obtains a target data block from the M target data blocks as the second data block, and a strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

7. The method of claim 6, wherein, The first device finds a second data block similar to the first data block in the first index according to the first information, including: The first device finds a second similar fingerprint same as the first similar fingerprint in the first index, the second similar fingerprint is a weak hash value of the second data block; The first device obtains a second address of the second data block in the first device from the first correspondence relationship according to the second similar fingerprint; The first device obtains the second data block according to the second address.

8. The method of any one of claims 6-7, wherein, The first compression information includes a first compression block and difference information, wherein the first compression block includes a difference part of the first data block and the second data block, and the difference information is used to describe the difference between the first data block and the second data block; so that the second device obtains the first data block according to the first compression block, the difference information and a reference block, wherein the reference block is a data block same as the second data block in the second device.

9. A data transmission method, characterized by, Including: The second device obtains first compression information from the first device, the first compression information is information obtained by the first device compressing a difference part of a first data block relative to a second data block, the first data block is a data block to be sent by the first device to the second device, and the second data block is a data block similar to the first data block among data blocks already sent by the first device to the second device, wherein the second data block is obtained based on information of the first data block and a first index, and the first index is used to record information of data blocks already sent by the first device to the second device; The second device obtains a reference block from the local according to the first compression information, and the reference block is same as the second data block in the first device; The second device obtains the first data block according to the first compression information and the reference block; The first index records a first correspondence relationship between a similar fingerprint of a data block and a second address of the data block in the second device; the second address is a second address of the second data block in the second device; the first correspondence relationship includes a correspondence relationship between a strong fingerprint and the second address, the strong fingerprint is a strong hash value of a data block, and the strong hash value is used to describe a data attribute of the corresponding data block; when the second device finds M target similar fingerprints same as a first similar fingerprint in the first index, the M is a positive integer greater than 1, the first similar fingerprint is a weak hash value of the first data block, and the method further includes: The second device obtains M target addresses corresponding to the M target similar fingerprints from the first correspondence relationship according to the M target similar fingerprints; The second device obtains M target data blocks according to the M target addresses; The second device obtains a strong fingerprint of the M target data blocks; The second device obtains a target data block from the M target data blocks as the second data block, and a strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

10. The method of claim 9, wherein, The first compression information includes a first compression block and difference information, wherein the first compression block includes a difference part of the first data block and the second data block, and the difference information is used to describe a difference between the first data block and the second data block; the second device obtains the first data block according to the first compression information and the reference block, including: The second device obtains the first data block according to the first compression block, the difference information and the reference block.

11. The method according to claim 9 or 10, characterized in that, The first compression information includes a storage address of the reference block in the second device, and the second device obtains the reference block from the local according to the first compression information, including: The second device obtains the reference block from the local according to the storage address of the reference block in the second device.

12. A data transmission system, characterized by The system includes a first device and a second device, wherein the second device is used to receive data sent by the first device, and the first device includes: A first sending unit, the first sending unit is used to send first information of a first data block to a second device, the first data block is a data block to be sent by the first device to the second device, and the first information is used to describe an attribute of the first data block; The second device includes: A finding unit, the finding unit is used to find second information of a second data block similar to the first data block from a first index according to the first information sent by the sending unit, and the first index is used to record information of data blocks already received by the second device from the first device; A second sending unit, the second sending unit is used to send the second information of the second data block found by the finding unit to the first device, and the second information is used to describe an attribute of the second data block; The first device further includes: an obtaining unit, configured to obtain a reference block from the local according to the second information, the reference block being the same as the second data block; a compression unit, configured to compress a difference part of the first data block relative to the reference block obtained by the obtaining unit to obtain first compression information; the first sending unit is further configured to send the first compression information to the second device; the second device further includes: a decompression unit, configured to obtain the first data block according to the first compression information and the second data block; wherein the first information includes a first similar fingerprint, the first similar fingerprint being a weak hash value of the first data block; the first index records a first correspondence relationship between a similar fingerprint of a data block and a second address of the data block in the second device; the second address is a second address of the second data block in the second device; the first correspondence relationship includes a correspondence relationship between a strong fingerprint and a similar fingerprint and the second address, the strong fingerprint being a strong hash value of a data block, the strong hash value being used to describe a data attribute of the corresponding data block; the finding unit of the second device is configured to, when the second device finds M target similar fingerprints same as the first similar fingerprint in the first index, the M being a positive integer greater than 1, obtain, according to the M target similar fingerprints, M target addresses corresponding to the M target similar fingerprints from the first correspondence relationship; obtain, according to the M target addresses, M target data blocks; obtain strong fingerprints of the M target data blocks; and obtain, from the M target data blocks, a target data block as the second data block, the strong fingerprint of the second data block being consistent with the strong fingerprint recorded in the first index.

13. A data transmission device, characterized in that, comprise: an obtaining unit, configured to obtain first information of a first data block, the first data block being a data block to be sent by the data sending device to a second device, the second device being configured to receive data blocks sent by the data sending device, the first information being used to describe an attribute of the first data block; a finding unit, configured to find a second data block similar to the first data block according to the first information obtained by the obtaining unit and a first index, the first index being used to record information of data blocks that have been sent by the data sending device to the second device; a compression unit, configured to compress a difference part of the first data block relative to the second data block found by the finding unit to obtain first compression information; a sending unit, configured to send the first compression information obtained by the compression unit to the second device, so that the second device obtains the first data block according to the first compression information. The first information includes a first similar fingerprint, the first similar fingerprint is a weak hash value of the first data block, and the first index records a first correspondence relationship between a similar fingerprint of a data block and a storage address of the data block in the data sending device; the first correspondence relationship is a correspondence relationship between a strong fingerprint and the similar fingerprint and the storage address, the strong fingerprint is a strong hash value of the data block, and the strong hash value is used to describe a data attribute of a corresponding data block; When the data sending device finds M target similar fingerprints same as the first similar fingerprint in the first index, M is a positive integer greater than 1, the data sending device obtains M target addresses corresponding to the M target similar fingerprints from the first correspondence relationship according to the M target similar fingerprints, the data sending device obtains M target data blocks according to the M target addresses, and the data sending device obtains a strong fingerprint of the M target data blocks. The data sending device obtains a target data block from the M target data blocks as the second data block, and a strong fingerprint of the second data block is consistent with a strong fingerprint recorded in the first index.

14. A data receiving device, characterized by Comprise: A receiving unit is configured to obtain first compressed information from a first device, the first compressed information being information obtained by the first device by compressing a difference part of a first data block relative to a second data block, the first data block being a data block to be sent by the first device to a data receiving device, and the second data block being a data block similar to the first data block and already sent by the first device to the data receiving device, wherein the second data block is obtained by searching based on information of the first data block and a first index, and the first index is configured to record information of data blocks already sent by the first device to the data receiving device; An obtaining unit is configured to obtain a reference block from a local device according to the first compressed information, the reference block being the same as the second data block in the first device; A decompression unit is configured to obtain the first data block according to the first compressed information received by the receiving unit and the reference block obtained by the obtaining unit. The first index records a first correspondence relationship between a similar fingerprint of a data block and a second address of the data block in the data receiving device; the second address is a second address of the second data block in the data receiving device; the first correspondence relationship includes a correspondence relationship between a strong fingerprint and the similar fingerprint and the second address, the strong fingerprint is a strong hash value of the data block, and the strong hash value is used to describe a data attribute of a corresponding data block; when the data receiving device finds M target similar fingerprints same as the first similar fingerprint in the first index, M is a positive integer greater than 1, the first similar fingerprint is a weak hash value of the first data block, and the data receiving device further comprises: The selecting unit is configured to: acquire M target addresses corresponding to the M target similar fingerprints from the first correspondence according to the M target similar fingerprints; acquire M target data blocks according to the M target addresses; acquire strong fingerprints of the M target data blocks; and acquire one target data block from the M target data blocks as the second data block, wherein the strong fingerprint of the second data block is consistent with the strong fingerprint recorded in the first index.

15. An electronic device, comprising: The electronic device comprises an interaction device, an input / output (I / O) interface, a processor and a memory, and the memory stores program instructions; The interaction device is configured to acquire an operation instruction input by a user; The processor is configured to execute the program instructions stored in the memory to perform the method according to any one of claims 1-5.

16. An electronic device, comprising: The electronic device comprises an interaction device, an input / output (I / O) interface, a processor and a memory, and the memory stores program instructions; The interaction device is configured to acquire an operation instruction input by a user; The processor is configured to execute the program instructions stored in the memory to perform the method according to any one of claims 6-8.

17. An electronic device, comprising: The electronic device comprises an interaction device, an input / output (I / O) interface, a processor and a memory, and the memory stores program instructions; The interaction device is configured to acquire an operation instruction input by a user; The processor is configured to execute the program instructions stored in the memory to perform the method according to any one of claims 9-11.

18. A computer-readable storage medium comprising instructions, wherein, When the instructions are run on the computer device, the computer device is caused to perform the method according to any one of claims 1-5.

19. A computer-readable storage medium comprising instructions, wherein, When the instructions are run on the computer device, the computer device is caused to perform the method according to any one of claims 6-8.

20. A computer-readable storage medium comprising instructions, wherein: When the instructions are run on the computer device, the computer device is caused to perform the method according to any one of claims 9-11.

Images