Encryption difference compressed data restoration method and device based on similarity perception

Abstract

The invention belongs to the technical field of computer storage, and particularly discloses an encryption difference compressed data restoration method and device based on similarity perception. According to the method and the device, the differential data block and the reference block which are subjected to aggregation storage are read from the continuous disk space at one time; in the data recovery process, pipelining processing is carried out on a difference decoding task and an erasure code decoding task; and performing data restoration according to the size of the data slice, the difference data block and the reference block by executing the pipelined difference decoding task and the pipelined erasure code decoding task. According to the mode, the whole aggregation storage data block containing the differential data block and the reference block is read from the continuous disk space, compared with respective reading, one-time extra disk overhead can be reduced, then the differential decoding task and the erasure code decoding task are subjected to pipelining processing, and the decoding efficiency is improved. And data recovery is carried out by executing the decoding task, so that the data recovery performance of the encryption difference compression system can be effectively improved.

Description

Technical Field

[0001] This application belongs to the field of computer storage technology, and more specifically, relates to a method and apparatus for repairing encrypted differential compressed data based on similarity awareness. Background Technology

[0002] With the development of technologies such as artificial intelligence and big data, the global data storage volume is growing rapidly, and the demand for data storage is increasing daily. However, traditional self-managed storage methods by enterprises can no longer meet their needs due to their high storage costs and complex management. Therefore, more and more enterprises are choosing to outsource their data storage to cloud storage service providers. This type of storage system is called an outsourced storage system, and they hope to ensure that their data is not stolen and to minimize storage costs as much as possible. To this end, encrypted differential compression systems have emerged, which can reduce storage redundancy to a minimum while ensuring data security.

[0003] With the ever-expanding volume of data, data corruption or loss is quite common in outsourced storage. This necessitates that outsourced storage systems possess excellent data recovery capabilities to ensure rapid data restoration in the event of corruption or loss.

[0004] However, current encrypted differential compression systems have the following problems when performing data repair: (1) Data repair requires two processes: differential decoding and erasure coding decoding. Erasure coding decoding can only be performed after all differential blocks have completed differential decoding, which affects the performance of data repair.

[0005] (2) The strategy for data storage is to randomly write to the storage backend data segment. For blocks that have undergone differential compression, this will likely result in the differential block and the base block not being in the same data segment. During data repair, an additional disk I / O operation is required, which reduces the differential decoding performance.

[0006] In summary, the aforementioned encrypted differential compression system exhibits poor performance in data repair. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this application is to provide a similarity-aware encrypted differential compression data repair method and apparatus, aiming to solve the problem of poor performance of existing encrypted differential compression systems in data repair.

[0008] To achieve the above objectives, in a first aspect, this application provides a similarity-aware encrypted differential compression data repair method, comprising: In response to a data repair command triggered by a user in the encrypted differential compression system, the differential data block and the base block, which have been aggregated and stored, are read from contiguous disk space at once. During the data repair process, the differential decoding task and erasure coding decoding task are pipelined, and the data slice size required for the pipeline is determined based on the file similarity at the time of data upload. By executing the differential decoding task and erasure coding decoding task after pipelined processing, data repair is performed based on the data slice size, the differential data block, and the reference block.

[0009] In one embodiment, prior to the step of reading the aggregated differential data block and the reference block from contiguous disk space at once, the method further includes: During data storage, retrieve the data block adjacent to the duplicate data block; The adjacent data blocks are marked as candidate reference blocks, and the candidate reference blocks are cached in the memory aggregation buffer; Obtain the differential data block to be stored, and calculate the similarity between the differential data block and each candidate reference block in the memory aggregation buffer; Identify target candidate reference blocks with similarity greater than a preset threshold, and cache the difference data blocks in a cache location adjacent to the target candidate reference blocks; When the data transfer is complete, the target candidate reference block and differential data block in the memory aggregation buffer are aggregated and stored in a contiguous disk space.

[0010] In one embodiment, the step of reading aggregated differential data blocks and a base block from contiguous disk space at once in response to a data repair command triggered by a user in an encrypted differential compression system includes: In response to a data repair command triggered by a user in the encrypted differential compression system, determine the data block reading parameters; In a single I / O operation, aggregated storage data blocks are read from contiguous disk space at once, based on the data block read parameters. The aggregated storage data block is divided according to the metadata to obtain the differential data block and the base block after aggregated storage.

[0011] In one embodiment, the step of pipelined processing of differential decoding and erasure coding decoding tasks during data repair includes: During the data repair process, obtain the size of the data slice decoded by the differential decoding task; When the size of the decoded data slice reaches the data slice size required by the pipeline, the decoded data slice is sent to the next storage backend, and erasure coding decoding task is performed based on the next storage backend; While performing erasure coding decoding, differential decoding continues until the size of the decoded data slice reaches the size required by the pipeline, thus completing the pipelined processing of differential decoding and erasure coding decoding tasks.

[0012] In one embodiment, the step of determining the required data slice size for the pipeline based on file similarity during data upload includes: When performing an erasure coding decoding task and a single-block differential decoding task, the erasure coding decoding time and the single-block differential decoding time are obtained respectively. During data upload, determine the number and size of similar blocks in the file; Calculate the average size of similar blocks based on the number and size of similar blocks in the file; The required data slice size for the pipeline is calculated based on the erasure coding decoding time, the single-block differential decoding time, the average size of similar blocks, and the file similarity during data upload.

[0013] In one embodiment, before the step of calculating the data slice size required for the pipeline based on the erasure coding decoding time, the single-block differential decoding time, the average size of similar blocks, and the file similarity at the time of data upload, the method further includes: Determine the file size when uploading data; The file similarity at the time of data upload is calculated based on the file size, the number of similar blocks in the file, and the average size of the similar blocks.

[0014] Secondly, this application provides a similarity-aware encrypted differential compression data repair device, comprising: The read module is used to read the aggregated differential data blocks and the base block from contiguous disk space in response to the data repair command triggered by the user in the encrypted differential compression system. The determination module is used to pipeline the differential decoding task and erasure coding decoding task during the data repair process, and to determine the data slice size required for the pipeline based on the file similarity at the time of data upload. The repair module is used to repair data based on the data slice size, the differential data block, and the reference block by executing the differential decoding task and erasure coding decoding task after pipelined processing.

[0015] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0016] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0017] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0019] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) This application introduces a pipeline in the data repair process of the encrypted differential compression system to pipeline the differential decoding task and the erasure coding decoding task. Specifically, after the differential decoding task is executed, the erasure coding decoding task can also be executed. At the same time, differential decoding can continue to be performed on the slice, thereby pipelined the differential decoding task and the erasure coding decoding task, which can effectively improve the data repair performance of the encrypted differential compression system from the perspective of efficiency.

[0020] (2) In the data storage process, this application aggregates the base block and differential data block into a contiguous disk space and marks them accordingly in the metadata. When data repair is required, the entire aggregated storage data block containing the differential data block and the base block is read from the contiguous disk space. Compared with reading them separately, this reduces an additional disk overhead, thereby effectively improving the differential decoding performance of the encrypted differential compression system.

[0021] In summary, this application responds to a data repair command triggered by a user in an encrypted differential compression system by reading the aggregated storage differential data block and the base block from contiguous disk space in one go. During the data repair process, the differential decoding task and the erasure coding decoding task are pipelined, and the data slice size required for the pipeline is determined based on the file similarity at the time of data upload. By executing the pipelined differential decoding task and erasure coding decoding task, data repair is performed based on the data slice size, the differential data block, and the base block. Through this method, when data repair is required, the encrypted differential compression system reads the entire aggregated storage data block containing the differential data block and the base block from contiguous disk space, reducing one additional disk overhead compared to reading them separately. Then, the differential decoding task and the erasure coding decoding task are pipelined, and data repair is performed by executing the decoding task, thereby effectively improving the data repair performance of the encrypted differential compression system. Attached Figure Description

[0022] Figure 1 This is one of the flowcharts of the encryption differential compression data repair method based on similarity awareness provided in the embodiments of this application; Figure 2 This is a schematic diagram of aggregated storage provided in an embodiment of this application; Figure 3 This is a schematic diagram of task pipeline provided in an embodiment of this application; Figure 4 This is the second flowchart of the encryption differential compression data repair method based on similarity awareness provided in the embodiments of this application; Figure 5 This is a schematic diagram of the module structure of the encryption differential compression data repair device based on similarity awareness provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0025] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0026] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0027] Based on this, embodiments of this application provide a method for repairing encrypted differential compressed data based on similarity awareness, referring to... Figure 1 , Figure 1 This is one of the flowcharts illustrating the similarity-aware encrypted differential compressed data repair method provided in this application embodiment. In this embodiment, the similarity-aware encrypted differential compressed data repair method includes steps S10 to S30: Step S10: In response to the data repair command triggered by the user in the encrypted differential compression system, read the aggregated differential data block and the base block from the contiguous disk space at once.

[0028] It should be noted that the data repair command refers to the command initiated by a user within the enterprise to perform data repair using the encrypted differential compression system. This data repair command can be triggered in the operation interface of the encrypted differential compression system. Since the storage base block and differential data block are aggregated in this embodiment, when data repair is required using differential data block and base block, the entire large block (differential data block and base block) stored after aggregation can be read from the contiguous disk space at once, instead of reading the differential data block and base block separately. In comparison, the reading method of this embodiment can reduce an additional disk I / O overhead.

[0029] Furthermore, before the step of reading the aggregated differential data block and the reference block from the contiguous disk space at once, the method further includes: during data storage, acquiring data blocks adjacent to the duplicate data block; marking the adjacent data blocks as candidate reference blocks and caching the candidate reference blocks in a memory aggregation buffer; acquiring the differential data block to be stored and calculating the similarity between the differential data block and each candidate reference block in the memory aggregation buffer; determining a target candidate reference block with a similarity greater than a preset threshold and caching the differential data block in a cache location adjacent to the target candidate reference block; and when the data transmission is completed, aggregating and storing the target candidate reference block and the differential data block in the memory aggregation buffer into the contiguous disk space.

[0030] It should be understood that, in order to ensure that the aggregated differential data blocks and base blocks can be read from contiguous disk space at once, this embodiment needs to aggregate and store the target candidate base block and differential data blocks into contiguous disk space during data storage. Since adjacent data blocks of duplicate data blocks are highly likely to be similar to other data blocks, this embodiment marks adjacent data blocks as candidate base blocks for duplicate data blocks during data storage. For example, refer to... Figure 2 , Figure 2 This is a diagram illustrating aggregated storage. Specifically, if data stream 1 is detected... In data blocks and data streams If data blocks are duplicated, then... Adjacent non-repeating blocks and Stored in the aggregation block buffer as a candidate reference block, when subsequent... and When the data block arrives, it is discovered and Data blocks and memory aggregation buffers and If the data blocks are similar, then these two blocks will be compared with... and They are stored separately in aggregate form.

[0031] Understandably, these candidate reference blocks will not be directly written to the hard drive. Instead, they will first be placed in a memory aggregation buffer. When the difference data block to be stored is similar to a candidate reference block (i.e., the similarity between the difference data block and each candidate reference block in the memory aggregation buffer exceeds a preset threshold), this candidate reference block will be used as the target candidate reference block. The target candidate reference block and the difference data block in the memory aggregation buffer will then be stored together. Furthermore, the memory aggregation buffer can be managed using the Least Recently Used (LRU) algorithm. When the memory aggregation buffer is full, the least recently used aggregated storage data block will be written to the hard drive. When the data transfer is complete, all data in the memory aggregation buffer will be written to the hard drive, and the difference data block to be stored and its target candidate reference block will be stored in adjacent locations on the disk so that they can be read in a single I / O operation, thus reducing an additional disk I / O overhead.

[0032] It is important to emphasize that, in order to effectively improve the accuracy of determining the target candidate benchmark blocks, this embodiment uses similarity to measure whether the differential data block is similar to each candidate benchmark block in the memory aggregation buffer. That is, when the similarity is greater than a preset threshold, it indicates similarity; conversely, when the similarity is less than or equal to the preset threshold, it indicates dissimilarity. The algorithm used to calculate the similarity can be a similarity hash algorithm, a machine learning-based similarity algorithm, etc.

[0033] Step S20: During the data repair process, the differential decoding task and erasure coding decoding task are pipelined, and the data slice size required for the pipeline is determined based on the file similarity at the time of data upload.

[0034] It should be understood that, in order to effectively improve the data repair performance of the encrypted differential compression system from an efficiency perspective, this embodiment pipelines the differential decoding task and the erasure coding decoding task during the data repair process. This allows the differential decoding task to be executed first, followed by erasure coding decoding to perform erasure coding decoding on the complete data slice obtained from the differential decoding. Simultaneously, differential decoding continues on the acquired differential data slice. Compared to existing encrypted differential compression systems, this embodiment does not require all differential blocks to be differentially decoded before erasure coding decoding can be performed.

[0035] Furthermore, the step of pipelined processing of differential decoding and erasure coding decoding tasks during data repair includes: obtaining the size of the data slice decoded by the differential decoding task during data repair; when the size of the decoded data slice reaches the data slice size required for the pipeline, sending the decoded data slice to the next storage backend, and performing the erasure coding decoding task based on the next storage backend; while performing the erasure coding decoding task, continuing to perform differential decoding until the size of the decoded data slice reaches the data slice size required for the pipeline, thereby completing the pipelined processing of differential decoding and erasure coding decoding tasks.

[0036] It is understandable that differential decoding and erasure coding decoding tasks need to be performed during data repair. In order to effectively improve the data repair performance of the encrypted differential compression system from the perspective of efficiency, this embodiment pipelines the differential decoding and erasure coding decoding tasks. Specifically, after obtaining the size of the data slice decoded by the differential decoding task, it is determined whether the size of the decoded data slice meets the data slice size required by the pipeline. If so, it indicates that the erasure coding decoding task needs to be performed. At this time, the decoded data slice is sent to the next storage backend. The erasure coding decoding task is performed based on the next storage backend to realize the erasure coding decoding of the data slice. At the same time, the differential decoding task will continue to be executed to perform differential decoding on subsequent data slices until the size of the decoded data slice meets the data slice size required by the pipeline again. The above steps are repeated to pipeline the differential decoding and erasure coding decoding tasks required for data repair.

[0037] For example, see reference. Figure 3 , Figure 3 This is a diagram illustrating task pipeline implementation, specifically: when it is necessary to... Retrieve data from the server and send it to When using the server, first start from The server extracts a partial differential data slice (tens or hundreds of KB) from the required data block. Then, it performs differential decoding on this data slice by executing a differential decoding task. It then determines whether the size of the decoded data slice (the complete data slice) meets the required data slice size for the pipeline. If so, the decoded data slice is sent to the next storage backend, and an erasure coding decoding task is performed based on the next storage backend. The result is then sent to... The server, at the same time, will also be from The server continues to acquire differential data slices and performs differential decoding by executing differential decoding tasks. In this way, the differential decoding task and erasure coding decoding task are pipelined. When using the encrypted differential compression system for data repair, the data repair performance of the encrypted differential compression system can be effectively improved from the perspective of efficiency.

[0038] Furthermore, the step of determining the required data slice size for the pipeline based on file similarity during data upload includes: acquiring the erasure coding decoding time and the single-block differential decoding time when performing an erasure coding decoding task and a single-block differential decoding task, respectively; determining the number and size of similar blocks in the file during data upload; calculating the average size of similar blocks based on the number and size of similar blocks in the file; and calculating the required data slice size for the pipeline based on the erasure coding decoding time, the single-block differential decoding time, the average size of similar blocks, and the file similarity during data upload.

[0039] It should be understood that erasure coding decoding time refers to the time required to perform one erasure coding decoding task. Similarly, single-block differential decoding time refers to the time required to perform one single-block differential decoding task. Having obtained the erasure coding decoding time and single-block differential decoding time, the required data slice size for the pipeline can be calculated by combining the average size of similar blocks and the file similarity during data upload. Specifically:

[0040] in, This indicates the size of the data slice required for the pipeline. Indicates the erasure coding decoding time. Indicates the single-block differential decoding time. This represents the average size of similar blocks. This indicates the file similarity when the data is uploaded.

[0041] Furthermore, the formula for calculating the data slice size required for the pipeline can be derived as follows: After obtaining the erasure coding decoding time and the single-block differential decoding time, define the decoding time ratio. Specifically:

[0042] in, This represents the number of similar blocks in a data slice, i.e.:

[0043] To make the segments of the production line more uniform, it should be made ,Right now:

[0044] At this point, the formula used to calculate the data slice size required for the pipeline can be determined.

[0045] Furthermore, before the step of calculating the data slice size required for the pipeline based on the erasure coding decoding time, the single-block differential decoding time, the average size of similar blocks, and the file similarity at the time of data upload, the method further includes: determining the file size at the time of data upload; and calculating the file similarity at the time of data upload based on the file size at the time of data upload, the number of similar blocks in the file, and the average size of similar blocks.

[0046] Understandably, when performing similarity detection on data, the detected similar blocks can be recorded, and the number of similar blocks in the file can be counted in real time. Then, the file similarity at the time of data upload can be calculated by combining the file size at the time of upload and the average size of the similar blocks. Specifically:

[0047] in, This indicates the file similarity during data upload. This indicates the number of similar blocks in the file. This represents the average size of similar blocks. This indicates the size of the file when the data is uploaded.

[0048] Step S30: By executing the differential decoding task and erasure coding decoding task after pipelined processing, data repair is performed based on the data slice size, the differential data block, and the reference block.

[0049] It should be understood that after obtaining the data slice size, differential data block, and base block, differential decoding repair can be performed by executing a pipelined differential decoding task, and erasure coding repair can be performed by executing a pipelined erasure coding decoding task. This can effectively improve the data repair performance of the encrypted differential compression system.

[0050] This embodiment responds to a data repair command triggered by a user in the encrypted differential compression system, reading the aggregated differential data block and the base block from contiguous disk space in one go. During the data repair process, the differential decoding task and the erasure coding decoding task are piped, and the data slice size required for the pipeline is determined based on the file similarity at the time of data upload. By executing the piped differential decoding task and the erasure coding decoding task, data repair is performed based on the data slice size, the differential data block, and the base block. Through this method, when data repair is required, the encrypted differential compression system reads the entire aggregated storage data block containing the differential data block and the base block from contiguous disk space, reducing one additional disk overhead compared to reading them separately. Then, the differential decoding task and the erasure coding decoding task are piped, and data repair is performed by executing the decoding task, thereby effectively improving the data repair performance of the encrypted differential compression system.

[0051] In one specific implementation, this application provides steps for determining the differential data block and the reference block. Please refer to... Figure 4 , Figure 4 This is the second flowchart illustrating the similarity-aware encrypted differential compression data repair method provided in this application embodiment. Step S10 includes steps S101 to S103: Step S101: In response to the data repair command triggered by the user in the encrypted differential compression system, determine the data block reading parameters.

[0052] It should be noted that the data repair command refers to the command initiated by users within an enterprise to repair data using the encrypted differential compression system. The data block read parameters refer to the parameters for reading aggregated storage data blocks from contiguous disk space at one time. These data block read parameters include, but are not limited to, read location and read method.

[0053] Step S102: Through a single I / O operation, aggregated storage data blocks are read from contiguous disk space in one operation according to the data block read parameters.

[0054] It is understandable that the aggregated storage data block refers to the entire large data block containing the differential data block and the base block. After determining the data block read parameters, the aggregated storage data block is read from the contiguous disk space in one I / O operation, thereby reducing the overhead of an additional disk I / O operation.

[0055] Step S103: Divide the aggregated storage data block according to the metadata to obtain the differential data block and the base block after aggregated storage.

[0056] It should be understood that when aggregated storage differential data blocks and base blocks are stored, corresponding marks are made in the metadata. Therefore, after reading aggregated storage data blocks from contiguous disk space at once, the metadata can be used to divide the aggregated storage data blocks into aggregated storage differential data blocks and base blocks.

[0057] This embodiment responds to a data repair command triggered by a user in the encrypted differential compression system, determines data block read parameters, and reads an aggregated storage data block from contiguous disk space in a single I / O operation based on these parameters. The aggregated storage data block is then divided according to metadata to obtain differential data blocks and a base block after aggregated storage. By determining the data block read parameters and reading the aggregated storage data block from contiguous disk space in a single I / O operation, an additional disk I / O overhead is reduced. Then, the metadata used during aggregated storage is used to divide the aggregated storage data block into differential data blocks and a base block, thereby effectively improving the accuracy and efficiency of data block partitioning.

[0058] The following describes the similarity-aware encrypted differential compressed data repair device provided in this application. The similarity-aware encrypted differential compressed data repair device described below corresponds to the similarity-aware encrypted differential compressed data repair method described above. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the module structure of the similarity-aware encrypted differential compression data repair device provided in this application embodiment, including: The read module T10 is used to read the aggregated differential data blocks and the base block from the contiguous disk space in response to the data repair command triggered by the user in the encrypted differential compression system.

[0059] The T20 module is used to pipeline the differential decoding task and erasure coding decoding task during the data repair process, and to determine the data slice size required for the pipeline based on the file similarity at the time of data upload.

[0060] The repair module T30 is used to repair data based on the data slice size, the differential data block, and the reference block by executing the differential decoding task and erasure coding decoding task after pipelined processing.

[0061] This embodiment responds to a data repair command triggered by a user in the encrypted differential compression system, reading the aggregated differential data block and the base block from contiguous disk space in one go. During the data repair process, the differential decoding task and the erasure coding decoding task are piped, and the data slice size required for the pipeline is determined based on the file similarity at the time of data upload. By executing the piped differential decoding task and the erasure coding decoding task, data repair is performed based on the data slice size, the differential data block, and the base block. Through this method, when data repair is required, the encrypted differential compression system reads the entire aggregated storage data block containing the differential data block and the base block from contiguous disk space, reducing one additional disk overhead compared to reading them separately. Then, the differential decoding task and the erasure coding decoding task are piped, and data repair is performed by executing the decoding task, thereby effectively improving the data repair performance of the encrypted differential compression system.

[0062] It is understood that the detailed functional implementation of each of the above modules can be found in the description of the aforementioned method embodiments, and will not be repeated here.

[0063] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0064] Based on the methods in the above embodiments, this application provides an electronic device, please refer to... Figure 6 , Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0065] It should be noted that the system may include: a processor 10, a communications interface 20, a memory 30, and a communication bus 40. The processor 10, communications interface 20, and memory 30 communicate with each other via the communication bus 40. The processor 10 can invoke logical instructions stored in the memory 30 to execute the methods described in the above embodiments.

[0066] Furthermore, the logical instructions in the aforementioned memory 30 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0067] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0068] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0069] It is understood that the processor in the embodiments of this application can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0070] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor.

[0071] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. Those skilled in the art will readily understand that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for repairing encrypted differential compressed data based on similarity awareness, characterized in that, include: In response to a data repair command triggered by a user in the encrypted differential compression system, the differential data block and the base block, which have been aggregated and stored, are read from contiguous disk space at once. During the data repair process, the differential decoding task and erasure coding decoding task are pipelined, and the data slice size required for the pipeline is determined based on the file similarity at the time of data upload. By executing the differential decoding task and erasure coding decoding task after pipelined processing, data repair is performed based on the data slice size, the differential data block, and the reference block.

2. The method as described in claim 1, characterized in that, Before the step of reading the aggregated differential data block and the base block from contiguous disk space at once, the method further includes: During data storage, retrieve the data block adjacent to the duplicate data block; The adjacent data blocks are marked as candidate reference blocks, and the candidate reference blocks are cached in the memory aggregation buffer; Obtain the differential data block to be stored, and calculate the similarity between the differential data block and each candidate reference block in the memory aggregation buffer; Identify target candidate reference blocks with similarity greater than a preset threshold, and cache the difference data blocks in a cache location adjacent to the target candidate reference blocks; When the data transfer is complete, the target candidate reference block and differential data block in the memory aggregation buffer are aggregated and stored in a contiguous disk space.

3. The method as described in claim 1, characterized in that, The step of reading the aggregated differential data block and the base block from contiguous disk space at once in response to a data repair command triggered by a user in the encrypted differential compression system includes: In response to a data repair command triggered by a user in the encrypted differential compression system, determine the data block reading parameters; In a single I / O operation, aggregated storage data blocks are read from contiguous disk space at once, based on the data block read parameters. The aggregated storage data block is divided according to the metadata to obtain the differential data block and the base block after aggregated storage.

4. The method as described in claim 1, characterized in that, The steps for pipelined processing of differential decoding and erasure coding decoding tasks during data repair include: During the data repair process, obtain the size of the data slice decoded by the differential decoding task; When the size of the decoded data slice reaches the data slice size required by the pipeline, the decoded data slice is sent to the next storage backend, and the erasure coding decoding task is performed based on the next storage backend; While performing erasure coding decoding, differential decoding continues until the size of the decoded data slice reaches the size required by the pipeline, thus completing the pipelined processing of differential decoding and erasure coding decoding tasks.

5. The method as described in claim 1, characterized in that, The step of determining the required data slice size for the pipeline based on file similarity during data upload includes: When performing an erasure coding decoding task and a single-block differential decoding task, the erasure coding decoding time and the single-block differential decoding time are obtained respectively. During data upload, determine the number and size of similar blocks in the file; Calculate the average size of similar blocks based on the number and size of similar blocks in the file; The required data slice size for the pipeline is calculated based on the erasure coding decoding time, the single-block differential decoding time, the average size of similar blocks, and the file similarity during data upload.

6. The method as described in claim 5, characterized in that, Before the step of calculating the required data slice size for the pipeline based on the erasure coding decoding time, the single-block differential decoding time, the average size of similar blocks, and the file similarity during data upload, the method further includes: Determine the file size when uploading data; The file similarity at the time of data upload is calculated based on the file size, the number of similar blocks in the file, and the average size of the similar blocks.

7. A similarity-aware encrypted differential compression data repair device, characterized in that, include: The read module is used to read the aggregated differential data blocks and the base block from contiguous disk space in response to the data repair command triggered by the user in the encrypted differential compression system. The determination module is used to pipeline the differential decoding task and erasure coding decoding task during the data repair process, and to determine the data slice size required for the pipeline based on the file similarity at the time of data upload. The repair module is used to repair data based on the data slice size, the differential data block, and the reference block by executing the differential decoding task and erasure coding decoding task after pipelined processing.

8. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-6.

Images