Capacity reduction in storage systems

Abstract

One aspect includes implementing capacity reduction in a storage system, including: for each of a candidate page and a target page in the storage system, identifying a subset of sectors having identical data or a minimum amount of different data; performing a bitwise exclusive OR (XOR) operation on sectors of the candidate page and the target page; determining an entropy from a result of the XOR operation. When the entropy is determined to be less than or equal to a threshold, one aspect includes constructing a reference page from XOR sectors containing results of the bitwise XOR operation, and performing a compression operation on the reference page.

Description

Background Technology

[0001] Data optimization techniques reduce system management and storage overhead costs. One optimization technique is compression, where the system re-encodes data using fewer bits than the original data, which can improve storage efficiency, transfer speed, and processing efficiency. Another optimization technique is deduplication, where the system eliminates duplicate data in the file system to improve storage efficiency, transfer speed, and processing efficiency. Data sent to the storage system can be segmented into fixed-size or variable-size fragments, where each fragment can be identified by an identifier (such as a hash signature or data hash). Once an identifier is generated, it is used to determine whether the data fragment already exists in the system. If so, it does not need to be stored again. While data compression techniques save storage space, they may involve overhead costs, such as increased input / output (IO) access overhead. Summary of the Invention

[0002] The present invention is provided to introduce, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to represent key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0003] According to various aspects of this disclosure, a method for capacity reduction in a storage system is provided. The method includes: for each of a candidate page and a target page in the storage system, identifying a subset of sectors having different data having the same data or a minimum threshold amount; performing a bitwise XOR operation on the sectors of the candidate page and the target page; storing the result of the bitwise XOR operation in an XOR sector; and determining an entropy based on the result of the XOR operation. The entropy indicates that corresponding data between the candidate page and the target page is different. When it is determined that the entropy is less than or equal to a threshold, the method includes constructing a reference page based on the XOR sector and performing the compression operation on the reference page.

[0004] According to various aspects of this disclosure, a system for capacity reduction in a storage system is provided. The system includes: a memory including computer-executable instructions; and a processor that executes the computer-executable instructions, which, when executed by the processor, cause the processor to perform operations. The operations include: for each of candidate pages and target pages in the storage system, identifying a subset of sectors having the same data or a minimum threshold amount of different data; performing a bitwise XOR operation on the sectors of the candidate pages and the target pages; storing the result of the compressed bitwise XOR operation in an XOR page; and determining an entropy based on the result of the XOR operation. The entropy indicates that corresponding data between the candidate pages and the target pages is different. When it is determined that the entropy is less than or equal to a threshold, the operations further include constructing a reference page based on the XOR sectors and performing the compression operation on the reference page.

[0005] According to various aspects of this disclosure, a computer program product for capacity reduction in a storage system is provided. The computer program product is embodied on a non-transitory computer-readable medium and includes instructions that, when executed by a computer, cause the computer to perform operations. The operations include: for each of a candidate page and a target page in the storage system, identifying a subset of sectors having the same data or a minimum threshold amount of different data; performing a bitwise XOR operation on the sectors of the candidate page and the target page; storing the result of the compressed bitwise XOR operation in an XOR page; and determining an entropy based on the result of the XOR operation. The entropy indicates that corresponding data between the candidate page and the target page is different. When it is determined that the entropy is less than or equal to a threshold, the method includes constructing a reference page based on the XOR sectors and performing the compression operation on the reference page. Attached Figure Description

[0006] Other aspects, features, and advantages of the claimed invention will become more apparent from the following detailed description, the appended claims, and the accompanying drawings, in which the same reference numerals identify similar or identical elements. Reference numerals introduced in connection with the drawings in the specification may be repeated in one or more subsequent drawings without further description in the specification to provide context for other features.

[0007] Figure 1 This is a diagram illustrating an example of a storage system based on the implementation plan;

[0008] Figure 2 It is a diagram that depicts an exemplary metadata data structure;

[0009] Figure 3It is a flowchart of the process based on the implementation plan;

[0010] Figures 4A to 4B It is a diagram describing pages in a storage system, and Figure 4C This refers to an exemplary XOR result sector based on the implementation scheme;

[0011] Figure 5 It is a flowchart of the process based on the further implementation plan; and

[0012] Figure 6 This is a diagram illustrating an example of a computing device according to the implementation plan. Detailed Implementation

[0013] Before describing embodiments of the concepts, structures, and techniques sought to be protected herein, some terminology is explained. The following description includes many terms whose definitions are generally known in the art. However, the following glossary of definitions is provided to clarify the following description and may aid in understanding the specification and claims.

[0014] As used herein, the term "storage system" is intended to be interpreted broadly to encompass, for example, private or public cloud computing systems used for storing data, as well as systems used for storing data including and excluding virtual infrastructure. As used herein, the terms "client," "host," and "user" are interchangeable to refer to any person, system, or other entity that uses the storage system to read / write data and makes requests to configure storage units within the storage system. In some embodiments, the term "storage device" may also refer to a storage array comprising multiple storage devices. In some embodiments, storage medium may refer to one or more storage media, such as hard disk drives, combinations of hard disk drives, flash memory storage devices, combinations of flash memory storage devices, combinations of hard disk drives, flash memory, and other storage devices, as well as other types and combinations of computer-readable storage media, including those yet to be conceived. Storage medium may also refer to both physical and logical storage media and may include multi-level virtual-to-physical mapping and may be or include an image or disk image. Storage medium may be computer-readable and may also be referred to herein as a computer-readable program medium. Furthermore, "storage unit" can refer to any storage unit, including those described above regarding storage devices, and includes storage volumes, logical drives, containers, or any storage unit exposed to clients or applications. A storage volume can be a logical storage unit that can be independently identified and addressed within the storage system.

[0015] In some implementations, the term "IO request" or simply "IO" can be used to refer to input or output requests, such as data read or write requests, or requests to configure and / or update storage unit characteristics. A characteristic can refer to any service that can be configured for the storage system.

[0016] In some implementations, storage device may refer to any non-volatile memory (NVM) device, including hard disk drives (HDDs), solid-state drives (SSDs), flash memory devices (e.g., NAND flash memory devices), and similar devices that can be accessed locally and / or remotely (e.g., via a storage connectivity network (SAN) (also referred to herein as a storage array network (SAN)).

[0017] In some implementations, a storage array (sometimes referred to as a disk array) can refer to a data storage system for block-based, file-based, or object-based storage, wherein the storage array may include, for example, dedicated storage hardware comprising rotating hard disk drives (HDDs), solid-state drives (SSDs), and / or all-flash drives. It should be understood that flash memory is a solid-state (SS) random access media type that, compared to a hard disk drive (HDD) with physical moving parts, can read any address range without latency penalty, as these physical moving parts need to be repositioned when reading from different address ranges, thus significantly increasing the latency of random I / O data.

[0018] In some implementations, the data storage entity and / or storage unit can be any one or more of a file system, object storage, virtualization device, logical unit, logical unit number, logical volume, logical device, physical device and / or storage medium.

[0019] In some implementations, a logical unit (LU) can be a logical entity provided by the storage system for accessing data from the storage system, and as used herein, a logical unit can be used interchangeably with a logical volume. In many implementations herein, LU or LUN (Logical Unit Number) can be used interchangeably. In some implementations, a LUN can be a logical unit number used to identify a logical unit; it can also refer to one or more virtual disks or virtual LUNs that can correspond to one or more virtual machines. In some implementations, a volume refers to a storage unit on a disk.

[0020] In some implementations, a storage cluster can refer to a configuration type that enables extended clustering (e.g., clusters distributed across different geographical locations). Storage cluster configurations allow entities to provide load balancing and migration between data centers.

[0021] In some implementations, Log Structured Storage (LSS) refers to a file system in which data and metadata are sequentially written to a circular buffer called the log. The Log Structured File System reclaims free space from the tail of the log to prevent the file system from filling up when the log head rolls back to merge with the tail. The tail can free up space and move forward by skipping data for which a newer version exists in the log in the more distant future. If no newer version exists, the data is moved and appended to the head.

[0022] In some implementations, a block refers to a sequence of bytes or bits with a maximum length (block size). The block size can be fixed or variable. A block device or block storage device can refer to an abstraction layer of hardware responsible for storing and retrieving specified data blocks. In some implementations, a file system can be based on a block device, which can be an abstraction layer of hardware responsible for storing and retrieving specified data blocks. Block storage devices can be abstracted by a file system or database management system (DBMS) for use by applications and end users. Physical or logical volumes accessed via block I / O can be devices within a server, directly attached via SCSI or Fibre Channel, or remote devices accessed via a storage area network (SAN) using protocols such as iSCSI.

[0023] While vendor-specific terminology may be used herein to facilitate understanding, it should be understood that the concepts, techniques, and structures sought to be protected herein are not limited to use with any particular commercial product. Furthermore, for the sake of clarity of this disclosure, well-known methods, procedures, circuits, components, and products are not described in detail herein.

[0024] The phrases “such as,” “for example,” “exemplary,” and variations thereof are used herein to describe non-limiting embodiments and to mean “serves as an example, instance, or illustration.” Any embodiment described herein via these phrases and / or variations is not necessarily to be construed as preferred or superior to other embodiments and / or to exclude features incorporated into other embodiments. Additionally, the term “optionally” is used herein to indicate that a feature or process, etc., is provided in some embodiments but not in others. Any particular embodiment of the invention may include multiple “optional” features unless such features conflict.

[0025] Modern log-structured storage systems (LSS) typically support deduplication methods to reduce physical storage capacity and achieve more logical capacity in the same given physical space. Deduplication can be performed on an entire page (e.g., 4K). However, in many cases, sub-sector deduplication is not performed for several reasons. For example, sub-sectors (less than 512MB) are very small pieces of data, meaning that searching for such matching fragments may require more resources in terms of metadata representation and CPU cycles than any benefit such reduction might offer. Moreover, compression, which is extremely common in storage systems, further reduces the need for sub-sector deduplication, as the difference between compressed sectors and compressed partial sectors may be smaller. Therefore, only a technique capable of sub-sector reduction is needed, which will consume very little metadata representation and CPU cycles.

[0026] Hash functions are typically used to find duplicates. A hash table is built to represent most of the stored pages. For each new page, a match lookup is performed via the hash table. The hash may be deterministic (e.g., a 20-byte SHA1 is widely considered a page-wide match) or non-deterministic, meaning that a hash alone is insufficient and a page-wide comparison is necessary. The same applies to sectors.

[0027] Partial page matching occurs when two pages have several identical sectors but one or more different sectors. The partial matching result is represented by metadata that stores and marks the identical sectors, and in some cases, the unique sectors (i.e., the different sectors).

[0028] Partial page matching (e.g., matching sectors between two pages) can be achieved using page comparison functions, where the two pages are examined sector by sector to determine which sectors contain the same data. In one implementation, an XOR operation is performed on each of the multiple distinct sector pairs between the two pages being compared. The result of the XOR operation (only one bit) can be stored instead of the original data, thus saving space.

[0029] Different sectors can originate from many factors, including timestamps, serial numbers, and other fields that represent a very small portion of the sector.

[0030] As noted above, the implementation described in this paper stores the XOR results of different sectors rather than unique data. In this way, considering that most sectors are identical (i.e., composed of many zeros), the XOR will produce very efficiently compressed sectors.

[0031] Now go to Figure 1 The system 100 for implementing sector-based capacity reduction in a storage system environment will now be described according to the implementation scheme. Figure 1The storage system 100 includes processing circuitry 102, a memory 104, and persistent storage device 106. In an embodiment, the persistent storage device 106 may be locally installed within the storage system 100, such as... Figure 1 As shown, or it may be an external or remote device that is communicatively coupled to the storage system 100 via a connection (not shown).

[0032] If the persistent storage device 106 is remote, it can be coupled to the storage system via known connections, interfaces, and network circuitry (e.g., Ethernet cable, Wi-Fi wireless connection, IP network, SCSI cable, SATA cable, Fibre Channel (FC) cable, etc.). If the connection is a network connection, the interface and network circuitry may include, for example, one or more Ethernet cards, cellular modems, FC adapters, Wi-Fi wireless network adapters, and / or other devices for connecting to a network. If the connection to the persistent storage device is a local storage connection, the interface and / or network interface circuitry may include, for example, SCSI, SAS, ATA, SATA, FC, and / or other similar controllers and ports.

[0033] Persistent storage device 106 may include any type of persistent storage device, such as hard disk drives, solid-state storage devices, flash drives, etc. Persistent storage device 106 stores user data as volumes 122A to 122n, which represent one or more logical disks accessible to the user. Volumes 122A to 122n are organized and managed with reference to metadata stored in various data structures, such as metadata structure 108 in memory 104. Figure 1 As shown, by way of a non-limiting example, the persistent storage device is implemented via storage array 120. Although data storage units are described herein as volumes in a storage array, it should be understood that other data storage units or object storage units can be used to achieve the advantages of the implementation.

[0034] Memory 104 can be any type of digital system memory, such as random access memory (RAM). Memory 104 stores the operating system (OS, not depicted) in operation (e.g., Linux, UNIX, Windows, MacOS, or similar operating systems). Memory 104 may also store the input / output (I / O) stack (not shown) in operation. The I / O stack allows processing of I / O requests (not depicted) from external hosts (and local applications) for the volume data 122A to 122n managed by storage system 100.

[0035] Memory 104 also stores index 110 for tracking metadata corresponding to the blocks being compressed. A hash table 111 is provided in memory 104. Hash table 111 contains the hash values ​​of pages stored in the system. For each new page, a match lookup is performed via the hash table. The hash may be deterministic (e.g., a 20-byte SHA1 is widely considered a page-wide match) or non-deterministic, meaning that a hash alone is insufficient and a page-wide comparison is required. The same applies to sectors.

[0036] Partial page matching occurs when two pages have several identical sectors but one or more different sectors. The partial match results are represented by metadata indicating the sectors that are identical, and only those unique sectors (i.e., the different sectors) are stored.

[0037] The storage system includes processing circuitry 102, which can be any type of processor or processor group configured to perform operations, such as a microprocessor, multi-core microprocessor, digital signal processor, system-on-a-chip, collection of electronic circuits, similar type of controller, or any combination thereof. It should be understood that the storage system 100 may also include other components known in the art, including interconnect circuitry. In operation, the processing circuitry 102 can execute one or more applications, which can be implemented in logic executable by the processing circuitry.

[0038] Processing circuitry 102 is configured to receive and serve input / output (IO) requests from one or more external or remote systems (not shown) communicatively coupled to storage system 100. Coupling can be implemented via any type of communication network (such as the Internet, local area network, wide area network) and can include wired and / or wireless technologies.

[0039] Volumes 122A to 122n can be addressed via logical addresses that are mapped to the array's physical addresses via a three-layer indirection scheme used to perform address translation during read and write operations on the array.

[0040] During deduplication, indirection layers regarding memory addressing can be used to convert or map between memory space layers. These include (i) a low-level mapping (leaf) containing pointers to data, (ii) a middle-level mapping containing pointers to the low-level mapping, and (iii) a top-level mapping containing pointers to the middle-level mapping. The low-level layer can be a Logical Block Address Space (LBA), the middle-level layer can be a Virtual Block Address Space (VLB), and the top-level layer can be a Physical Address Space (PLB). Each of these address spaces reflects a set of addresses. Mapping can be implemented using a data structure that takes logical addresses and uses pointers to point to VLB addresses, which in turn point to the physical address (PLB) where some data (e.g., such as a page) resides.

[0041] like Figure 2 As shown, the metadata data structure (which corresponds to) Figure 1 The metadata structure 108 (which can be implemented in a Log Structured Storage (LSS) system) includes a three-tier indirection scheme for accessing user data. The three-tier scheme includes a Logical Block Address (LBA) layer (i.e., the leaf layer), a Virtual Block (VLB) layer, and a Physical Block (PLB) layer. The leaves are the Logical Block Address (LBA) representation layers, which are typically organized as a tree, where each leaf corresponds to a specific LBA range. In a Log Structured Storage system, data is stored in contiguous data blocks (called PLBs, e.g., 2MB blocks). Actual user data pages reside within the PLBs and are referenced by the VLBs.

[0042] Deduplication can be achieved using a unique hash representing a page. The system may include a deduplication index that represents the majority hash (and thus the most unique capacity of the volume cluster). Partial page matching occurs when a subpage (e.g., several sectors of a page) matches between two pages (e.g., some pages are unique and some pages are common).

[0043] Figure 2 The diagram illustrates the logical representation of metadata in data structure 200 for two partially matching pages (P1 and P2, 210 and 220, respectively) in the cluster, demonstrating the process of performing full-page deduplication on the cluster. Figure 2 As shown, two pages, P1 and P2, are maintained separately (i.e., the pages have no common parts maintained by the system). The common parts of the pages (i.e., representing duplicate data) are shown as 212 and 222. Page P1 also contains data 214 that is not shared with P2. Similarly, page P2 contains data 224 that is not shared with page P1. Therefore, the data stored in the portion of page P1 (214) is not a copy of the portion of the data stored in P2 (224). Figure 2 In the metadata structure 200, for page P1 (210), the page's logical address #1 points to VLB entry #1, which in turn points to page P1. Virtual Logical Block (VLB) entries #1 and #2 correspond to... Figure 1 VLB 114 in. Similarly, in Figure 2 In the metadata structure 200, for page P2 (220), the page's logical address #2 points to VLB entry #2, which in turn points to page P2. Logical addresses #1 and #2 correspond to... Figure 1 The logical block address (LBA) in the document is 112. Pages P1 and P2 correspond to... Figure 1 The physical block address in the code is PLB 116.

[0044] In one non-limiting embodiment described herein, a technique is provided for identifying different sectors between two pages and storing an information bit generated by the XOR operation per byte in each sector through an XOR operation performed on the different sectors. The information bit indicates whether a byte in the first page is the same as a corresponding byte in the second page. If the number of outputs returning True or '1' (or something other than '0') in the XOR operation exceeds a predefined threshold number (e.g., 90% of the total number of bytes in the sector), compression is performed on the XORed page instead of the original page. Otherwise, if the threshold is not met, compression is performed on the original page. Those skilled in the art will understand that alternatives to the above embodiments can be used. For example, good compressibility may still be possible even if all bits are '1' (e.g., the page and its bitwise NOT).

[0045] Different sectors can arise from a number of factors, including timestamps, serial numbers, and other fields that represent a very small portion of the sector. This can include areas where the same data is found but differs due to, for example, timestamps or checksum fields that vary by a few bytes, resulting in sectors that are nearly identical or substantially identical but not exactly the same.

[0046] As noted above, the implementation described herein stores the compressed XOR results of different sectors rather than unique data. In this way, considering that most sectors are identical (i.e., the XOR result page consists of many zeros), the XOR will produce very efficiently compressed sectors.

[0047] Now go to Figure 3 and Figures 4A to 4C The process 300 for implementing sub-sector-based capacity reduction and exemplary pages 400A, 400B and XOR sector 400C for implementing sub-sector-based capacity reduction will now be described in the implementation. Figure 3 The process 300 assumes that a request or triggering event has caused the system to begin comparing two pages in the storage system (referred to in this document as the first (or candidate) page and the second (or target) page). Figure 3 The process described also assumes that the comparison of the two pages results in a significant number of byte matches between the pages. Pages 400A and 400B are each, for example, 4K in size and contain 8 sectors, each sector (e.g., representative sectors 404A and 404B) containing 512 bytes.

[0048] In box 302, process 300 identifies the same and / or nearly identical sectors for the candidate page and the target page. Figure 4A and Figure 4B Two pages, 400A and 400B, are shown, where a set of sectors 402A and 402B are identical or substantially similar. As a non-limiting example, this can be achieved by accessing ( Figure 1The process involves using hash table 111 to perform a lookup of the hash value of the sector corresponding to the page, or using a similarity hash associated with similar (not just identical) sectors or pages. The lookup of a sector produces a hash value for that sector, which can be compared to determine if a match or similarity exists. If one or more sectors exist and the actual pages are compared, these pages can be considered partial page matches for a given sector. It should be understood that the process can be extended to identify not only identical sectors but also nearly identical sectors, such as those differing by only a small number or percentage of bytes.

[0049] In box 304, procedure 300 performs a bitwise XOR operation on different (or all) sectors of the page (in terms of similarity), and in box 306 stores the result of the XOR operation in an XOR structure. Figure 4C An exemplary XOR sector 400C with fields of bytes (B1 to Bn) is shown. XOR sector 400C refers to an intermediate structure used during the process of determining which sector (i.e., the original page sector or the XOR sector) to use to build a new reference page and for performing compression.

[0050] The process described in boxes 310-316 provides a non-limiting example of a heuristic method for selecting XOR sectors or raw sectors. The process determines entropy based on the result of the XOR operation, which can be used to determine which sector to use for the reference page. Entropy can refer to the predictability of data, such as detectable patterns in the data or their absence. For example, how likely is it that if a piece of data appears on a page, the next piece of data will appear? If there is little predictability, it means there are few detectable patterns in the data, meaning it is not suitable for compression (e.g., when there are many zero bytes in a page, the page is determined to have low entropy). Since the process deals with lossless compression (the performance of compression algorithms is limited by the entropy of the page), however, since we can find similar pages to XOR the page with, we are able to reduce the page entropy without losing data, thereby allowing lossless compression to be more efficient (because the new page has lower entropy). It will be understood that other ways of selecting sectors can be employed to achieve the advantages of the implementation described herein. For example, the process can use a threshold regarding the number of non-zero bytes to determine the number of non-zero bytes in each sector to determine whether to use raw sectors from candidate pages or XOR sectors to construct the reference page. The non-zero bytes reflect the differences between the two pages. Therefore, in box 308, process 300 determines which sector to use for the reference page, as described in boxes 310-316.

[0051] In box 310, process 300 determines the entropy based on the result of the XOR operation. The determined entropy can be a quantifiable value that reflects detectable patterns in the data.

[0052] In box 312, process 300 determines whether the entropy is less than or equal to a predefined threshold. The threshold can be the number of bytes or a percentage of the total number of bytes. If the entropy is equal to or less than the threshold, it means that a small number of bytes differ between the pages. In this case, process 300 uses XOR sectors to construct a reference page (and compresses it later in box 314). Otherwise, if the entropy exceeds the threshold, process 300 uses the original sectors (sectors from the candidate page comparison) to construct the reference page and compresses it later in box 316. In box 318, compression is performed on the reference page (the combined page).

[0053] Sub-sector-based capacity reduction offers many benefits. For example, a single bit is sufficient to indicate unique (candidate) or XORed data; therefore, very little metadata is required. Additionally, the process involves minimal computation, as the XOR operation is readily implemented. Furthermore, adjacent sectors typically have the same high similarity probability, making byte-by-byte comparisons advantageous.

[0054] Now go to Figure 5 The process 500 for decompressing a page (e.g., a compressed page from Figure 4) will now be described according to the implementation scheme.

[0055] In box 502, procedure 500 attempts to read the user page. In box 504, procedure 500 determines whether the user page is an XOR sector. If so, procedure 500 reads the user page and the corresponding reference page in box 506. In box 508, procedure 500 decompresses the XOR sector and the reference page, and in box 510, procedure 500 returns the decompression result to the requester of the user page.

[0056] In box 512, when it is determined that the user page is not an XOR sector (from box 504), process 500 reads the user page in box 514, decompresses the user page in box 516, and returns the decompression result to the requester in box 518.

[0057] Figure 6 An exemplary computer 600 (e.g., physical or virtual) capable of performing at least a portion of the processes described herein is illustrated. Computer 600 includes a processor 602, volatile memory 604, non-volatile memory 606 (e.g., hard disk or flash memory), output device 607, and graphical user interface (GUI) 608 (e.g., mouse, keyboard, monitor). Non-volatile memory 606 stores computer instructions 612, operating system 616, and data 618. In one example, computer instructions 612 are executed by processor 602 outside of volatile memory 604. In one embodiment, article 620 includes non-transitory computer-readable instructions.

[0058] The processing can be implemented in hardware, software, or a combination of both. The processing can be implemented in a computer program that executes on a programmable computer / machine, each programmable computer / machine including a processor, storage media or other article of manufacture readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and one or more output devices. Program code can be applied to data input using the input devices to perform the processing and generate output information.

[0059] The system can be processed at least in part via a computer program product (e.g., in a machine-readable storage device) to be executed or controlled by a data processing device (e.g., a programmable processor, a computer, or multiple computers). Each such program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, the program can be implemented in assembly language or machine language. The language can be a compiled language or an interpreted language, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program can be deployed to execute on one or more computers located at a single site or distributed across multiple sites and interconnected by a communication network. The computer program can be stored on a storage medium or device (e.g., a CD-ROM, hard disk, or magnetic disk) readable by a general-purpose or special-purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer. The processing can also be implemented as a machine-readable storage medium configured with a computer program, wherein, upon execution, instructions in the computer program cause the computer to run.

[0060] The processing can be performed by one or more programmable processors that execute one or more computer programs to perform system functions. All or part of the system can be implemented as special-purpose logic circuits (e.g., FPGA (Field Programmable Gate Array) and / or ASIC (Application-Specific Integrated Circuit)).

[0061] Having described exemplary embodiments of the invention, it will now be apparent to those skilled in the art that other embodiments incorporating their concepts may also be used. The embodiments contained herein should not be limited to those disclosed, but should be limited only by the spirit and scope of the appended claims. All publications and references cited herein are expressly incorporated herein by reference in their entirety.

[0062] Elements of the different embodiments described herein can be combined to form other embodiments not specifically set forth above. The various elements described in the context of a single embodiment may also be provided individually or in any suitable sub-combination. Other embodiments not specifically described herein are also within the scope of the appended claims.

Claims

1. A method for reducing capacity in a storage system, the method comprising: For each of the candidate pages and the target pages in the storage system, identify a subset of sectors that have the same data or at least a specified amount of different data; Perform a bitwise XOR operation on the sectors of the candidate page and the target page; The result of the XOR operation is stored in an XOR page, which includes one information bit per byte of each sector generated by the XOR operation, and the sectors in the XOR page are XOR sectors. For each XOR sector, the entropy is determined based on the result of the XOR operation; When it is determined that the entropy is less than or equal to the threshold, a reference page is constructed based on the XOR sector containing the result of the bitwise XOR operation, wherein compression is performed on the reference page; When it is determined that the entropy is greater than the threshold: The reference page is constructed based on the corresponding sector of the candidate page; as well as The compression is performed on the reference page; The entropy indicates that the corresponding data between the candidate page and the target page is different.

2. The method of claim 1, wherein determining the entropy based on the result of the XOR operation comprises determining the number of non-zero bytes in each sector based on the result of the XOR operation; When the number of non-zero bytes is determined to be less than or equal to the threshold, the reference page is constructed based on the XOR sector containing the result of the bitwise XOR operation; and When the number of non-zero bytes is determined to be greater than the threshold, the reference page is constructed based on the corresponding sector of the candidate page.

3. The method of claim 1, wherein the candidate page and the target page are determined to be partially matched regarding the same data or the minimum amount of different data.

4. The method of claim 1, wherein identifying the subset of sectors having the same data or the minimum specified amount of different data comprises retrieving the identifier of the sector from a hash table and comparing the hash values ​​of the sectors, wherein when the hash values ​​are the same, the data in the sector is considered a match.

5. The method of claim 1, wherein performing the bitwise XOR operation on the sector includes performing the bitwise XOR operation on the different data.

6. The method of claim 1, wherein performing the bitwise XOR operation on the sector comprises performing the bitwise XOR operation on all sectors having the minimum specified amount of distinct data.

7. The method of claim 1, further comprising: Attempting to read the user page; as well as When it is determined that the user page is the XOR sector: Read the user page and the corresponding reference page; Decompress the XOR sector and the reference page; as well as The decompression result is returned to the requester.

8. The method of claim 7, further comprising: When it is determined that the user page is not the XOR sector: Read the user page; Decompress the user page; as well as The result of decompressing the user page will be returned to the requester.

9. A system for implementing capacity reduction in a storage system environment, the system comprising: The memory includes computer-executable instructions; as well as A processor that executes computer-executable instructions, which, when executed by the processor, cause the processor to perform operations including: For each of the candidate pages and the target pages in the storage system, identify a subset of sectors that have the same data or at least a specified amount of different data; Perform a bitwise XOR operation on the sectors of the candidate page and the target page that are not in the subset; The result of the XOR operation is stored in an XOR page, which includes one information bit per byte of each sector generated by the XOR operation, and the sectors in the XOR page are XOR sectors. For each XOR sector, the entropy is determined based on the result of the XOR operation; When it is determined that the entropy is less than or equal to the threshold, a reference page is constructed based on the XOR sector containing the result of the bitwise XOR operation, wherein compression is performed on the reference page; When it is determined that the entropy is greater than the threshold: The reference page is constructed based on the corresponding sector of the candidate page; and Perform a compression operation on the reference page; The entropy indicates that the corresponding data between the candidate page and the target page is different.

10. The system of claim 9, wherein determining the entropy based on the result of the XOR operation comprises determining the number of non-zero bytes in each sector based on the result of the XOR operation; When the number of non-zero bytes is determined to be less than or equal to the threshold, the reference page is constructed based on the XOR sector containing the result of the bitwise XOR operation; and When the number of non-zero bytes is determined to be greater than the threshold, the reference page is constructed based on the corresponding sector of the candidate page.

11. The system of claim 9, wherein the candidate page and the target page are determined to be partially matched regarding the same data or the minimum amount of different data.

12. The system of claim 9, wherein identifying the subset of sectors having the same data or the minimum specified amount of different data comprises retrieving the identifier of the sector from a hash table and comparing the hash values ​​of the sectors, wherein when the hash values ​​are the same, the data in the sector is considered to be a match.

13. The system of claim 9, wherein performing the bitwise XOR operation on the sector includes performing the bitwise XOR operation on the different data.

14. The system of claim 9, wherein performing the bitwise XOR operation on the sector comprises performing the bitwise XOR operation on all sectors having different data having the minimum specified amount.

15. A computer program product for implementing capacity reduction in a storage system environment, the computer program product being embodied on a non-transitory computer-readable medium, and the computer program product comprising instructions that, when executed by a computer, cause the computer to perform operations, the operations including: For each of the candidate pages and the target pages in the storage system, identify a subset of sectors that have the same data or at least a specified amount of different data; Perform a bitwise XOR operation on the sectors of the candidate page and the target page that are not in the subset; The result of the XOR operation is stored in an XOR page, which includes one information bit per byte of each sector generated by the XOR operation, and the sectors in the XOR page are XOR sectors. For each XOR sector, the entropy is determined based on the result of the XOR operation; When it is determined that the entropy is less than or equal to the threshold, a reference page is constructed based on the XOR sector containing the result of the bitwise XOR operation, wherein compression is performed on the reference page; When it is determined that the entropy is greater than the threshold: The reference page is constructed based on the corresponding sector of the candidate page; and Perform the compression on the reference page; The entropy indicates that the corresponding data between the candidate page and the target page is different.

16. The computer program product of claim 15, wherein determining the entropy based on the result of the XOR operation comprises determining the number of non-zero bytes in each sector based on the result of the XOR operation; When the number of non-zero bytes is determined to be less than or equal to the threshold, the reference page is constructed based on the XOR sector containing the result of the bitwise XOR operation; and When the number of non-zero bytes is determined to be greater than the threshold, the reference page is constructed based on the corresponding sector of the candidate page.

17. The computer program product of claim 15, wherein the candidate page and the target page are determined to be partially matched with respect to the same data or a minimum amount of different data.

Images