Encoding for Data Recovery in Storage Systems

Abstract

A method for reading from a storage medium to recover a group of information sectors, each information sector including a corresponding information payload. The medium stores redundant data, the redundant data comprising a plurality of separate redundant codes for the group, each code being a linear sum of terms, each term in the sum being an information payload from a different corresponding information sector of the information sectors in the group, the different corresponding information sectors being weighted by a corresponding coefficient from a coefficient set for the redundant code. The method comprises, after the redundant data has been stored on the medium: identifying a set of k' information sectors to be recovered; selecting k' redundant codes from the redundant codes; determining a block matrix E for the k' information sectors using the k' sets of coefficients for the selected codes; determining a matrix D that is the matrix inverse of E; and recovering the k' information payloads from the inverse matrix D.

Description

Background Art

[0001] Most large-scale digital data storage media systems use redundancy to help correct the more likely types of errors. This can exist within a single data sector, across multiple data sectors within a single medium, or across multiple media. For example, within a single sector, it is common for hard drives to use LDPC (Low Density Parity Check) as a method of error correction for small-scale bit errors. It is also common to use RAID (Redundant Array of Independent Disks) across media to protect against individual disk failures.

[0002] Methods and systems for recovering from sector errors within a single medium are less frequently used and tend to be more primitive. For example, on disk drives, it is common to reserve a small portion of the medium as a spare and use one of these locations as a replacement if a pre-allocated area of a particular sector is deemed to have failed. This can be problematic if the spare area is too large or too small, as it must be selected in advance and can also lead to unpredictable access latency. In another example, on tape-based systems, it is common to perform a read-after-write check to see if the write was successful, and if not, to write an additional copy of the sector. This can be problematic because it can lead to unpredictable capacity within the tape.

[0003] More complex and efficient schemes are not feasible in conventional storage systems because they must handle situations where data is overwritten or where all the data to be written to the entire medium is not available at the same time. Furthermore, they are often unnecessary due to the need for protection across multiple media, such as due to the relatively high failure rates of disk drives and tapes.

[0004] A WORM (Write Once Read Many) storage system is a form of storage in which all data is written once in a single operation. Although classic optical media such as CDs and DVDs are both WORM and write all data in a single operation, they are not prone to using complex media-wide redundancy systems because of the need to provide data at a fixed rate when reading, such as to an audio or video playback device, and / or the need to keep the cost of playback devices very low in consumer scenarios.

[0005] Another known type of optical WORM storage uses quartz glass as a storage medium. Information is imprinted on the structure with the help of an ultrafast laser (usually a femtosecond laser). This laser has the ability to direct a large amount of energy in a very confined space, changing the structure of the glass in that area in a controlled and permanent way, and thus storing information there. Some of these systems allow data to be stored in three dimensions on the medium, in which case the location of a given bit or symbol can be called a voxel. Reading then works by shining light onto a specific part of the glass using a polarization-sensitive microscope and inferring the data written to that area by measuring specific properties of the observed light. Summary of the Invention

[0006] According to one aspect disclosed herein, a computer-implemented method for reading from a storage medium to recover a group of information sectors, each information sector including a corresponding information payload, is provided. The storage medium stores redundant data, the redundant data including a plurality of separate redundant codes for the group, each redundant code being a linear sum of terms, each term in the sum being an information payload of a different corresponding information sector from the information sectors in the group, the different corresponding information sectors being weighted by a corresponding coefficient from a set of coefficients for the redundant code. The method comprises, after the redundant data has been stored on the storage medium, performing a decoding process comprising: identifying a set of k' information sectors from the group from which the corresponding information payload is to be recovered based on the redundant data; selecting k' redundant codes from the redundant codes; determining a block matrix E, wherein each matrix column includes corresponding coefficients for a different one of the k' information sectors and each matrix row includes a set of coefficients for a different one of the k' redundant codes, and vice versa; determining a matrix D that is a matrix inverse of E; and i =∑ j (d i，j ·r j ) to recover k' information payloads, where v i is the information payload, i is the index indicating the corresponding information sector, and j is the index indicating each redundant code r j The index of d i,j is a matrix element of D, and ∑ j is the sum of k' redundant codes, the calculation being performed for each information sector i among the k' information sectors.

[0007] This summary is provided to introduce a selection of concepts in a simplified form that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] To assist in understanding the embodiments of the present disclosure and to show how such embodiments may be implemented, reference is made, by way of example only, to the accompanying drawings, in which:

[0009] Figure 1 is a schematic representation of a data storage scheme,

[0010] Figure 2 is a flow chart of a method of storing data in a medium including redundant data,

[0011] Figure 3 is a flow chart of a method of reading data from a medium including recovery based on redundant data, and

[0012] Figure 4 A scheme with both rows and columns of redundant data is schematically shown. DETAILED DESCRIPTION

[0013] The following describes a network storage and recovery method with specific, but not exclusive, application to WORM (Write Once Read Many) storage systems, such as quartz glass, where all data is written at once in a single operation and where the media is particularly durable. This allows for better in-media redundancy and error recovery methods and systems. While the embodiments may be described with respect to quartz glass media, the disclosed methods can also be applied to any WORM media where all data to be written is available for processing at the time of writing.

[0014] More generally, the disclosed methods can be used with any media, including more traditional optical discs, magnetic media, or electronic storage media. However, in practice, they tend to be used to read and write small portions of data in real time or on the fly. For example, although classic optical media such as CDs and DVDs are both WORM and write all data in a single operation, they do not tend to use complex media-wide redundancy systems because of the need to provide data to, for example, audio or video playback devices at a fixed rate when reading, and / or the need to keep the playback devices very low cost in consumer scenarios. The disclosed methods have particular, but not exclusive, applicability to long-term storage systems, such as in archival storage systems, where it is acceptable to write relatively large amounts of data at a time and read relatively large amounts of data at a time if recovery is required. The disclosed methods can be used, for example, in quartz glass data repositories and / or in cloud-based archival systems.

[0015] The encoding method divides the data to be stored into sectors, which can be called content sectors or information sectors. The information sectors are used to construct random linear combinations of the information in these sectors. These combinations can be called recovery sectors or redundant sectors. Both information sectors and redundant sectors can be stored on the storage medium. Alternatively, the redundant information can be made known to the reader via a separate medium. In either case, during the reading process, a method is applied to determine which sectors have been correctly read and which sectors are missing. The correctly read redundant sectors are then used to recover the lost information sectors by reversing the random linear combinations. This process is similar to inverting a system of linear equations.

[0016] As an optional optimization, information sectors can be grouped and redundant encoding and decoding operations can be applied independently to each group. This approach reduces the computational overhead of the aforementioned encoding and decoding with minimal degradation in error correction performance. Alternatively, all sectors on the entire medium can be encoded as a single group.

[0017] The set of coefficients used for the linear combination can be predetermined. As another optional extension to increase coding flexibility, the coefficients can be stored along with the coding sector. However, this would have a relatively large storage cost. Therefore, to reduce overhead, the coefficients can instead be generated from a deterministic pseudo-random process. Therefore, it is sufficient to store the seed that initializes the random process and the index of the random process that generates the coefficients.

[0018] Codecs are a common technique for providing reliability in both storage and network applications. Codecs differ in how they combine the original content to construct the encoded information. These differences are intended to balance reliability performance, codec overhead, encoding and decoding computational complexity, and other factors. The method disclosed herein is similar to the network codec technology disclosed in US Pat. No. 7,756,051, used to encode data to be transmitted over a network for content distribution, rather than storage. However, existing network coding techniques are "non-systematic," meaning that the payload data itself is encoded using the same coding scheme that adds redundancy, and decoding is always required to extract the payload information. In contrast, the method disclosed herein is "systematic," meaning that the payload remains unencoded using a coding scheme that adds redundancy (even if the payload happens to be encoded using some other, orthogonal, lower-level coding scheme, such as for compression and / or encryption purposes rather than redundancy). Instead, the redundant data is separated from the payload information, for example, appended in a separate redundant sector, or even transmitted on a separate medium. In the systematic case, the corresponding decoding method is only required to recover from errors in the payload information, not necessarily for every read.

[0019] Compared to other coding techniques, network coding offers very good reliability performance for a wide range of failure scenarios (i.e., decoding can be performed with high probability with moderate storage overhead) and also allows the construction of an arbitrary number of coded sectors. To provide these benefits, computational performance may be sacrificed. This disclosure expands the scope of existing network coding-based coding schemes to include storage applications.

[0020] Therefore, the disclosed method for constructing error correction codes provides reliable data recovery for applications such as archival storage and can provide quantifiable reliability guarantees. In embodiments, it also reduces storage overhead compared to existing systems.

[0021] In an embodiment, nearby storage elements (each storing a single bit or other such basic data symbol) can be grouped together to define a sector. Sectors typically store tens of kilobytes of data and are generally of the same size. As with all storage systems, elements within a sector may contain errors when written or read due to write and read imperfections. To address this issue, redundant information is stored in each sector, which allows errors within the sector to be detected and, typically, corrected. In addition, an integrity check can be stored in the form of a hash or checksum to determine whether the aforementioned correction was successful. There are many known algorithms for determining integrity. Some available bytes within the sector can be reserved to store integrity check information. After reading all elements within the sector and checking integrity, it can then be determined whether the sector as a whole was read correctly (with a very high probability) or whether there are errors within the sector. The currently disclosed decoding method can then recover the sector containing the error.

[0022] The method includes generating (and, in one embodiment, storing on the medium) specially crafted codes that can be used to recover erroneous sectors. In one embodiment, these redundant codes can be stored in their own redundant sectors on the medium. Thus, some sectors can be reserved on the medium for storing special redundant sectors instead of user data. To recover a lost user sector, the redundant sector is combined with other user sectors that have been correctly read.

[0023] In one implementation, any redundant sector can be used to recover any user sector. In other words, all user data stored across the entire medium can be used to construct all redundant sectors, so each redundant sector can help recover each user sector. Even if this approach is possible, in practice, it may be computationally infeasible. Therefore, as an optimization, the user sectors can be divided into groups instead, and the encoding method constructs group-specific redundant sectors. In this case, recovery is performed only within a given group, combining user sectors from a single group with redundant sectors from the same group to recover lost user sectors (of that group). The size of these groups (the number of user and redundant sectors) is a design parameter that depends on the expected sector read failure rate, the desired reliability target, and other practical issues.

[0024] Now refer to Figures 1 to 4 Some example embodiments of the techniques of this disclosure are discussed in more detail.

[0025] Figure 1 A scheme for arranging data into sectors on a storage medium is shown. The data is stored by a storage computer. The storage medium may be an integral part of the storage computer, or may be an external or removable medium. The storage computer may include one or more computer units (in one or more housings) located in one or more locations. A reader computer may read the data and recover from errors. The reader computer may be the same computer as the storage computer or a different computer, or, if implemented on multiple computer units, may share one or more identical units. In the case of multiple computer units, appropriate distributed computing techniques will be familiar to those skilled in the art. The or each computer unit of the storage and reading computers includes one or more processors that respectively execute the disclosed storage and reading methods. The or each processor may include one or more cores. Examples of such processors include general-purpose CPUs (central processing units); and coprocessors, accelerator processors, or specialized processors, such as dedicated memory processors, or repurposed GPUs (graphics processors), DSPs (digital signal processors), or cryptographic processors.

[0026] In an embodiment, the storage medium is a quartz glass-based storage medium. However, this is not limiting. More generally, the storage medium can be any WORM (write-once, read-many) medium, or ROM (read-only memory), or even rewritable (but non-volatile) memory. The difference between ROM and WORM is that ROM is written to in the factory, while WORM can only be written once in the field by the end user (consumer). The storage medium can be an optical medium such as quartz glass or an optical disk (e.g., a CD or DVD); a magnetic medium such as a hard drive, floppy disk, or magnetic tape; or an electronic medium such as a hardwired ROM, EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), or flash memory or a solid-state drive. Other types of storage media may also be familiar to those skilled in the art, and most generally, the disclosed methods are applicable to any type of computer-readable medium for storing data.

[0027] In an embodiment, the storage medium may comprise a single, continuous piece of writable material, such as a single sheet of quartz glass or a single magnetic disk. Alternatively, the medium may comprise more than one piece of writable material, such as multiple sheets of glass or multiple magnetic disks. In the latter case, in addition to the redundancy of a large number of sectors within a platter, the disclosed scheme may also provide redundancy across multiple individual pieces of writable material ("platters"). This application space may be particularly important, for example, in archival storage.

[0028] Regardless of the form of the medium, the information to be stored is in the information sector group S i , i = 1 ... n. In the notation used in this paper, i represents the corresponding information sector S in the group i The index i is represented by n, and there are n information sectors in the group. For notational purposes, the index i is represented as ranging from 1 to n to indicate that it can take values indicating any of the n information sectors. This does not necessarily mean that the actual numerical parameter used by the reading or writing computer to index the sector will numerically range from 1 to n (for example, in practice this may be in binary form ranging from 0 to n-1). It will be understood that this is just a convenient mathematical or symbolic representation.

[0029] The group of n information sectors can be all the information sectors on a medium (e.g., glass, disk, or tape), or it can be just one of a plurality of groups of information sectors on the medium. In the latter case, the following scheme can be applied to just one of the groups, or to each of some or all of the groups independently. The following describes storage and encoding within a given group.

[0030] Each information sector S i Represents a separate portion of data. Each information sector S iStored in different corresponding physical sectors of the storage medium. In an embodiment, each physical sector in the physical sector can be the same size as each other. In an embodiment, the information sectors stored on the physical sector can each have the same size as each other. In an embodiment, the physical sectors within a group can be continuous with each other, or can be interleaved with sectors of one or more other groups. Each physical sector includes a plurality of storage elements, each for storing a single basic symbol (e.g., a bit) of data. For example, in a 3D quartz glass medium, these would be single voxels. In an embodiment, the storage elements within a given physical sector can form a continuous series or array of elements, such as a continuous rectangle or rectangular cuboid of voxels. However, this is not necessary, and the sector can be just a logical sector that is not bound to the underlying physical layout of the physical storage elements.

[0031] Each information sector S i At least the information payload v i , i.e. data content (user data). This is the actual data that the user wishes to store. In an embodiment, each information payload may include a vector of constituent information values and is treated as a vector of constituent information values. Therefore, the information payload v i It may also be referred to as an information vector in this article.

[0032] The reading computer will also need some mechanism to detect each information sector S i Information payload v i To achieve this, the corresponding error detection code z i With information payload v i In an embodiment, this is associated with its corresponding information payload v i Stored together in each single information sector S i For example, append to the corresponding payload v i However, alternatively, it could in principle be stored elsewhere on the same medium, or even on a different medium on the reading computer. Regardless of where it is stored, the error recovery code z i A small piece of redundant data, such as a parity bit or a checksum, that allows a reading computer to detect (but not necessarily correct) errors in the payload and error detection codes when reading the payload. i This can include errors in initially placing the message payload v i Errors that occur when writing to the media, or errors that occur due to degradation on the media in the time between writing and reading, or read errors that occur during the reading process, or a combination of any two or more of these.

[0033] In addition to the information sector S iand error detection code z i In addition, multiple redundant codes r are calculated j , j = 1 ... k. These enable recovery of the information payload v in the event of any type of error (writing, degradation or reading) detected while reading i One or more information payloads in , for example based on (multiple) error detection codes or even just a complete read failure. In an embodiment, each information payload may include a vector of constituent elements and may be processed as a vector of constituent elements. Thus, the recovery code r j It may also be referred to herein as a redundancy vector.

[0034] Note: Error detection code z i are also redundant data, but they are not included for the purpose of error recovery and, in embodiments, do not allow error recovery, or at best only allow error recovery for information subject to simpler or more limited errors (if used for this purpose). To distinguish the error detection code, the redundant code r j Also called an error recovery code.

[0035] In the notation used herein, j represents the number of sectors of information S for a given i The redundancy code (recovery code) of the group r j The index of a corresponding redundant code in the group, and there are k redundant codes for this group. For the purpose of notation, the index j is represented from 1 to k to indicate that it can take values indicating any k redundant codes. This does not necessarily mean that the index used by the reading or writing computer to index the redundant code r j The actual numerical parameter of ranges numerically from 1 to k (eg, in practice this may range from 0 to k-1 in binary). It will be appreciated that this is merely a convenient mathematical or symbolic representation.

[0036] In an embodiment, the redundant code r j Sets are stored in sectors with information S i On the same storage medium, in at least one separate redundant sector R. This may be a separate physical sector of the storage medium. In some such embodiments, each code is stored in a different corresponding redundant sector R. i In some embodiments, each redundant code r j Can be combined with each information vector v i or the size of the information sector. In an embodiment, each redundant sector R i Each information sector S i Alternatively, each redundant code r j The size of the information vector v may be different from (eg, smaller than) i Or information sector S iThe size of the redundant sectors R may be such that more than one code can be stored in each redundant sector R. In other alternatives, it is not excluded that the redundant codes are not stored in separate sectors and may be stored in the information sector S. i Distributed between.

[0037] Note that for a given information sector S i A group does not necessarily have the same number or redundancy r as the information sectors in the group j (k is not necessarily equal to n), and in a given group of redundant codes r j With information sector S i There is no one-to-one mapping between . Instead, as will be discussed in more detail later, each redundant code r j is the n information sectors S in the group i The minimum information payload v i Generally speaking, the total number k of redundant codes may be smaller than or the same as the number of data items n, or in an embodiment, the number k of redundant codes may also be larger than the number of data items n.

[0038] In addition, for calculating the redundant code r j (See below) j,i It may be stored on the medium together with the code itself, for example in (multiple) redundant sectors R. In this case, the reading computer will read the coefficients from the medium and use them to calculate the redundant code r j Alternatively, the coefficients may be determined according to a predetermined deterministic process, such as a pseudo-random process, and only an indication of the process used may be stored in the medium, for example also in the redundant sector(s) R. In this case, a reading computer would read the indication from the medium, use it to determine the process used to determine the coefficients, use this process to determine the coefficients themselves, and use these coefficients to calculate the redundant code r j The indication may, for example, comprise a seed for the pseudo-random process. Alternatively, it may also comprise an indication of which algorithm, from among a plurality of available forms of algorithms, is to be used as the predetermined process. Alternatively, the form of the algorithm may be assumed by the reading computer.

[0039] In another variation, the redundant code r does not need to be j or coefficient c j,iAny instructions for determining the coefficients or processes may be written to the storage medium itself. Alternatively, any of these may be transmitted to the reading computer via a separate medium (e.g., a communication channel). For example, they may be published, or may be sent specifically to the reading computer, for example, over a network or on a separate storage medium such as a dongle. In the case where the storage and reading computers are the same computer, communication may simply involve storing the code, coefficients, and / or instructions in question locally on the computer, for example, on a local hard drive.

[0040] Figure 2 A coding method that can be applied to a storage computer is shown for use with reference to Figure 1 Store data in the described form and determine the redundancy code r j The method is performed by software stored in a memory and running on at least one processor of a storage computer. The memory may include one or more storage units, which may include, for example, the sectors S described above with respect to which information is stored. i The storage medium may be any type of medium discussed, and / or different types, such as RAM (random access memory), etc.; and it may be the same or a separate unit from the storage medium, or a combination thereof.

[0041] At step 210, the method determines n information sectors S to be written to a storage medium (eg, quartz glass). i group, i=1...n. This step involves determining at least n corresponding information payloads v i , each sector S i A. This is the user information (ie, content) that the method uses to store and protect with redundant code. Step 210 may also include: i Generate the corresponding error detection code z i (E.g. one or more checksum bytes.) Alternatively, these could be stored elsewhere on the medium, or in principle even on a different medium, although this would make error detection slower.

[0042] Steps 220 to 230 then generate redundant codes, for example to be stored in redundant sector(s) R. The method groups the information bytes into k redundant codes or codewords r j , j = 1 ... k (e.g. a codeword may consist of 2 bytes). These may be described as random linear codes or linear independent codes, for reasons explained later.

[0043] At step 220, the method selects coefficients for performing the linear combination. To accommodate multiple information payloads v per group i The possibility of error in the method will generate multiple redundant codes r1, r2, ..., rk Therefore, at step 220, the method generates j=1..k redundant codes r j Each redundant code in selects a different coefficient set c j，i Each set (each for the corresponding code r j ) includes n non-zero coefficients c j，i , a non-zero coefficient for i=1...n information sectors S i Each information sector in . For example, these coefficients can be randomly selected. These sets together can be described as coefficients c j，i A k×n matrix C of , where j=1...k and i=1...n.

[0044]

[0045] Each row corresponds to k redundant codes r j , j = 1...k different sets of coefficients for a different redundant code. The k sets should be linearly independent of each other, or their probabilities should be within some tolerable threshold when randomly selected. The process of selecting these coefficients will be discussed in more detail later.

[0046] Each column in the matrix C corresponds to a matrix for n information sectors S i , i=1...n are coefficients for weighting the data of different corresponding sectors.

[0047] At step 230, the method calculates each redundant code r in the redundant code for a given j j As a linear sum:

[0048] r=c1·v1+c2·v2+…+c n ·v n (2)

[0049] In other words:

[0050] r1=c 1，1 v1+c 1，2 v2+…+c 1，n ·v n

[0051] r2=c 2，1 ·v1+c2·v2+…+c 2，n ·v n

[0052] …

[0053] r k =c k，1 v1+c k，2 v2+…+c k，n ·v n(3)

[0054] or:

[0055] r j =c j，1 v1+c j，2 v2+…+c j，n ·v n，j =1...k (3a)

[0056] Each redundant code r j is the sum of n entries, one entry for each of the n information sectors S in the group being encoded i Each information sector in. By i Take the multiplicand and multiply it by the corresponding coefficient c of the information sector j，i As shown above, in the embodiment, the multiplicand of each term is simply the information payload v i The error detection code z can be ignored in the construction and recovery of redundant codes. i (e.g., checksum bytes); these are used to check for errors in the sector, but are not used for the cross-sector redundancy code described herein. j However, in an alternative embodiment, it is not excluded that i Additional data, such as the corresponding error detection code z i , can be included in the corresponding multiplicand of each term.

[0057] In an embodiment, each information payload v in the group i can include and be treated as a vector of information values (elements), e.g., a vector of individual bits or bytes. Similarly, each redundant code r j A vector of redundant elements may be included and treated as a vector of redundant elements. Thus, the information payload and the redundancy code may be referred to as an information vector and a redundancy vector, respectively. However, it should be understood that this is not limiting, and in other variations below, the information payload v i and / or redundant code r j Each of can be considered as a single scalar value.

[0058] In a preferred embodiment, the information payload v i and redundant code r j Each of them is a vector, coefficient c j,i Each coefficient in is a scalar.

[0059] Observe that the above addition and multiplication operations are performed in a finite field. For example, if the coefficients and codewords each consist of 2 bytes (16 bits), then the appropriate field for the operation is, for example, the Galois field GF (216 ). Otherwise, the symbolic representation follows standard algebraic rules: the constants c1, c2, …, c n and the corresponding vectors v1, v2, ..., v n Each element of is multiplied; furthermore, vector addition is performed element-by-element.

[0060] Note also that, in general, k may be less than, equal to, or greater than n, depending on the implementation.

[0061] Since there are usually multiple information vectors v in a group i The method generates multiple redundant vectors r1, r2, ..., r for a given group. k As mentioned above, the corresponding coefficients are expressed as for the redundant vector r j c i,1 , c i,2 ,…,c i,n , where j ranges from 1 to k. All c j,i The value of is non-zero. In addition, the value of c for different coefficient sets j =[c j,1 ,c j,2 ,…,c j,n ] are linearly independent of each other, where each set is for the redundant code r j A set of coefficients of a redundant code in , i.e. a row of the matrix in formula (1), corresponds to a row in formula (3). j＝1 The coefficient set c1=[c 1,1 ,c 1,2 ,…,c 1,n ] and each coefficient set c2=[c 2,1 ,c 2,2 ,…,c 2,n ]…c k =[c k,1 ,c k,2 ,…,c k,n ] etc. (Note that the vectors in this paper are different types of vectors that encode user or redundant data - in this paper, vector c j Represents a set of coefficients, each information vector v i＝1…n one).

[0062] "Linearly independent" means that a set cannot be created from a linear combination of other sets. That is, for any given set c for a given set of information sectors (i.e., the set of coefficients used in Equation 3), j＝a =[c a,1 ,c a,2 ,…,c a,n ], there is no scalar factor set β such that c j＝a =(β1·c1)+…+(βa-1 ·c a-1 )+(β a+1 ·c a+1 )+…+(β k ·c k ), where c j represents [c j,1 , c j,2 , …, c j,n . This condition is equivalent to saying that each set of coefficients adds a new redundant message to the redundant code. If a set of coefficients in the set of coefficients does not satisfy this condition, the corresponding redundant code generated from this set of coefficients does not add new redundant information to the code, so the information payload v i is one less than the redundant code.

[0063] Assuming the linear independence condition is satisfied, k redundant vectors can be used to recover the user information v i from any k lost user sectors S i . Similarly, if there are k' < k lost user sectors, any k' redundant vectors can be used to reconstruct the lost user sectors (with the aid of the n - k' user sectors that have been read without error). The condition for the set of coefficients c j to be linearly independent is equivalent to a matrix of row rank k in equation (1).

[0064] To achieve linear independence, the coefficients of the set can be selected according to a pseudorandom process. This alone does not strictly ensure linear independence. However, this means that the set is linearly independent within a certain threshold probability range. In some embodiments, the method can simply include pseudorandomly selecting the set of coefficients and not checking the linear dependence condition, assuming that these sets may be linearly independent within a tolerable threshold probability, and if they are proven not to be linearly independent, the result is tolerable (i.e., there is one less error information sector S i in the group of n sectors that can be corrected). In other words, simply randomly generate the coefficients and then hope for the best. However, alternatively, a selection process that ensures the set of coefficients is linearly independent can be used to select the set of coefficients. For example, this can include pseudorandomly selecting them and then checking that the selected set is linearly independent, and if not, using a pseudorandom number generator to reselect one, some, or all of the coefficients until the linear independence condition is satisfied. Another possibility is to use a pre-designed set of coefficients.

[0065] At step 240, the information sector S i is written to the storage medium. The (multiple) redundant sectors R can also be written to the medium, and in an embodiment, this can also include the coefficients or the seed of the process for generating the coefficients (see below). Alternatively, some or all of this redundant data can be separately transmitted to the reading computer. Furthermore, it will be understood that Figure 2 This is given only as an example, and in other variations of the method, the information sector S may be stored before step 220 or 230. i Write to storage media. The order of steps is not important unless there are dependencies in the information generated.

[0066] Reference again Figure 1 and formula (3), note that the redundant code r is used to encode j The encoding scheme is a systematic encoding scheme. That is, it makes the payload information v i itself is not transformed, and the redundant information r j With payload information v i Separate. In contrast, in non-systematic encoding schemes, such as those previously used in network coding, as disclosed in US Pat. No. 7,756,051, the encoding mathematically transforms the payload itself. In other words, the redundancy is distributed across the entire information sector or packet. In previous non-systematic approaches for network encoding and decoding, all content received at the receiving end is encoded, and decoding is required to read each packet. In the systematic approach disclosed herein for storage, only erroneous sectors need to be decoded, and the computational overhead scales with the number of lost sectors.

[0067] In the case of a system, the information payload v i The user data may be in the clear, or may be transformed by some lower coding layer (such as for media compression and / or encryption), but either way, it is not transformed by a coding method or scheme at the layer where the redundancy codec is added (i.e., generating said redundancy codes / vectors), and the redundancy codes are not stored in physical storage elements of the same medium as or overlapping with the payload (e.g., they are separate voxels).

[0068] However, please note that the scope of this disclosure is not limited to the systematic case. In alternative embodiments, non-systematic cases can also be used for storage. In the non-systematic case, this means that only redundant codes will be stored on the storage medium, and no "original" information sectors will be stored on the medium. In this case, as long as at least n codes are read correctly, the information sectors will be completely recovered from the codes.

[0069] In the case of system or non-system, the redundancy vector r is increased j＝1…k The number k of redundancy vectors increases the probability of successful recovery of user data (information sectors). However, this also increases the computational cost (for creating the redundancy vectors). The choice of k is a system parameter that can be specified at design time.

[0070] Given a choice of the number of information sectors per group (n) and the number of redundant vectors (k), it is desirable to choose a coefficient set c that guarantees (or within a tolerable probability)j is linearly independent of c i,j appropriate value of .

[0071] A mechanism is also needed to convert the code j and coefficient c j,i The data is then transferred to the reading computer that will perform the decoding. One approach is to fix the coefficient values and make them known to the process of generating the redundant vector and the process of recovering the lost vector. This method requires fixing n·k coefficients, which actually requires a lot of space to store these values.

[0072] An alternative is to store the coefficients together with the redundant vector on the storage medium, for example in (multiple) redundant sectors R. In this method, in the data corresponding to the redundant vector c j Some space is reserved in the redundant sectors of the j,1 , c i,2 ,…,c k,n . This will require additional overhead for each vector.

[0073] A third alternative is to design a deterministic process that generates the coefficients and has a short description. One implementation is to use a well-known algorithm to generate pseudo-random numbers. In this approach, the designer will determine the algorithm and initial seed for generating the pseudo-random number sequence (and thus the coefficients). The designer will preferably check that the generated random numbers satisfy the linear independence assumption, for example by checking that the first n·k codewords define a matrix of the form (1) and that the matrix has row order k. This check only needs to be done once for each seed. In this approach, no additional information needs to be stored in each sector. However, it will be required to be able to map the physical location to a specific sector. That is, to be able to identify the location of each information or redundant sector on the glass. For each sector read, the decoding method on the reading computer will need to be able to derive the group to which the sector belongs, and whether the sector is the i-th information sector or the i-th redundant sector. This can be achieved by fixing the location in the medium to groups and sectors.

[0074] In other variations of any of the above, the redundant codes, coefficients and / or seeds (or other such indicators of a deterministic process) may be transmitted to the reading computer via a separate medium, such as on an accompanying dongle or over a network communication channel.

[0075] Another remark concerns the error detection code (e.g., checksum) added to each sector. i In an embodiment, a sector contains information and redundant codewords, as well as additional information that can be used to detect errors in the sector. i. In such embodiments, these checksums are preferably not used to create redundant codes. Instead, in some such embodiments, the bytes of the redundant sector R are first calculated, and then these bytes can be used to calculate the checksum(s) of the corresponding redundant sector(s).

[0076] Figure 3 Illustrates a decoding method that can be applied to a storage computer for reading data from a medium as described in the reference Figure 1 including using a redundant code r j to recover an error sector. The method is performed by software stored in a memory and running on at least one processor of the reading computer. The memory may include one or more storage units, which may include, for example, any type of medium discussed above with reference to the storage medium storing information sectors S i and / or different types, such as RAM (random access memory), etc.; and it may be the same or separate unit as the storage medium, or a combination thereof.

[0077] At step 310, the method begins with the process of reading a sector from a storage medium (e.g., glass). After reading the sector S i (or actually R), the method uses the corresponding error detection code z i (e.g., checksum) to determine whether the sector is error-free. The process also knows the group to which the sector belongs, as well as the position of the sector within the group (whether it is the i-th information sector or a redundant sector). Alternatively or additionally, an error can be detected when a complete read of the sector fails. In the case of error detection using the code z i , a zeroed assumed value is successfully read from each bit or symbol of the sector, but at least one of these values is detected as incorrect based on the redundant information in the corresponding error detection code z i ; while in the case of a read failure, a zeroed value cannot be read from at least one bit or symbol in the sector.

[0078] In either case, assume that after reading, the method fails to correctly read k' < k information sectors S i (for groups i = 1...n), and it has at least successfully read k' redundant codes (thus, has correctly read n - k' information sectors S i ). Then, the recovery process proceeds as follows.

[0079] At step 320, the method subtracts the correct information sectors from each of the redundant codes. Assume that the information vector v j has been correctly read for n - k' values of i. This means that the method updates each redundant vector r j as follows:

[0080] r j ←r j -c j，i ·v i (4)

[0081] where j now uses k' redundant codes r in the recovery j index in . For ease of notation, this may be expressed as j=1…k'. It will be understood that this is again just a convenient mathematical notation and does not restrict the form taken by the actual numerical parameters used to reference the code on the reading computer. Furthermore, strictly speaking, this is not necessarily the same sequence of j values used to count through the k redundant codes during encoding. As a matter of notation, the new index may alternatively be labeled j', but the simpler j notation is adopted hereinafter. In either case, this notation is not intended to indicate that the k' redundant codes used for decoding are the first (lowest) k' index codes in the sequence of k redundant codes indexed during encoding (it could be any k' of them, not necessarily the first k' in the sequence indexed during encoding).

[0082] From this point in the manual, r j refers to the updated redundant code after step 320, and j refers to the index of the k' updated redundant code r used in the recovery. j The index of .

[0083] At step 330, the method proceeds by removing the information sectors S corresponding to the correct i The matrix of formula (1) is updated by adding the columns (those determined to be correctly read at step 310). In doing so, the matrix (1) is transformed into a k'×k' matrix. This reduced matrix may be labeled E herein. During the reduction process, a mapping of the index of the original block to the updated index is also stored.

[0084] At step 340, the method inverts the reduced matrix to produce D = E -1 . Inverse matrix E -1 Is to make E·E -1 =1 matrix.

[0085] At step 350, the method recovers the lost information sectors by performing the following operations:

[0086]

[0087] For the i-th error sector S i , which is the sum of all j of the k' redundant codes used in the recovery. For the incorrect or unsuccessfully read information sector S determined to be erroneous at step 310 iThe summation is performed separately for each information sector i in .

[0088] Finally, v is converted using the mapping from step 330 above. i Remap to the appropriate lost sector.

[0089] The redundant code r used in the above decoding method j and coefficient c j,i The value of may be transferred to the reading computer process by any of the means previously discussed; or by i The data may be included together on the storage medium itself (eg, on one or more redundant sectors R), or transmitted via a separate medium, or a combination of these methods.

[0090] Note that the above has been described for the systematic case, where the n information sectors themselves are stored on the storage medium, and recovery is performed only for the minimum number required to recover the lost or erroneous information sectors (i.e., the number of recovery codes k' used in the recovery is equal to the number of information sectors to be recovered). The codes that are both present and error-free are simply read directly from the medium. However, in principle the method can be used for any k'×k' matrix, where k' is the number of redundant codes used in decoding and is also equal to the number of information sectors that the decoding method expects to read. In the extreme case, the method can even be applied to the non-systematic case, where only redundant codes are stored on the medium (no information sectors), and all desired information sectors are recovered from the code. In this case, there is no step 320 (Formula 4) of updating the code based on the correctly read information sectors, and the information vector v is recovered entirely from the k' redundant codes. i , where, in this case, k' is simply the number of information vectors to be recovered, not the number of losses or errors, and r used in decoding j There are simply j=1...k' codes used in decoding.

[0091] There are several parameters to consider in the encoding scheme that will affect decoding.

[0092] One is the ratio of k (the number of redundancies provided) to n (the number of information sectors to be encoded). If the coefficients are chosen pseudo-randomly without actively ensuring that they are linearly independent, this will have an impact on the chances of constructing an efficient code.

[0093] In any computer, a given value must be represented in a finite field (also called a Galois field) of size L, and in an embodiment, addition is performed in a wraparound (modulo) manner within the field. For example, if the field size L is 8 bits, then after 255, adding 1 wraps around to 0.

[0094] Let n be the number of information blocks. When constructing a new code, a random number c is selected i combined with v i (see also Equation 2). The number c i can be any number in the field other than zero. Therefore, there are L = 2 16 - 1 ways to pick c i . There are L n ways to pick the n values of c i .

[0095] If k (< n) codes have been selected, the probability that a randomly generated code (as described above) depends on the existing codes is given below (i.e., the new code is not good because it does not have the linearly independent property).

[0096] The existing codes can generate (i.e., span) (L + 1) k codes. This is because the existing codes r1, r2,..., r k can be linearly combined as d1·r1 +... + d k ·r r for random coefficients d1,..., d k from the field. Since the values of d i can take on zero values, the number of linear combinations of the codes is (L + 1) k .

[0097] Therefore, the probability that a randomly generated coefficient vector (i.e., code) is one of the coefficient vectors spanned by the existing codes is: (L + 1) k / L n ≈ L k-n = 1 / (2 16 - 1) n-k . When k is close to n, the probability of a wrong selection increases. However, in an embodiment, k can be less than 10% - 20% of n, and n can be in the thousands. Therefore, the probability of a wrong selection is small.

[0098] Note: Different from other redundant coding and decoding schemes for storage, in this scheme, n is not limited by the field size L. In other existing redundant schemes, n + k is limited to be less than L. However, in this scheme, n + k can be greater than L. The number of codes that can be generated is limited to n. The random construction of the code works better for larger fields (see the denominator in the above equation), but the influence of n is still more significant.

[0099] Another parameter to consider is the group size. In particular, the advantage of using a larger group size (larger n) is that it increases the chance of recovery.

[0100] In a group with n information sectors and k associated redundant codes, there will be a maximum number of errors that can be tolerated (at least n of the n information sectors and k codes will need to be successfully read). For example, if there are 8 information sectors and 2 associated redundant codes, the system can tolerate up to 2 errors in the 10-sector content consisting of 8 information sectors and 2 redundant sectors and still recover the entire group. That is, the total number of information sectors and codes that are successfully read is at least the number n of information sectors that were originally written to the media in the group.

[0101] However, errors in information sectors stored on a storage medium are random. With a finite group size, there's always the unfortunate chance of having more erroneous sectors than the designed number of redundant codes. With small group sizes, this chance can be quite high. For example, in the example above, statistical fluctuations in the number and distribution of errors could easily result in a given group having three or more erroneous information sectors, making it unrecoverable. The larger the group size, the smaller the chance of being more unfortunate than planned. That is, as n approaches infinity, the probability of a group being unrecoverable for a given number of associated redundant codes approaches the theoretical statistical value.

[0102] Again, assume that n is the number of information blocks per group, and that k redundant blocks are generated per group. Furthermore, assume that p is the probability of correctly reading a (information or coding) block. On average, we expect (1-p)·(n+k) failures. As long as the number of failed blocks is less than k, then (according to the linear independence assumption in the construction of the coding blocks) it should be possible to reconstruct the lost block. If more than k failures are observed, the decoder will be unable to recover at least one block of the group.

[0103] The probability of failure is P fail ≤ exp[-(n+k)·D(n / (n+k)||p)], where D[a||p] = a·log(a / p) + (1-a)·log((1-a) / (1-p)). (This can be derived from the tail bound of the binomial distribution, assuming k / n > 1-p; the D(a||p) function is the relative entropy.) The probability of failure is observed to decrease exponentially with the group size n. Therefore, larger n is better.

[0104] However, there is also the D(·||·) term. This requires that for a reasonable amount of overhead, n should be in the order of thousands compared to the expected failure probability (1-p) to ensure a very low failure probability. In storage, it is desirable to have a very low failure probability (P fail It should be 10 -x , for as large x as possible), since failure to retrieve data is practically unacceptable. Thus, in embodiments very large group sizes may be desired.

[0105] In some embodiments, the group size may even be all information sectors as a whole.In some embodiments, the group may span multiple sheets of writable material (eg, multiple sheets of glass).

[0106] Now refer to Figure 4 Other optional optimizations are discussed. This provides for faster recovery of relatively small groups of solutions and / or incremental use of solutions.

[0107] The above recovery process is applicable to the entire sector group S i (i=1...n). This means that, to recover a sector, if any n sectors are successfully read out of a total of n data sectors and k redundant sectors (or more generally, any n information vectors and / or redundant codes are successfully read out of n information vectors and k redundant codes), then the method is able to recover the data. This method works best when the goal of the read process is to recover the entire contents stored in the storage medium (e.g., glass). This may be acceptable for applications such as archival storage, where only occasional reads are required and slow recovery times are acceptable. However, in some cases, it may be desirable to read only a subset of the data correctly, and to do so more quickly.

[0108] To reduce the workload of recovering a subset of sectors, the information sectors can be organized as follows: Figure 4 The matrix form shown in the example (information sectors are indicated by boxes). Each row (redundant sector r x,i,j ) and each column (redundant sector r y,i,j ) calculates the redundant sectors. For example, r x,1,1 to r x,1,k’ You can use as input v 1,1 to v 1,n and constructed in the same way as before; and similarly, for r y,1,1 to r y,k’,1 , using input v 1,1 , v 2,1 ,…,v n,1 In the example shown, there are m rows and m columns (a square matrix), but note that more generally the number of rows can be different from the number of columns.

[0109] Using smaller groups allows for faster recovery. It also allows for incremental use of the scheme in encodings. For example, a column with its redundancy can be written once, and long after many columns have been written, the row redundancy can be written to provide additional redundancy protection. The row redundancy can even be chosen based on the actual error rates observed on the earlier columns.

[0110] In order to recover the information vector v i,j , we can use the redundant vector from row i or column j. Observe that the number of information vectors above is m·m=m2 , and the number of redundant vectors is 2·m·k'.

[0111] Before describing in detail the process of recovering sectors for this embodiment, a few observations are made.

[0112] To combine this method with the previous section ( Figures 1 to 3 ) basic scheme, considering n = m 2 or The basic scheme will need to read between n and n+k sectors to recover the lost sector; the scheme in this section can use row redundant or column redundant sectors to Recovering a large amount of workload.

[0113] ·refer to Figure 4 The example described uses a square m×m matrix: the number of rows is equal to the number of columns (m). The number of redundant sectors for rows and columns is assumed to be the same (k'). Alternative implementations may use different dimensions in rows and columns, depending on the size of the rows or columns and other parameters of the system, and may also adjust the number of redundant sectors per row or column appropriately.

[0114] In the above, linear coding is used to construct both row and column redundancy vectors. Other methods can be used to construct row redundancy sectors or columns (or indeed both). That is, to recover lost blocks from either rows or columns, linear codes can be used on both, or a linear code can be used to recover lost blocks for each row and a different redundancy coding scheme can be used to recover blocks for each column, or vice versa.

[0115] exist Figure 4 In this arrangement, the process of reading a sector is as follows. First, try to read it directly. If successful, stop. The cost of the read is the cost of reading one sector. If that fails, try to recover from the row (or column). The cost of reading a sector in terms of m+1 to m+k (depending on how many sectors are in error). If that also fails, try to recover using column redundancy (or row redundancy). This will add another cost, which is between m+1 and m+k sectors read.

[0116] In case the recovery using row and column redundancy fails, the recovery continues in an iterative process. Observe that the recovery using both row and column redundancy fails, so there are multiple sectors in error. Make a list of the lost sectors and pick one of these sectors, say v i,j’ , and try to recover it using the information in column j' (observing that v i,j’ With the target sector v i,j In the same line). j' refers to the sector containing v i,j’ Column (where v i,j’If successful, try to recover the original sector v i,j , or repeat for another lost sector. If recovery is not possible i,j’ , then add all missing sectors from column j' to the list and repeat for another sector.

[0117] Once the process has been resumed, it is allowed to resume v i,j or the process has read all information sectors v and all redundant sectors r and d and it is still impossible to recover v i,j In the latter case, the recovery fails.

[0118] Note that in the worst case, the performance of the table arrangement is comparable to the baseline scheme in the previous section, i.e., the process will need to read all n = m 2 sectors and all redundant sectors. However, in common cases, the error sector should be read sectors are then decoded.

[0119] As another optional optimization, additionally or independently of the reference Figure 4 With the described optimization, information from different recovery coding groups may be stored physically interleaved with each other on the storage medium (ie, their actual physical locations are spatially interleaved).

[0120] Defects in the writing and reading process can cause errors in correctly read sectors. These errors can occur independently of each other, meaning that the probability of failure of a sector is the same for all sectors and a failure in one sector does not change the probability of failure in any other (adjacent) sectors. Errors can also be spatially correlated, for example, when a defect in reading or writing affects many "adjacent" sectors, i.e., sectors that are close to each other in physical space. Figures 1 to 4 The methods discussed can handle any error pattern. However, the performance of the decoding depends on the number of errors in the group: the cost of decoding increases with the number of erroneous sectors. Therefore, it may be desirable to avoid a situation where errors affect many sectors from the same group. Instead, it may be preferable to distribute these errors evenly across the groups.

[0121] It is expected that correlated errors may affect sectors that are physically close in the physical space of the material. To break these correlations, sectors from different groups can be interleaved, preferably maximizing the spatial distance between sectors from the same group. The exact layout depends on the physical properties of the medium.

[0122] Figure 4The examples are described using groups arranged in rows and columns, but more generally the same principles can be applied to any arrangement of information sectors arranged into overlapping storage subsets (overlapping groups of information sectors). Redundant codes are generated per subset (group), i.e., the redundant codes can be used to recover information sectors of the same subset, but cannot be used directly to recover information sectors that are not part of the same subset. Subsets can have overlapping information sectors; for example, subset A may contain information sectors s2, s3, s7, and s8, while subset B may contain s1, s2, s3, and s6. The decoding process works iteratively by identifying subsets that can be recovered and then using the (recovered) information sectors from these subsets to recover more subsets.

[0123] As an example, assume that a redundant code r is generated for subset A. A , and generate similar redundant code r for subset B B , and information sectors s2 and s6 are lost (all other s i 、r A and r B are all received correctly). Since there are two lost sectors in subset B, and B has only one redundant sector, it is impossible to recover subset B. However, there is enough redundancy to use subset A to recover s2. Then, subset B can be used to recover s6 using the reconstructed s2. It can be observed that recovery is possible even when s2 and s3 are lost: even if neither subset can be used to recover separately, r A and r B can be simplified to rewrite r as depending only on s2 and s3 (using the same procedure as in Equation 4) A and r B to depend only on s2 and s3, and then restore s2 and s3 using these relationships.

[0124] The motivation behind creating subsets is to be able to recover lost sectors of information using fewer sectors (i.e. local). This speeds up the decoding phase, for example by decoding smaller groups and by taking advantage of the placement of sectors from the same subset in nearby locations on the storage medium. (The runtime advantage comes at the expense of some codec efficiency).

[0125] The case of product codes (i.e., when we divide the information sectors into columns and rows) is a special case of the above scheme, where the subsets defined by the columns (rows) do not share any information sectors, and each subset defined by the columns has exactly one information sector shared with each subset defined by the rows.

[0126] To improve reliability, some storage systems place copies of the content at geographically different sites (data centers); for example, sites on different continents. Existing storage systems typically store identical copies of both the information and the redundant sectors at these sites. As another extension, the above method can be used to store different sets of redundant codes at each site. In other words, unique redundant codes are generated per site, for example by using a unique random seed per site to create a coefficient matrix to generate coefficients. In the event of severe errors at multiple sites, where the sectors correctly read by each site are insufficient to recover the original content, the redundant codes from multiple sites can be combined to increase the probability of successful recovery.

[0127] This scheme is equivalent to: assuming s sites, generating s·k redundant codes and assigning k redundant codes to each site. The storage overhead per site is the same as in the previously described procedure. However, if redundant codes are allowed to be combined across sites, the probability of successful recovery is equivalent to that of using s·k redundant codes.

[0128] It will be understood that the above-described embodiments have been described by way of example only.

[0129] More generally, according to one aspect disclosed herein, there is provided a computer-implemented method of reading from a storage medium to recover a group of information sectors, each information sector including a respective information payload; wherein the storage medium stores redundant data, the redundant data comprising a plurality of separate redundant codes for the group, each redundant code being a linear sum of terms, each term in the sum being the information payload of a different respective information sector from the information sectors in the group, the different respective information sectors being weighted by respective coefficients of a coefficient set for the redundant code; and wherein the method comprises, after the redundant data is stored on the storage medium, performing a decoding process, the decoding process comprising:

[0130] - identifying k' sets of information sectors from said group from which the corresponding information payload is to be recovered based on the redundant data;

[0131] - Select k' redundant codes from the redundant codes;

[0132] - determining a block matrix E, wherein each matrix column comprises a corresponding coefficient of a different one of the k' information sectors and each matrix row comprises a set of coefficients of a different one of the k' redundant codes, or vice versa;

[0133] - determining that the matrix D is the matrix inverse of E; and

[0134] - By calculating v i =∑ j (d i，j ·r j) to recover k' information payloads, where v i is the information payload, i is the index indicating the corresponding information sector, and j is the index indicating each redundant code r j The index of d i,j is a matrix element of D, and ∑ j is the sum of k' redundant codes, the calculation being performed for i of each of the k' information sectors.

[0135] In an embodiment, some or all of the information sectors are also stored on the storage medium. This is the systematic case. Alternatively, the method can be used in a non-systematic case, in which case only the redundant code is stored on the medium instead of the information sectors, and the information payload is fully recovered from the redundant code.

[0136] In an embodiment, the identification may include: identifying which information sectors in the information sector group are lost or erroneous, do not exist on the storage medium, or have information payloads containing errors. i , and identifies which information sectors are present and error-free, are found to be present on the storage medium and have information payloads v that do not contain errors i , wherein the k′ information sectors include lost information sectors and / or erroneous information sectors of the group.

[0137] In an embodiment, the k' information sectors may be only the lost and / or erroneous sectors of the group. Alternatively, it is not excluded that k' may be larger than strictly necessary, for example up to the same number of codes k for the group stored on the storage medium, even if there are fewer than k lost and erroneous sectors. However, this would be more computationally intensive than required for recovery.

[0138] In an embodiment, the method may include: before said recovery, performing r j ←r j -(c j，i ·v i ), to update each of the k' redundant codes, where c j,i is the coefficient corresponding to the i-th information sector and the j-th redundant code; wherein the redundant code r used in the recovery j is the updated redundant code.

[0139] In an embodiment, the set of coefficients may not be stored on a storage medium, and instead the method may include:

[0140] - reading from a storage medium an indication of a predetermined deterministic process for determining a set of coefficients and determining the set of coefficients using said process based on the indication, or

[0141] - receiving the coefficient set via a separate medium, or

[0142] - receiving, via a separate medium, an indication of a predetermined deterministic process for determining a set of coefficients, and determining the set of coefficients using said process based on the indication.

[0143] The separate medium may be another digital or computer-readable storage medium (e.g., a dongle) or a network (e.g., the Internet). As another alternative, the separate medium may even include another form of medium, such as paper or printed media, or an audible. Regardless of the method employed, receiving may include receiving the coefficients in a communication specifically addressed or sent to a reading computer, or alternatively via publication. For example, the coefficients may be published online.

[0144] In an embodiment, the coefficient sets of different redundant codes may be linearly independent of each other.

[0145] In an embodiment, the method may comprise an initial stage of storing the redundant code on a storage medium prior to said decoding process.

[0146] In an embodiment, the initial storage stage may include selecting coefficients according to a process that ensures that coefficient sets of different redundant codes are linearly independent of each other.

[0147] In an embodiment, the initial storage phase may include pseudo-randomly selecting coefficients.

[0148] In an embodiment, the information sector group may be all information sectors on the storage medium.

[0149] In an alternative embodiment, the group of information sectors may be one of a plurality of groups of information sectors on a storage medium, and the method may be applied individually to each of the groups.

[0150] In some such embodiments, information sectors from different groups may be physically interleaved on the storage medium.

[0151] In an embodiment, the group of information sectors may be one of a first group of information sectors stored on the storage medium and a second group of information sectors stored on the storage medium that overlaps with the first group, the group including some but not all identical information sectors, each of the first group and the second group being associated with a corresponding redundant code set. In such an embodiment, the method may include: when initially unable to recover information payloads of all information sectors in the first group based on the corresponding redundant code set associated with the first group, recovering information payloads of information sectors in the second group based on the corresponding redundant code set associated with the second group, thereby recovering at least one payload of information sectors overlapping with the first group; and subsequently recovering the first group based on the redundant sectors associated with the first group and the at least one recovered information sector in the first group.

[0152] The other of the first and second groups of information sectors may be encoded according to the same redundancy scheme or a different redundancy scheme.

[0153] In an example application of the various techniques disclosed herein, the storage medium may include a quartz glass storage medium.

[0154] In an embodiment, the storage medium may include multiple separate pieces of writable material, wherein the information sector and / or redundant code spans across the multiple pieces of writable material.

[0155] For example, the storage medium may comprise a plurality of individual disks, or even a plurality of storage units housed in a single housing. In an embodiment, the individual pieces of material have the same media type, but alternatively, it is not excluded that they may comprise different media types, such as glass and magnetic, etc.

[0156] Some or all of the information sectors may be replicated across each piece of material (a copy of the same information on each piece of material), or some information codes may be stored only on one piece of material, while other information sectors are stored only on another piece of material. In the case where redundant codes are stored on the media, some or all of the redundant codes may be replicated across each piece of material (a copy of the same code on each piece of material), or some information codes may be stored on one piece of material, while other information sectors are stored on another piece of material.

[0157] In some embodiments, these pieces of material may even be distributed across multiple different data centers in multiple different geographic locations.

[0158] Each information sector may also include a corresponding error detection code. In an embodiment, the multiplicand may be only the corresponding information payload, rather than the corresponding error detection code. Alternatively, it is not excluded that the multiplicand includes both the corresponding information payload and the corresponding error detection code.

[0159] The error may occur when storing one or more information payloads on the storage medium during writing, or may occur due to degradation after writing but before reading. The error may be detected during reading based on an error detection code. Once the error is detected, it may be recovered based on a redundancy code.

[0160] In an embodiment, each of said information payloads is a vector of information elements, each of said coefficients is a scalar, and said multiplication comprises an element-wise multiplication of a respective scalar coefficient of a respective information payload with each information element of the respective information payload.

[0161] In an embodiment, the sets of coefficients may be selected according to a process that ensures the sets are linearly independent of each other. Alternatively, the coefficients may be selected according to a pseudo-random process (thereby inherently having some likelihood that the sets are linearly independent of each other).

[0162] In an embodiment, the storage medium can be a glass-based storage medium, such as a quartz glass storage medium. Alternatively, the storage medium can be another form of optical storage medium such as an optical disc, or a magnetic storage medium, an optical storage medium such as a disk or tape, or an electronic storage medium such as an EEPROM or flash memory.

[0163] In an embodiment, the storage medium may be a write-once, read-many (WORM) storage medium.

[0164] In an embodiment, the method may be used for archival storage.

[0165] In an embodiment, the information payload may include plaintext user data.Alternatively, the information payload may be encoded by a lower layer encoding scheme.

[0166] In an embodiment, the redundant code may be stored on the storage medium, for example in one or more redundant sectors separate from the information sectors. Alternatively, the redundant code need not be stored on the medium.

[0167] For example, this may include publishing a code or instruction of the predetermined process, or transmitting them to a designated party or parties via a communication channel separate from the storage medium. The predetermined process may, for example, include a deterministic pseudo-random process, and the instruction may include at least a seed for the pseudo-random process.

[0168] The information sectors of the group may be interleaved with the information sectors of one or more other groups.Alternatively, the information sectors of the group may be physically contiguous on the storage medium.

[0169] Error detection may be based on corresponding error detection codes.

[0170] The method may further include directly reading non-erroneous payload values without requiring recovery.

[0171] In an embodiment, each of the elements of the inverse matrix D is a scalar, and each of the redundant codes is a vector. i =∑ j (d i，j ·r j The product “·” in ′) can be an element-wise multiplication.

[0172] According to another aspect disclosed herein, there is provided a computer program embodied on a computer-readable storage, the computer program comprising code configured to perform the method of any embodiment disclosed herein when executed on one or more processing units.

[0173] According to another aspect, a computer system is provided, comprising a memory and a processing device, the memory comprising one or more storage units, the processing device comprising one or more processing units, the memory storing code arranged to be run on the processing device, wherein the code is configured to perform a method according to any embodiment disclosed herein.

[0174] Other variations or uses of the disclosed technology may become apparent to those skilled in the art once given the disclosure herein.The scope of the present disclosure is not limited by the described embodiments but only by the claims that follow.

Claims

1. A computer-implemented method of reading from a storage medium to recover a group of information sectors, each information sector including a corresponding information payload; wherein the storage medium stores redundant data, the redundant data comprising a plurality of separate redundant codes for the group, each redundant code being a linear sum of terms, each term in the sum being an information payload from a different respective one of the information sectors in the group, the different respective information sectors being weighted by a respective coefficient from a set of coefficients for the redundant code, wherein the coefficients are selected according to a pseudorandom process; as well as The method includes, after the redundant data has been stored on the storage medium, performing a decoding process, comprising: identifying k' sets of information sectors from the group from which the corresponding information payload is to be recovered based on the redundant data; Select k' redundant codes from the redundant codes; determining a block matrix E, wherein each matrix column of the block matrix E comprises the corresponding coefficients of a different one of the k′ information sectors and each matrix row comprises the set of coefficients of a different one of the k′ redundant codes, or vice versa; Determine that matrix D is the matrix inverse of E; and By calculating v i =∑ j (d i，j ·r j ) to recover the k' information payloads, where v i is the information payload, i is an index indicating the corresponding information sector, and j is an index indicating each redundant code r j The index of d i,j is the matrix element of D, and ∑ j is the sum of the k′ redundant codes, the calculation being performed for i of each information sector of the k′ information sectors.

2. The method of claim 1, wherein some or all of the information sectors are also stored on the storage medium.

3. The method according to claim 2, wherein the identification comprises: Identifying which information sectors of the group of information sectors are missing or erroneous, are not present on the storage medium, or have information payloads that contain errors i , and identifying which information sectors are present and non-erroneous, are found to be present on said storage medium and have information payloads v that do not contain errors i , wherein the k′ information sectors include the lost information sectors and / or the erroneous information sectors of the group.

4. The method according to claim 3, wherein: The k' information sectors are only the lost sectors and / or the erroneous sectors of the group.

5. The method according to claim 3 or 4, comprising: Prior to said recovery, by performing r j ←r j -(c j，i ·v i ) to update each of the k' redundant codes, where c j,i is the coefficient corresponding to the i-th information sector and the j-th redundant code; The redundant code r used in the recovery j is the updated redundant code.

6. A method according to any preceding claim, wherein the coefficient set is not stored on the storage medium and instead the method comprises: - reading from the storage medium an indication of a predetermined deterministic process for determining the set of coefficients and determining the set of coefficients using the process based on the indication, or - receiving the set of coefficients via a separate medium, or - receiving, via a separate medium, an indication of a predetermined deterministic process for determining the set of coefficients, and determining the set of coefficients using the process based on the indication.

7. A method according to any preceding claim, wherein the coefficient sets for the different redundancy codes are linearly independent of each other.

8. A method according to any preceding claim, comprising: An initial stage of storing the redundant codes on the storage medium prior to the decoding process, wherein the initial storage stage comprises selecting the coefficients according to a process that ensures that the coefficient sets for the different redundant codes are linearly independent of each other.

9. A method according to any preceding claim, comprising: The redundant code is stored in an initial stage on the storage medium before the decoding process, wherein the initial storage stage includes pseudo-randomly selecting the coefficients according to the pseudo-random process.

10. A method according to any preceding claim, wherein: The information sector group is all the information sectors on the storage medium.

11. The method according to any one of claims 1 to 10, wherein the group of information sectors is one of a plurality of groups of information sectors on the storage medium, and the method is applied to each of the groups individually.

12. A method according to any preceding claim 11, wherein The information sectors from different ones of the groups are physically interleaved on the storage medium.

13. A method according to any preceding claim, wherein: The information sector group is one of a first group of information sectors stored on the storage medium and a second group of information sectors stored on the storage medium that overlaps with the first group and includes some but not all of the same information sectors, each of the first group and the second group being associated with a corresponding redundant code set; wherein the method comprises: recovering the information payloads of the information sectors in the second group based on the corresponding redundancy code set associated with the second group when the information payloads of all the information sectors in the first group cannot be initially recovered based on the corresponding redundancy code set associated with the first group, thereby recovering at least one of the payloads of the information sectors that overlap with the first group; and The first group is then recovered based on redundant sectors associated with the first group and at least one recovered information sector in the first group.

14. A method according to any preceding claim, wherein the storage medium comprises a plurality of separate pieces of writable material, wherein the information sectors and / or redundant codes span the plurality of pieces of writable material.

15. A computer program embodied on a computer readable storage and comprising code configured to perform the method of any preceding claim when run on one or more processing units.

Images