Compression / decompression using index correlating uncompressed / compressed content

Abstract

Compression of data that permits direct reconstruction of arbitrary portions of the uncompressed data. Also, the direct reconstruction of arbitrary portions of the uncompressed data. Conventional compression is done such that decompression has to begin either at the very beginning of the data, or at particular intervals (e.g., at block boundaries - every 64 kilobytes) within the data. However, the principles described herein permit decompression to begin at any point within the compressed data, without having to decompress any prior portion of the file. Thus, the principles described herein permit random access of the compressed data. In accordance with the principles described herein, this is accomplished by using an index that correlates positions within the uncompressed data with positions within the compressed data.

Description

BACKGROUND

[0001] In the information age, the ability to store and communicate data is essential.Notwithstanding, there will always be finite limits in our ability to practically store andcommunicate larger quantities of data. Accordingly, information that can be representedusing one format in a larger number of bits (i.e., uncompressed data) is often encoded intoa form that uses a fewer number of bits (i.e., compressed form) prior to storage orcommunication. However, it also takes some amount of time and processing power toperform compression and decompression.

[0002] Typically, compression can be accomplished by removing redundancy in theinformation. As an example, if a text file is being compressed, it is possible and highlylikely that certain words or phrases repeatedly appear in the uncompressed text file. Forexample, in this very patent application, certain words will be repeated with varying degreesof frequency. Compression could involve using a word dictionary that replaces certaincommonly used human-readable text sequences with smaller non-human-readablereplacements. Another way is to use a combination of literal text and copy instructions inthe compressed form. As an example, the LZ4 compression method compresses byreplacing the text by a combination of literals and copy instructions from prior portions ofthe uncompressed text.

[0003] The subject matter claimed herein is not limited to embodiments that solve anydisadvantages or that operate only in environments such as those described above. Rather,this background is only provided to illustrate one exemplary technology area where someembodiments describe herein may be practiced.BRIEF SUMMARY

[0004] This Summary is provided to introduce a selection of concepts in a simplifiedform that are further described below in the Detailed Description. This Summary is notintended to identify key features or essential features of the claimed subject matter, nor is itintended to be used as an aid in determining the scope of the claimed subject matter.

[0005] The principles described herein relate to compression of data being done in away that permits direct reconstruction of arbitrary portions of the uncompressed data.Conventional compression is done such that decompression has to begin either at the verybeginning of the data, or at particular large intervals (e.g., at block boundaries - every 64kilobytes) within the data. However, the principles described herein permit decompressionto begin at any point within the compressed data, without having to decompress any priorportion of the data. Thus, the principles described herein permit rapid random access of thecompressed data. In accordance with the principles described herein, this is accomplishedby using an index that correlates positions within the uncompressed data with positionswithin the compressed data.

[0006] Additional features and advantages will be set forth in the description whichfollows, and in part will be obvious from the description, or may be learned by the practiceof the teachings herein. Features and advantages of the invention may be realized andobtained by means of the instruments and combinations particularly pointed out in theappended claims. Features of the present invention will become more fully apparent fromthe following description and appended claims, or may be learned by the practice of theinvention as set forth hereinafter.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to describe the manner in which the above-recited and other advantagesand features can be obtained, a more particular description of the subject matter brieflydescribed above will be rendered by reference to specific embodiments which are illustratedin the appended drawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting in scope, embodimentswill be described and explained with additional specificity and details through the use of theaccompanying drawings in which:

[0008] Figure 1 illustrates an example computing system m which the principlesdescribed herein may be employed;

[0009] Figure 2 illustrates a flowchart of a method for compressing so as to permitreconstruction of arbitrary portions of the uncompressed data, in accordance with theprinciples described herein;

[0010] Figure 3 illustrates an environment in which compression and decompressioncan occur and will be frequently referred to when describing the compression of Figure 2;

[0011] Figure 4 shows an environment that is similar to the environment of Figure 3,except that the compressed data is shown having multiple sequential compressed segmentsas an example of the compressed data;

[0012] Figure 5 illustrates an environment that is similar to the environment of Figure4, except that the compressed segments are illustrated as including literal parts and copyinstruction parts, the copy instruction parts instructing to copy from prior or currentcompressed segments;

[0013] Figure 6 illustrates a flowchart of a method for decompressing arbitrary portionsof compressed data, in accordance with the principles described herein.

[0014] Figure 7 A illustrates an example compressed segment and is used to describe adecompression in a first example in which one arbitrary portion of the uncompressed datais accessed; and

[0015] Figure 7B illustrates an example compressed segment and is used to describe adecompression in a second example in which another arbitrary portion of the uncompresseddata is accessed.DETAILED DESCRIPTION

[0016] The principles described herein relate to compression of data being done in away that permits direct reconstruction of arbitrary portions of the uncompressed data.Conventional compression is done such that decompression has to begin either at the verybeginning of the data, or at particular large intervals (e.g., at block boundaries - every 64kilobytes) within the data. However, the principles described herein permit decompressionto begin at any point within the compressed data, without having to decompress any priorportion of the data. Thus, the principles described herein permit rapid random access of thecompressed data. In accordance with the principles described herein, this is accomplishedby using an index that correlates positions within the uncompressed data with positionswithin the compressed data.

[0017] Whatever compression technique is used, the compression tracks correlationsbetween at least some positions in the uncompressed data and corresponding positions inthe compressed data. The compression then constructs an index that records the correlationbetween the positions in the uncompressed data and the corresponding positions in thecompressed data. An index that includes these correlations is constructed and associatedwith the compressed data, so that the index is available for construction of arbitrary portionsof the uncompressed data from the index and from the compressed data. For example, theindex could include an entry for every regular interval of bytes in the uncompressed data(e.g., for each uncompressed byte address location that is a multiple of 256).

[0018] As an example, in one embodiment, the compression occurs one compressedsegment at a time, so that the compressed data includes a sequence of compressed segments.In that case, the index could include (for every so many uncompressed bytes) the identity ofthe compressed segment as well as the uncompressed offset within that compressed segmentthat represents that information (e.g., segment portion 23, 5 uncompressed bytes in).

[0019] This permits random access during decompression at any defined start positionof a piece ofuncompressed data to be accessed. Given that start position, the decompressionuses the index to identify a corresponding position in the compressed data. Thedecompression then navigates to that position, and from there traverses an appropriatenumber of further uncompressed bytes to find the beginning of the uncompressed portion toaccess. The decompression then begins decompressing until an appropriate size ofuncompressed portion is formed.

[0020] As an example, suppose that the uncompressed data to access is fromuncompressed byte 8250 to uncompressed byte 8260 (so 11 uncompressed bytes long).Suppose further that the compressed data is formed of a sequence of compressed segments,and that there is an index entry at regular intervals of 256 uncompressed bytes. Thedecompression will use the index entry for uncompressed byte address 8192 to identify thecompressed segment and uncompressed byte offset that corresponds to uncompressed byteaddress 8192. Suppose that that index entry identifies compressed segment 23 and 5uncompressed bytes in. The decompression would obtain compressed segment 23, andtraverse 63 bytes from the beginning of the compressed segment (potentially reaching intosubsequent compressed segments) to reach the start position. Then, the decompressionwould decompress from that position for 11 more uncompressed bytes to extract the desiredportion.

[0021] Because the principles described herein are performed in the context of acomputing system, some introductory discussion of a computing system will be describedwith respect to Figure 1. Computing systems are now increasingly taking a wide variety offorms. Computing systems may, for example, be handheld devices, appliances, laptopcomputers, desktop computers, mainframes, distributed computing systems, data centers, oreven devices that have not conventionally been considered a computing system, such aswearables (e.g., glasses). In this description and in the claims, the term "computing system"is defined broadly as including any device or system (or a combination thereof) that includesat least one physical and tangible processor, and a physical and tangible memory capable ofhaving thereon computer-executable instructions that may be executed by a processor. Thememory may take any form and may depend on the nature and form of the computingsystem. A computing system may be distributed over a network environment and mayinclude multiple constituent computing systems.

[0022] As illustrated in Figure 1, in its most basic configuration, a computing system100 includes at least one hardware processing unit 102 and memory 104. The processingunit 102 includes a general-purpose processor. Although not required, the processing unit102 may also include a field programmable gate array (FPGA), an application specificintegrated circuit (ASIC), or any other specialized circuit. In one embodiment, the memory104 includes a physical system memory. That physical system memory may be volatile,non-volatile, or some combination of the two. In a second embodiment, the memory is nonvolatilemass storage such as physical storage media. If the computing system is distributed,the processing, memory and / or storage capability may be distributed as well.

[0023] The computing system 100 also has thereon multiple structures often referred toas an "executable component". For instance, the memory 104 of the computing system 100is illustrated as including executable component 106. The term "executable component" isthe name for a structure that is well understood to one of ordinary skill in the art in the fieldof computing as being a structure that can be software, hardware, or a combination thereofFor instance, when implemented in software, one of ordinary skill in the art wouldunderstand that the structure of an executable component may include software objects,routines, methods (and so forth) that may be executed on the computing system. Such anexecutable component exists in the heap of a computing system, in computer-readablestorage media, or a combination.

[0024] One of ordinary skill in the art will recognize that the structure of the executablecomponent exists on a computer-readable medium such that, when interpreted by one ormore processors of a computing system (e.g., by a processor thread), the computing systemis caused to perform a function. Such structure may be computer readable directly by theprocessors (as is the case if the executable component were binary). Alternatively, thestructure may be structured to be interpretable and / or compiled (whether in a single stage orin multiple stages) so as to generate such binary that is directly interpretable by theprocessors. Such an understanding of example structures of an executable component is wellwithin the understanding of one of ordinary skill in the art of computing when using theterm "executable component".

[0025] The term "executable component" is also well understood by one of ordinaryskill as including structures, such as hard coded or hard wired logic gates, that areimplemented exclusively or near-exclusively in hardware, such as within a fieldprogrammable gate array (FPGA), an application specific integrated circuit (ASIC), or anyother specialized circuit. Accordingly, the term "executable component" is a term for astructure that is well understood by those of ordinary skill in the art of computing, whetherimplemented in software, hardware, or a combination. In this description, the terms"component", "agent", "manager", "service", "engine", "module", "virtual machine" or thelike may also be used. As used in this description and in the case, these terms (whetherexpressed with or without a modifying clause) are also intended to be synonymous with theterm "executable component", and thus also have a structure that is well understood by thoseof ordinary skill in the art of computing.

[0026] In the description that follows, embodiments are described with reference to actsthat are performed by one or more computing systems. If such acts are implemented insoftware, one or more processors (of the associated computing system that performs the act)direct the operation of the computing system in response to having executed computer-executable instructions that constitute an executable component. For example, suchcomputer-executable instructions may be embodied on one or more computer-readablemedia that form a computer program product. An example of such an operation involves themanipulation of data. If such acts are implemented exclusively or near-exclusively inhardware, such as within a FPGA or an ASIC, the computer-executable instructions may behard-coded or hard-wired logic gates. The computer-executable instructions (and themanipulated data) may be stored in the memory 104 of the computing system 100.Computing system 100 may also contain communication channels 108 that allow thecomputing system 100 to communicate with other computing systems over, for example,network 11 0.

[0027] While not all computing systems require a user interface, in some embodiments,the computing system 100 includes a user interface system 112 for use in interfacing with auser. The user interface system 112 may include output mechanisms 112A as well as inputmechanisms 112B. The principles described herein are not limited to the precise outputmechanisms 112A or input mechanisms 112B as such will depend on the nature of thedevice. However, output mechanisms 112A might include, for instance, speakers, displays,tactile output, virtual or augmented reality, holograms and so forth. Examples of inputmechanisms 112B might include, for instance, microphones, touchscreens, virtual oraugmented reality, holograms, cameras, keyboards, mouse or other pointer input, sensors ofany type, and so forth.

[0028] Embodiments described herein may comprise or utilize a special-purpose orgeneral-purpose computing system including computer hardware, such as, for example, oneor more processors and system memory, as discussed in greater detail below. Embodimentsdescribed herein also include physical and other computer-readable media for carrying orstoring computer-executable instructions and / or data structures. Such computer-readablemedia can be any available media that can be accessed by a general-purpose or specialpurposecomputing system. Computer-readable media that store computer-executableinstructions are physical storage media. Computer-readable media that carry computerexecutableinstructions are transmission media. Thus, by way of example, and not limitation,embodiments of the invention can comprise at least two distinctly different kinds ofcomputer-readable media: storage media and transmission media.

[0029] Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM,or other optical disk storage, magnetic disk storage, or other magnetic storage devices, orany other physical and tangible storage medium which can be used to store desired programcode means in the form of computer-executable instructions or data structures and whichcan be accessed by a general-purpose or special-purpose computing system.

[0030] A "network" is defined as one or more data links that enable the transport ofelectronic data between computing systems and / or modules and / or other electronic devices.When information is transferred or provided over a network or another communicationsconnection (either hardwired, wireless, or a combination of hardwired or wireless) to acomputing system, the computing system properly views the connection as a transmissionmedium. Transmission media can include a network and / or data links which can be used tocarry desired program code means in the form of computer-executable instructions or datastructures and which can be accessed by a general-purpose or special-purpose computingsystem. Combinations of the above should also be included within the scope of computerreadablemedia.

[0031] Further, upon reaching various computing system components, program codemeans in the form of computer-executable instructions or data structures can be transferredautomatically from transmission media to storage media (or vice versa). For example,computer-executable instructions or data structures received over a network or data link canbe buffered in RAM within a network interface module (e.g., a "NIC"), and then beeventually transferred to computing system RAM and / or to less volatile storage media at acomputing system. Thus, it should be understood that storage media can be included incomputing system components that also (or even primarily) utilize transmission media.

[0032] Computer-executable instructions comprise, for example, instructions and datawhich, when executed at a processor, cause a general-purpose computing system, specialpurposecomputing system, or special-purpose processing device to perform a certainfunction or group of functions. Alternatively, or in addition, the computer-executableinstructions may configure the computing system to perform a certain function or group offunctions. The computer executable instructions may be, for example, binaries or eveninstructions that undergo some translation (such as compilation) before direct execution bythe processors, such as intermediate format instructions such as assembly language, or evensource code.

[0033] Although the subject matter has been described in language specific to structuralfeatures and / or methodological acts, it is to be understood that the subject matter defined inthe appended claims is not necessarily limited to the described features or acts describedabove. Rather, the described features and acts are disclosed as example forms ofimplementing the claims.

[0034] Those skilled in the art will appreciate that the invention may be practiced innetwork computing environments with many types of computing system configurations,including, personal computers, desktop computers, laptop computers, message processors,hand-held devices, multi-processor systems, microprocessor-based or programmableconsumer electronics, network PCs, minicomputers, mainframe computers, mobiletelephones, PDAs, pagers, routers, switches, datacenters, wearables (such as glasses) andthe like. The invention may also be practiced in distributed system environments where localand remote computing system, which are linked (either by hardwired data links, wirelessdata links, or by a combination of hardwired and wireless data links) through a network,both perform tasks. In a distributed system environment, program modules may be locatedin both local and remote memory storage devices.

[0035] Those skilled in the art will also appreciate that the invention may be practicedin a cloud computing environment. Cloud computing environments may be distributed,although this is not required. When distributed, cloud computing environments may bedistributed internationally within an organization and / or have components possessed acrossmultiple organizations. In this description and the following claims, "cloud computing" isdefined as a model for enabling on-demand network access to a shared pool of configurablecomputing resources (e.g., networks, servers, storage, applications, and services). Thedefinition of"cloud computing" is not limited to any of the other numerous advantages thatcan be obtained from such a model when properly deployed.

[0036] For the processes and methods disclosed herein, the operations performed in theprocesses and methods may be implemented in differing order. Furthermore, the outlinedoperations are only provided as examples, an some of the operations may be optional,combined into fewer steps and operations, supplemented with further operations, orexpanded into additional operations without detracting from the essence of the disclosedembodiments.

[0037] In accordance with the principles described herein, a mechanism is describedthat allows any arbitrary portion ofuncompressed data to be accessed from compressed data.Part of this mechanism involves the way the uncompressed data is compressed, which isdescribed by way of example with respect to Figures 2 through 5. Thereafter, a mechanismto access a portion of the uncompressed data from that compressed data will then bedescribed with respect to Figures 6 through 7B.

[0038] First, compression is described. Figure 2 illustrates a flowchart of a method 200for compressing so as to permit reconstruction of arbitrary portions of the uncompresseddata, in accordance with the principles described herein. Figure 3 illustrates an environment300 in which compression and decompression can occur and will be frequently be referredto when describing the compression of Figure 2. As an example, the compression of Figure2 may be performed by a computing system, such as the computing system 100 ofFigure 1.In that case, the method 200 may be performed by an executable component, such as theexecutable component 106 ofFigure 1.

[0039] The method includes generating compressed data from uncompressed data (act201), and while doing so, tracking correlations between at least some positions in theuncompressed data and corresponding positions in the compressed data (act 202). Thecompression also constructs an index that records these correlations (act 203). Thecompression also associates the index with the compressed data (act 204) so that the indexis available during decompression. The decompression then can use the index to reconstructarbitrary portions of the uncompressed data from the compressed data.

[0040] As an example, in Figure 3, the uncompressed data 310 is compressed intocompressed data 320. The compressed data 310 could be a file, a database, or any otherdata structure that is suitable for accessing arbitrary portions of the data structure.Furthermore, the compression formulates an index 330 of correlations between positions ofthe compressed data 310 and corresponding compressed positions.

[0041] As an example, the index 330 includes correlation entries 331 that include entries331A through 331N that correlate respective uncompressed positions A through N withinthe uncompressed data 310 with corresponding compressed positions within the compresseddata 320. The index 330 is associated with the compressed data 320 as represented by theline 321. Compression of the uncompressed data 310 into compressed data 320 and theassociated index 330 is generally represented by arrow 311. On the other hand,decompression of arbitrary portions of the uncompressed data 310 from the compressed data320 and associated index 330 is generally represented by arrow 312. The combination ofthe compressed data 320 and the index 330 fully represent content of the uncompressed data310, albeit in compressed form.

[0042] In one example, there is a correlation for positions at fixed intervals of theuncompressed data. For example, Figure 3 illustrates that there are several locations Athrough N at fixed intervals within the uncompressed data 310. In one embodiment, thefixed interval is spaced by a fixed amount that falls between 2 / \6 and 2 / \9 uncompressedbytes, inclusive. It is advantageous to have that fixed interval be a binary power (i.e., 2 / \mwhere "m" is a whole number) ofuncompressed bytes. This would allow the decompressionto rapidly find the appropriate entry corresponding to any uncompressed byte address withinthe index 330 using the most significant bits (all bits other than them least significant bits)in the uncompressed byte address. This would also allow the decompression to rapidly findan uncompressed byte offset from the interval boundary using the m least significant bits.

[0043] In a simple example, if the interval was every 32 (2 / \5) uncompressed bytes andthere were only 8 (2 / \3) intervals altogether, each uncompressed byte can be uniquelyidentified using a single 8-bit number. If the uncompressed byte address were binary101001100, then the three most significant bits 101 can be used to determine that theuncompressed byte is within the sixth (101) interval (i.e., 000 would be the first interval),and is at seventh uncompressed byte (00110) uncompressed bytes in that interval (note00000 would be the first uncompressed byte in that interval). Thus, use of intervals in binarypowers assists with rapid use of the index during decompression.

[0044] In one more specific embodiment, the fixed interval is between 2 / \6 and 2 / \9uncompressed bytes, inclusive. A fixed interval smaller than 2 / \6 uncompressed byteswould cause the index to be a significant percentage of the uncompressed data size, therebyworking against the objective of achieving a good compression ratio. As an example, ifeach index entry were two bytes, and the uncompressed interval was 32 (or 2 / \5)uncompressed bytes, the index would be roughly 15 percent of the original data size, andthat is just for the index. On the other hand, if the spacing is 1024 (or 2 / \1 0) uncompressedbytes, the average number ofuncompressed bytes to traverse to get to a compressed positionthat corresponds to the beginning of the interval (a maximum of 1024 bytes, but an averageof about 512 bytes) would be large, thereby increasing latency for accessing arbitraryportions of the uncompressed data. Thus, the fixed interval is some binary power ofuncompressed bytes between 2 / \6 and 2 / \9. It is possible that a central portion of this range2 / \7 or 2 / \8 uncompressed bytes for the fixed interval may provide a sweet spot balancingcompression ratio and decompression latency.

[0045] Figure 4 shows an environment 400 that is similar to the environment 300 ofFigure 3, except that the environment 400 shows the compressed data 420 as an example ofthe compressed data 320. The compressed data 420 includes a sequence 430 of compressedsegments. While this sequence 430 could include any number of compressed segments, byway of illustration and example only, the sequence 430 is shown as including compressedsegments 431 through 437. In such a case, each of the correlation entries 331A through331D can include an identity of the compressed segment and an uncompressed offset. Eachcompressed segment compresses a range of uncompressed bytes. The uncompressed byteoffset represents an uncompressed byte position within that range. As an example, acompressed segment 23 perhaps represents a compressed form of the information fromuncompressed byte address 8187 to 8264. If the index entry is for uncompressed byteposition 8192 (which is a multiple of 256), then the uncompressed offset would representthe fifth byte from the beginning because 8192 minus 8187 is equal to 5.

[0046] Herein, an "identity" of a compressed segment is any value that uniquelyidentifies a compressed segment amongst the sequence of compressed segments. As anexample, that value could be an address of the beginning of the compressed segment. Thus,in this description and in the claims, an "identity" of a compressed segment is broad enoughto encompass all ways to uniquely identify a compressed segment, including addresses ofthe compressed segment. Use of addresses as the identifier is advantageous in that oncethe identifier address is known, the compressed segment can be obtained without furtherlookup of another address.

[0047] Figure 5 illustrates an environment 500 that is similar to the environment 400 ofFigure 4, except that the environment 500 shows the compressed data 520 as an example ofthe compressed data 420, and the compressed segments 531 through 537 as an example ofthe respective compressed segments 431 through 437. In the example of Figure 5, each ofat least some of the compressed segments 531 through 537 includes a literal part(represented in Figure 5 as the part of the compressed segments that are not filled in), and acopy instruction part (represented in Figure 5 as the part of the compressed segments thathave leftward-leaning hash marks).

[0048] The literal part includes literal uncompressed bytes of the uncompressed data.These are parts where no substantial redundant information was found to take advantage ofin order to compress the data. In Figure 5, each of the compressed segments 531 through537 includes a literal part constituting bytes copied from the uncompressed data. Forinstance, compressed segment 531 is entirely a literal part, and compressed segments 532through 537 each include a respective literal part 532A through 537A.

[0049] The copy instruction part includes instructions to copy from one or more byteranges of one or more prior compressed segments in the sequence of compressed segments.The copy instruction part provides instructions to copy that will be followed duringdecompression, and includes a location within the compressed data to copy from as well asthe number ofbytes to copy. This copy instruction part enables compression to occur as thecopy instruction is represented more compactly than the bytes that will be copied in orderto decompress. In the illustrated example, the compressed segments 532 through 537includes corresponding copy instruction parts 532B through 537B. As an example, asrepresented by arrow 540, the copy instruction part 536B of the compressed segment 536includes an instruction to copy from the compressed segment 434. Thus, in the example ofFigure 5, during compression of each compressed segment one at a time, the compressionlooks for redundant information in the compressed data, and represents that informationwith an instruction to copy, rather than including all of those bytes.

[0050] An example will now be described in which the uncompressed data is text.Suppose that the following text is to be compressed.{ "name": "Scott", "city": "Redmond", "lastSignedOn": "2020-05-05" } ,{ "name": "Amy", "city": "Salt Lake City", "lastSignedOn": "2020-04-25" } ,

[0051] This could potentially be compressed as in the following Table 1, where eachrow represents a distinct compressed segment, and the left column represents literals, whilethe right column represents the text that is formulated by a copying from a previouscompressed segment (a higher row when represented in table form), or from the literal partof the current compressed segment.Literal Copy[ {"name": "Scott", "city "· "Redmond " "'lastSignedOn "· "2020-05-05" } ' { "name": "Amy ","city": "Salt Lake City"," lastSignedOn"·" 2020-04-25 "},Table 1

[0052] The copy instruction part instructs to copy from one or more previOuscompressed segments in the sequence of compressed segments. However, the compressionensures that the one or more prior compressed segments from which bytes can be copied areconfined to be no earlier in the sequence of compressed segments than a predeterminednumber of bytes. For instance, this limitation in the copying is represented in Figure 5 bybracket 550. The predetermined number of bytes may be kept low enough that the bytesmay be read from the sequence 530 in a single read request, thereby enabling fasterdecompression. Furthermore, this allows the decompression process to be bound. As anaside, a smart compressor could improve compression ratio by putting the literal"lastSignedOn": "2020-05-05", } , earlier, so that it could later copy reusable parts of thisliteral.

[0053] In one embodiment, the predetermined number of bytes is a binary power (i.e.,of size 2 / \n where "n" is a whole number). The lower "n" is, the less opportunity forcompression, but the lower the memory needed to keep the number of bytes in order tocompress and decompress. On the other hand, the higher "n" is, the more opportunity thereis for compression, but the higher the memory needed to compress and decompress. If "n"is very high, it is possible that multiple read requests will be needed to access compresseddata during any given reconstruction of arbitrary portions of the uncompressed data. Thus,there is a balance between compression ratio on the one hand, and on the other hand,memory usage and access latency. In one embodiment, "n" is at least ten to ensure a goodcompression ratio. In another embodiment, "n" is at most fifteen in order to ensurepopulation of memory during a single read request. In yet another embodiment, "n" is awhole number between ten and fifteen, inclusive.

[0054] Although the principles described herein show each compressed segment asincluding at most one literal part and one copy instruction part, in some embodiments, acompressed portion can include multiple literals and copy instructions parts, or may includezero literal parts (only copy instruction part(s)) or zero copy instruction parts (only literalpart(s)). However, sufficient header information is used for each part that thedecompression can identify the boundary between literal parts and copy instruction parts.Thus, given the uncompressed byte address, the decompression process can still traverse thecompressed segment counting uncompressed bytes that it traverses in order to find anappropriate position to begin decompression from.

[0055] Now that an example compression mechanism has been described with respectto Figures 2 through 5, a decompression method will now be described with respect toFigure 6. Figure 6 illustrates a flowchart of a method 600 for decompressing arbitraryportions of compressed data, in accordance with the principles described herein. As anexample, the decompression of Figure 6 may be performed by a computing system, such asthe computing system 100 of Figure 1. In that case, the method 600 may be performed byan executable component, such as the executable component 106 ofFigure 1.

[0056] The decompression can be done starting from an identified start position of theuncompressed data. Accordingly, the method 600 begins by identifying a start position ofa portion of the uncompressed data to be accessed (act 601). Rather than having this startposition necessarily be at the beginning of the uncompressed data, or at the beginning ofany given block, the start position can be from any byte of the uncompressed data. Thus,the identified start position can be a byte address of the uncompressed data.

[0057] The method 600 also includes accessing an index associated with thecompressed data (act 602). This index correlates between locations in the compressed dataand corresponding locations in the uncompressed data. In the example of Figures 3 through5, this index is the index 330, which includes correlations 331A through 331N betweenpositions A through N of the compressed data 310 and corresponding positions within thecompressed data 320 for Figure 3, compressed data 420 for Figure 4, and compressed data520 for Figure 5. There is no temporal dependency between the identification of the startposition (act 601) and the accessing of the index (act 602). Accordingly, acts 601 and 602are shown in parallel.

[0058] The decompression method 600 process then finds a correlation within the indexthat includes a corresponding position in the uncompressed data that is prior to the identifiedstart position of the portion of the uncompressed data to access (act 603). Thedecompression method 600 then uses that correlation to find an exact position in thecompressed data that corresponds to the identified start position of the portion ofuncompressed data to access (act 604). From there, the decompression method 600decompresses from the found exact position until the entire portion of uncompressed datais decompressed (act 605). The end of the decompression process does not need to be at theend of the compressed data, nor at any boundary between blocks. Rather, the end of theportion of the uncompressed file that is being accessed may be any byte address. Thus, anyarbitrary portion of the uncompressed data can be accessed directly from the compresseddata.

[0059] This might best be illustrated by way of an example. Suppose that the index isstructured to include the information in the following Table 2.Uncompressed Byte Address Compressed Segment ID Uncompressed Bytes FromStart To Traverse0 0 0256 4 15512 9 28768 15 331024 17 21280 22 241536 26 161792 29 42048 33 212304 37 32560 45 192816 47 63072 50 73328 55 25Table 2This example includes an entry every 256 uncompressed bytes, and tells how to get to thecorresponding position in the compressed portion. Specifically, to get to a particularcompressed position that corresponds to the uncompressed byte address of the entry, theprocess would begin from the identified compressed segment and then traverse a certainnumber of uncompressed bytes.

[0060]

[0061] FIRST EXAMPLEIn a first example, the arbitrary portion of the uncompressed data to access is 6bytes beginning at uncompressed byte address 1806. Accordingly, the uncompressed byteaddress to begin decompression from is identified as uncompressed byte address 1806 (act601). Based on this uncompressed byte address start position, the decompression then usesthe index (e.g., the information from Table 2) to identify a corresponding position in theuncompressed data that is prior to the identified start position of the portion of theuncompressed data to access (act 603). In this example, uncompressed byte address 1792is just prior to the identified start uncompressed address of 1806.

[0062] Accordingly, the entry correlating the uncompressed byte address 1792 is usedto find the exact position in the compressed data that corresponds to the identified startposition of uncompressed byte address 1806 (act 604). Referring to Table 1, thedecompression can identify that uncompressed byte address 1792 (and thus compressedsegment 29) is at or just prior to the exact position that corresponds to the identified startposition (act 521) (uncompressed byte address 1806).

[0063] Furthermore, since the decompression would have to traverse 4 uncompressedbytes into the compressed segment to get to the position corresponding to uncompressedbyte address 1792, and since the start position byte address 1806 is even 14 uncompressedbytes beyond that, the decompression determines to traverse 18 bytes into the compressedsegment 29 to get to the exact start position of the decompression. "Traversing"uncompressed bytes in compressed data means that decompression counts the number ofuncompressed bytes it encounters, but does not need to actually know what those traverseduncompressed bytes are. Thus, such traversing does not need to actually decompressanything.

[0064] To traverse through compressed segments that include copy instruction parts, thedecompression process would load the predetermined number of bytes (see bracket 550 ofFigure 5) prior to the current compressed segment). From that point, the decompressionwould be applied to the next 6 bytes. The loaded predetermined number of bytes (seebracket 550 of Figure 5) will also assist in decompressing should decompression involveinterpreting a copy instruction part.

[0065] Figure 7 A illustrates an example compressed segment 29 in further detail andwill be used to illustrate the first example. In this example, the compressed segment 29includes 30 bytes of literal bytes 29A copied directly from the uncompressed data 310during compression, as well as 10 additional bytes of copy instruction 29B (from byte 30 tobyte 40). As previously described, in this example, the compressed segment has theuncompressed byte address 1792 that corresponds to information 4 bytes into compressedportion 29, which is the same as the information that is 4 bytes into the literal part 29A ofthe address portion, as represented by vertical line 601A. Furthermore, to get touncompressed byte address 1806 that corresponds to information where decompression isto start from, the decompression actually traverses 18 bytes (4 bytes plus 14 bytes) into thecompressed segment 29, which is the same as the information that is 18 bytes into the literalpart 29A, as represented by the vertical line 602A. This vertical line (at bytes address 18 inthe literal part 29A) is the exact position at which decompression is to start.

[0066] Accordingly, the decompression proceeds by copying bytes from byte 18 in theliteral part 29A (from line 602A). Furthermore, since the next six bytes of informationhappen to all be within the literal part 29A, the decompression of the six bytes ofuncompressed data involves simply copying from bytes 18 to 24 of the literal part 29A ofthe compressed segment 29. These bytes are represented by the region with the leftward-leaning line fills in Figure 6A between lines 602A and 603A.

[0067] SECOND EXAMPLE

[0068] The first example was quite simple as all of the uncompressed bytes to berandomly accessed from the compressed data happen to all have been bytes that wereliterally copied directly from the uncompressed data in the first place. The second exampleis slightly more complicated and will involve decompression by copying from the literalpart 29A as well as partially following a copy instruction in a copy instruction part 29B ofthe compressed segment 29.

[0069] In a second example, the uncompressed byte address to begin decompressionfrom is identified as uncompressed byte address 1806 (act 601), the same as in the firstexample. However, the uncompressed portion to access is now 40 uncompressed bytes long.The same as in the first example, the decompression traverses to the position 18 bytes intothe compressed segment 29 to begin decompression as again represented by line 602A.Accordingly, the first 12 uncompressed bytes can be decompressed simply by copying theremainder (from byte 18 to byte 30) of the literal part 29A of the compressed segment 29.

[0070] However, 28 more uncompressed bytes still are to be decompressed from thecompressed portion in order to total 40 uncompressed bytes. Accordingly, thedecompression follows the copy instruction of the copy instruction part 29B. In thisexample, suppose that the copy instruction instructs to copy 44 bytes from a particularposition of a prior compressed segment. As illustrated in Figure 6B, the copied portion 29Crepresents bytes 30 to 74 that would be copied should compressed portion 29 bedecompressed in its entirety (thus the 40 bytes of the compressed portion plus some headers,would decompress into 74 uncompressed bytes). However, since only 28 moreuncompressed bytes are needed (ending at line 603B), the decompression only follows thecopy instruction with respect to the first 28 bytes (from byte 30 to byte 58).

[0071]

[0072] THIRD EXAMPLEIn a third example, the uncompressed byte position to begin with isuncompressed byte position 1846. The decompression first seeks 54 uncompressed bytesfrom the uncompressed byte 1792, or in other words 58 uncompressed bytes from thebeginning of the compressed segment 29. As apparent from the discussion of the secondexample, decompression would begin from uncompressed byte 58 found in the copiedportion 29C. If ten uncompressed bytes were to be obtained, then uncompressed bytes 58through 68 would be obtained from the copied portion 29C. If 20 uncompressed bytes wereto be obtained, then all of the remaining uncompressed bytes 58 through 74 would beobtained as part of decompressing 16 of those 20 bytes, and then the first 4 bytes of theliteral part of the next compressed segment 30 would then be copied to complete theextraction.

[0073] Accordingly, the principles described herein allow the ability to access anyposition of compressed data without having to decompress any part of the compressed dataexcept for the portion of the compressed data that actually contains the information to beaccessed in uncompressed form. Thus, the principles described herein enable random accessof compressed data, opening up a completely new technical usage for compressed data.

[0074] The present invention may be embodied in other specific forms withoutdeparting from its spirit or characteristics. The described embodiments are to be consideredin all respects only as illustrative and not restrictive. The scope of the invention is, therefore,indicate by the appended claims rather than by the foregoing description. All changes whichcome within the meaning and range of equivalency of the claims are to be embraced withintheir scope.

Claims

1. A method for compressing data so as to permit reconstruction of arbitrary portions of the uncompressed data, the method comprising: generating the compressed data from the uncompressed data; while generating the compressed data, tracking correlation between at least some positions in the uncompressed data and corresponding positions in the compressed data; constructing an index that records the correlations between the at least some positions in the uncompressed data and the corresponding positions in the uncompressed data; and associating the index with the compressed data so that the index is available for reconstruction of arbitrary portions of the uncompressed data from the index and compressed data.

2. The method in accordance with Claim 1, the index having correlations for positions at fixed intervals of the uncompressed data.

3. The method in accordance with Claim 2, the fixed intervals being spaced by a fixed amount that falls between 2 / \6 and 2 / \9 uncompressed bytes, inclusive.

4. The method in accordance with Claim 1, the compressed data comprising a sequence of compressed segments.

5. The method in accordance with Claim 4, the correlation within the index including, for each of the at least some positions in the uncompressed data, the following: an identity of the compressed segment and an uncompressed offset.

6. The method in accordance with Claim 4, at least some of the sequence of compressed portions comprising: a literal part that includes literal uncompressed bytes of the uncompressed data, and including a copy instruction part that includes instructions to copy from one or more byte ranges of one or more prior compressed segments in the sequence of compressed segments, the one or more prior compressed segments being confined to be no earlier in the sequence of compressed segments than a predetermined number of bytes.

7. The method in accordance with Claim 6, the predetermined number ofbytes being of size 2 / \n bytes, where n is a whole number no greater than fifteen.

8. The method in accordance with Claim 6, the predetermined number of bytes being of size 2 / \n bytes, where n is a whole number that is at least ten.

9. The method in accordance with Claim 8, the predetermined number ofbytes being of size 2 / \n bytes, where n is a whole number from ten to fifteen, inclusive.

10. A method for decompressing arbitrary positions of compressed data, the method comprising: identifying a start position of the portion of the uncompressed data to access from the compressed data; accessmg an index associated with the compressed data, the index recording correlations between at least some positions in the uncompressed data and the corresponding positions in the uncompressed data; finding a correlation within the index that includes a corresponding position in the uncompressed data that is prior to the identified start position of the portion of the uncompressed data to access; using the correlation to find an exact position m the compressed data that corresponds to the identified start position of the portion of uncompressed data to access; and decompressing the compressed data from the found exact position until the entire portion ofuncompressed data is decompressed.