Compression system with longest match processing for generating compressed data

Abstract

An electronic device executes a method for generating compressed data from raw data, the electronic device including a compression subsystem having a comparator, a history buffer, a match detector, and a command generator. Beginning at each cycle of a clock, the compression subsystem processes search strings copied from raw data to generate commands for compressed data. For processing each search string, the comparator compares each of N substrings from the search string to stored data from the history buffer to find matches between the string and the stored data. The match detector then determines a longest match for each of the substrings. The command generator next selectively outputs commands for the compressed data based on the longest matches for the substrings.

Description

Background Technology

[0001] Related technologies

[0002] Some electronic devices perform operations to compress data, such as user or system files, data streams, or sequences. Electronic devices can compress data to reduce its size, enabling more efficient storage in memory, allowing data to be transferred between electronic devices over a network, and so on. For example, electronic devices can use dictionary encoding standards such as Lempel Ziv-77 (LZ77), LZ78, or Lempel-Ziv-Welch (LZW) to generate compressed data from raw data (i.e., uncompressed data).

[0003] While data compression can improve the efficiency of storing and processing data, many devices use software (i.e., software routines, applications, etc.) to compress data. Using software to compress data typically requires a general-purpose processor, such as a central processing unit (CPU), to perform a large number of compression operations and associated memory accesses. Due to the large number of compression-related operations and memory accesses, using software to compress data is inefficient. Attached Figure Description

[0004] Figure 1 A block diagram illustrating compressed data according to some implementation schemes is presented.

[0005] Figure 2 A block diagram illustrating text writing commands according to some implementation schemes is presented.

[0006] Figure 3 A block diagram illustrating a string copy command according to some implementation schemes is presented.

[0007] Figure 4 A diagram is presented illustrating the raw data, search string, and multiple substrings according to some implementation schemes.

[0008] Figure 5 A block diagram illustrating an electronic device according to some embodiments is presented.

[0009] Figure 6 A block diagram illustrating a compression subsystem according to some implementation schemes is presented.

[0010] Figure 7A A block diagram is presented illustrating the selective output of commands for compressing data according to some implementation schemes.

[0011] Figure 7B A block diagram is presented illustrating the selective output of commands for compressing data according to some implementation schemes.

[0012] Figure 8A timeline diagram of the operations performed by the compression subsystem during the clock cycle is presented according to some implementation schemes.

[0013] Figure 9 A flowchart illustrating a process for compressing raw data to generate commands for compressed data, according to some implementation schemes, is presented.

[0014] Figure 10 A block diagram is presented illustrating a process for selectively outputting commands from a command generator according to some implementation schemes.

[0015] Throughout the accompanying drawings and description, similar reference numerals refer to the same elements. Detailed Implementation

[0016] The following description is presented to enable any person skilled in the art to make and use the described embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications. Therefore, the described embodiments are not limited to the embodiments shown, but are to be given the widest scope consistent with the principles and features disclosed herein.

[0017] the term

[0018] In the following description, various terms are used to describe the implementation scheme. The following is a simplified and general description of one of these terms. It should be noted that the term may have important additional aspects, which are not stated herein for clarity and brevity, and therefore this description is not intended to limit the terminology.

[0019] Functional Block: A functional block refers to a group, set, and / or collection of one or more interconnected circuit elements (such as integrated circuit elements, discrete circuit elements, etc.). Circuit elements are "interconnected" because they share at least one property. For example, interconnected circuit elements may be included in, fabricated on, or otherwise coupled to a particular integrated circuit chip or portion thereof, may participate in performing a given function (computing or processing function, memory function, etc.), may be controlled by a common control element and / or common block, etc. A functional block may include any number of circuit elements, from a single circuit element (e.g., a single integrated circuit logic gate) to millions or billions of circuit elements (e.g., an integrated circuit memory).

[0020] Compressed data

[0021] In the described embodiments, operations are performed to generate compressed data from raw data. Generally, compressed data is the output of one or more compression and / or other operations on raw data, which cause at least some of the raw data to be replaced by other values ​​that are commanded and / or can be used to recreate the raw data. In the described embodiments, various types of data can be compressed, including user or system files (e.g., audio and / or video files, document files, executable files, operating system files, etc.), data streams or sequences (e.g., audio and / or video data streams, data sequences received via a network interface, etc.), data captured from sensors (e.g., cameras and / or microphones, thermometers, vibration sensors, etc.), etc. In the described embodiments, numerous compression standards, algorithms, or formats, or combinations thereof, can be used to compress data, including dictionary encoding standards such as Lempel Ziv-77 (LZ77), LZ78, or Lempel-Ziv-Welch (LZW).

[0022] As used herein, the terms “compressed data” and “compression” broadly apply to operations on raw data that result in at least some of the raw data being replaced by commands and / or other values ​​that can be used to recreate the raw data. As mentioned above, these operations include various encoding standards, algorithms, or formats, or combinations thereof. Therefore, these terms should not be interpreted as limited to operations such as dictionary-encoded compression and / or other operations that can sometimes be considered “compression” operations.

[0023] Figure 1 A block diagram illustrating compressed data according to some implementation schemes is presented. For example, in Figure 1 As can be seen, compressed data 100 includes a set of data 102 (i.e., series, sequences, etc.), which may be files, data streams, or sequences, or be included in files, data streams, or sequences. Generally, each command in command 102 includes, identifies, and / or refers to information that can be used to obtain and / or generate strings (i.e., bits, characters, numbers, etc.), also referred to as "symbols," which will be appended to the recreated original data during the decompression operation. In some embodiments, each command includes a set of bits organized to represent and / or identify the string that can be used to obtain and / or generate the command. In some embodiments, the information in a command refers to information outside the command itself, such as information about a separate source from which the command value is to be obtained, information that will be used for calculations associated with the command, references to tables, records, or other data structures, etc.

[0024] In some implementations, the commands in the compressed data include text writing commands and string copy commands. Figure 2A block diagram illustrating a text write command 200 according to some embodiments is presented. The text write command 200 includes a command identifier (CMD) that, when processed in a decompression subsystem, causes the decompression subsystem to acquire a text (text) of a specified length (e.g., in bits, bytes, etc.) and append the text to the recreated original data. (Although shown as...) Figure 2 The text is written into part of command 200, but in some implementations, the text may be stored separately, such as in another part or block of compressed data 100. Figure 1 In the example above, the L1 literal write command causes the decompression subsystem to retrieve the first byte string, such as the character 'A', and append that first byte string to the recreated output data. The "append" instruction, as used herein, adds a given string to the previously recreated original data (using...). Figure 1 The command (not shown) is used to recreate the sequence at the end, or as the initial string in the newly recreated data (when the command is the first command in the compressed data). Continue Figure 1 In the example, the L2 to L5 literal write commands cause the second to fifth byte strings, such as the characters B, C, D, and E, to be retrieved and appended to the recreated original data at the corresponding positions.

[0025] Figure 3 A block diagram illustrating a string copy command 300 according to some embodiments is presented. The string copy command 300 includes a command identifier (CMD) that, when processed in a decompression subsystem, causes the decompression subsystem to retrieve a string of a specified length (length, e.g., in bits, bytes, etc.) from the previously recreated original data, backwards at a distance (distance), and appends the data to the recreated original data. Generally, the string copy command enables the decompression subsystem to copy strings of various lengths (possibly from the minimum to the maximum string length) from previously recreated original data and append copies of the strings to the end of the recreated original data. Figure 1For example, the S1 string copy command causes the decompression subsystem to copy a string (i.e., B and C) consisting of text appended to the recreated original data starting three bytes back from the previously recreated original data by the text write commands L2 and L3, and appends the copied string to the recreated original data (after the text appended to the recreated data by the text write command L4). After processing the S1 string copy command, the recreated original data is as follows: ABCDBC. The S2 string copy command causes the decompression subsystem to obtain a copy of the string consisting of the L4 text and the S1 string (i.e., DBC) and add this copy of the string to the recreated original data. After processing the S2 string copy command, the recreated original data is as follows: ABCDBCEDBC. The S3 string copy command causes the decompression subsystem to obtain a copy of the string consisting of the L5 text and the S2 string (i.e., EDBC) and add this copy of the string to the recreated original data. After processing the S3 string copy command, the recreated original data is as follows: ABCDBCEDBCEDBC. (The text continues with further details about the recreated original data.) Figure 1 As seen in the example, the string copy command can be used to copy both the text and the previous string, individually or in combination, and append them to the recreated original data.

[0026] Although a specific sequence of commands is shown in compressed data 100, in some embodiments, the compressed data may include commands of different sequences and / or types. Generally, compressed data 100 includes sufficient commands to implement the operations described herein. Additionally, although various fields and information are shown as included in text write command 200 and string copy command 300, in some embodiments, text write command 200 and string copy command 300 include different information and / or information arranged differently. Generally, text write command 200 and string copy command 300 include sufficient information to implement the operations described herein. Furthermore, although in the description… Figure 1 One-byte literals are used as examples, but in some implementations, literals of other lengths may be used. For example, in some implementations, literals up to two bytes are permitted. Additionally, in some implementations, a minimum string length is used for string copy commands based on compression standards, algorithms, or formats, or combinations thereof, and / or for other reasons. For example, in some implementations, the minimum string length is three bytes (and therefore, the shortest permitted string is three bytes).

[0027] In some implementations, string copy commands are permitted to copy strings from a "dictionary" defined by and therefore including a finite amount of recreated original data. In these implementations, for a string copy command, there are limitations on how far back in the recreated original data the string can be retrieved before it is no longer part of the dictionary. For example, in some implementations, the dictionary comprises a sliding window of previously recreated original data of 32kB, 1kB, or another amount. In other words, and assuming a 64kB dictionary, even in the case of more than 64kB of previously recreated original data, only the most recently generated 64kB of strings from the recreated original data can be retrieved for the string copy command.

[0028] In some implementations, during the compression operation, the electronic device generates compressed data (e.g., compressed data 100) from the original data by using a sliding window (e.g., a 32kB sliding window) to navigate through the original data in a segment and determining a dictionary for each segment that includes the data in that segment. As the compressed data for each segment, the electronic device then generates a sequence of text write commands that append the text values ​​of unique strings to the recreated original data, and / or generates string copy commands that retrieve a copy of the string from the dictionary (i.e., in the previously recreated original data) and append the copy of the string to the recreated original data. In some implementations, and as described above, during the decompression operation, commands are retrieved from the compressed data and used to recreate the original data using the same dictionary / sliding window / segment.

[0029] Overview

[0030] In the described implementation, the electronic device performs operations for creating compressed data from raw data. The electronic device includes a compression subsystem function block that acquires and processes search strings from the raw data to generate commands for compressing the data. During operation, in each clock cycle, the compression subsystem copies a new search string from the raw data. The compression subsystem extracts multiple substrings from each search string. For example, in some implementations, the compression subsystem extracts four eight-byte substrings from an eleven-byte substring, where each substring includes eight different sequential bytes from the raw data. Then, in subsequent clock cycles, the compression system processes each of the substrings to determine the longest match of the substring from stored data in a history buffer, and selectively outputs commands for compressing the data based on the longest match of the substring.

[0031] The compression subsystem includes a comparator, a history buffer, a match detector, and a command generator, each of which is a functional block performing operations for processing substrings. The comparator receives individual substrings extracted from the search string (e.g., extracted by a string processor in the compression subsystem) as input. The comparator then uses a set of comparison elements (e.g., combinational logic, etc.) in the comparator to compare each substring extracted from the search string with each possible match in the stored data in the history buffer. For example, assuming a ten-byte substring, when the history buffer contains thirteen bytes (i.e., bytes 0 to 12) of stored data, the comparator compares each substring with each of four possible ten-byte sequences of the stored data (i.e., bytes 0 to 9, 1 to 10, 2 to 11, and 3 to 12). The comparison elements in the comparator return the result of each comparison, indicating the length of the match when a match was found or when no match was found. For example, in some implementations, for each comparison, the comparison element returns a bit sequence (e.g., a bit vector) indicating whether a match was found (or not found) for each byte. It should be noted that a "match" of the bytes storing data is found when each bit in the comparison bytes of the substring has the same value.

[0032] The comparator forwards the comparison results (i.e., the indications of matches) to the match detector. The match detector processes the indications of matches for each substring in the corresponding match pipeline individually and substantially in parallel (i.e., simultaneously or nearly simultaneously) to determine the longest match of the substring. Within each match pipeline, there are one or more stages of the longest match selector function block, each stage reducing the number of longest matches forwarded to the next stage for comparison, and the final stage returning the identifier of the single longest match of the substring. For example, suppose there are three stages of longest match selectors in the match pipeline. The first stage of the longest match selector might process K matches (e.g., 512, 1000, etc.) and return K / X partial matches (e.g., 128, 200, etc.). For example, for each group of four matches in the received matches, the first stage of the longest match selector compares two pairs of matches separately to determine the longest match pair to proceed to the next pair of comparisons (and eliminating two shorter matches from further comparisons), the next pair of comparisons returning the single longest match of that group of four matches. Continuing the example, the second matching level processes K / X partial matches from the first level and returns (K / X) / Y partial matches (e.g., 32, 20, etc.), and the third longest matching level processes (K / X) / Y partial matches from the second level and returns a single longest match. The longest matches returned from the third and final levels are the longest matches for the corresponding substrings among all matches received from the comparator. The match detector then forwards information about the longest match from each matching pipeline to the command generator, including an indication of the match length (e.g., a bit sequence representing the match) and the location or address in the history buffer where the stored data associated with the longest match will be found.

[0033] The command generator receives the longest match from the match detector and selectively outputs one or more commands for compressing data based on the longest match. For this operation, the command generator determines the text write command and / or string copy command to be used based on the longest match. The command generator then outputs the text write command and / or string copy command or otherwise makes the text write command and / or string copy command available to external entities (e.g., memory, processor, network interface, etc.). The command generator outputs commands "selectively" because whether a command is output for a given longest match depends on the length of the longest match found for the previous substring, and a command may not be output for every longest match / substring. The operations performed by the command generator for selectively outputting commands are described in more detail below.

[0034] By using a hardware compression subsystem to compare substrings with stored data and generate commands based on the comparison, the described implementation uses hardware to efficiently and quickly perform compression operations that existing devices perform using software. Compared to performing the same operation using software entities, the compression subsystem is faster and more efficient (e.g., requiring fewer memory accesses, using less power, etc.). Furthermore, using the compression subsystem frees up other functional blocks in the electronic device (e.g., processor, memory, etc.) to perform other operations. Therefore, the compression subsystem improves the overall performance of the electronic device, which in turn increases user satisfaction.

[0035] Search string and substring

[0036] In the described implementation, the compression subsystem performs operations to generate compressed data using a search string and a substring copied from the original data. Figure 4 A block diagram illustrating raw data 400, search string 402, and substrings 404 to 410 according to some embodiments is presented. Raw data 400 is or includes data to be used to generate compressed data and can be any type of data on which or on which the operations described herein can be performed. For example, raw data 400 can be or includes one or more of the following: a file, a data stream received via a network interface, data acquired from a sensor or other data source, data output by a function block in an electronic device, etc. Raw data 400 comprises a sequence of bytes, with only the leftmost byte labeled as byte 412.

[0037] Search string 402 is a copy of a portion of the original data 400. In each clock cycle, the compression subsystem copies multiple bytes of the original data 400 to generate search string 402. (As in...) Figure 4 As can be seen, search string 402 is a copy of the bytes from the original data 400, starting from start byte (SB) 414 and including sequential bytes (i.e., all bytes in order) up to end byte (EB) 416. After copying the bytes from the original data 400 during a clock cycle, the compression subsystem increments start byte 414 to a new position (in Figure 4 This is shown as the next start byte (NSB) 418, preparing to copy bytes from the original data 400 (i.e., from the next start byte 418 to the next end byte (NEB) 420) to the next search string in the next clock cycle. In this way, the compression subsystem advances through the original data 400 with each clock cycle, thereby copying the corresponding search string.

[0038] Substrings 404 to 410 are copies of search string 402, which are then processed in the compression subsystem to generate different parts of the commands used to compress the data. In each clock cycle, while copying search string 402 from the original data 400, the compression subsystem generates substrings 404 to 410 by copying the corresponding bytes of search string 402. The compression subsystem then processes substrings 404 to 410 to find the longest match among them (if any such match exists), and selectively generates commands for compressing the data (e.g., text write commands and string copy commands) based on the matches found for substrings 404 to 410.

[0039] In the described implementation, substrings 404 to 410 have a specified length in bytes, which is... Figure 4 The substring is represented as eight bytes. In some implementations, the specified length of substrings 404 to 410 is selected and set by the system designer or another entity. For example, the byte length of substrings 404 to 410 may be set based on a trade-off between the speed and efficiency of data compression and the size, complexity, and / or power consumption of the processing circuitry in the compression subsystem. Furthermore, the compression subsystem processes a specified number of substrings substantially in parallel (i.e., entirely or partially simultaneously) to generate commands. Figure 4 In the example, it is assumed that the number of substrings processed in parallel is four, and therefore four substrings 404 to 410 are generated from search string 402. In some implementations, the specified number of substrings is selected and set based on the number and / or arrangement of processing circuitry in the compression system. In some implementations, the number and / or arrangement of processing circuitry in the compression system is selected, at least in part, based on the desired number of strings to be processed substantially in parallel.

[0040] Given that substrings 404 to 410 are eight bytes long and there are four substrings, the length of search string 402 is sufficient to extract four distinct sequential substrings, and is therefore eleven bytes. From the eleven bytes of search string 402, substring 404 is the first sequential substring (i.e., a sequence of bytes that is not exactly the same as the other strings in substrings 406 to 410), consisting of a copy of the first eight bytes from search string 402 starting from start byte 414. Substrings 406 to 410 are the second through fourth sequential substrings, consisting of eight bytes copied from search string 402 starting at an offset that is progressively larger than the start byte 414. The progressively larger offset is one byte for substring 406, two bytes for substring 408, and three bytes for substring 410.

[0041] As described above, after copying search string 402 from the original data 400, the compression subsystem increments the start byte 414 to prepare for retrieving the next search string. In some implementations, the number of bytes incremented by the start byte 414 is based on the number of substrings to be processed substantially in parallel within the compression subsystem. For Figure 4 In the example, the number of substrings, and therefore the number of bytes that the starting byte 414 advances in each clock cycle, is four. Thus, the next starting byte (i.e., the next starting byte 418) is four bytes away from the starting byte 414. By advancing the starting byte 414 in this way, the compression subsystem eventually processes the substrings starting from each byte of the original data 400.

[0042] In the described implementation, during each clock cycle, the compression subsystem shifts processed bytes from the raw data 400 and adds processed bytes to the history buffer. Generally, the history buffer stores a specified amount of processed / previous raw data (e.g., 1kB, 32kB, etc.) to serve as a dictionary for the compression subsystem. In other words, the history buffer stores a specified amount of processed previous raw data, allowing string copy commands to return from this raw data to copy the string. In some implementations, the number of processed bytes shifted from the raw data 400 and added to the history buffer is equal to the number of substrings and the number of bytes incremented from the starting byte 414. Figure 4 In the example, the number of bytes is four, and therefore, in each clock cycle, four bytes of data are shifted from the original data 400 and added to the history buffer.

[0043] Despite about Figure 4 Examples of the number of bytes in substrings 404 to 410 and search string 402, and the number of substrings processed substantially in parallel, are described, along with other values ​​depending on this. However, in some implementations, different numbers of bytes exist in substrings 404 to 410 and search string 402, and / or different numbers of substrings are processed substantially in parallel. Generally, the described implementations can be used for search strings and substrings of any length and / or for any number of substrings processed substantially in parallel. Additionally, while to explain the various operations (e.g., the generation of substrings 404 to 410), in Figure 4 While a separate search string 402 is used elsewhere in this document, in some implementations, it may not be copied and / or may not exist and / or not be used. In these implementations, various operations can be performed directly using the original data 400, such as directly copying substrings 404 to 410 from the original data 400.

[0044] Electronic devices

[0045] Figure 5 A block diagram illustrating an electronic device 500 according to some embodiments is presented. (See also...) Figure 5 As can be seen, electronic device 500 includes processor 502 and memory 504. Processor 502 is a functional block that performs calculations, compression, and other operations in electronic device 500. Processor 502 includes processing subsystem 506 and compression subsystem 508. Processing subsystem 506 includes one or more functional blocks that perform general-purpose calculations, compression, and other operations, such as a central processing unit (CPU) core, a graphics processing unit (GPU) core, an embedded processor, and / or an application-specific integrated circuit (ASIC).

[0046] Compression subsystem 508 is a function block that performs operations to generate compressed data from raw data. Generally, compression subsystem 508 takes raw data (e.g., files, data streams, etc.) as input and returns commands to be included in the compressed data (i.e., commands from the dictionary encoding standard being used to compress the raw data) as output. As described above, the commands in the compressed data can then be processed in a software application in a decompression subsystem (not shown) and / or electronic device 500 or another electronic device to recreate the original data. Based on the arrangement of the function blocks in compression subsystem 508 (e.g., multiple individual comparators, matching pipelines, etc.), compression subsystem 508 generates and outputs up to a specified number of commands per time period (e.g., 1 to 4 commands per clock cycle, 1 to 5 commands per 333ps, etc.). Compression subsystem 508 is described in more detail below.

[0047] Memory 504 is a functional block that performs operations on a memory (e.g., main memory) in electronic device 500. Memory 504 includes: memory circuitry (i.e., storage elements, access elements, etc.) for storing data and instructions for use by the functional blocks in electronic device 500; and control circuitry for handling access to data and instructions in the memory circuitry (e.g., reading, writing, checking, deleting, invalidating, etc.). The memory circuitry in memory 504 includes computer-readable memory circuitry, such as fourth-generation double data rate synchronous dynamic random access memory (DDR4 SDRAM), static random access memory (SRAM), or combinations thereof.

[0048] Electronic device 500 is shown using a specific number and arrangement of elements (e.g., functional blocks and devices such as processor 502, memory 504, etc.). However, for illustrative purposes, electronic device 500 is simplified. In some embodiments, electronic device 500 contains a different number or arrangement of elements. For example, electronic device 500 may include a power subsystem, a display, etc. Generally, electronic device 500 includes sufficient elements to perform the operations described herein.

[0049] Despite the compression subsystem 508 in Figure 5 While shown as being included in processor 502, in some embodiments, compression subsystem 508 is a separate and / or independent functional block. For example, compression subsystem 508 may be implemented on a separate integrated circuit chip or the like (either independently or together with supporting circuit elements and functional blocks). Generally, in the described embodiments, compression subsystem 508 is suitably housed in electronic device 500 to perform the operations described herein.

[0050] Electronic device 500 may be any electronic device that performs data compression or other operations, or may be included in any electronic device that performs data compression or other operations. For example, electronic device 500 may be an electronic device or may be included in an electronic device, such as a desktop computer, laptop computer, wearable electronic device, tablet computer, smartphone, server, artificial intelligence device, virtual or augmented reality device, network appliance, toy, audiovisual equipment, home appliance, controller, vehicle, etc., and / or combinations thereof.

[0051] Compression Subsystem

[0052] In the described implementation, the compression subsystem performs operations to generate compressed data from the raw data. In other words, the compression subsystem generates a sequence of commands based on the raw data according to an encoding standard such as a dictionary encoding standard, which is then used as the compressed data. Figure 6 A block diagram illustrating a compression subsystem 508 according to some embodiments is presented. (See also...) Figure 6 As can be seen, the compression subsystem 508 includes a string processor 600, a history buffer 602, a comparator (COMP) 604, a match detector 606, and a command generator 608.

[0053] String processing program 600 is a function block within compression subsystem 508 that performs operations to process raw data 610. The operations performed by string processing program 600 include generating a search string 612 and providing it to comparator 604. As described above, search string 612 is a copy of multiple bytes from the raw data 610 processed in compression subsystem 508 to generate command 614 for compressing data. To generate search string 612, string processing program 600 maintains a record of the starting byte in the raw data 610 (e.g., similar to starting byte 414). In each cycle of clock 616, string processing program 600 uses the record of the starting byte to copy a certain number of bytes from an identified position in the raw data 610 to generate search string 612. String processing program 600 then provides search string 612 to comparator 604 for processing. As described above, the number of bytes in search string 612 depends on the number and length of substrings to be processed in compression subsystem 508. For example, when processing four substrings of sixteen bytes in length in the compression subsystem 508, the string processor 600 copies nineteen bytes from the original data 610 for use in searching string 612.

[0054] Another operation performed by string processor 600 is to shift data from original data 610 to generate shifted data 618, and then add shifted data 618 to history buffer 602. In each cycle of clock 616, after providing search string 612 to comparator 604, string processor 600 shifts multiple bytes from original data 610, thereby generating shifted data 618. String processor 600 then adds shifted data 618 to history buffer 602, where shifted data 618 is used as part of a dictionary to generate commands for compressing data (described in more detail below). For this operation, string processor 600 (or history buffer 602) shifts shifted data 618 to a lower position in history buffer 602, which causes existing data (if any data exists in history buffer 602) to shift from a lower position to a higher position in history buffer 602. As used herein, “shifting” data bytes involves removing a certain number of data bytes from the sequence of original data 610 in a first-in, first-out (FIFO) order (because compression subsystem 508 receives the bytes of original data 610 in sequence), leaving the remaining original data 610 for operation in subsequent cycles of clock 616. In this description, the bytes shifted from search string 612, and more generally the data stored in history buffer 602, may be referred to as “previously processed data,” etc., because the shifted bytes had already begun processing as part of the search string before being shifted, as described herein. As mentioned above, the number of bytes that string processing program 600 shifts from original data 610 depends on the number of substrings to be processed in compression subsystem 508. For example, when four substrings are to be processed in compression subsystem 508, string processing program 600 shifts original data 610 by four bytes in each cycle of clock 616.

[0055] Another operation performed by the string processing program 600 is to increment the start byte after providing the search string 612 to the comparator 604 and to shift data from the original data 610 in each cycle of clock 616. For this operation, the string processing program 600 increments the recorded start byte by a specified number of bytes, which updates / sets the start byte in preparation for copying bytes from the original data 610 to the search string 612 in subsequent cycles of clock 616. By incrementing the start byte, the string processing program 600 ensures that the compression subsystem 508 progresses through the original data 610 (over multiple clock cycles) to generate commands for compressing the data. As described above, the number of bytes the string processing program 600 increments the start byte depends on the number of substrings to be processed in the compression subsystem 508. For example, when four substrings are to be processed in the compression subsystem 508, the string processing program 600 increments the start byte by four bytes in each cycle of clock 616.

[0056] History buffer 602 is a functional block that performs operations for storing previously processed data for use by comparator 604 and possibly other entities. History buffer 602 includes storage elements 620 for storing and accessing data in history buffer 602; these storage elements are sequential logic storage elements, such as flip-flops, latches, etc. The sequential logic storage elements in history buffer 602 are close to and coupled to comparator 604 via relatively direct signal lines. For example, in some embodiments, sequential logic storage element 620 is close to or adjacent to comparator 604 in a semiconductor layout in which storage element 620 and comparator 604 are fabricated, and coupled to comparator 604 via relatively direct communication lines in the semiconductor layout. For this reason, comparator 604 can quickly (e.g., within a single cycle of clock 616) access previously processed data in storage element 620 to perform string comparisons or other operations.

[0057] During operation, the history buffer 602 (and more specifically, storage element 620) acts as a shift chain, such that the history buffer 602 receives shifted data 618 from the string processor 600 and shifts the shifted data 618 onto existing data in storage element 620 in a first-in, first-out (FIFO) order. In other words, the history buffer 602 shifts existing data (if any) sequentially from lower-position storage elements to higher-position storage elements, and the shifted data 618 is stored in the lower-position storage elements freed by shifting the existing data. In this way, data in the history buffer 602 "flows" (i.e., shifts) through the history buffer 602 in a given direction. As a result of this operation, a given set of lowest-position storage elements 620 (e.g., the lowest N bytes) should hold the latest shifted data after each clock cycle. Once the shifted data 618 has been shifted into storage element 620, the data is available to the comparator 604 and possibly other entities.

[0058] In some implementations, the history buffer 602 is used as a dictionary according to the encoding standard used in the compression subsystem 508, and therefore stores only a limited amount of previously processed data (e.g., 1kB, 32kB, etc.). In these implementations, the storage element 620 has a limited storage capacity to store the previously processed data. When the storage element 620 is full of previously processed data, the next shift operation for shifting in new shift data 618 will cause some of the existing data in the history buffer 602 to be discarded or dropped (i.e., shifted out of the highest position of the storage element 620). In this way, when the history buffer 602 is full, the incoming shift data “push” the existing data out of the history buffer 602.

[0059] Comparator 604 is a functional block that performs the following operation: compares a substring from search string 612 with stored data from history buffer 602 (i.e., previously processed data stored) to find a match (if any) between the stored data and the substring. Recall that, as described above, each substring is a copy of search string 612 of a specified number of bytes starting at a corresponding byte offset from the starting byte in search string 612. In operation, each substring is compared with every possible sequence of stored data in history buffer 602 to determine if a match exists, and if so, the length of the match (i.e., the number of matching bytes) is determined. This means that, starting from the first byte of stored data, each substring is compared with the sequence of data (or string) at each byte offset in storage element 620. For example, if history buffer 602 stores 1000 bytes of data, the substring length is sixteen bytes, and the minimum match length is three bytes (depending on the encoding standard, etc.), 997 individual comparisons will be performed. In other words, the given 16-byte substring is compared with the 16-byte stored data starting from the first byte of the stored data, with the 16-byte stored data starting from the second byte of the stored data, with the 16-byte stored data starting from the third byte of the stored data, and so on, up to the 997th byte of the stored data (the remaining bytes are not compared due to the string length).

[0060] Comparison element (CE) 622 is a functional block that is, as or includes, an array, tree, or other arrangement of combinational logic (e.g., AND gates, etc.) and / or other elements (e.g., complex logic gates, latches, etc.) that perform individual (i.e., bit-by-bit) comparisons between a substring and stored data. Comparison element 622 returns the result of each comparison, indicating the length of the match when a match was found or when no match was found. For example, in some embodiments, for each comparison, comparison element 622 returns a bit sequence (e.g., a bit vector) indicating whether a match was found (or not found) for each byte. For example, in some embodiments, and continuing with the sixteen-byte substring example, if a match is found between the first eight bytes of a given substring and the stored data from history buffer 602, and thus an eight-byte match is found, the bit sequence returned by comparison element 622 would be 1111111100000000. As another example, in some embodiments, comparison element 622 returns a count of a series of matching bytes.

[0061] Match detector 606 is a function block that performs operations to determine the longest match found for each substring or to determine that no match was found for the substring based on indications of matches received from comparator 604. Match detector 606 includes a plurality of matching pipelines 624 to 630, which respectively include longest match selector (LM SEL) stages 632 to 638, 640 to 646, 648 to 654, and 656 to 662. Each matching pipeline is used to reduce the longest match found by comparator 604 for the corresponding substring until the final stage in the matching pipeline (e.g., the longest match selector stage 638 in matching pipeline 624) outputs a single longest match. The longest match output by the final stage is the longest match among the matches received from comparator 604 in terms of the bytes of the match between the corresponding substring and the stored data (or one selected from two or more longest matches of the same length). In some implementations, in addition to the longest match, each match pipeline tracks and outputs the location of the stored data in storage element 620 where the longest match was found and / or other information about the longest match (e.g., absolute or relative address, byte offset, etc.). In these implementations, the location of the stored data of the longest match may be saved using a tag or other identifier that is propagated along with the match and whose location can be determined based on the source used for the match in comparator 604 (i.e., a specific comparison element).

[0062] To reduce the number of matches when the longest match is found, the longest match selection stage in the matching pipeline compares a specified number of match groups with each other and selects the longest match based on the comparison. For stages other than the final stage, the results of the comparisons (i.e., the “local” longest match) are provided to the next stage for comparison (and reduction). Thus, the number of match groups provided from each stage is smaller than the number of match groups received by that stage. For example, each longest match selection stage in each matching pipeline (e.g., longest match selection stages 632 to 638, etc.) may perform a match reduction of 4:1, 16:1, or other sizes. For example, assuming that for each substring, comparator 604 provides 1024 matches to match detector 606, then the first longest match selection stage in the given matching pipeline can reduce the received 1024 matches to 256 matches (i.e., 4:1 reduction), the second longest match selection stage can reduce the 256 matches from the first stage to 64 matches, the third longest match selection stage can reduce the 64 matches from the second stage to 16 matches, and the final matching stage can reduce the 16 matches from the third stage to a single longest match (i.e., 16:1 reduction). Each longest match selection stage uses one or more internal longest match selector function blocks (and possibly multiple comparison steps) to compare groups of matches to find the longest match in each group. For example, 4:1 reduction might involve the internal longest match selector function block performing a 4:1 comparison step, two parallel 2:1 comparison steps, followed by a third 2:1 comparison step, and so on.

[0063] Command generator 608 is a function block that performs operations to output commands based on the longest match received from the longest match pipeline and related information. In other words, command generator 608 uses the longest match to determine the commands to be output for compressed data. Generally, commands are used for compressed data according to encoding standards, such as text write commands and string copy commands for dictionary compression standards. Commands from compressed data can then be processed separately in decompression systems, software applications, etc., to recreate the original data.

[0064] Command generator 608 "selectively" outputs commands for compressing data based on the longest match of a substring. Selectively outputting commands based on the longest match means that the longest match output from the match pipeline can, but is not required to, be used to generate commands for compressing data. Whether a given longest match is used to generate a command depends on the previous longest match in the same or an earlier clock cycle, and therefore the command. A sufficiently long longest match in a match pipeline prevents subsequent longest matches in some or all match pipelines from being used to generate commands in a given clock cycle and possibly one or more later clock cycles, as described in more detail below.

[0065] Figures 7A to 7BA block diagram is presented illustrating the selective output of commands for compressing data according to some implementation schemes. For Figures 7A to 7B In the example, each square represents the longest match output from one of the matching pipelines 624 to 630 in each cycle of clock 616, from the first clock cycle (i.e., clock cycle 700) to the final clock cycle (i.e., clock cycles 702 to 712) and including that final cycle. In other words, in each of clock cycles 700 to 712, each of the matching pipelines 624 to 630 provides the longest match output representing the length of the maximum match found by that matching pipeline, and these matches are... Figures 7A to 7B The values ​​are shown using squares (for clarity, the longest match value is not shown in some of the squares). Additionally, for... Figures 7A to 7B Assuming the substring is 16 bytes long, but this is not required.

[0066] for Figures 7A to 7B Recall the examples in the text, such as... Figure 4 As shown, the starting byte of each substring is offset by one byte from the next substring, and thus, in a given clock cycle (e.g., clock cycle 700, etc.), the substring for which a match is processed in match pipeline 624 includes all bytes of the substring for which a match is processed in match pipeline 626, except for the last byte, and so on for the substrings for which a match is processed in match pipelines 626 to 630.

[0067] Regarding command generation, in each clock cycle, command generator 608 considers the longest match output from matching pipelines 624 to 630 in the order from matching pipeline 624 to matching pipeline 630. Generally, given the aforementioned overlap in the substrings for which the longest matches are processed in matching pipelines 624 to 630, when a longest match longer than one byte is found in matching pipeline 624 in clock cycle 700, the longest match includes at least the first byte of the substring processed in matching pipeline 626 in clock cycle 700. In this case, any longest match found in matching pipeline 626 in clock cycle 700 cannot be used to generate a command because of the longest match found in matching pipeline 624. This is true because the next longest match (i.e., the longest match after the longest match found in matching pipeline 624) does not begin until after the last byte of the longest match found in matching pipeline 624, and therefore cannot begin from the first byte of the substring processed in matching pipeline 626. In fact, it is possible that the longest match in matching pipeline 624 is long enough (i.e., four bytes) that the next longest match will not begin from the starting byte of any other substring for which the longest match is output from matching pipelines 626 to 630 in clock cycle 700. It is also possible that the longest match found in matching pipeline 624 is long enough (i.e., five or more bytes) that the longest match will not begin from the starting byte of any substring for which the longest match is output from matching pipelines 626 to 630 in clock cycles 702 or 704 to 706. In some implementations, a longest match of sixteen bytes (significantly longer matches, as described below) is permitted. When a sixteen-byte longest match is found in matching pipeline 624 in clock cycle 700, the next longest match will not begin from the starting byte of any other substring for which the longest match is currently output from matching pipelines 626 to 630. Figure 7A The longest match is determined by the first byte of any substring processed in the matching pipeline 624 to 630, meaning that the next longest match will not be found in clock cycles 702 to 706. When a longest match is unavailable due to a previous longest match of sufficient length, it is discarded and not used to generate commands. Therefore, commands are selectively output (i.e., output or not output) from the command generator based on the longest match of a substring.

[0068] for Figure 7AThe example in the text assumes that in clock cycle 700, no match is found for the substring processed in each of the matching pipelines 624 to 628, and / or the longest match is found to be less than a threshold number of bytes (e.g., three bytes, etc.). This can happen when the substring does not match from the first byte (e.g., does not match a substring not yet encountered in the previously processed raw data) or only matches a few bytes in the stored data. In this case, for each of the matching pipelines 624 to 630, the command generator 608 outputs a literal write command to the compressed data for writing only the first byte of the substring. In other words, for each of the three one-byte literals in the first byte of each of these substrings, the command generator 608 outputs three separate literal write commands. It should be noted that since a literal is only one byte, the literal write commands do not affect the bytes of subsequent substrings, and the command generator 608 can generate commands for the longest match from subsequent matching pipelines.

[0069] continue Figure 7A Suppose that in clock cycle 700, a nine-byte (9B) longest match is found for the substring being processed in matching pipeline 630. In other words, starting from the first byte of the sixteen-byte substring being processed in matching pipeline 630, the longest match (though possibly not the only match) found in the stored data is the first nine bytes of the substring. This means that the nine-byte string at a given position in the previously processed raw data (i.e., the dictionary) in storage element 620 can be used to recreate the substring. Therefore, command generator 608 sends a string copy command to the compressed data output to copy the nine-byte string from the position in the previously processed raw data / dictionary.

[0070] Since the longest match in matching pipeline 630 during clock cycle 700 is nine bytes long, at least the first byte of each of the next eight substrings processed in matching pipelines 624 to 630 is included in the longest match. Therefore, the longest matches from these substrings become unavailable, as indicated by the shading in the next eight longest matches returned during clock cycles 702 to 704. From the perspective of clock cycle 700, the longest matches found in matching pipelines 624 to 630 during clock cycle 706 are marked with a question mark. This is to indicate that these longest matches are candidates based on their generation command. This is true because it is not yet known (i.e., during clock cycle 700) that the substrings processed by matching pipelines 624 to 630 to generate the longest match output in clock cycle 706 overlap with the longest matches of the previous substrings.

[0071] It should be noted that although speculative computation of substrings in clock cycles 702-704 is lost because the longest match is discarded / ignored by command generator 608 due to the nine-byte match in matching pipeline 630 in clock cycle 700, progress was made because a relatively long longest match was found in matching pipeline 630 in clock cycle 700. Furthermore, if no longest match was found in clock cycle 700 and / or the longest match was shorter, some or all of the longest matches found in clock cycles 702-704 might already be useful.

[0072] For clock cycles 702 and 704, all longest matches from match pipelines 624 to 630 are discarded, and no commands are output. This is similar to the behavior during clock cycles 706 to 712. Figure 7B As shown, the longest match of sixteen bytes (16B) was found in the match pipeline 624 during clock cycle 706. In this case, in some implementations, since the match can be found in... Figure 7B The clock cycle continues beyond the indicated range, and therefore the maximum longest match length (e.g., 255 bytes) can be reached depending on the encoding standard used; therefore, command generator 608 does not immediately generate a command. Instead, command generator 608 records the occurrence of a sixteen-byte match and its position in storage element 620, and monitors subsequent clock cycles to determine whether the match continues, i.e., whether one or more subsequent substrings encounter additional matching bytes after the matching bytes in the substring from which the longest match was found in the match pipeline 624 during clock cycle 706. When it is later found that the longest match is only sixteen bytes (and therefore no other matches are found), command generator 608 outputs a string copy command to copy the sixteen-byte string from the position in the previously processed original data / dictionary. Otherwise, if the longest match is longer than sixteen bytes, command generator 608 outputs a string copy command to copy the bytes of the entire matched string from the position in the previously processed original data / dictionary.

[0073] Despite Figure 6 The compression subsystem 508 and matching detector 606 in this document present a specific number and arrangement of functional blocks, but in some embodiments, the compression subsystem 508 and / or matching detector 606 contain different numbers and / or arrangements of functional blocks. For example, in some embodiments, the matching detector 606 includes different numbers of matching pipelines. Generally, the compression subsystem 508 and matching detector 606 include sufficient functional blocks to perform the operations described herein.

[0074] Although the longest matches from matching pipelines 624 to 630 are described as being used to generate commands sequentially from matching pipelines 624 to 630, this is not mandatory. In some implementations, the command generator 608 selectively uses the longest matches to generate commands based on the relative length of the longest matches, which helps improve compression. In these implementations, the command generator 608 determines the length of the longest match for each of the matching pipelines 624 to 630 and then discards shorter longest matches early to generate string copy commands for subsequent longer longest matches. For example, when matching pipelines 624 to 626 have three-byte and twelve-byte longest matches, if the three-byte longest match of matching pipeline 624 is used to generate the string copy command, then the twelve-byte match in pipeline 626 will be skipped, as described above. In these implementations, based on the relative length of the longest match, command generator 608 issues a text write command for matching pipeline 624, thereby discarding the longest match of matching pipeline 624, and generating a 12-byte string copy command for the longest match from matching pipeline 626.

[0075] Operations performed during a clock cycle

[0076] In some implementations, the compression subsystem uses the edges, levels, transitions, etc. of the clock signal during the clock cycle to control the performance of the operations used to generate compressed data from the raw data. Figure 8 A timeline diagram is presented showing the operations performed by the compression subsystem during a clock cycle according to some implementation schemes. For Figure 8 In the example, it is assumed that the compression subsystem 508 performs the operation, and therefore the clock is clock 616. However, in order to... Figure 8 On a single page, only two of the four longest-match selection levels of the compression subsystem 508 are shown.

[0077] A clock 616 is typically a signal that cycles between two voltage levels as time progresses (from left to right, as indicated by the time arrows), i.e., cyclically rising and falling. Figure 8 As shown, clock 616 generates clock cycles 800 to 810 as time progresses. Within compression subsystem 508, clock 616 controls when functional blocks and / or circuit elements begin and / or perform operations and synchronize those operations. In other words, the edges, levels, transitions, etc., of clock 616 trigger the operations of functional blocks and circuit elements in compression subsystem 508 and synchronize those operations.

[0078] The compression subsystem 508 is divided into multiple levels, blocks, or stages by circuit elements (e.g., latches, flip-flops, transmission gates, switches, etc.) controlled by clock 616. Each level, block, or stage performs its operation during each clock cycle. Therefore, information can flow from a given level, block, or stage to the next level, block, or stage in consecutive clock cycles. Figure 8 In the example above, in clock cycle 800, string processor 600 copies search string 612 from raw data 610 and provides search string 612 to comparator 604, which generates substrings from search string 612. In clock cycle 802, the substrings are compared with stored data from storage element 620 in history buffer 602 in comparison element 622 to find matches. In clock cycle 804, a first longest match selection stage in the corresponding matching pipeline of match detector 606 processes the matches of each substring from comparator 604 to determine the local longest match of that substring. In clock cycle 806, a second longest match selection stage in the corresponding matching pipeline of match detector 606 processes the local longest matches of each substring from the first matching stage to determine the longest match of that substring. In clock cycle 808, command generator 608 uses the longest match from each of the matching pipelines to selectively output commands for compressing data.

[0079] To keep the function blocks and circuit elements in the compression subsystem 508 busy, processing of the next search string 612 begins in clock cycle 802 and continues through clock cycle 810. In other words, after the string processor 600 and comparator 604 have completed their operations in clock cycle 800, the string processor 600 and comparator 604 perform the same operations on the new search string. Similar to the search string described above, the new search string advances through levels, blocks, or stages in the compression subsystem 508, but one clock cycle later in each level, block, or stage. Although Figure 8 Not shown, but this mode of operation continues until all the raw data has been processed in the compression subsystem 508, where the compression subsystem 508 begins processing a new search string 612 from the raw data 610 in each clock cycle.

[0080] Although specific operations are shown as being performed within a corresponding clock cycle, in some embodiments, different operations are performed and / or different operations are performed in different clock cycles. For example, in some embodiments, copying a search string, generating a substring, and comparing may occur in the same clock cycle. Generally, in the described embodiments, the cycle of clock 616 is used to trigger / cause operations in compression subsystem 508 and to synchronize the operations.

[0081] The process of compressing input data.

[0082] In the described implementation, a compression subsystem (e.g., compression subsystem 508) including a string processor, a history buffer, a comparator, a match detector 606, and a command generator (e.g., string processor 600, history buffer 602, comparator 604, match detector 606, and command generator 608) performs operations to compress raw data (e.g., raw data 610) to generate a command (e.g., command 614) for compressing the data. Figure 9 A flowchart illustrating a process for compressing raw data to generate commands for compressed data, according to some implementation schemes, is presented. It should be noted that... Figure 9 The operations shown are presented as general examples of operations performed by some implementations. Operations performed by other implementations include different operations, operations performed in different orders, and / or operations performed by different entities or functional blocks.

[0083] for Figure 9 The illustrated process assumes that the compression subsystem receives raw data from a source (such as a file, data stream, etc.). In other words, individual bytes of the raw data are received by the compression subsystem and are therefore available for operation by functional blocks within the compression subsystem. In some embodiments, the compression subsystem simply (and may only) compresses the raw data provided to it by another entity (e.g., a processing subsystem, memory) to generate commands for the raw data. In other words, in these embodiments, the compression subsystem does not perform any operations other than outputting commands generated based on the raw data provided to it. However, in some embodiments, the compression subsystem includes functional blocks for performing other operations such as retrieving the raw data from memory and / or other locations or entities (e.g., a processing subsystem, network interface, I / O device, etc.), storing the generated commands in memory, and / or providing the commands to other entities.

[0084] for Figure 9 In the example above, at least some of the functional blocks in the compression subsystem perform certain operations "substantially in parallel" because these operations are performed at approximately the same time (e.g., within the same period of a clock). For example, the comparison operation in step 904 can be performed substantially in parallel for each substring and the stored data from the history buffer. As another example, the determination of the longest match for each substring in step 906 can be performed substantially in parallel.

[0085] Figure 9The process begins when the string processor obtains the search string (e.g., search string 612) from the raw data (step 900). For this operation, the string processor copies a specified number of bytes from the raw data to be processed in the compression subsystem. The string processor then provides the search string to a comparator, which generates a number of substrings from the search string (step 902). During this operation, the comparator generates each substring by copying different sequences of a given number of bytes from the search string. The first substring (e.g., substring 404) includes a given number of bytes from the search string, starting from the starting byte of the search string (e.g., search string 402), and each subsequent substring (e.g., substrings 406 through 410) includes a given number of bytes from the search string, starting from an increasingly larger offset in bytes from the starting byte.

[0086] Next, the comparator compares each substring with the stored data from the history buffer in the corresponding comparison element to find a match between the substring and the stored data (step 904). For this operation, assuming the substring is N bytes long, the individual comparison elements compare each possible N-byte sequence in the stored data (i.e., byte sequences 0 to N-1, 1 to N, 2 to N+1, etc.) with each substring in essentially in parallel. Each individual internal element provides an output indicating whether a match was found (or not found), how long the match is, and the position of the matching data in the history buffer. For example, in some implementations, each internal element outputs a representation of a match or lack thereof, such as a numerical value, bit sequence, match length, and offset, as well as a value indicating the position in the history buffer.

[0087] Depending on the size of the history buffer and the amount of data stored therein, the comparator can contain a large number of comparison elements and thus return a large number of match identifiers. From these matches, the longest match is determined so that the appropriate command for the substring is output from the command generator. Therefore, the comparator provides the matches to the match detector. The match detector uses the corresponding match pipeline to determine the longest match for each substring (step 906). As mentioned above, each match pipeline includes one or more longest match selection stages (e.g., longest match selection stages 632 to 638), where matches are compared by length with other matches to eliminate shorter matches until only a single longest match of the substring remains.

[0088] Based on the longest match of each substring, the command generator selectively outputs commands for compressing the data (step 908). For this operation, the command generator outputs at least one command based on the longest match, and may output as many commands as the substring. Figure 10A block diagram illustrating a process for selectively outputting commands according to some implementation schemes is presented. It should be noted that... Figure 10 The operations shown are presented as general examples of operations performed by some implementations. Operations performed by other implementations include different operations, operations performed in different orders, and / or operations performed by different entities or functional blocks.

[0089] As in Figure 10 As can be seen, the process begins when the command generator acquires a count value from an earlier clock cycle or sets the count value to zero during the clock cycle (i.e., at the start of the current clock cycle) (step 1000). For this operation, it is assumed that the command generator stores a count of the longest matching bytes used in the current clock cycle or the previous clock cycle for command generation. The count is then used to control which subsequent longest matches are discarded and which are used to generate commands.

[0090] Next, the command generator skips each subsequent pipeline in a loop, starting from the first matching pipeline, and decrements the count value until the count value equals zero or there are no remaining matching pipelines and therefore the longest match must be skipped (step 1002). For this operation, the command generator skips the longest match from the matching pipeline, such as the one described above for... Figure 7A As described above, the longest match from the skipped match pipeline is discarded. When all match pipelines are skipped (step 1004), the command generator terminates processing the match pipeline without outputting a command for that clock cycle (step 1006). This result may occur when the longest match from the previous clock cycle is long enough to make all the longest matches in that clock cycle unavailable, as also as... Figure 7A As shown.

[0091] On the other hand, when the count is zero and one or more matching pipelines are not skipped (step 1004), one or more pipelines (and therefore the longest match) are still being processed in that clock cycle. In this case, when the length of the longest match in the next matching pipeline in a specified order (e.g., cyclically) is shorter than a threshold or no match is found, the command generator outputs a text write command for the next matching pipeline and sets the count to zero (step 1008). Otherwise, when the length of the longest match in the next matching pipeline in a specified order is longer than a threshold, the command generator outputs a string copy command associated with the longest match and sets the count to the length of the longest match in bytes minus one. Setting the count in this way allows subsequent matching pipelines in that clock cycle and subsequent clock cycles to be skipped based on the length of the longest match used for the string copy command, or not skipped when outputting a text write command. The command generator then returns to the skip operation, i.e., returns to step 1002 to process the count and possibly skip matching pipelines, etc.

[0092] In some embodiments, at least one electronic device (e.g., electronic device 500) uses code and / or data stored on a non-transitory computer-readable storage medium to perform some or all of the operations described herein. More specifically, when performing the described operations, at least one electronic device reads code and / or data from the computer-readable storage medium and executes the code and / or uses the data. The computer-readable storage medium can be any means, medium, or combination thereof that stores code and / or data for use by the electronic device. For example, the computer-readable storage medium can include, but is not limited to, volatile and / or non-volatile memories, including flash memory, random access memory (e.g., eDRAM, RAM, SRAM, DRAM, DDR4 SDRAM, etc.), non-volatile RAM (e.g., phase-change memory, ferroelectric random access memory, spin-torque random access memory, magnetoresistive random access memory, etc.), read-only memory (ROM), and / or magnetic or optical storage media (e.g., disk drives, magnetic tape, CDs, DVDs, etc.).

[0093] In some implementations, one or more hardware modules perform the operations described herein. For example, a hardware module may include, but is not limited to, one or more processors / cores / central processing units (CPUs), application-specific integrated circuit (ASIC) chips, neural network processors or accelerators, field-programmable gate arrays (FPGAs), compression and / or encoding subsystems, computing units, embedded processors, graphics processing units (GPUs) / graphics cores, pipelines, accelerated processing units (APUs), function blocks, controllers, accelerators, and / or other programmable logic devices. When such a hardware module is activated, it performs some or all of the operations. In some implementations, the hardware module includes one or more general-purpose circuits configured to perform operations by executing instructions (program code, firmware, etc.).

[0094] In some embodiments, data structures representing some or all of the structures and mechanisms described herein (e.g., electronic device 500 or a portion thereof) are stored on a non-transitory computer-readable storage medium, including databases or other data structures that can be read by the electronic device and used directly or indirectly to manufacture hardware including the structures and mechanisms. For example, the data structures may be behavioral-level descriptions or register-transfer-level (RTL) descriptions of hardware functionality in a high-level design language (HDL) (such as Verilog or VHDL). This description can be read by a synthesis tool that synthesizes the description to generate a netlist comprising a list of gate / circuit elements from a synthesis library representing the functionality of the hardware including the structures and mechanisms described above. The netlist can then be placed and routed to generate a dataset describing the geometry to be applied to a mask. The mask can then be used in various semiconductor manufacturing steps to produce one or more semiconductor circuits (e.g., integrated circuits) corresponding to the structures and mechanisms described above. Alternatively, the database on the computer-accessible storage medium may be a netlist (with or without a synthesis library) or a dataset (as needed), or Graphical Data System (GDS) II data.

[0095] In this specification, variables or unspecified values ​​(i.e., general descriptions of values ​​in the absence of specific instances of values) are represented by letters such as N, M, and X. As used herein, although similar letters may be used in different places in this specification, the variables and unspecified values ​​in each case are not necessarily the same; that is, some or all of the general variables and unspecified values ​​may be expected to have different variables and values. In other words, the N and any other letters used to represent variables and unspecified values ​​in this specification are not necessarily related to each other.

[0096] As used herein, the expression “etc.” is intended to represent one and / or case, that is, an equivalent of “at least one” of the elements associated with “etc.” in the list. For example, in the statement “the electronic device performs a first operation, a second operation, etc.”, the electronic device performs at least one of the first operation, the second operation, and other operations. Furthermore, the elements associated with “etc.” in the list are merely examples from a set of examples, and in some embodiments, at least some of these examples may not appear.

[0097] The foregoing description of embodiments is presented for illustrative and descriptive purposes only. This foregoing description is not intended to be exhaustive or to limit the embodiments to the disclosed form. Accordingly, many modifications and variations will be apparent to those skilled in the art. Furthermore, the foregoing disclosure is not intended to limit the embodiments. The scope of the embodiments is defined by the appended claims.

Claims

1. An electronic device for generating compressed data from raw data, the electronic device comprising: A compression subsystem, comprising a comparator, a history buffer, a match detector, and a command generator; At the start of each clock cycle, the compression subsystem is configured to process a search string to generate commands for the compressed data, the search string comprising multiple bytes from the original data, wherein when processing the search string: The comparator is configured to compare each of the N substrings from the search string with stored data from the history buffer to find a match between the substring and the stored data, each substring having a specified length in bytes and including bytes from different ordered sequences of the search string; The matching detector is configured to: in multiple stages of a corresponding matching pipeline of the substring, eliminate matches from the multiple matches of the substring based on comparisons of the lengths of multiple matches to determine the longest match of each of the multiple matches of the substring, wherein the final stage of the corresponding matching pipeline outputs the longest match of the substring; and The command generator is configured to selectively output commands for the compressed data based on the longest match of the substring.

2. The electronic device as claimed in claim 1, wherein: The comparator includes a comparator element; and When each of the substrings is compared with the stored data from the history buffer to find the match, the comparator is configured to: The substring and the comparison string from the stored data are provided as input to the corresponding element in the comparison element, each comparison string having the specified length in bytes and including bytes from different ordered sequences of the stored data; as well as Receive an identifier indicating the length of each match as the output of the corresponding element.

3. The electronic device as claimed in claim 2, wherein: The identifier includes a bit sequence, which comprises individual bits of each byte of the substring; and The corresponding element is configured to output a first value for each bit in which the corresponding bytes of the substring and the comparison string match, and a second value for each bit in which the corresponding bytes of the substring and the comparison string do not match.

4. The electronic device as claimed in claim 1, wherein: The matching detector includes N matching pipelines, each of which includes K-level longest matching selectors arranged in sequence; and When determining the longest match for each of the substrings, the matching detector is configured to: In the corresponding pipeline of the matching pipeline, the match of the substring is received from the comparator; and In each stage of the corresponding pipeline in the matching pipeline, the local longest match is selected from the individual match groups and provided as the match for the next stage of the matching pipeline, wherein the final stage of the stage selects the longest match from the final and smallest individual match group.

5. The electronic device of claim 4, wherein when the command for the compressed data is selectively output based on the longest match of the substring, the command generator is configured to: Starting with the first match pipeline, skip match pipelines that would not be processed in the clock cycle due to the previous longest match in a cyclical order; When all of the matching pipelines are skipped, the matching pipeline processing is terminated within the clock cycle without outputting a command; When one or more matching pipelines are not skipped and therefore still need to be processed, for the next matching pipeline in the specified order, the command is output based on the length of the longest match of the next matching pipeline; as well as Based on the length of the longest match in the next matching pipeline, the remaining matching pipelines are processed or skipped during the clock cycle.

6. The electronic device of claim 5, wherein the designated order is the following: Cyclic order; or The order is determined based on the longest match of each of the matching pipelines that still needs to be processed.

7. The electronic device of claim 5, wherein when the command is output, the command generator is configured to: When no match is found or the longest match is shorter than the threshold length, output a text write command; and When the longest match is longer than the threshold length, a string copy command is output.

8. The electronic device of claim 4, wherein when selectively outputting the command for the compressed data based on the longest match of the substring, the command generator is configured to: Obtain a count value from an earlier period of the clock, if any, or set the count value to zero; Starting from the first matching pipeline, skip each subsequent pipeline in a loop order and decrement the count value until the count value equals zero or there are no remaining matching pipelines to skip; When there are no remaining matching pipelines to skip, the output command is terminated within the clock cycle and the current value of the count is retained in subsequent clock cycles. as well as When the count value is zero and one or more matching pipelines have not been skipped: When the longest match in the next matching pipeline in the specified order is shorter than a threshold or no match is found, a text write command for the next matching pipeline is output and the count value is set to zero; and When the length of the longest match in the next matching pipeline in the specified order is longer than the threshold, output a string copy command associated with the longest match and set the count value to be equal to the length of the longest match in bytes minus one. as well as Return to the skip operation.

9. The electronic device as claimed in claim 1, wherein: The search string comprises multiple bytes, which are N-1 bytes longer than the specified length (in bytes) of the substring; and The substrings include each different order of substrings from the search string, wherein the first substring in the substrings starts from the first byte of the search string, and each subsequent substring in the substrings starts from an increasingly longer offset in bytes from the first byte.

10. The electronic device of claim 1, wherein the compression subsystem further comprises: A string processing program, configured to: [operate on] each cycle of the clock. The search string is obtained from the raw data; as well as The search string is provided to the comparator.

11. The electronic device of claim 10, wherein after the search string is provided to the comparator in each cycle of the clock, the string processing procedure is configured to: Shift N bytes of data out of the original data; and Add the N bytes of the data to the stored data in the history buffer.

12. The electronic device of claim 10, wherein: The search string comprises multiple bytes from the original data, starting with a first byte in the original data: and After the search string is provided to the comparator in each cycle of the clock, the string processing procedure is configured to: The starting byte is incremented by N bytes, thereby configuring the compression subsystem to process the next search string in the next cycle of the clock.

13. The electronic device of claim 1, further comprising: A receiving entity, configured to receive the command from the compression subsystem and use the command for one or more subsequent operations.

14. A method for creating compressed data from raw data in an electronic device, the electronic device including a compression subsystem, the compression subsystem including a comparator, a history buffer, a match detector, and a command generator, the method comprising: At the beginning of each clock cycle, the compression subsystem processes a search string to generate a command for the compressed data. The search string includes multiple bytes from the original data. Processing the search string includes: The comparator compares each of the N substrings from the search string with stored data from the history buffer to find a match between the substring and the stored data. Each substring has a specified length in bytes and includes bytes from different ordered sequences from the search string. The longest match of each substring is determined by the matching detector at multiple stages of the corresponding matching pipeline of the substring, based on comparisons of the lengths of multiple matches, by eliminating matches from the multiple matches of the substring, wherein the final stage of the corresponding matching pipeline outputs the longest match of the substring; and The command generator selectively outputs commands for the compressed data based on the longest match of the substring.

15. The method of claim 14, wherein: The comparator includes a comparator element; and Comparing each of the substrings with the stored data from the history buffer to find the match includes matching each of the substrings: The substring and the comparison string from the stored data are provided as input to the corresponding element in the comparison element, each comparison string having the specified length in bytes and including bytes from different ordered sequences of the stored data; as well as Receive an identifier indicating the length of each match as the output of the corresponding element.

16. The method of claim 15, wherein: The identifier includes a bit sequence, which comprises individual bits of each byte of the substring; and The corresponding element outputs a first value for each bit where the corresponding bytes of the substring and the comparison string match, and a second value for each bit where the corresponding bytes of the substring and the comparison string do not match.

17. The method of claim 14, wherein: The matching detector includes N matching pipelines, each of which includes K-level longest matching selectors arranged in sequence; and Determining the longest match for each of the substrings includes: The matching of the substring is received from the comparator in the corresponding pipeline of the matching pipeline; as well as In each stage of the corresponding pipeline in the matching pipeline, the local longest match is selected from the individual match groups and provided as the match for the next stage of the matching pipeline, wherein the final stage of the stage selects the longest match from the final and smallest individual match group.

18. The method of claim 17, wherein selectively outputting the command for the compressed data based on the longest match of the substring comprises within the period of the clock: Starting with the first match pipeline, skip match pipelines that would not be processed in the clock cycle due to the previous longest match in a cyclical order; When all of the matching pipelines are skipped, the matching pipeline processing is terminated within the clock cycle without outputting a command; When one or more matching pipelines are not skipped and therefore still need to be processed, for the next matching pipeline in the specified order, the command is output based on the length of the longest match of the next matching pipeline; as well as Based on the length of the longest match in the next matching pipeline, the remaining matching pipelines are processed or skipped during the clock cycle.

19. The method of claim 18, wherein outputting the command comprises: When no match is found or the longest match is shorter than the threshold length, output the text write command; as well as When the longest match is longer than the threshold length, a string copy command is output.

20. The method of claim 17, wherein selectively outputting the command for the compressed data based on the longest match of the substring comprises within the period of the clock: Obtain a count value from an earlier period of the clock, if any, or set the count value to zero; Starting from the first matching pipeline, skip each subsequent pipeline in a loop order and decrement the count value until the count value equals zero or there are no remaining matching pipelines to skip; When there are no remaining matching pipelines to skip, the output command is terminated within the clock cycle and the current value of the count is retained in subsequent clock cycles. as well as When the count value is zero and one or more matching pipelines have not been skipped: When the longest match in the next matching pipeline in the specified order is shorter than a threshold or no match is found, a text write command for the next matching pipeline is output and the count value is set to zero; and When the length of the longest match in the next matching pipeline in the specified order is longer than the threshold, output a string copy command associated with the longest match and set the count value to be equal to the length of the longest match in bytes minus one. as well as Return to the skip operation.

21. The method of claim 14, wherein: The search string comprises multiple bytes, which are N-1 bytes longer than the specified length (in bytes) of the substring; and The substrings include each different order of substrings from the search string, wherein the first substring in the substrings starts from the first byte of the search string, and each subsequent substring in the substrings starts from an increasingly longer offset in bytes from the first byte.

22. The method of claim 14, wherein the compression subsystem further comprises: A string processing program, wherein in each cycle of the clock, the string processing program: The search string is obtained from the raw data; as well as The search string is provided to the comparator.

23. The method of claim 22, wherein after providing the search string to the comparator in each cycle of the clock, the string processing program further performs the following operations: Shift N bytes of data out of the original data; and Add the N bytes of the data to the stored data in the history buffer.

24. The method of claim 22, wherein: The search string comprises multiple bytes from the original data, starting with a first byte in the original data: and After providing the search string to the comparator in each cycle of the clock, the string processing program also performs the following operations: The starting byte is incremented by N bytes, thereby configuring the compression subsystem to process the next search string in the next cycle of the clock.

Images