Data compression system, method of lossy data compression and method of data compression

Abstract

A method for lossy data compression, the method comprising: receiving raw data at a storage device; receiving a request to compress a flag; accessing an on-board data compression algorithm library including various data compression algorithms respectively corresponding to lossy data compression schemes; selecting one of the data compression algorithms based on a plurality of parameters; running the selected data compression algorithm online such that when the raw data is received, the raw data is compressed by the storage device, and then the raw data is stored on the storage device as compressed data; or running the selected data compression algorithm offline such that the raw data is stored at the storage device, thereafter compressed by the storage device according to the selected data compression algorithm, and the raw data is re-saved at the storage device as compressed data. Also provided are a data compression system and a method of data compression.

Description

[0001] [Cross-reference to related applications]

[0002] This application claims priority and rights to U.S. Provisional Patent Application No. 62 / 542,008, filed August 7, 2017, and also to U.S. Provisional Patent Application No. 62 / 522,423, filed June 20, 2017, and is related to U.S. Non-Provisional Patent Application No. 15 / 688,779, filed August 28, 2017, which also claims priority and rights to the aforementioned U.S. Provisional Patent Application No. 62 / 522,423. The full text of the above U.S. Provisional Patent Application and U.S. Non-Provisional Patent Application is incorporated herein by reference. Technical Field

[0003] Some embodiments of this disclosure generally relate to a driving device including a data compression scheme. Background Technology

[0004] Data compression is a well-known and useful technique for improving the efficient use of data storage capacity, such as hard storage devices. However, some data compression schemes or algorithms are more effective for certain types of data than for others. Data compression can be referred to as "lossless" or "lossy" data compression. In "lossless" data compression, the accurate original data can be obtained from the compressed version, while in the case of "lossy" data compression, the accurate input data cannot be recovered from the compressed version.

[0005] Generally, data compression occurs at the host, which then sends the compressed data as a compressed data file to the drive (e.g., a hard disk drive (HDD) or a solid state drive (SSD)). However, on-drive data compression involves the drive receiving the raw, uncompressed data file from the host, compressing the data, and then storing the compressed data file on the drive.

[0006] Previous on-drive data compression methods "clumsily" compressed data, or performed unintelligent / overly simplistic data compression, using the same data compression method or algorithm to compress every raw data file received by the drive. The drive then stored the compressed data as a compressed data file on the drive. In other words, unintelligent on-drive data compression meant that the drive took every piece of data received from the host, compressed each piece of data using the same data compression algorithm, and then stored the compressed data file on the drive.

[0007] When using non-smart data compression, the driver ignores distinctive characteristics of the data, such as whether the data has been compressed by the host; whether the data was previously encrypted; whether the driver user wants to compress the data, to what extent, or based on what parameters; and various other factors. Additionally, when using non-smart data compression, the driver ignores the fact that some types of data are easier to compress using one or more specific data compression schemes compared to other types.

[0008] Previous versions of data compression, often described as "clumsy" or unintelligent, (e.g., generic compression), were based on the assumption that users, when trying to store data on their drives, did not want to use lossy compression schemes that could lead to the loss of important data. Consequently, these unintelligent compression schemes always involved lossless compression, not lossy compression. In other words, because lossy compression is harmful to certain types of data (e.g., databases, working data files, etc.) and can potentially cause data corruption, only lossless compression schemes were previously implemented to maintain data integrity.

[0009] Furthermore, previous lossy data compression schemes were always performed by the host, and the data was not compressed by the drive. Instead, the drive simply received and stored the compressed data files sent from the host. The host typically corresponds to the central processing unit (CPU) / computer processor of the user attempting to compress and store data. Unfortunately, the computational operations associated with this lossy data compression often consume a large number of CPU cycles and also a significant amount of energy.

[0010] The above information is disclosed only to enhance understanding of the background of this disclosure, and therefore may contain information that does not constitute prior art. Summary of the Invention

[0011] Embodiments of this disclosure provide a driver that can intelligently perform lossy data compression without the assistance of a host.

[0012] According to one embodiment, a method for lossy data compression is provided, the method comprising: receiving raw data at a storage device; receiving a compression request flag; accessing an onboard data compression algorithm library, the onboard data compression algorithm library containing various data compression algorithms respectively corresponding to lossy data compression schemes; selecting one of the data compression algorithms based on multiple parameters; running the data compression algorithm online such that when the raw data is received, the raw data is compressed by the storage device, and then the raw data is stored as compressed data on the storage device; or running the data compression algorithm offline such that the raw data is stored at the storage device, and thereafter the storage device compresses the raw data according to the selected data compression algorithm, and the raw data is re-stored as compressed data at the storage device.

[0013] The parameters may be included in an enhanced key of the key value pair corresponding to the original data, or in a Quality of Service (QoS) preference data file associated with the storage device.

[0014] The method may also include prompting the user who requests compression of the raw data to provide the parameters.

[0015] The method may further include: reading metadata contained in the original data corresponding to an object signature or object extension; accessing an object signature dictionary or an object extension dictionary on the storage device to determine the data type of the original data based on the object signature or the object extension, respectively; and selecting the data compression algorithm based on the determined data type.

[0016] According to another embodiment, a data compression method is provided, the method comprising: receiving raw data at a storage device; compressing the raw data using the storage device according to a lossy data compression scheme to generate compressed data; and storing the compressed data on the storage device.

[0017] The method may further include using the storage device to determine a data compression scheme for compressing the raw data.

[0018] The method may further include: establishing a Quality of Service (QoS) preference data file containing information related to a compression scheme and parameters; identifying an object file corresponding to the original data by the storage device; and determining the data compression scheme for compressing the original data based on the information and the object file.

[0019] The method may further include: reading metadata of the key of the key value pair corresponding to the original data; selecting a data compression algorithm from a data compression algorithm library on the storage device based on the metadata; and executing the data compression scheme for compressing the original data according to the selected data compression algorithm.

[0020] The original data may include object signatures or object extensions, and the method further includes: accessing an object signature dictionary or an object extension dictionary on the storage device to determine the data type of the original data based on the object signature or the object extension, respectively; and performing the data compression scheme based on the determined data type.

[0021] The method may further include sending the raw data from the host computer to the storage device according to a user request.

[0022] The method may further include: providing a menu for prompting a user to provide information; and determining a data compression algorithm for compressing the raw data based on the information provided by the user.

[0023] The method may further include storing the raw data on the storage device; estimating the available time or resources corresponding to the storage device; and compressing the raw data when it is determined that sufficient time and resources are available.

[0024] According to another embodiment, a data compression system is provided, the data compression system comprising: a host; and a storage device configured to receive raw data from the host, and including a memory and a processor, the processor being configured to execute instructions, the instructions, when executed by the processor, causing the processor to: cause the storage device to compress the raw data according to a lossy data compression scheme to produce compressed data; and cause the storage device to store the compressed data on the storage device.

[0025] When executed by the processor, the instructions may also cause the processor to enable the storage device to determine a data compression scheme for compressing the raw data.

[0026] When executed by the processor, the instructions may also cause the processor to cause the storage device to: access a Quality of Service (QoS) preference data file containing information related to a compression scheme and parameters; identify an object file corresponding to the original data; and determine, based on the information and the object file, the data compression scheme for compressing the original data.

[0027] When executed by the processor, the instructions may also cause the processor to cause the storage device to: read metadata of the key of the key value pair corresponding to the original data; select a data compression algorithm from a data compression algorithm library based on the metadata; and execute the data compression scheme for compressing the original data according to the selected data compression algorithm.

[0028] The raw data may include object signatures or object extensions, and the instructions, when executed by the processor, may also cause the processor to cause the storage device to: access an object signature dictionary or an object extension dictionary on the storage device to determine the data type of the raw data based on the object signature or the object extension, respectively; and execute the data compression scheme based on the determined data type.

[0029] When executed by the processor, the instructions also cause the processor to enable the storage device to receive the raw data from the host computer upon user request.

[0030] When executed by the processor, the instructions may also cause the processor to: enable the host to provide a menu for prompting the user who is requesting compression of the raw data to provide information; and enable the storage device to determine a data compression algorithm for compressing the raw data based on the information provided by the user.

[0031] When executed by the processor, the instructions may also cause the processor to: store the raw data on the storage device; estimate the available time or resources corresponding to the storage device; and compress the raw data when sufficient time and resources are available.

[0032] The driver disclosed herein, according to embodiments, is capable of receiving raw data and performing lossy data compression on said raw data, thereby allowing a user to change the control level of specific parameters corresponding to data compression. Therefore, by offloading lossy data compression activities that might otherwise impose a substantial burden on the user system, the driver can cyclically free up the host CPU. In some embodiments, the driver may implement additional or expanded hardware resources, or may implement operating system (OS) level and / or driver level changes. Attached Figure Description

[0033] Some embodiments can be understood in more detail by reading the following description in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A comparison is shown between examples of conventional methods for lossy data compression and examples of lossy data compression methods according to embodiments of the present disclosure.

[0035] Figure 2 This invention discloses a lossy compression driver that can perform intelligent lossy data compression at the driver without the assistance of a host, according to an embodiment of the present disclosure.

[0036] [Explanation of reference numerals in the attached figures]

[0037] 1: Key;

[0038] 2: Value;

[0039] 3: Data compression predictor;

[0040] 4: Object extended dictionary;

[0041] 5: Object signature dictionary;

[0042] 6: Entropy estimation algorithm library;

[0043] 7: Compressor;

[0044] 11: Receiver switch;

[0045] 12: Exit switch;

[0046] 13: Output;

[0047] 14: Service quality preference data file;

[0048] 15: Onboard device performance profile / Device performance profile;

[0049] 16: Compression algorithm performance table;

[0050] 17: Data compression algorithm library / onboard encoding library;

[0051] 110, 120: Lossy data compression schemes;

[0052] 112: Host;

[0053] 113: Compression / Data Compression / Lossless Data Compression;

[0054] 114: Compressed data file;

[0055] 116: Driver;

[0056] 118: Raw data file / raw data;

[0057] 122: Host / Main Computer / Main Central Processing Unit;

[0058] 123: Compression / Lossy data compression / Lossy compression / Data compression;

[0059] 124: Compressed data file / Compressed data / Compressed lossy data file;

[0060] 125: Store / Save / Resave;

[0061] 126: Drive / Intelligent Lossy Data Compression Drive / Storage Device / Intelligent Compression Drive;

[0062] 128: Uncompressed data / raw data file / raw data / data file / raw uncompressed data / data / object raw data file. Detailed Implementation

[0063] The features of the present invention and its implementation methods can be more readily understood by referring to the following detailed description of the embodiments and accompanying drawings. In the following, embodiments will be described in more detail with reference to the accompanying drawings. However, the present invention can be implemented in various different forms and should not be considered as limited to the embodiments shown herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey to those skilled in the art the various aspects and features of the invention. Therefore, processes, elements, and techniques that are not essential for a full understanding of the various aspects and features of the invention by those skilled in the art will not be described. Unless otherwise stated, the same reference numerals denote the same elements throughout all drawings and written description, and therefore will not be repeated. Furthermore, for clarity, components unrelated to the description of the embodiments will not be shown. In the drawings, the relative sizes of various elements, layers, and regions may be exaggerated for clarity.

[0064] In the following description, various specific details are set forth for illustrative purposes to provide a thorough understanding of various embodiments. However, it will be apparent that various embodiments may be practiced without these specific details or using one or more equivalent configurations. In other instances, well-known structures and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring the various embodiments.

[0065] It should be understood that when an element, layer, area, or component is described as being "on," "connected to," or "coupled to" another element, layer, area, or component, the element, layer, area, or component may be directly located on, directly connected to, or directly coupled to the other element, layer, area, or component, or there may be one or more intermediate elements, layers, areas, or components. However, "directly connected / directly coupled" refers to a component being directly connected or directly coupled to another component without any intermediate components. Similarly, other expressions describing the relationships between components, such as "between," "immediately between," "adjacent to," and "directly adjacent to," can be interpreted similarly. Additionally, it should be understood that when an element or layer is referred to as being “between” two elements or layers, the element or layer may be the only element or layer between the two elements or layers, or there may be one or more intermediate elements or layers.

[0066] The terminology used herein is for the purpose of illustrating particular embodiments only and is not intended to limit the invention. Unless the context clearly indicates otherwise, the singular form “a (a and an)” as used herein is intended to include the plural form as well. It should also be understood that when the terms “comprises,” “have,” and “includes” are used in this specification, they indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The term “and / or” as used herein includes any and all combinations of one or more of the related listed items.

[0067] The terms “substantially,” “about,” and similar terms used herein are used as approximations, not as terms of degree, and are intended to take into account the inherent biases of measured or calculated values ​​known to those skilled in the art. “About” or “approximately” as used herein includes the stated value and means within an acceptable range of deviation from a particular value as determined by those skilled in the art, taking into account the measurement and the errors associated with the measurement of a particular quality (i.e., limitations of the measurement system). For example, “about” may mean within one or more standard deviations, or within ±30%, 20%, 10%, or 5% of the stated value. Additionally, the use of “may” when describing embodiments of the invention refers to “one or more embodiments of the invention.” The terms “use,” “using,” and “used” as used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Furthermore, the term “exemplary” is intended to refer to an instance or illustration.

[0068] When an embodiment can be implemented in different ways, a particular process sequence may be performed differently from the sequence described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of the described process.

[0069] Various embodiments are illustrated herein with reference to sectional views, which are schematic illustrations of embodiments and / or intermediate structures. Therefore, variations relative to the illustrated shapes are expected due to, for example, manufacturing techniques and / or tolerances. Furthermore, the specific structural or functional descriptions disclosed herein are merely illustrative and intended to illustrate embodiments according to the concepts of this disclosure. Therefore, the embodiments disclosed herein should not be considered as limited to specific illustrated shapes of individual regions, but should include shape deviations caused, for example, by manufacturing processes. For instance, an implantation region illustrated as rectangular should generally have circular or curved features and / or a gradient of implantation concentration at its edges rather than a binary change from the implantation region to the non-implantation region. Similarly, a buried region formed by implantation may induce some implantation in the region between the buried region and the surface traversed during implantation. Therefore, the regions illustrated in the figures are schematic and their shapes are not intended to illustrate the actual shapes of the regions of the device and are not intended to be limiting. Additionally, as those skilled in the art will recognize, various modifications can be made to the illustrated embodiments without departing from the spirit and scope of this disclosure.

[0070] Electronic devices or electrical devices and / or any other related devices or components according to embodiments of the invention described herein can be implemented using any suitable hardware, firmware (e.g., application-specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, various components of these devices can be formed on an integrated circuit (IC) chip or on separate IC chips. Furthermore, various components of these devices can be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a substrate. Additionally, various components of these devices can be processes or threads that run in one or more computing devices by one or more processors, execute computer program instructions, and interact with other system components for performing the various functionalities described herein. The computer program instructions are stored in memory that can be implemented in a computing device using standard memory devices such as, for example, random access memory (RAM). Computer program instructions may also be stored in other non-transitory computer-readable media, such as compact disc read-only memory (CD-ROM), flash drives, or similar components. Furthermore, those skilled in the art will recognize that, without departing from the spirit and scope of exemplary embodiments of the present invention, the functionality of various computing devices can be combined or integrated into a single computing device, or the functionality of a particular computing device can be distributed across one or more other computing devices.

[0071] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms (e.g., those defined in common dictionaries) should be interpreted as having the same meaning as their meaning in the context of the relevant art and / or in this specification, and should not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.

[0072] As mentioned above, traditionally, non-intelligent data compression performed by hard storage devices without host assistance is always in the form of lossless compression. Furthermore, traditionally, all lossy data compression is performed by the host / host computer / host CPU, not by the drive. However, there are scenarios where different types of data are suited to higher levels of compression, in which lossy data compression is acceptable or even desirable (e.g., when editing and / or storing music files, video files, or other multimedia files). Additionally, it may be useful to compress the original data file at the drive while freeing up resources that would otherwise be consumed by the host CPU for compression.

[0073] Embodiments of this disclosure provide a storage device (e.g., a solid-state drive (SSD)) capable of performing lossy data compression and storage without the assistance of a corresponding host during data compression, thereby freeing up the host's central processing unit for other user activities.

[0074] Figure 1 A comparison is shown between examples of conventional methods for lossy data compression and examples of lossy data compression methods according to embodiments of the present disclosure.

[0075] Reference Figure 1 As explained above, the conventional drive 116 does not perform lossy data compression without the assistance of the host 112. Instead, as... Figure 1 As shown, the lossy data compression scheme 110 is performed through the following steps: providing the original data file 118 to the host 112 before compressing the data 113; and sending the compressed data file 114 to the drive 116 for storage, which exhausts the system resources of the host 112. That is, conventionally, if the original data file 118 (e.g., an original sound / audio data file in Pulse Code Modulation (PCM) format) is to be compressed, the user instructs the host 112 to compress the original data file 118 113 into a compressed data file 114 (e.g., compressed into a Motion Picture Experts Compression Standard Audio Layer 3 (MP3) data file), and the host 112 then performs data compression 113 to create the compressed data file 114. Afterwards, the host 112 sends the compressed data file 114 to the drive 116 for storage.

[0076] Therefore, while prior art enabled drive 116 to perform lossless data compression 113 on raw data 118 (which would preserve data fidelity), drive 116 could not perform lossy data compression. Lossy data compression would traditionally consume a significant amount of processor loops on the corresponding host machine (e.g., on host 112) when the loss of data fidelity is acceptable and permissible to the user, or when the user requests it (e.g., when the user prefers high-volume lossy data compression, such as in the case of editing music and video).

[0077] In view of the above, it would be beneficial if the task of lossy data compression could be offloaded from the host located in the user's main central processing unit. Therefore, after receiving the raw, uncompressed data / raw data file 128 from the host / host computer / host central processing unit 122, the drive 126 (e.g., a hard storage device or solid-state drive (SSD)) according to embodiments of this disclosure performs one or more lossy data compression schemes 120 and compresses the raw data file 128 123 to produce a compressed data file 124, which is then stored / saved 125 on the drive 126.

[0078] According to this embodiment, a user can instruct the host 122 (e.g., by instructing the central processing unit to set a "request compression" flag) to compress 123 the original data 128 (e.g., when the user has finished editing a video or audio data file that will be saved in compressed form on the drive 126). This enables the drive 126 to automatically compress 123 the original data file 128 using lossy data compression to generate a compressed data file 124 on the drive 126 without using the resources of the host 122. The intelligent lossy data compression drive 126 can automatically compress data by analyzing the data (e.g., by analyzing the type of data file, available data compression methods and algorithms, user parameters, and quality of service (QoS) requirements) to identify situations suitable for lossy data compression 123. The drive 126 can then perform lossy data compression 123 according to the selected data compression scheme and save 125 the compressed data file 124 on the drive 126. It should be noted that the driver 126 may compress the data 128 128 123 immediately after receiving the data 128, or may store the data in an uncompressed format for a short period of time before compressing the data 123 and storing the compressed data 124 125 on the driver 126.

[0079] Figure 2This invention discloses a lossy compression driver that can perform intelligent lossy data compression at the driver without the assistance of a host, according to an embodiment of the present disclosure.

[0080] Reference Figure 2 As described above, a user can send the raw data file 128 (e.g., a raw audio data file in PCM format) directly from the host 122 to the drive 126. The drive 126 can then recognize the raw data file 128 as suitable for conversion into a lossy, compressed data file 124. The drive 126 can then perform lossy data compression 123 to create a compressed data file 124 (e.g., an MP3 data file) within the drive 126, thereby converting the raw data file 128 into a compressed lossy data file 124, and then storing / saving the compressed lossy data file 124 on the drive 126 itself. Therefore, this embodiment is a storage device (e.g., a drive) 126 capable of automatically or semi-automatically performing well-controlled lossy data compression 123, thereby freeing up the CPU clock cycles and resources that would otherwise be used by the host 122 to perform data compression 113 for other activities.

[0081] The "smart compression" drive 126 may include command and control protocols for data management and relatively enhanced on-disk resources (e.g., on a solid-state drive). For example, unlike a conventional hard drive, drive 126 may include a data compression algorithm library 17, which includes various lossy data compression mechanisms. The onboard encoding library / data compression algorithm library 17 may contain code corresponding to various data compression algorithms used to perform various lossy data compression schemes and encoding schemes. Drive 126 may also include a Quality of Service (QoS) preference data file 14 expanded to include lossy data compression parameters. Drive 126 can use these parameters to determine how to compress the raw data 128.

[0082] Each lossy data compression request from a user is ultimately associated with the original data file 128 to be compressed 123. The data compression request may contain a direct command explicitly specifying the file format to be used as the result of compression 123, or a direct command explicitly specifying the parameters used to instruct the driver 126 to perform compression 123. Alternatively, the data compression request may correspond to embedded commands contained in the key metadata, and the driver 126 may determine how to perform compression 123 based on the key metadata.

[0083] When drive 126 compresses a file in a lossy manner 123, drive 126 can use information such as requesting a data compression algorithm or a final format to determine which compression scheme to use (e.g., selecting a data compression algorithm from the data compression algorithm library 17). That is, drive 126 can use various information to know that drive 126 should use a certain lossy data compression algorithm to compress the original data file 128, or to know that drive 126 should ultimately save the compressed data file 124 according to a specific lossy storage format (e.g., MP3 data file, Joint Picture Experts Group (JPEG) data file, Moving Picture Experts Group (MPEG) data file, etc.).

[0084] Alternatively, driver 126 can use parameters and settings (e.g., stored in a quality of service preference data file 14) to determine when and how to perform lossy compression on data. For example, the quality of service file may instruct all files in a certain extended format to be compressed 123 into JPEG data files. The quality of service file may also contain compression details of the compression algorithm, allowing the user to select or fine-tune specific data compression schemes or quality settings, set subsampling levels, or control various other parameters.

[0085] Therefore, driver 126 can use the information to understand the required data compression format and associated parameters for any given raw data file received. According to embodiments of this disclosure, the data compression format and parameters can be delivered and determined as one or more of at least three different options to enable driver 126 to know this information, as will be further described below.

[0086] As another example of an enhancement to driver 126, driver 126 may include encoding acceleration hardware and may be capable of performing deduplication online or offline, and also capable of lossless data compression online or offline. When lossy data compression 123 is performed in online operation, the data is compressed 123 when driver 126 receives the original data file 128. When lossy data compression 123 is performed in offline mode, the uncompressed data 128 may be stored on driver 126 for an indeterminate amount of time. Then, when driver 126 determines that there is sufficient time / resources, or when driver 126 decides to work with the uncompressed data 128 stored on driver 126, the data file 128 may be compressed 123 and may be re-saved 125 to driver 126 as a compressed data file 124. Whether driver 126 decides to perform online or offline operation may be based on an estimate made by driver 126 corresponding to the amount of available time and / or system resources required for compression.

[0087] This estimation can be based on machine learning predictions and can be determined based on analysis of the onboard device performance profile 15 of driver 126. For example, device performance profile 15 may include information corresponding to a record of the physical capabilities currently in use by driver 126 (e.g., free memory, processor capacity, etc.), and compression algorithm performance table 16 may be used to predict the amount of free resources that a given task may require. Based on the information contained in device performance profile 15 and compression algorithm performance table 16, driver 126 may determine, based on a judgment of the type of the raw data file 128 to be compressed, which compression algorithm from data compression algorithm library 17 will most effectively utilize available device resources.

[0088] The compression algorithm performance table 16 may contain information about various compression algorithms (e.g., the average compression ratio of each compression algorithm, the throughput of each compression algorithm, the computational and memory resource requirements of each compression algorithm, etc.), and may be able to select a specific data compression algorithm from the data compression algorithm library 17 based on file type and available throughput (which can be determined by the quality of service requirements from the quality of service preference data file 14), and may be able to select device capabilities based on the device performance profile 15.

[0089] Other enhancements to driver 126 may include an entropy estimation algorithm library 6, an object signature dictionary 5, and / or an object extension dictionary 4, which are capable of identifying different types of raw media data files (e.g., by recognizing the extensions or signatures of the raw data files).

[0090] Entropy estimation algorithm library 6 may contain code for testing the entropy of arbitrary sampled data to determine the compression level of data file 128. Entropy-based calculations can be used to search for parts of value 2 to predict the extent to which the file can be compressed. Entropy-based calculations are based on the principle that entropy data or random data is more difficult to compress (e.g., already compressed files have higher entropy properties, and uncompressed files are more likely to have more compressible ordered data). Object signature dictionary 5 may contain known object / file signatures that correspond to corresponding compression techniques. Object extension dictionary 4 may contain a series of known object / file extensions and assessments of how compressible these types of objects are for each of the various compression techniques. Object extension dictionary 4 provides this information to data compression predictor 3 for compression. Therefore, object extension dictionary 4 and object signature dictionary 5 can provide information to data compression predictor for compression (partly by detecting files that have already been losslessly compressed and not compressing them again).

[0091] Therefore, when the raw data file 128 is received, the various components of the driver 126 can be used to identify the type of the raw data 128 to be compressed. This type can influence the operation of the data compression predictor 3 and the compressor 7, which in turn can be used to select an appropriate data compression algorithm from the data compression algorithm library 17. For example, the data compression predictor 3 can use information contained in the object extended dictionary 4, the object signature dictionary 5, and / or the entropy estimation algorithm library 6 to determine or help determine the compression factor of value 2.

[0092] Subsequently, compressor 7 can perform data compression based on the indicated parameters. Data compression predictor 3 and compressor 7 can identify the data file, determine its type, and intelligently apply the most effective lossy data compression algorithm based on factors such as: direct user requirements, transient performance of driver 126, quality of service requirements based on the data file 14 (which can be used to determine which data compression scheme should be used to compress the data), and historical data compression algorithm performance stored in compression algorithm performance table 16. Data compression predictor 3 and compressor 7 can also subsequently determine if a better data compression algorithm is available (e.g., if more processor space becomes available, more buffer space becomes available, etc.).

[0093] Once the data compression predictor 3 determines how to compress the original data file 128, it provides information to the receive switch 11, which then switches between: outputting data (e.g., value 2) to the compressor 7; or bypassing the compressor 7 by sending data to the output switch 12 for output 13 (if it is decided not to compress, for example when a file type not identified as requiring lossless compression is received). Thus, the data compression predictor 3 determines how compression should occur, and the receive switch 11 either sends data to the compressor 7 or sends data to the output switch 12 for output 13.

[0094] The following presents three options for determining an appropriate compression scheme and parameters for compressing the raw data file 128 received by the driver 126.

[0095] As a first option, the data compression format and parameters can be determined as the explicit part of the enhanced save command.

[0096] In a conventional system, each raw data file 128 saved to drive 126 may include a final save order to drive 126, thereby instructing drive 126 to explicitly instruct a certain raw data file 128 to be saved using a certain data compression type and certain data compression parameters.

[0097] In contrast, using this option, when a user instructs the host 122 to command the drive 126 to save the original data file 128 as a compressed data file 124, the data compression format and parameters can be explicitly determined as part of the save command. Of the three options presented herein, this option is generally compatible with block-based storage devices (because, according to the second option below, block-based storage devices do not have key value metadata options). However, implementing this first option may require changes to the drive / operating system layer of a conventional block-based data file system.

[0098] Changes to the driver / operating system layer may include scanning all user activity that causes write commands to drive 126 to determine if there is an intent or opportunity to perform lossy compression 123 on the original data file 128 (e.g., the user intends to save a bitmap as a JPEG, or intends to move a bitmap from an uncompressed drive to drive 126). Another change to the driver / operating system layer may include interrupting the data compression request to confirm the user's intent and prompting the user to provide information about the parameters of compression 123 (e.g., the user can interact with a menu at host 122 to allow the user to select parameters regarding lossy data compression 123). Additionally, additional modified commands may be used to transmit identity and parameter information and enable drive 126 to identify which logical block addresses (LBAs) are associated with the target data file.

[0099] After obtaining information associated with one or more of the proposed changes to the driver / operating system layer, the raw data file 128 can be sent to the driver 126, and the operating system and driver can forward user-selected information from the host 122 to the driver 126. This option may also implement certain enhanced protocols for communication with the driver 126.

[0100] As a second option, the data compression format and associated parameters can be explicitly encoded in key 1 of key 1 and value 2 ((KV) pair). Driver 126 can respond to enhancement commands or command sets, or can detect the enrichment key 1 associated with the original data 128, wherein the writing of lossy compressed data is accompanied by a "request compression" flag, data compression format identification, and / or data compression format parameters.

[0101] In a KV-based system according to this option, information regarding the data compression format and parameters can be stored in the metadata portion of key 1, which indicates how the data should be stored. For example, key 1 may contain metadata indicating that the compressed data file 124 should be stored as a JPEG data file using a certain data compression setting and a certain subsample size. For example, the data compression predictor 3 and compressor 7 of driver 126 can use the information contained in key 1 (e.g., metadata) to determine whether data file 128 should be modified / compressed and how data compression should be performed.

[0102] Additionally, key 1 may contain file / object extensions indicating the type of data file 128, which can provide information about whether and how compression is performed. A set of known object extensions can be maintained in object extension dictionary 4, and data compression predictor 3 can use information from object extension dictionary 4 to determine the type of data compression 123.

[0103] Key 1 may also contain a file / object signature, i.e., a line of code that provides information about the type of data file 128 stored in the value 2 of the KV pair. A series of signatures for known data types can be maintained in the object signature dictionary 5, and information from the object signature dictionary 5 can be provided to the data compression predictor 3 to determine the type used for compression 123.

[0104] Therefore, driver 126 can receive key 1 to indicate that the received data is a raw media data file, and key 1 may contain parameters on how the data should be compressed. Thus, object feature dictionary 5 or object extended dictionary 4 can detect that the received raw data file 128 corresponding to value 2 should be compressed by reading the information / metadata of key 1, and can use the metadata to determine which data compression algorithm from data compression algorithm library 17 should be used.

[0105] Driver 126 can read the metadata portion of key 1 to determine how to compress the data 123. When data file 128 is sent as a KV object to KV storage, KV storage can read key 1, identify and read the metadata portion, and determine that driver 126 is instructed to use the indicated data compression format and parameters to save 125 the compressed data file 124 with the selected type of lossy data compression 123. By using the information contained in the metadata portion, driver 126 can use the specified data compression algorithm and parameters to compress the associated original data file 128 123.

[0106] Therefore, the data compression format and parameters can be an explicit part of the KV storage system. The key 1 of the KV pair may contain metadata containing hidden information, where parameters can be encoded to enable the driver 126 to recognize compressible data. This user-level entry of metadata information can be accomplished using the KV storage software stack. That is, the user can tag the data file 128 that they want to compress 123 with lossy tags, and the information corresponding to these tags is included in the key 1 of the KV pair.

[0107] This option provides users with "fine-grained" control, is easy to implement, and can be implemented with very minor changes. With this option, users understand that when a certain type of raw data file is saved to drive 126, the data will be compressed.

[0108] As a third option, the data compression format and parameters can be implicitly configured as part of a larger Quality of Service (QoS) preference data file 14. The QoS preference data file 14 can be generated (e.g., by a user) and constructed using drive 126 to specify exact or general lossy compression and storage options. The QoS preference data file 14 may contain settings for specific and / or global implementations of the data compression strategy (e.g., to indicate the default data compression type and parameters to be applied to all data files 128 of a given type, to require a backup copy of the original data file 128 to be reserved, to block or delay lossy data compression 123 under given conditions, etc.).

[0109] The Quality of Service (QoS) preference data file 14 may contain information used by the driver 126 to identify the original data file 128 (e.g., information in key 1, data file extension, header signature, etc.). After the driver 126 identifies a media data file compatible with the QoS preference data file 14, the options contained in the QoS preference data file 14 can be automatically implemented by the smart driver 126. The driver 126 can use the information in the QoS preference data file 14 to select and automatically apply appropriate data compression algorithms and parameters. In some embodiments, global QoS requirements can override or delay conflicting user instructions or key-field designations.

[0110] Therefore, driver 126 can be configured such that when a user causes host 122 to send a specific type of raw data file 128 to driver 126, driver 126 can compress 123 and save 125 the data in a corresponding manner according to the quality of service preference data file 14. This option theoretically reduces user control after initial configuration of general rules, which can be applied to thousands of data files 128, thus eliminating the need for some fine-grained control from the user. However, this option enables global data compression control, allowing the quality of service preference data file 14 to be globally configured with a data compression mechanism, and reducing intrusive user interaction. It should be noted that driver 126 according to this embodiment may include additional hardware resources, such as a faster controller processor, a larger write buffer, a larger controller processor random access memory allocation, and / or one or more of a data compression / encoding accelerator.

[0111] Therefore, the above embodiments provide a self-sufficient storage device capable of intelligent lossy data compression.

[0112] The above are illustrative examples and should not be construed as limiting the exemplary embodiments. Although several exemplary embodiments have been set forth, those skilled in the art will readily understand that many modifications may be made to the exemplary embodiments without substantially departing from the novel teachings and advantages of the exemplary embodiments. Therefore, all such modifications are intended to be included within the scope of the exemplary embodiments as defined by the claims. In the claims, the means-attached-function clauses are intended to cover the structures described herein that perform the functions, and not only structural equivalents but also equivalent structures. Therefore, it should be understood that the above are illustrative examples and should not be construed as limiting oneself to the specific embodiments disclosed, and modifications to the disclosed exemplary embodiments and other exemplary embodiments are also intended to be included within the scope of the appended claims. The inventive concept is defined by the foregoing claims, the equivalents of which are included herein.

Claims

1. A method for lossy data compression, characterized in that, The method includes: Receive raw data at the storage device; Receive compression requests; Access to a data compression algorithm library, the data compression algorithm library containing data compression algorithms corresponding to lossy data compression schemes; One of the data compression algorithms is selected based on the metadata associated with the key in the key value pair corresponding to the original data; Estimate the amount of available resources required for compression, wherein the amount of available resources required for compression includes the amount of time required for compression, the amount of available system resources required for compression, or a combination thereof; and Based on the estimated amount, the selected data compression algorithm is run in either the first or second mode; The storage device is configured to execute the data compression algorithm according to either the first mode or the second mode. In the first mode, the selected data compression algorithm is executed online, such that the received raw data is received by the storage device, compressed by the storage device, and stored in the storage device as compressed data. In the second mode, the selected data compression algorithm is executed offline, such that the raw data is stored at the storage device, compressed by the storage device according to the selected data compression algorithm, and re-stored at the storage device as compressed data.

2. The method according to claim 1, characterized in that, It also includes prompting users who request compression of the raw data to provide the metadata.

3. The method according to claim 1, characterized in that, Also includes: Read the metadata associated with the original data, which corresponds to an object signature or object extension; Access the object signature dictionary or the object extension dictionary on the storage device to determine the data type of the original data based on the object signature or the object extension, respectively; as well as The data compression algorithm is selected based on the determined data type.

4. A data compression method, characterized in that, The method includes: Receive raw data at the storage device; The original data is compressed using the storage device according to a lossy data compression scheme to produce compressed data; The compressed data is stored on the storage device; Read the metadata associated with the key in the key value pair corresponding to the original data; A data compression algorithm is selected from the data compression algorithm library on the storage device based on the metadata; Estimate the amount of available resources required for compression, wherein the amount of available resources required for compression includes the amount of time required for compression, the amount of available system resources required for compression, or a combination thereof; Based on the estimated amount, a data compression scheme for compressing the original data is executed online or offline according to the selected data compression algorithm; and When the lossy data compression scheme is executed online, the original data is compressed when it is received by the storage device.

5. The method according to claim 4, characterized in that, It also includes using the storage device to determine the data compression scheme for compressing the raw data.

6. The method according to claim 5, characterized in that, Also includes: Create a Quality of Service (QoS) preference data file containing information related to the compression scheme and parameters; The storage device identifies the object file corresponding to the original data; as well as The data compression scheme for compressing the original data is determined based on the information and the object file.

7. The method according to claim 4, characterized in that, The original data includes object signatures or object extensions, and the method further includes: Access the object signature dictionary or the object extension dictionary on the storage device to determine the data type of the original data based on the object signature or the object extension, respectively; and The data compression scheme is executed based on the determined data type.

8. The method according to claim 4, characterized in that, It also includes sending the raw data from the host computer to the storage device according to a user request.

9. The method according to claim 8, characterized in that, Also includes: Provide a menu to remind users to provide information; as well as The data compression algorithm used to compress the original data is determined based on the information provided by the user.

10. The method according to claim 4, characterized in that, It also includes storing the original data on the storage device; Estimate the available time or resources corresponding to the storage device; and When sufficient time and resources are available, the raw data is compressed.

11. A data compression system, characterized in that, include: Host; as well as A storage device, configured to receive raw data from the host, and comprising: Memory; and A processor is configured to execute instructions that, when executed by the processor, cause the processor to: The storage device compresses the original data according to a lossy data compression scheme to produce compressed data; The storage device stores the compressed data on the storage device. Read the metadata of the key corresponding to the key value pair of the original data; Select a data compression algorithm from the data compression algorithm library based on the metadata; Estimate the amount of available resources required for compression, wherein the amount of available resources required for compression includes the amount of time required for compression, the amount of available system resources required for compression, or a combination thereof; Based on the estimated amount, the data compression scheme for compressing the original data is executed online or offline according to the selected data compression algorithm; and When the lossy data compression scheme is executed online, the original data is compressed when it is received by the storage device.

12. The system according to claim 11, characterized in that, When executed by the processor, the instructions also cause the processor to cause the storage device to determine the data compression scheme for compressing the raw data.

13. The system according to claim 12, characterized in that, When executed by the processor, the instruction also causes the processor to access the storage device: Access to a Quality of Service (QoS) preference data file containing information about compression schemes and parameters; Identify the object file corresponding to the original data; as well as The data compression scheme for compressing the original data is determined based on the information and the object file.

14. The system according to claim 12, characterized in that, The original data includes object signatures or object extensions, and The instructions, when executed by the processor, also cause the processor to access the storage device: Access the object signature dictionary or the object extension dictionary on the storage device to determine the data type of the original data based on the object signature or the object extension, respectively; as well as The data compression scheme is executed based on the determined data type.

15. The system according to claim 11, characterized in that, When executed by the processor, the instructions also cause the processor to enable the storage device to receive the raw data from the host computer upon user request.

16. The system according to claim 15, characterized in that, When the instruction is executed by the processor, it also causes the processor to: The host provides a menu to alert the user who is requesting compression of the raw data. as well as The storage device determines a data compression algorithm for compressing the raw data based on information provided by the user.

17. The system according to claim 11, characterized in that, When executed by the processor, the instruction also causes the processor to access the storage device: The original data is stored on the storage device; Estimate the available time or resources corresponding to the storage device; as well as When sufficient time and resources are available, the raw data is compressed.

Images