Data writing method and system and data reading method and system

Abstract

The invention provides a data writing method and system and a data reading method and system, which can be applied to the technical field of data storage and the technical field of data compression. The data writing method comprises the following steps: in response to a received writing request for target business data issued by an application layer, calling a compression engine to perform data compression on the target business data to obtain target compressed data; and based on predefined metadata, writing the target compressed data into a target storage area in the storage medium according to a compression mode adopted for performing data compression on the target business data.

Description

Technical Field

[0001] The present invention relates to the fields of data storage technology and data compression technology, and in particular to a data writing method and system, and a data reading method and system. Background Art

[0002] With the rapid development of information technology, data volumes are exploding. This is especially true in emerging fields like artificial intelligence and big data computing. The storage and management of massive amounts of data have become significant challenges for enterprises. To effectively address this challenge, data compression technology has emerged. Through methods like data compression and data deduplication, it reduces data storage space requirements, thereby lowering storage costs for enterprises.

[0003] During the implementation of this invention, it was discovered that during data compression, metadata must be used to record the compression algorithm corresponding to each piece of compressed data to facilitate subsequent data decompression. Because metadata can be very large, this creates significant storage pressure. Furthermore, during decompression, the compression algorithm used must be determined for each piece of data, increasing system overhead. Summary of the Invention

[0004] In view of the above problems, the present invention provides a data writing method and system, and a data reading method and system.

[0005] According to a first aspect of the present invention, a data writing method is provided, comprising: in response to receiving a write request for target business data issued by an application layer, calling a compression engine to compress the target business data to obtain target compressed data; based on predefined metadata, according to the compression method used to compress the target business data, writing the target compressed data to a target storage area in a storage medium, wherein the metadata represents a mapping relationship between multiple storage areas included in the storage medium and multiple compression methods.

[0006] The second aspect of the present invention provides a data reading method, which includes: in response to receiving a read request for target business data issued by an application layer, based on storage address information related to the target business data included in the read request, reading target compressed data obtained by compressing the target business data from a target storage area among multiple storage areas included in a storage medium; based on predefined metadata, determining a target compression method corresponding to the target storage area, wherein the metadata represents a mapping relationship between multiple storage areas included in the storage medium and multiple compression methods, and the compressed data in the same storage area is obtained based on the same compression method; calling a decompression engine to use a target decompression method corresponding to the target compression method to decompress the target compressed data to obtain the target business data.

[0007] The third aspect of the present invention provides a data writing system, which includes: an application layer for initiating a write request for target business data; a compression engine for compressing the target business data to obtain target compressed data; a storage medium, including multiple storage areas associated with multiple compression methods; a first processor for calling the compression engine to perform data compression based on the write request, and based on predefined metadata, according to the compression method used by the compression engine to compress the target business data, writing the target compressed data to the target storage area in the storage medium, wherein the metadata represents the mapping relationship between the multiple storage areas included in the storage medium and the multiple compression methods.

[0008] The fourth aspect of the present invention provides a data reading system, which includes: an application layer for initiating a read request for target business data; a storage medium including multiple storage areas associated with multiple compression methods; a second processor for reading target compressed data obtained by compressing the target business data from a target storage area among the multiple storage areas based on storage address information related to the target business data included in the read request; determining a target compression method corresponding to the target storage area based on predefined metadata, wherein the metadata represents a mapping relationship between multiple storage areas included in the storage medium and multiple compression methods, and the compressed data in the same storage area is obtained based on the same compression method; a decompression engine for decompressing the target compressed data to obtain the target business data using a target decompression method corresponding to the target compression method based on a decompression instruction issued by the second processor.

[0009] The fifth aspect of the present invention provides a data writing device, comprising: a compression module, for responding to a write request for target business data issued by the application layer, calling a compression engine to compress the target business data to obtain target compressed data; a writing module, for writing the target compressed data into a target storage area in a storage medium based on predefined metadata and the compression method used for compressing the target business data, wherein the metadata represents a mapping relationship between multiple storage areas included in the storage medium and multiple compression methods.

[0010] The sixth aspect of the present invention provides a data reading device, comprising: a reading module, for responding to a read request for target business data issued by an application layer, and based on storage address information related to the target business data included in the read request, reading target compressed data obtained by compressing the target business data from a target storage area among multiple storage areas included in a storage medium; a determination module, for determining a target compression method corresponding to the target storage area based on predefined metadata, wherein the metadata represents a mapping relationship between multiple storage areas included in the storage medium and multiple compression methods, and the compressed data in the same storage area is obtained based on the same compression method; a decompression module, for calling a decompression engine to use a target decompression method corresponding to the target compression method to decompress the target compressed data to obtain target business data.

[0011] The seventh aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0012] The eighth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0013] The ninth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0015] Figure 1 A diagram showing application scenarios of a data writing method and system, and a data reading method and system according to an embodiment of the present invention is shown;

[0016] Figure 2 A flow chart of a data writing method according to an embodiment of the present invention is shown;

[0017] Figure 3 An example of the data volume of metadata corresponding to each storage area according to an embodiment of the present invention is shown;

[0018] Figure 4 A schematic diagram illustrating data compression and storage of service data based on the data type of the service data according to an embodiment of the present invention is shown;

[0019] Figure 5shows an architecture diagram of a storage medium implemented as a three-level hybrid storage according to an embodiment of the present invention;

[0020] Figure 6 A flow chart of a data reading method according to an embodiment of the present invention is shown;

[0021] Figure 7A A structural diagram of a data writing system according to an embodiment of the present invention is shown;

[0022] Figure 7B Shown is a structural diagram of a data reading system according to an embodiment of the present invention;

[0023] Figure 8 It shows a structural block diagram of a data writing device according to an embodiment of the present invention;

[0024] Figure 9 A structural block diagram of a data reading device according to an embodiment of the present invention is shown;

[0025] Figure 10 A block diagram of an electronic device suitable for implementing a data writing method and a data reading method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0029] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] An embodiment of the present invention provides a data writing method, comprising: in response to receiving a write request for target business data issued by an application layer, calling a compression engine to compress the target business data to obtain target compressed data; based on predefined metadata, according to the compression method used to compress the target business data, writing the target compressed data to a target storage area in a storage medium, wherein the metadata represents a mapping relationship between multiple storage areas included in the storage medium and multiple compression methods.

[0031] Figure 1 The application scenario diagram of the data writing method and system, and the data reading method and system according to the embodiments of the present invention is shown.

[0032] like Figure 1 As shown, an application scenario 100 according to this embodiment may include a business system 101 and a storage system 102. Business system 101 represents the application layer, and storage system 102 further includes a processor, a compression / decompression engine, and a storage medium. Business system 101 and storage system 102 are connected via a network. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0033] The business system 101 includes a business system host, which generates business data during the execution of business and issues data writing or reading requests.

[0034] The processor may be of various types, for example, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), etc.

[0035] The compression / decompression engine is used to perform data compression or decompression tasks, compressing business data to obtain compressed data, and decompressing compressed data to restore business data. The compression / decompression engine can be the same processing engine or two different processing engines, such as a compression engine and a decompression engine.

[0036] The compression / decompression engine may adopt an external compression card, such as a GPU, a Field-Programmable Gate Array (FPGA), and the like.

[0037] The storage medium may be any type of storage medium, and may be a single storage medium, such as a cache or a hard disk; or a mixed storage medium, such as including three levels of storage, such as a first-level cache, a second-level cache, and a storage pool.

[0038] In the scenario of an embodiment of the present invention, the processor is used to generate a corresponding data processing strategy based on the data write or read request issued by the business system 101, and call the compression / decompression engine to perform data compression or data decompression tasks, and store the compressed data to a storage medium, or read compressed data from the storage medium for decompression by the compression / decompression engine.

[0039] For example, upon receiving a data write request from the business system 101, the processor sends the business data sent by the business system 101 to the compression / decompression engine, and sends a compression instruction to the compression / decompression engine to call the compression / decompression engine to compress the business data to obtain compressed data. The compressed data is then stored in a storage medium.

[0040] Furthermore, when the storage medium is implemented as a hybrid storage medium, such as the above-mentioned three-level storage, the compressed data can be written to the disk in the first-level cache, second-level cache, and storage pool step by step, that is, first written to the first-level cache, then written to the second-level cache, and finally written to the storage pool.

[0041] For another example, when receiving a data reading request issued by the business system 101, the processor reads the compressed data from the storage medium and sends a decompression instruction to the compression / decompression engine to call the compression / decompression engine to decompress the compressed data, and returns it to the business system 101 after obtaining the business data.

[0042] Furthermore, when the storage medium is implemented as a hybrid storage medium, such as the above-mentioned three-level storage, it can be read from the first-level cache first. If there is no compressed data in the first-level cache, the compressed data can be read from the second-level cache. If there is no compressed data in the second-level cache, the compressed data can be read from the storage pool.

[0043] The following will be based on Figure 1 The scene described by Figures 2 to 5 The data writing method according to the embodiment of the present invention is described in detail.

[0044] Figure 2 A flow chart of a data writing method according to an embodiment of the present invention is shown.

[0045] like Figure 2 As shown, the data writing method of this embodiment includes operations S201 to S202.

[0046] In operation S201 , in response to receiving a write request for target business data issued by an application layer, a compression engine is called to compress the target business data to obtain target compressed data.

[0047] In operation S202, based on predefined metadata and according to the compression method used to compress the target business data, the target compressed data is written to the target storage area in the storage medium, wherein the metadata represents the mapping relationship between multiple storage areas included in the storage medium and multiple compression methods.

[0048] According to an embodiment of the present invention, the system architecture to which the above data writing method can be applied can refer to Figure 1 The system shown includes an application layer (ie, business system 101), a processor, a compression engine, and a storage medium. The above data writing method can be executed by the processor.

[0049] After the application layer initiates a write request for the target business data, the processor may call the compression engine to compress the target business data and write the compressed target compressed data into the storage medium.

[0050] The storage medium may be any type of storage medium, and may be a single storage medium, such as a cache or a hard disk; or a hybrid storage medium, such as including three levels of storage: a first-level cache, a second-level cache, and a storage pool.

[0051] Among them, the compression engine compresses the currently incoming target business data by adaptively using the most suitable compression method at the moment, such as selecting the most suitable compression method for the current scenario from a plurality of predetermined compression methods.

[0052] For example, depending on the size of the target business data, fast compression is used for large blocks of data (such as larger than 1M), and deep compression is used for small blocks of data (such as smaller than 50K).

[0053] For example, according to the current load of the processor, when the current load is large, fast compression (or no compression) is used, and when the current load is small, deep compression is used.

[0054] For example, according to the priority corresponding to the current target business data, fast compression (or no compression) is used when the priority is high, deep compression is used when the priority is low, and so on.

[0055] Therefore, for a continuous data flow, different business data can be compressed using different compression methods.

[0056] Since the compressed data needs to be decompressed during the subsequent data reading phase, the compression method used for each business data item must be recorded in advance to facilitate the use of the corresponding decompression method during decompression. This information is recorded in metadata. If each business data item is recorded, the metadata volume will be very large. On the one hand, since metadata is stored in memory, memory space is relatively limited, resulting in significant storage pressure. On the other hand, during decompression, the compression algorithm used must be determined for each data item, increasing system overhead.

[0057] In order to solve the above technical problems, the above method of the embodiment of the present invention divides the storage medium into multiple storage areas in advance, binds the corresponding relationship between the compression method and the storage area, and uses metadata to record this corresponding relationship. When storing data, the data obtained using different compression methods are stored in different storage areas in a one-to-one correspondence. In this way, the metadata only needs to record one piece of information for each storage area to indicate the compression method corresponding to the data in the storage area, so as to realize the information recording of all data in the storage area, and the volume of metadata is greatly reduced. Subsequently, when reading data, all data read from a specific storage area are bound to the same compression method, and there is no need to judge each data item.

[0058] Figure 3 An example of the data volume of metadata corresponding to each storage area according to an embodiment of the present invention is shown.

[0059] like Figure 3 As shown, it is assumed that there are three optional compression methods for compressing business data: compression method A: no compression; compression method B: fast compression; compression method C: deep compression.

[0060] The storage medium is pre-divided into three storage areas: the first storage area, the second storage area, and the third storage area. The space corresponding to the address of 0-1TB is allocated to the first storage area, the space corresponding to the address of 1-2TB is allocated to the second storage area, and the space corresponding to the address of 2-3TB is allocated to the third storage area.

[0061] Metadata is used to map the three storage areas to the three compression methods: the first storage area is bound to compression method A, the second storage area is bound to compression method B, and the third storage area is bound to compression method C. In other words, uncompressed data is stored in the first storage area, data using fast compression is stored in the second storage area, and data using deep compression is stored in the third storage area.

[0062] Among them, it is assumed that the amount of metadata required to record a mapping relationship is 4B. Since the data in the first storage area is not compressed, there is no need to record its compression method. Therefore, the size of the metadata for this storage area is 0B; for the data in the second storage area and the third storage area, only 4B of metadata is required to record their compression methods.

[0063] If the relevant method is adopted, a mapping relationship is recorded for each business data. Assuming that the size of each business data is 4KB, 4B metadata is required for each 4KB block; however, if the method of the embodiment of the present invention is adopted, only 4B metadata is required for each 1TB partition, saving 99.99% of metadata space.

[0064] According to an embodiment of the present invention, a storage medium is pre-divided into multiple storage areas using the method of an embodiment of the present invention. During data storage, data obtained using different compression methods is stored in separate storage areas in a one-to-one correspondence. The corresponding relationship between the compression method and the storage area is bound, and metadata is used to record this correspondence. In this way, metadata only needs to record one piece of information for each storage area, eliminating the need to use metadata to record the compression method used for each piece of data. This significantly reduces the metadata volume, solves the metadata expansion problem, and reduces storage pressure. Furthermore, when subsequently reading data, all data read from a specific storage area is bound to the same compression method, eliminating the need to judge each piece of data, reducing system overhead and improving system responsiveness.

[0065] According to an embodiment of the present invention, calling the compression engine to compress the target business data may be performed according to the data type. Based on this, calling the compression engine to compress the target business data to obtain the target compressed data may include the following operations.

[0066] First, the data type of the target business data is determined. Then, a compression engine is invoked to compress the target business data based on the data type, generating target compressed data. For example, mappings between multiple data types and multiple compression methods can be pre-assigned, allowing specific compression methods to be used for specific data types.

[0067] Determining the data type of the target business data includes: obtaining attributes of the target business data, wherein the attributes include at least one of the following: continuity of the write address, data volume, and compression rate; and determining the data type of the target business data based on the attributes of the target business data.

[0068] There are many methods for distinguishing different types of data based on the above attributes, including but not limited to the following methods.

[0069] For example, data written continuously may be determined as one type of data, while data written non-continuously, i.e., randomly, may be determined as another type of data. For example, data with a data volume greater than a predetermined threshold may be determined as one type of data, while data with a data volume less than a predetermined threshold may be determined as another type of data. For example, data with a compression ratio greater than a predetermined threshold may be determined as one type of data, while data with a compression ratio less than a predetermined threshold may be determined as another type of data. For example, data written continuously and with a data volume greater than a predetermined threshold may be determined as one type of data, while data written randomly and with a data volume less than a predetermined threshold may be determined as another type of data; and so on.

[0070] According to an embodiment of the present invention, the compression method in the related art is single, and the same compression algorithm is used for all business data, resulting in invalid compression and waste of computing power. For example, data with a high compression rate is still compressed, and even performance is affected due to redundant processing, resulting in low compression efficiency.

[0071] The method of the embodiment of the present invention provides differentiated compression method selection for different types of data, dynamically adjusts the compression method according to the different data types, solves the problems of single data reduction method and mismatch between data reduction strategy and data type, avoids invalid compression, and achieves the optimal balance between compression efficiency and performance.

[0072] According to embodiments of the present invention, continuously writing data places higher demands on system responsiveness. Larger data volumes require longer compression calculations, while data with higher compression ratios tends to be less compressible. By comprehensively defining data types based on these performance characteristics, a more accurate compression method can be adapted. Therefore, write address continuity, data volume, and compression ratio can be used as comprehensive indicators to distinguish data types.

[0073] Based on this, according to the attributes of the target business data, determining the data type of the target business data includes: when the continuity of the write address of the target business data meets the predetermined conditions, the data volume is greater than the predetermined data volume threshold, and the compression rate is less than the predetermined compression rate threshold (such as 50%), the target business data is determined to be the first type of data; when the continuity of the write address of the target business data does not meet the predetermined conditions, and the data volume is less than or equal to the predetermined data volume threshold, the target business data is determined to be the second type of data.

[0074] In addition, data with a compression rate greater than a predetermined compression rate threshold (such as 90%) may be defined as a third category of data.

[0075] Specifically, continuously written large blocks of data (≥1MB) with an entropy value below a threshold can be identified as first-type data, such as streaming media data; for example, video data, image data, and file data. Randomly written small blocks of data (≤64KB) can be identified as second-type data, such as database data. For example, operational data during database addition, deletion, modification, and query operations. Data with a compression ratio greater than a predetermined compression ratio threshold (e.g., 90%) can be defined as compressed data.

[0076] Among them, the compression rate of data can be determined by calculating the entropy value of the data. The higher the entropy value, the more random the data is and the lower the compressibility is; conversely, the lower the entropy value, the more regular the data is and the higher the compressibility is.

[0077] Among them, the entropy value of data with a data volume less than or equal to the predetermined data volume threshold is difficult to calculate accurately due to its small data volume. Therefore, in order to improve the accuracy of data classification, the reference indicator of the entropy value is discarded for this type of data, and only the continuity of the write address and the data volume are used to distinguish them.

[0078] According to an embodiment of the present invention, based on the above-defined data type, calling a compression engine to perform data compression on target business data based on the data type of the target business data includes the following method.

[0079] When the data type of the target business data is the first type of data, the compression engine is called to compress the target business data using the first compression method.

[0080] When the data type of the target business data is the second type of data, the compression engine is called to compress the target business data using the second compression method.

[0081] In the case where the data type of the target service data is the third type of data, no compression is performed.

[0082] The first compression method has a faster compression speed than the second compression method, but a lower compression ratio than the second compression method. The first compression method is fast compression, which has a faster compression speed but a lower compression ratio; the second compression method is deep compression, which has a slower compression speed but a higher compression ratio.

[0083] For the first compression method, for example, the following compression algorithms may be used but are not limited to: LZ4 (Lempel-Ziv4), Snappy (Snappy compression algorithm), LZO (Lempel-Ziv-Oberhumer); for the second compression method, for example, the following compression algorithms may be used but are not limited to: Zstandard, Brotli (lossless data compression algorithm), LZMA (Lempel-Ziv-Markov chain algorithm).

[0084] According to an embodiment of the present invention, continuous writing of data places higher demands on the system's write response performance. If deep compression is used, large continuous spaces will be occupied for a long time, affecting storage utilization and response speed; the processing time of data compression calculations with larger data volumes will be longer. If deep compression is used, it will take a long time to process the data, and the system response speed will be slower; data with a higher compression ratio is not highly compressible. If it is still compressed, it will cause a waste of compression resources. By comprehensively considering these performance factors to define the data type and adapting a more appropriate compression method, it is possible to take into account both system performance and compression efficiency, and achieve a balance between compression efficiency and performance.

[0085] According to an embodiment of the present invention, a method for implementing data compression and writing in combination with different data types may be: calling a compression engine to compress business data according to the data type, using different compression methods for different data types, and further using metadata to record the binding relationship between multiple compression methods and storage areas, and based on the metadata, storing the data obtained using different compression methods in different storage areas one by one.

[0086] Furthermore, it is possible to establish one-to-one corresponding task queues for various data types and compression methods, distribute business data of different data types to different data queues, and compress the data in a specific task queue using a specific compression method, and store the compressed data in a specific storage area. Metadata can record the storage area ID and the associated task queue, thus linking the compression method and storage area. For example, metadata can be a pre-configured three-level mapping relationship between data type, data queue, compression method, and storage area.

[0087] Calling the compression engine to compress the target business data based on the data type of the target business data includes: adding the target business data to a target task queue in multiple task queues based on the data type of the target business data, wherein the multiple task queues are used to store business data of multiple predetermined data types and are associated with multiple compression methods; calling the compression engine to read the target business data from the target task queue, and using the target compression method associated with the target task queue to compress the target business data.

[0088] Furthermore, after the target service data is compressed, the obtained target compressed data is written into a target storage area corresponding to the target task queue / target compression mode.

[0089] Figure 4 A schematic diagram illustrating a principle of compressing and storing service data based on the data type of the service data according to an embodiment of the present invention is shown.

[0090] like Figure 4 As shown, one-to-one corresponding task queues are established for various data types / compression methods, such as data type A corresponds to task queue A, data type B corresponds to task queue B, data type C corresponds to task queue C, and so on.

[0091] After receiving business data, the processor first classifies the data, determines the data type, and distributes business data of different data types to different data queues. Data type A is sent to task queue A, data type B is sent to task queue B, data type C is sent to task queue C, and so on.

[0092] The compression methods corresponding to different task queues are pre-defined. Task queue A corresponds to compression method A, task queue B corresponds to compression method B, task queue C corresponds to compression method C, and so on.

[0093] The compression engine is called to read business data from the task queue and compress it using the compression method bound to the task queue. For example, data read from task queue A is compressed using compression method A, data read from task queue B is compressed using compression method B, data read from task queue C is compressed using compression method C, and so on.

[0094] Finally, based on the association relationship defined by metadata (metadata can record the association relationship between storage areas and task queues, realizing the association and binding of compression methods and storage areas), the compressed data corresponding to different task queues are stored in different storage areas. The compressed data A corresponding to task queue A is stored in the first storage area, the compressed data B corresponding to task queue B is stored in the second storage area, and the compressed data C corresponding to task queue C is stored in the third storage area...

[0095] According to an embodiment of the present invention, by increasing the task queue, it is realized that different data types, different compression methods, and different storage spaces are associated and bound, and data distribution, task distribution, and storage allocation are organized rationally and orderly. By increasing the task queue, on the one hand, the task queue acts as a middleware to realize the decoupling of upstream and downstream. After the current business data is issued, the processor can distribute it to the task queue and can execute the processing and analysis of the next data without waiting for the compression engine to complete the compression of the previous data. It can be seen that the processor assigns tasks and the compression engine compresses data asynchronously, which improves the system processing performance as a whole. On the other hand, by increasing the task queue, it is convenient to establish the binding of the task queue and the compression method. Like this, the compression engine does not need to judge the compression method of each data one by one, thereby improving the system processing performance.

[0096] According to an embodiment of the present invention, another method for achieving data compression and writing in combination with different data types may be: calling a compression engine to compress business data according to the data type, using different compression methods for different data types, and further using metadata to record the binding relationship between multiple compression methods and storage areas, and based on the metadata, storing the data obtained using different compression methods in different storage areas one by one.

[0097] Furthermore, the compression engine can be configured to include multiple compression processing units, each configured to execute various compression methods corresponding to business data of various predetermined data types. For example, compression processing unit A executes compression method A on data of data type A, compression processing unit B executes compression method B on data of data type B, compression processing unit C executes compression method C on data of data type C, and so on. The compressed data is then stored in a specific storage area. Metadata can record the association between the storage area and the compression processing unit, effectively binding the compression method to the storage area.

[0098] Assume that the data type of the target business data is a target type among multiple predetermined data types; call the compression engine to compress the target business data based on the data type of the target business data, including: determining a target compression processing unit corresponding to the target type from multiple compression processing units; calling the target compression processing unit to compress the target business data using a target compression method corresponding to the target type.

[0099] For example, the metadata pre-configures a three-level mapping relationship of data type-compression method / compression processing unit-storage area.

[0100] After the business data flow is sent down, the processor classifies the business data into data type A, data type B, data type C, etc.; according to the configured three-level mapping relationship, the processor calls compression processing unit A to perform compression method A on the data of data type A, calls compression processing unit B to perform compression method B on the data of data type B, calls compression processing unit C to perform compression method C on the data of data type C, etc., and stores the compressed data in a specific storage area.

[0101] According to the embodiments of the present invention, by providing multiple compression processing units, different data types, different compression methods, and different storage spaces can be associated and bound, rationally and orderly organizing data distribution, task distribution, and storage allocation. Furthermore, multiple compression tasks can be processed in parallel, improving system processing performance.

[0102] According to an embodiment of the present invention, in the process of calling the compression engine to compress the target business data based on the data type of the target business data, the compression method can also be dynamically adjusted according to the business priority and the processor load.

[0103] Based on this, calling the compression engine to compress the target business data based on the data type of the target business data includes: obtaining the business priority corresponding to the target business data and the processor utilization; when the business priority corresponding to the target business data belongs to the predetermined priority range and the processor utilization is greater than the predetermined utilization threshold, calling the compression engine to use the first compression method to compress the target business data, or not to compress the target business data; for example, if the target business data is high priority data and the processor utilization is greater than the predetermined utilization threshold (such as greater than 70%), fast compression is used.

[0104] When the business priority corresponding to the target business data does not fall within the predetermined priority range, or the processor utilization is less than or equal to the predetermined utilization threshold, the compression engine is called to compress the target business data using the second compression method; for example, if the target business data is low priority data and the processor utilization is less than the predetermined utilization threshold (such as less than 50%), deep compression is used.

[0105] This allows for dynamic adjustments to compression policies based on service priority and processor load, improving the response speed of high-priority services. For example, data partitions for high-priority services can be automatically switched to fast compression mode, freeing up computing power to ensure high-priority services are served.

[0106] Furthermore, data compression of the target business data based on the data type of the target business data may be performed by dynamically adjusting the predetermined compression method corresponding to the data type according to the business priority corresponding to the target business data and the utilization of the processor during the compression process. Specific implementation methods may include but are not limited to the following. The first implementation method is: when the data type of the target business data is the second type of data (database type, the pre-configured compression method is deep compression), the business priority corresponding to the target business data falls within the predetermined priority range, and the utilization of the processor is greater than the first utilization threshold, the compression engine is called to use the first compression algorithm (fast compression) to compress the target business data.

[0107] In this way, for high-priority services, the processor increases (such as CPU utilization > 70%), and automatically switches the data of high-priority services from deep compression to fast compression mode, releasing computing power to ensure high-priority services.

[0108] The second implementation method is: when the data type of the target business data is the second type of data (database type, the pre-configured compression method is deep compression), and the business priority corresponding to the target business data does not belong to the predetermined priority range or the processor utilization is between the second utilization threshold and the first utilization threshold, the compression engine is called to use the second compression algorithm (deep compression) to compress the target business data.

[0109] In this mode, if the service priority is not high and the processor pressure is moderate (for example, 30% < CPU utilization < 70%), no adjustment is made and compression is performed according to the preset compression mode.

[0110] The third implementation method is: when the data type of the target business data is the first type of data (streaming media type, the pre-configured compression method is fast compression), the business priority corresponding to the target business data belongs to the predetermined priority range, and the processor utilization is greater than the first utilization threshold, the target business data is not compressed.

[0111] In this way, for high-priority services, the processor load increases (such as CPU utilization > 70%), and the original fast compression mode is changed to no compression, freeing up computing power to ensure high-priority services.

[0112] Among them, the fourth implementation method is: when the data type of the target business data is the first type of data (streaming media type, the pre-configured compression method is fast compression), and the business priority corresponding to the target business data does not belong to the predetermined priority range or the processor utilization is between the second utilization threshold and the first utilization threshold, the compression engine is called to use the first compression algorithm (fast compression) to compress the target business data.

[0113] In this mode, if the service priority is not high and the processor pressure is moderate (for example, 30% < CPU utilization < 70%), no adjustment is made and compression is performed according to the preset compression mode.

[0114] The fifth implementation method is: when the data type of the target business data is the first type of data (streaming media type, the pre-configured compression method is fast compression), and the business priority corresponding to the target business data does not belong to the predetermined priority range or the processor utilization is less than the second utilization threshold, the compression engine is called to use the second compression algorithm (deep compression) to compress the target business data.

[0115] In this way, for low-priority services and reduced processor load (such as CPU utilization less than 30%), the original fast compression mode is changed to deep compression, which can ensure high data compressibility without affecting business performance.

[0116] According to an embodiment of the present invention, the storage medium used in the data writing method of the above embodiment can be implemented as any type of storage medium, and can be implemented as a single storage medium, such as a cache or a hard disk; or it can be implemented as a hybrid storage medium, for example, including a cache and a hard disk.

[0117] Furthermore, when the storage medium is implemented as a hybrid storage medium, it can be further implemented as a three-level hybrid storage, such as a first-level cache, a second-level cache, and a storage pool.

[0118] Furthermore, the data read and write speed supported by the first-level cache is greater than that of the second-level cache, and the data read and write speed supported by the second-level cache is greater than that of the storage pool; the capacity of the first-level cache is smaller than that of the second-level cache, and the capacity of the second-level cache is smaller than that of the storage pool.

[0119] Figure 5 The diagram shows an architecture in which a storage medium according to an embodiment of the present invention is implemented as a three-level hybrid storage.

[0120] like Figure 5 As shown, the first-level cache can be implemented as a volatile memory, the second-level cache can be implemented as a non-volatile memory, and the storage pool can be implemented as a non-volatile hard disk.

[0121] The first-level cache can be implemented as, for example, random access memory (RAM), dynamic random access memory (DRAM), or static random access memory (SRAM).

[0122] The second-level cache can be implemented as, for example, Storage Class Memory (SCM).

[0123] The storage pool can be implemented as, for example, a hard disk drive (HDD), a solid state drive (SSD), or a solid state hybrid drive (SSHD).

[0124] The compressed data obtained by using different compression algorithms is eventually written into multiple storage areas in the storage pool.

[0125] Furthermore, when the storage medium is implemented as a hybrid storage medium, such as the above-mentioned three-level storage, the compressed data can be written to the disk in the first-level cache, second-level cache, and storage pool step by step, that is, first written to the first-level cache, then written to the second-level cache, and finally written to the storage pool.

[0126] For example, writing the target compressed data to the target storage area in the storage medium includes: first, writing the target compressed data to the first-level cache; then, based on the capacity of the first-level cache and / or the storage time of the target compressed data in the first-level cache, if a first transfer condition is met, the target compressed data is transferred from the first-level cache to the second-level cache. For example, if the amount of data in the first-level cache is full and / or the storage time of the target compressed data in the first-level cache is greater than a first time threshold, the target compressed data is transferred from the first-level cache to the second-level cache; finally, based on the capacity of the second-level cache and / or the storage time of the target compressed data in the second-level cache, if a second transfer condition is met, the target compressed data is transferred from the second-level cache to the target storage area in the storage pool. For example, if the amount of data in the second-level cache is full and / or the storage time of the target compressed data in the second-level cache is greater than a second time threshold, the target compressed data is transferred from the second-level cache to the target storage area in the storage pool.

[0127] According to an embodiment of the present invention, the data first stored in the first-level cache is business data generated in the most recent time period and is read most frequently by users. Therefore, the data in the first-level cache is considered hot data. The data in the second-level cache is next in line, belonging to business data generated in a relatively recent time period. The frequency of user reads is also relatively high, but less than that of the first-level cache. Therefore, the data in the second-level cache is also considered hot data. In comparison, the data in the storage pool is read least frequently and is considered cold data.

[0128] According to an embodiment of the present invention, since the data reading and writing speed of the first-level cache - volatile memory is the fastest, data falls into the first-level cache first, which facilitates the rapid writing of hot data, and subsequent reading of hot data from the first-level cache is also faster, and data response performance can be improved.

[0129] However, due to the volatility of data in volatile memory, data must ultimately be stored in non-volatile storage. Since volatile memory generally has a smaller storage capacity, hot data may not be fully stored in the L1 cache. When the volatile memory is full, some data must be transferred to disk. Furthermore, if partial data is directly stored in a non-volatile hard drive, some recently generated hot data may be written to the hard drive. The slower read and write speeds of hard drives can reduce the read and write responsiveness of hot data.

[0130] To address this issue, non-volatile memory is added between the main memory and the hard drive as a second-level cache. The second-level cache supports data read and write speeds greater than the storage pool, but has a larger capacity than the first-level cache. This allows hot data to be removed from the first-level cache without having to be written directly to the hard drive. Instead, it can be temporarily stored in the second-level cache. The second-level cache temporarily stores large amounts of hot data, maximizing hot data retention and accelerating hot data read and write performance. Furthermore, the second-level cache uses non-volatile memory, which prevents data loss during power outages and improves the security of this data. This three-level storage mechanism also improves overall cache space utilization.

[0131] According to an embodiment of the present invention, in the above-mentioned data writing method, after receiving a write request for the target business data issued by the application layer, the data storage mechanism based on the above-mentioned three-level storage can be a method as described in the above-mentioned embodiment, and the target compressed data after the target business data is compressed will be written to the disk step by step through the first-level cache, the second-level cache and the storage pool.

[0132] Another implementation method may be: only retaining the original target business data in the first-level cache, and implementing step-by-step write-down of the compressed target business data from the second-level cache to the storage pool.

[0133] Specifically, writing the target compressed data into the target storage area in the storage medium includes: writing the target business data into the first-level cache, and writing the target compressed data into the second-level cache; when the storage time of the target business data in the first-level cache is greater than the third time threshold, deleting the target business data in the first-level cache.

[0134] When the data volume of the secondary cache is full and / or the storage time of the target compressed data in the secondary cache is greater than a second time threshold, the target compressed data is transferred from the secondary cache to the target storage area in the storage pool.

[0135] Another embodiment may be to write the compressed target business data to disk in a sequential manner through the first-level cache, the second-level cache, and the storage pool, and retain the original target business data in the first-level cache for a predetermined period of time. For example, the target business data may be written to the first-level cache in response to receiving a write request; and the target business data may be deleted from the first-level cache when the target compressed data is transferred from the first-level cache to the second-level cache, or when the storage time of the target business data in the first-level cache exceeds a third time threshold.

[0136] According to an embodiment of the present invention, the selection of the above-mentioned implementation methods can be dynamically adjusted in combination with the real-time utilization of the first-level cache.

[0137] For example, when the utilization rate of the first-level cache is high (for example, greater than 70%), the target compressed data after compressing the target business data can be temporarily stored only in the first-level cache, and the target compressed data can be written to the disk step by step through the first-level cache, second-level cache and storage pool; this method can save storage space and extend the storage time of hot data in the first-level cache.

[0138] For example, when the utilization rate of the first-level cache is low (for example, less than 30%), the target business data and target compressed data can be temporarily stored in the first-level cache at the same time, and the target compressed data can be written to the disk step by step through the first-level cache, second-level cache and storage pool; this method can fully utilize the storage space of the first-level cache to meet different types of user data requests.

[0139] For example, if the utilization rate of the first-level cache is between the maximum and minimum thresholds (greater than 30% and less than 70%), only the original target business data can be retained in the first-level cache, and the compressed target business data can be stored in the storage pool in a step-by-step manner from the second-level cache. This method allows for temporary storage of uncompressed data in the first-level cache, with the most recent target business data being treated as hot data. When a read request for the target business data arrives, the target business data in the first-level cache can be directly returned, resulting in a faster response and no decompression required, further improving response performance.

[0140] Based on the system architecture of the data writing method of the above embodiment, a second aspect of the present invention provides a data reading method.

[0141] Figure 6 A flow chart of a data reading method according to an embodiment of the present invention is shown.

[0142] like Figure 6 As shown, the data reading method of this embodiment includes operations S601 to S603.

[0143] Operation S601, in response to receiving a read request for target business data issued by the application layer, based on the storage address information related to the target business data included in the read request, read the target compressed data obtained by compressing the target business data from the target storage area among the multiple storage areas included in the storage medium.

[0144] Operation S602, based on predefined metadata, determine the target compression method corresponding to the target storage area, wherein the metadata represents the mapping relationship between multiple storage areas included in the storage medium and multiple compression methods, and the compressed data in the same storage area is obtained based on the same compression method.

[0145] Operation S603: calling a decompression engine to decompress the target compressed data using a target decompression method corresponding to the target compression method to obtain target service data.

[0146] According to an embodiment of the present invention, the system architecture to which the above data writing and reading method can be applied can refer to Figure 1 The system shown includes an application layer (ie, business system 101), a processor, a decompression engine, and a storage medium. The above data reading method can be executed by the processor.

[0147] After the application layer initiates a read request for the target business data, the processor can read the target compressed data obtained by compressing the target business data from the storage medium, call the decompression engine to decompress the target business data, and return the decompressed target business data to the application layer.

[0148] The above method determines the target decompression method for the target compressed data directly based on predefined metadata. During data storage, data generated using different compression methods are stored in separate storage areas. The metadata records the compression method associated with each storage area. When reading data, all data from a specific storage area is bound to the same compression method, eliminating the need to determine the compression method for each data item. This eliminates the need to determine the compression algorithm for each data request, reducing system overhead.

[0149] According to an embodiment of the present invention, the storage medium used in the data writing method of the above embodiment can be implemented as any type of storage medium, and can be implemented as a single storage medium, such as a cache or a hard disk; or it can be implemented as a hybrid storage medium, for example, including a cache and a hard disk.

[0150] The hybrid storage medium includes a predetermined cache and a storage pool, and a plurality of storage areas are located in the storage pool.

[0151] The storage address information includes: a first address of the target compressed data in a predetermined cache, and a second address of the target compressed data in a storage pool.

[0152] Based on the storage address information related to the target business data included in the read request, reading the target compressed data obtained by compressing the target business data from the target storage area among the multiple storage areas included in the storage medium includes: when it is determined based on the first address that the target compressed data exists in the predetermined cache, reading the target compressed data from the predetermined cache; when it is determined based on the first address that the target compressed data does not exist in the predetermined cache, reading the target compressed data from the storage area based on the second address.

[0153] According to an embodiment of the present invention, during the data writing phase, compressed data is written to disk in a predetermined cache and storage pool in a sequential manner, i.e., first written to the predetermined cache and then transferred to the storage pool. The specific method for performing the writing operation can be found in the description of the aforementioned embodiment and will not be repeated here.

[0154] According to an embodiment of the present invention, the data first stored in the predetermined cache is business data generated in the most recent time period and is most frequently accessed by users. Therefore, the data in the predetermined cache is considered hot data. In contrast, the data in the storage pool is business data generated earlier and is least frequently accessed, thus being considered cold data.

[0155] During the writing process, the target compressed data is temporarily stored in a predetermined cache for a period of time. Therefore, when reading data, the data can be read from the cache first. If the target compressed data does not exist in the predetermined cache, the target compressed data is read from the storage area. The data in the cache is faster, which can improve the request response speed.

[0156] According to an embodiment of the present invention, further, when the storage medium is implemented as a hybrid storage medium, it can be further implemented as a three-level hybrid storage. For example, it can be implemented as a level 1 cache, a level 2 cache, and a storage pool. The implementation methods of the level 1 cache, the level 2 cache, and the storage pool can be referred to the description of the previous embodiment and will not be repeated here.

[0157] Furthermore, the data read and write speed supported by the first-level cache is greater than that of the second-level cache, which in turn supports a greater data read and write speed than the storage pool. The capacity of the first-level cache is smaller than that of the second-level cache, which in turn is smaller than that of the storage pool. For example, the first-level cache can be implemented as volatile memory, the second-level cache can be implemented as non-volatile memory, and the storage pool can be implemented as a non-volatile hard disk.

[0158] In the process of reading compressed data from the storage medium, when the storage medium is implemented as a hybrid storage medium, such as the three-level storage mentioned above, it can be judged level by level, and priority is given to reading from the first-level cache. If there is no compressed data in the first-level cache, the compressed data is read from the second-level cache. If there is no compressed data in the second-level cache, the compressed data is read from the storage pool.

[0159] Specifically, the predetermined cache includes a first-level cache and a second-level cache, and the first address includes a first-level cache address of the target compressed data in the first-level cache and a second-level cache address of the target compressed data in the second-level cache.

[0160] Reading target compressed data obtained by compressing target business data from a target storage area among multiple storage areas included in a storage medium includes: reading the target compressed data from the first-level cache when it is determined based on the first-level cache address that the target compressed data exists in the first-level cache; reading the target compressed data from the second-level cache based on the second-level cache address when it is determined based on the first-level cache address that the target compressed data does not exist in the first-level cache; and reading the target compressed data from the storage area based on the second address when it is determined based on the first address that the target compressed data does not exist in the predetermined cache.

[0161] According to an embodiment of the present invention, the data first stored in the first-level cache is business data generated in the most recent time period and is read most frequently by users. Therefore, the data in the first-level cache is considered hot data. The data in the second-level cache is next in line, belonging to business data generated in a relatively recent time period. The frequency of user reads is also relatively high, but less than that of the first-level cache. Therefore, the data in the second-level cache is also considered hot data. In comparison, the data in the storage pool is read least frequently and is considered cold data.

[0162] According to an embodiment of the present invention, since the data reading and writing speed of the first-level cache - volatile memory is the fastest, data is read from the first-level cache first, and the data response performance is accelerated. However, since the storage capacity of volatile memory is generally small, it may be impossible for all hot data to be temporarily stored in the first-level cache. A non-volatile memory is added between the memory and the hard disk as a second-level cache, and the hot data is first transferred to the second-level cache. The data reading and writing speed supported by the second-level cache is greater than the storage pool, but the capacity is greater than the first-level cache. In this way, if the requested hot data does not exist in the first-level cache, it can still be read from the second-level cache, and only cold data is read from the storage pool, which speeds up the reading and writing performance of hot data. This three-level access mechanism improves the utilization rate and reading speed of the cache space as a whole.

[0163] According to an embodiment of the present invention, during the data writing process, one implementation method may be to retain only the original target business data in the first-level cache, and to implement the step-by-step write of the target compressed data from the second-level cache to the storage pool. Another implementation method may be to implement the step-by-step write of the target compressed data from the first-level cache, the second-level cache, and the storage pool, and to retain the original target business data in the first-level cache for a predetermined period of time.

[0164] The original target business data is retained in the first-level cache. In this way, the latest target business data is used as hot data. When a read request for the target business data comes, the target business data in the first-level cache can be directly returned.

[0165] Based on this, the storage address information also includes: the temporary storage address of the target business data in the first-level cache; the above-mentioned data reading method includes: before reading the target compressed data from the storage pool, when it is determined that the target business data exists in the first-level cache based on the temporary storage address, the target business data is read from the first-level cache and returned to the application layer.

[0166] According to an embodiment of the present invention, when a read request for target business data comes, the target business data in the first-level cache can be directly returned, which not only has a fast response speed but also does not require decompression, further improving the response performance.

[0167] Based on the above data writing method, a third aspect of the present invention provides a data writing system. Figure 7A FIG. 4 shows a structural diagram of a data writing system according to an embodiment of the present invention.

[0168] like Figure 7A As shown, the data writing system includes an application layer 701 , a first processor 702 - 1 , a compression engine 703 - 1 , and a storage medium 704 .

[0169] The application layer 701 is used to initiate a write request for target business data. The compression engine 703-1 is used to compress the target business data to obtain target compressed data.

[0170] Storage medium 704 includes multiple storage areas associated with multiple compression methods. First processor 702-1 is configured to, based on a write request, invoke a compression engine to perform data compression and, based on predefined metadata and the compression method used by the compression engine to compress the target business data, write target compressed data to a target storage area in the storage medium. The metadata indicates a mapping between the multiple storage areas included in the storage medium and the multiple compression methods.

[0171] The data interaction process based on this data writing system is as follows: Application layer 701 initiates a write request for business data. First processor 702-1, based on the business data included in the request, issues a compression instruction to compression engine 703-1 and sends the business data. Compression engine 703-1 compresses the business data based on the compression instruction to obtain compressed data. First processor 702-1 writes the compressed data to storage medium 704. The specific implementation method for first processor 702-1 performing data writing can be found in the description of operations S201 and S202 in the previous embodiment and will not be repeated here.

[0172] Furthermore, the storage medium 704 is implemented as a hybrid storage medium, for example, including three levels of storage: a primary cache, a secondary cache, and a storage pool. The compressed data can be written to the primary cache, the secondary cache, and the storage pool in a sequential manner, that is, first written to the primary cache, then to the secondary cache, and finally to the storage pool.

[0173] Based on the above data reading method, the fourth aspect of the present invention provides a data reading system, Figure 7B FIG. 4 shows a structural diagram of a data reading system according to an embodiment of the present invention.

[0174] like Figure 7B As shown, the data reading system includes an application layer 701 , a second processor 702 - 2 , a decompression engine 703 - 2 , and a storage medium 704 .

[0175] The application layer 701 is used to initiate a read request for target business data. The storage medium 704 includes multiple storage areas associated with multiple compression methods.

[0176] The second processor 702-2 is used to read the target compressed data obtained by compressing the target business data from the target storage area among the multiple storage areas based on the storage address information related to the target business data included in the read request; and determine the target compression method corresponding to the target storage area based on predefined metadata, wherein the metadata represents the mapping relationship between the multiple storage areas included in the storage medium and the multiple compression methods, and the compressed data in the same storage area is obtained based on the same compression method.

[0177] The decompression engine 703 - 2 is configured to decompress the target compressed data to obtain target business data using a target decompression method corresponding to the target compression method based on the decompression instruction issued by the second processor.

[0178] The data exchange process based on this data reading system is as follows: Application layer 701 initiates a read request for service data. Second processor 702-2, based on the storage address information included in the request, reads compressed data obtained by compressing the service data from storage medium 704. Second processor 702-2 issues a decompression instruction to decompression engine 703-2 and sends the compressed data. Decompression engine 703-2 decompresses the compressed data based on the compression instruction to obtain the service data. Second processor 702-2 then returns the service data to application layer 701. The specific implementation method for second processor 702-2 reading compressed data from storage medium 704 can be found in the description of operations S601 through S603 in the previous embodiment and will not be repeated here.

[0179] Furthermore, the storage medium 704 is implemented as a hybrid storage medium, for example, including three levels of storage: a first-level cache, a second-level cache, and a storage pool. When reading compressed data from the storage medium 704, a level-by-level determination can be made, with priority given to reading from the first-level cache. If the compressed data does not exist in the first-level cache, the compressed data is read from the second-level cache. If the compressed data does not exist in the second-level cache, the compressed data is read from the storage pool.

[0180] Based on the above data writing method, the present invention also provides a data writing device. Figure 8 The device is described in detail.

[0181] Figure 8 A structural block diagram of a data writing device according to an embodiment of the present invention is shown.

[0182] like Figure 8 As shown, the data writing device 800 of this embodiment includes a compression module 801 and a writing module 802 .

[0183] The compression module 801 is configured to, in response to receiving a write request for target business data issued by the application layer, call a compression engine to compress the target business data to obtain target compressed data;

[0184] The writing module 802 is used to write the target compressed data into the target storage area in the storage medium based on the predefined metadata and the compression method used to compress the target business data, wherein the metadata represents the mapping relationship between multiple storage areas included in the storage medium and multiple compression methods.

[0185] Based on the above data reading method, the present invention also provides a data reading device. Figure 9 The device is described in detail.

[0186] Figure 9 A structural block diagram of a data reading device according to an embodiment of the present invention is shown.

[0187] like Figure 9 As shown, the data reading device 900 of this embodiment includes a reading module 901 , a determination module 902 , and a decompression module 903 .

[0188] The reading module 901 is used to respond to a read request for target business data issued by the application layer, and based on the storage address information related to the target business data included in the read request, read the target compressed data obtained by compressing the target business data from the target storage area among the multiple storage areas included in the storage medium.

[0189] Determination module 902 is used to determine the target compression method corresponding to the target storage area based on predefined metadata, wherein the metadata represents the mapping relationship between multiple storage areas included in the storage medium and multiple compression methods, and the compressed data in the same storage area is obtained based on the same compression method.

[0190] The decompression module 903 is used to call the decompression engine to decompress the target compressed data using the target decompression method corresponding to the target compression method to obtain the target business data.

[0191] According to an embodiment of the present invention, any multiple modules among the compression module 801, the write module 802, the read module 901, the determination module 902, and the decompression module 903 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present invention, at least one of the compression module 801, the write module 802, the read module 901, the determination module 902, and the decompression module 903 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the compression module 801, the writing module 802, the reading module 901, the determination module 902, and the decompression module 903 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0192] Figure 10 A block diagram of an electronic device suitable for implementing a data writing method and a data reading method according to an embodiment of the present invention is shown.

[0193] like Figure 10As shown, an electronic device 1000 according to an embodiment of the present invention includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 1001 may also include onboard memory for caching purposes. Processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0194] Various programs and data required for the operation of electronic device 1000 are stored in RAM 1003. Processor 1001, ROM 1002, and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations according to the method flow of an embodiment of the present invention by executing the programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also perform various operations according to the method flow of an embodiment of the present invention by executing the programs stored in the one or more memories.

[0195] According to an embodiment of the present invention, electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to bus 1004. Electronic device 1000 may also include one or more of the following components connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or modem. Communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1010 as needed, so that computer programs read from the removable media can be installed into storage section 1008 as needed.

[0196] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0197] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include ROM 1002 and / or RAM 1003 described above, and / or one or more memories other than ROM 1002 and RAM 1003.

[0198] The embodiments of the present invention further include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to cause the computer system to implement the method provided by the embodiments of the present invention.

[0199] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when executed by the processor 1001. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0200] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0201] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009 and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0202] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0203] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0204] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0205] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data writing method, characterized in that the method comprises: In response to receiving a write request for target business data issued by the application layer, calling a compression engine to compress the target business data to obtain target compressed data; Based on predefined metadata, the target compressed data is written into the target storage area in the storage medium according to the compression method used to compress the target business data, wherein the metadata represents the mapping relationship between multiple storage areas included in the storage medium and multiple compression methods.

2. The method according to claim 1, characterized in that The storage medium includes a first-level cache, a second-level cache, and a storage pool, and the multiple storage areas are located in the storage pool; Writing the target compressed data into a target storage area in a storage medium includes: Writing the target compressed data into the first-level cache; When a first transfer condition is met, transferring the target compressed data from the first-level cache to the second-level cache; When a second transfer condition is met, the target compressed data is transferred from the secondary cache to the target storage area in the storage pool.

3. The method according to claim 2, wherein: The data read and write speed supported by the first-level cache is greater than that of the second-level cache, and the data read and write speed supported by the second-level cache is greater than that of the storage pool; The capacity of the first-level cache is smaller than that of the second-level cache, and the capacity of the second-level cache is smaller than that of the storage pool.

4. The method according to claim 3, wherein: The first-level cache includes volatile memory; The secondary cache includes a non-volatile memory; The storage pool includes a non-volatile hard disk.

5. The method according to claim 2, wherein: When a first transfer condition is met, transferring the target compressed data from the first-level cache to the second-level cache includes: transferring the target compressed data from the first-level cache to the second-level cache when a data amount of the first-level cache is full and / or a storage time of the target compressed data in the first-level cache is greater than a first time threshold; When the second transfer condition is met, the target compressed data is transferred from the secondary cache to the target storage area in the storage pool, including: when the data amount of the secondary cache is full and / or the storage time of the target compressed data in the secondary cache is greater than a second time threshold, the target compressed data is transferred from the secondary cache to the target storage area in the storage pool.

6. The method according to claim 2, characterized in that The method further comprises: In response to receiving the write request, writing the target service data into the first-level cache; When the target compressed data is transferred from the first-level cache to the second-level cache, or when the storage time of the target business data in the first-level cache is greater than a third time threshold, the target business data in the first-level cache is deleted.

7. The method according to claim 1, wherein calling a compression engine to compress the target business data to obtain target compressed data comprises: Determining the data type of the target business data; The compression engine is called to perform data compression on the target business data based on the data type of the target business data to obtain target compressed data.

8. The method according to claim 7, wherein determining the data type of the target service data comprises: Acquiring attributes of the target service data, wherein the attributes include at least one of the following: continuity of write addresses, data volume, and compression rate; The data type of the target business data is determined according to the attributes of the target business data.

9. The method according to claim 8, wherein determining the data type of the target business data according to the attributes of the target business data comprises: If the continuity of the write address of the target business data meets a predetermined condition, the data volume is greater than a predetermined data volume threshold, and the compression rate is less than a predetermined compression rate threshold, determining the target business data as first type data; When the continuity of the write address of the target business data does not meet a predetermined condition and the data volume is less than or equal to a predetermined data volume threshold, the target business data is determined to be second type data.

10. The method according to claim 9, wherein calling the compression engine to compress the target service data based on the data type of the target service data comprises: In a case where the data type of the target service data is the first type of data, calling the compression engine to compress the target service data using a first compression method; In a case where the data type of the target service data is the second type of data, calling the compression engine to compress the target service data using the second compression method; The compression speed of the first compression method is greater than that of the second compression method, and the compression ratio of the first compression method is less than that of the second compression method.

11. The method according to claim 7, wherein calling the compression engine to compress the target service data based on the data type of the target service data comprises: Obtaining the service priority and processor utilization corresponding to the target service data; In a case where the service priority corresponding to the target service data falls within a predetermined priority range and the utilization rate of the processor is greater than a predetermined utilization rate threshold, calling the compression engine to compress the target service data using a first compression method; In a case where the service priority corresponding to the target service data does not fall within a predetermined priority range, or the utilization rate of the processor is less than or equal to the predetermined utilization rate threshold, calling the compression engine to perform data compression on the target service data using a second compression method; The compression speed of the first compression method is greater than that of the second compression method, and the compression ratio of the first compression method is less than that of the second compression method.

12. The method according to claim 7, wherein calling the compression engine to compress the target service data based on the data type of the target service data comprises: adding the target business data to a target task queue among a plurality of task queues based on a data type of the target business data, wherein the plurality of task queues are used to store business data of a plurality of predetermined data types and are associated with a plurality of compression methods; The compression engine is called to read the target service data from the target task queue, and the target service data is compressed using a target compression method associated with the target task queue.

13. The method according to claim 7, wherein the data type of the target business data is a target type among multiple predetermined data types; the compression engine comprises multiple compression processing units, and the multiple compression processing units are configured to perform multiple compression methods corresponding to the business data of the multiple predetermined data types; Calling the compression engine to compress the target service data based on the data type of the target service data includes: determining a target compression processing unit corresponding to the target type from the plurality of compression processing units; The target compression processing unit is called to perform data compression on the target service data using a target compression method corresponding to the target type.

14. A data reading method, characterized in that: The method comprises: In response to receiving a read request for target service data issued by the application layer, based on storage address information related to the target service data included in the read request, read target compressed data obtained by compressing the target service data from a target storage area among a plurality of storage areas included in a storage medium; Determining a target compression method corresponding to the target storage area based on predefined metadata, wherein the metadata represents a mapping relationship between multiple storage areas included in the storage medium and multiple compression methods, and compressed data in the same storage area is obtained based on the same compression method; The decompression engine is called to decompress the target compressed data using a target decompression method corresponding to the target compression method to obtain the target business data.

15. The method according to claim 14, characterized in that The storage medium includes a predetermined cache and a storage pool, and the plurality of storage areas are located in the storage pool; The storage address information includes: a first address of the target compressed data in a predetermined cache, and a second address of the target compressed data in the storage pool; Reading target compressed data obtained by compressing the target business data from a target storage area among a plurality of storage areas included in a storage medium based on storage address information related to the target business data included in the read request includes: In a case where it is determined based on the first address that the target compressed data exists in the predetermined cache, reading the target compressed data from the predetermined cache; If it is determined based on the first address that the target compressed data does not exist in the predetermined cache, the target compressed data is read from the storage area based on the second address.

16. The method according to claim 15, characterized in that The predetermined cache includes a first-level cache and a second-level cache, and the first address includes a first-level cache address of the target compressed data in the first-level cache and a second-level cache address of the target compressed data in the second-level cache; Reading the target compressed data from the predetermined cache includes: In a case where it is determined based on the first-level cache address that the target compressed data exists in the first-level cache, reading the target compressed data from the first-level cache; If it is determined based on the L1 cache address that the target compressed data does not exist in the L1 cache, the target compressed data is read from the L2 cache based on the L2 cache address.

17. The method according to claim 16, wherein: The data read and write speed supported by the first-level cache is greater than that of the second-level cache, and the data read and write speed supported by the second-level cache is greater than that of the storage pool; The capacity of the first-level cache is smaller than that of the second-level cache, and the capacity of the second-level cache is smaller than that of the storage pool.

18. The method according to claim 16, characterized in that The storage address information also includes: a temporary storage address of the target service data in the first-level cache; The method further comprises: Before reading the target compressed data from the storage pool, if it is determined based on the temporary storage address that the target service data exists in the first-level cache, the target service data is read from the first-level cache and then returned to the application layer.

19. A data writing system, characterized in that: The system comprises: The application layer is used to initiate write requests for target business data; A compression engine, configured to compress the target service data to obtain target compressed data; a storage medium comprising a plurality of storage areas associated with a plurality of compression methods; The first processor is used to call the compression engine to perform the data compression based on the write request, and based on predefined metadata, write the target compressed data into the target storage area in the storage medium according to the compression method used by the compression engine to compress the target business data, wherein the metadata represents the mapping relationship between multiple storage areas included in the storage medium and multiple compression methods.

20. A data reading system, characterized in that: The system comprises: The application layer is used to initiate read requests for target business data; a storage medium comprising a plurality of storage areas associated with a plurality of compression methods; a second processor configured to read target compressed data obtained by compressing the target business data from a target storage area among the multiple storage areas based on storage address information related to the target business data included in the read request; and determine a target compression method corresponding to the target storage area based on predefined metadata, wherein the metadata indicates a mapping relationship between the multiple storage areas included in the storage medium and the multiple compression methods, and the compressed data in the same storage area is obtained based on the same compression method; A decompression engine is used to decompress the target compressed data to obtain the target business data using a target decompression method corresponding to the target compression method based on the decompression instruction issued by the second processor.

Images