Data processing method and system

Abstract

The embodiment of the present specification provides a data processing method and system, wherein the data processing method comprises: receiving a data storage request, wherein the data storage request carries initial to-be-stored data; determining a data type of the initial to-be-stored data, and caching the initial to-be-stored data in a case where it is determined that the data type meets a preset caching condition; in a case where it is determined that the current to-be-stored data meets a preset storage condition, converting the current to-be-stored data into target to-be-stored data, and storing the target to-be-stored data into a first type data storage module, wherein the current to-be-stored data comprises the initial to-be-stored data, thereby achieving the purpose of reducing the service life attenuation speed of the first type data storage module while ensuring the normal operation of the first type data storage module.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing method. Background Technology

[0002] With the continuous development of computer technology, different enterprises have different requirements for storage media when storing data. Some enterprises choose lower-cost storage media with relatively short lifespans in order to save storage costs. However, storage media with short lifespans generally have relatively few read / write cycles, and their lifespan will decrease as the number of read / write cycles increases. Therefore, how to provide a method that can ensure the normal operation of storage media and reduce the rate of lifespan degradation has become an urgent problem to be solved. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0005] Receive a data storage request, the data storage request carrying initial data to be stored;

[0006] Determine the data type of the initial data to be stored, and cache the initial data to be stored if the data type meets the preset caching conditions;

[0007] If it is determined that the cached current data to be stored meets the preset storage conditions, the current data to be stored is converted into target data to be stored, and the target data to be stored is stored in the first type of data storage module, wherein the current data to be stored includes the initial data to be stored.

[0008] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0009] The receiving module is configured to receive a data storage request, the data storage request carrying initial data to be stored.

[0010] The caching module is configured to determine the data type of the initial data to be stored, and cache the initial data to be stored if the data type meets a preset caching condition.

[0011] The storage module is configured to, when it is determined that the cached current data to be stored meets the preset storage conditions, convert the current data to be stored into target data to be stored, and store the target data to be stored in the first type of data storage module, wherein the current data to be stored includes the initial data to be stored.

[0012] According to a third aspect of the embodiments of this specification, a data processing method is provided, applied to a first type of data storage module, comprising:

[0013] Receive the target data to be stored, wherein the target data to be stored is the target data to be stored in the above data processing method, and the first type of data storage module is the first type of data storage module in the above data processing method;

[0014] Determine the corresponding initial metadata for the target data to be stored, and determine data storage units for the target data to be stored and the corresponding metadata based on the data format of the target data to be stored;

[0015] Based on the space size of the data storage unit, the initial metadata is converted into target metadata;

[0016] The target data to be stored and the corresponding target metadata are stored in the data storage unit.

[0017] According to a fourth aspect of the embodiments of this specification, a data processing apparatus is provided, applied to a first type of data storage module, comprising:

[0018] The receiving module is configured to receive the target data to be stored, wherein the target data to be stored is the target data to be stored in the above data processing method, and the first type of data storage module is the first type of data storage module in the above data processing method.

[0019] The determination module is configured to determine the corresponding initial metadata for the target data to be stored, and to determine the data storage unit for the target data to be stored and the corresponding metadata based on the data format of the target data to be stored;

[0020] The conversion module is configured to convert the initial metadata into target metadata based on the space size of the data storage unit;

[0021] The storage module is configured to store the target data to be stored and the corresponding target metadata into the data storage unit.

[0022] According to a fifth aspect of the embodiments of this specification, a data processing system is provided, comprising a data processing module and a first type of data storage module, wherein...

[0023] The data processing module is configured to receive a data storage request, the data storage request carrying initial data to be stored; determine the data type of the initial data to be stored, and cache the initial data to be stored if the data type meets a preset caching condition; and convert the cached current data to be stored into target data to be stored if the cached current data to be stored meets a preset storage condition, and send the target data to the first type of data storage module, wherein the current data to be stored includes the initial data to be stored.

[0024] The first type of data storage module is configured to receive the target data to be stored sent by the data processing module and store the target data to be stored.

[0025] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0026] Memory and processor;

[0027] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0028] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0029] According to an eighth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0030] The data processing method provided in this specification includes: receiving a data storage request, wherein the data storage request carries initial data to be stored; determining the data type of the initial data to be stored, and caching the initial data to be stored if the data type satisfies a preset caching condition; and converting the cached current data to be stored into target data to be stored if the cached current data to be stored satisfies a preset storage condition, and storing the target data to be stored in a first type of data storage module, wherein the current data to be stored includes the initial data to be stored.

[0031] Specifically, upon receiving initial data to be stored, this method, if the data type of the initial data to be stored meets preset caching conditions, does not immediately write the initial data to be stored into the first type of data storage module. Instead, it caches the initial data to be stored, thus avoiding the problem of a short lifespan caused by frequent data writes to the first type of data storage module, thereby improving its lifespan. Furthermore, if the current data to be stored meets the preset storage conditions, the cached current data to be stored is written to the first type of data storage module, ensuring that the first type of data storage module performs its data storage function. This achieves the goal of ensuring the normal operation of the first type of data storage module while reducing the rate of decay in its lifespan. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification;

[0033] Figure 2 This is a flowchart illustrating the processing procedure of a data processing system provided in one embodiment of this specification;

[0034] Figure 3 This is a schematic diagram of a TLC data storage format provided in one embodiment of this specification;

[0035] Figure 4 This is a schematic diagram of a QLC data storage format in a data processing system provided in one embodiment of this specification;

[0036] Figure 5 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0037] Figure 6 This is a flowchart of another data processing method provided in one embodiment of this specification;

[0038] Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0039] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0040] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0042] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0043] Chunk: refers to a unit of file slicing on a single engine of a distributed storage system.

[0044] IU: refers to the Indirection Unit data structure of the SSD FTL mapping table.

[0045] OP: refers to the additional Nand Flash space that each SSD (Solid State Drive) needs to reserve for internal GC data migration.

[0046] WA: refers to the write amplification factor of an SSD. In a completely random write mode, there is a fixed curve relationship between WA and OP. The WA for strictly sequential writes is always 1. However, in the real world, there is no strictly sequential write. Therefore, more often, we need to focus on the WA under random write IO patterns.

[0047] DWPD (Diskful Writes Per Day) refers to the number of times the entire capacity of an SSD can be written per day within its expected lifespan.

[0048] FTL: The Flash Translation Layer (FTL) is one of the most important aspects of a flash memory controller. By translating logical addresses from the host to physical addresses on the flash memory, SSDs can perform wear leveling.

[0049] Indirection Unit: refers to the smallest management unit used by the FTL layer to manage the mapping relationship between data on the physical medium and logical address data.

[0050] Nand Flash: Nand-Flash memory is a type of flash memory that uses a non-linear macrocell pattern internally, providing a cost-effective solution for realizing large-capacity solid-state memory.

[0051] GC: It is an abbreviation for Garbage Collection, a fundamental technology of solid-state drives (SSDs) that has a direct impact on the performance and lifespan of SSDs.

[0052] IO pattern: refers to the pattern characteristics of IO workload, which is generally described by information such as IO size, access location, access timing, and the ratio of read to write.

[0053] QLC SSD (Quad Level Cell SSD): A type of solid-state drive (SSD) where each cell can store 4 bits of data.

[0054] TLC SSD (Triple Level Cell SSD): A type of solid-state drive (SSD) where each cell can store 3 bits of data.

[0055] With the continuous development of computer technology, different enterprises have different requirements for storage media. Some enterprises, in order to save storage costs, will choose storage media with lower costs and relatively shorter lifespans. Storage media with shorter lifespans generally have relatively fewer read / write cycles, and their lifespan will decrease as the number of read / write cycles increases. This storage media can be understood as solid-state drives (SSDs) used for storing data; for example, this storage media could be a QLC SSD.

[0056] In cloud storage scenarios, some enterprises use TLC SSDs for data storage. However, TLC SSDs are relatively expensive. Therefore, some enterprises are starting to consider using QLC SSDs in cloud storage scenarios. QLC SSDs have a certain cost advantage over TLC SSDs. However, although QLC SSDs are more cost-effective, their hardware design results in a shorter lifespan compared to TLC SSDs. Without specific usage modifications, unsuitable use cases or inappropriate usage methods will waste the cost advantage of QLC SSDs.

[0057] Based on this, this specification provides a solution using a QLC SSD, which is a dual-in-one SSD that combines 3D Xpoint (a storage technology, and a class of products using 3D XPoint storage technology are collectively referred to as Optane) memory and QLC NAND flash memory (Nand Flash using QLC media) on a single SSD.

[0058] In this solution, the solid-state drive uses both Optane and QLC media. Optane media provides high-speed, small-block random write performance, while QLC media provides larger capacity to reduce costs.

[0059] However, the drawback of this solid-state drive solution is that Optane and QLC media are integrated on a single drive, and users cannot flexibly configure the space capacity ratio of Optane and QLC media according to actual needs; at the same time, users cannot use either media alone, resulting in strong coupling between Optane and QLC media.

[0060] Based on this, two data processing methods are provided in this specification. This specification also relates to a data processing system, a data processing device, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0061] Figure 1 This diagram illustrates a data processing system according to an embodiment of the present specification. The system includes a data processing module 104 and a first type data storage module 106, wherein...

[0062] The data processing module 104 is configured to receive a data storage request, the data storage request carrying initial data to be stored; determine the data type of the initial data to be stored, and cache the initial data to be stored if the data type meets a preset caching condition; and convert the cached current data to be stored into target data to be stored if the cached current data to be stored meets a preset storage condition, and send the target data to the first type data storage module 106, wherein the current data to be stored includes the initial data to be stored.

[0063] The first type of data storage module 106 is configured to receive the target data to be stored sent by the data processing module 104 and store the target data to be stored.

[0064] The data processing module 104 can be understood as a module in a data processing system that processes data. In practical applications, the data processing module 104 can be a CPU, a program, code running on the CPU, compiled code, etc., and this specification does not impose specific limitations on it. For example, the data processing module 104 can be a block storage service (BlockServer).

[0065] The first type of data storage module 106 can be understood as a module in a data processing system used to store target data to be stored. It should be noted that the first type of data storage module 106 has the characteristics of low cost, but relatively short service life and relatively low performance. For example, the first type of data storage module 106 can be a QLC SSD.

[0066] This data storage request can be understood as a request sent by the data storage requester 102 to instruct the data processing module 104 to process the data. For example, this data storage request can be an I / O request.

[0067] The data storage requester 102 can be understood as a module that needs to store data. In practical applications, the data storage requester 102 includes, but is not limited to, terminals, clients, virtual machines, secure containers, servers, databases, etc.

[0068] The initial data to be stored can be understood as the data received by the data processing module 104 that needs to be processed. It should be noted that this initial data to be stored can be of any type, and this specification does not specifically limit it. For example, the initial data to be stored can be multimedia data, files, code data, etc. Correspondingly, the target data to be stored can be understood as the data obtained after processing the initial data to be stored. In practical applications, to avoid frequent data writing to the first type of data storage module 106, which would reduce its lifespan, the data processing system caches and accumulates the received data through the data processing module 104. When the size of the cached data exceeds a preset size threshold, the data processing module 104 processes the cached data (this data processing includes, but is limited to, data splitting, data EC calculation, data compression, etc.) to obtain data that can be stored in the first type of data storage module 106.

[0069] In practical applications, this data storage system can be used in different scenarios, and the data processing module 104 and the first type of data storage module 106 will differ depending on the scenario in which the data storage system is applied. Specifically, when the data storage system is applied to a distributed storage scenario, the data storage system can be understood as a distributed storage system. Correspondingly, the data processing module 104 can be understood as a server that processes data in the distributed storage system, a server running block storage services in the distributed storage system, etc.; the first type of data storage module 106 can be understood as a storage resource pool in the distributed storage system used to store the target data to be stored, wherein the storage resource pool is composed of multiple storage media; the storage media can be the media used to store data, and the specific configuration can be set according to the actual application scenario. For example, the storage media can be QLC SSD. When the storage media is QLC SSD, the storage resource pool can be a QLC resource pool composed of multiple QLC SSDs. It should be noted that the first type of data storage module 106 can be deployed on a machine independent of the data processing module 104, that is, the QLC resource pool can be deployed independently on a machine.

[0070] In another scenario, the data storage system can be a system within a server used for data storage. Correspondingly, the data processing module 104 can be understood as a module within the server used for data processing, such as the CPU on the server, programs running on the server, or block storage services within the server; this specification does not impose specific limitations on these. Correspondingly, the first type of data storage module 106 can be understood as a solid-state drive deployed on the server for storing data; for example, the first type of data storage module 106 can be understood as a QLC SSD.

[0071] Specifically, the data processing module 104 in the data processing system provided in this specification is capable of receiving data storage requests, wherein the data storage request can be sent by the data storage requester 102, and the data storage request carries initial data to be stored.

[0072] After receiving a data storage request, the data processing module 104 can determine the data type of the initial data to be stored. This data type can be understood as information characterizing the type of the initial data to be stored. For example, the data type can be the data source of the initial data to be stored; initial data from different data sources are different types of data. For instance, initial data to be stored sent by a client can be of the client type; initial data to be stored sent by a server can be of the server type, etc. Alternatively, the data type can also be a multimedia data type, code type, log type, file type, etc.

[0073] After determining the data type of the initial data to be stored, the data processing module 104 can determine whether the data type meets the preset caching conditions. If so (i.e., if the data type meets the preset caching conditions), the data processing module 104 caches the initial data to be stored. Determining that the data type meets the preset caching conditions can be understood as the data processing module 104 determining that the data type meets the preset caching conditions if the data source is consistent with a preset data source. The preset data source can be understood as a pre-defined data storage requester 102 that needs to cache the data it sends. For example, the data processing module 104 can be pre-set to cache the initial data to be stored sent by the server and write the cached data to the first type of data storage module 106. Furthermore, the data processing module 104 can also be pre-set not to cache the initial data to be stored sent by the client.

[0074] In addition, the data processing module 104 caches the initial data to be stored, which can be understood as the data processing module 104 storing the initial data to be stored in memory; for example, the block storage service client caches and accumulates the data in memory.

[0075] After caching the initial data to be stored, the data processing module 104 will convert the current data to be stored into the target data to be stored if it determines that the cached current data to be stored meets the preset storage conditions.

[0076] The determination that the current data to be stored in the cache meets the preset storage conditions can be understood as the data processing module 104 determining the size of the current data to be stored in the cache, and determining that the current data to be stored in the cache meets the preset storage conditions if the size of the data is greater than the preset cache threshold. The preset cache threshold can be set according to the actual application scenario. For example, the preset cache threshold can be set to 1GB, 2GB, etc.

[0077] Alternatively, determining that the cached current data to be stored meets the preset storage conditions can also be understood as the data processing module 104 determining the cache time of the cached current data to be stored, and if the cache time is greater than the preset time threshold, then determining that the cached current data to be stored meets the preset storage conditions.

[0078] Furthermore, when the data processing module 104 stores the initial data to be stored in memory, the determination that the current data to be stored in the cache meets the preset storage conditions can also be understood as the data processing module 104 determining the memory occupancy rate of the current data to be stored in memory, and determining that the current data to be stored in memory meets the preset storage conditions when the memory occupancy rate is greater than the preset ratio threshold.

[0079] Specifically, the data processing module 104 converts the current data to be stored into target data to be stored. This can be understood as the data processing module 104 determining the data format corresponding to the target data to be stored; and based on this data format, dividing the current data to be stored into at least two target data sets. The data format can be set according to the actual application scenario; for example, the data format can be a 64KB data block. In other words, the data processing module 104 divides the cached current data to be stored into multiple 64KB data blocks.

[0080] Meanwhile, in the process of dividing the current data to be stored into at least two target data to be stored based on the data format, the data processing module 104 can also first divide the current data to be stored based on the data format to obtain at least two candidate data to be stored; then calculate the corresponding data security code for each candidate data to be stored; and then form at least two target data to be stored based on the at least two candidate data to be stored and the corresponding data security code.

[0081] After the data processing module 104 obtains the target data to be stored, it can send the target data storage module to the first type of data storage module 106.

[0082] After receiving the target data to be stored sent by the data processing module 104, the first type of data storage module 106 can calculate the corresponding initial metadata for the target data to be stored, and allocate corresponding data storage units for the target data to be stored and the corresponding metadata based on the data format of the target data to be stored.

[0083] Subsequently, the first type of data storage module 106 can convert the initial metadata into target metadata based on the space size of the data storage unit. This process involves the first type of data storage module 106 determining the metadata storage unit corresponding to the initial metadata from the data storage units; then determining the remaining storage space from the metadata storage units based on the data size of the initial metadata; then determining the corresponding fill data for the initial metadata based on the size of the remaining storage space; and finally constructing the target metadata based on the initial metadata and the corresponding fill data.

[0084] After determining the target metadata, the first type of data storage module 106 stores the target data to be stored and the corresponding target metadata into the data storage unit.

[0085] In one embodiment provided in this specification, the data processing system further includes a second type of data storage module. This second type of data storage module can be understood as a module in the data processing system used to store the initial data to be stored. It should be noted that the cost of this second type of data storage module is relatively higher than that of the first type of data storage module 106, but its lifespan is longer and its performance is relatively higher. For example, the second type of data storage module can be an Optane SSD. In practical applications, when the data storage system is used in a distributed storage scenario, the second type of data storage module can be understood as a storage resource pool used to store the initial data to be stored, and the storage medium constituting this storage resource pool can be an Optane SSD. When the storage medium is an Optane SSD, the storage resource pool can be an Optane resource pool composed of multiple Optane SSDs. It should be noted that the second type of data storage module can be deployed on a machine independent of the data processing module 104 and / or the first type of data storage module 106.

[0086] Meanwhile, when the data storage system is a system used for data storage in a server, the second type of data storage module can be a solid-state drive deployed on the server for storing data; for example, OptaneSSD.

[0087] Based on this, when the data processing module 104 determines that the data type does not meet the preset caching conditions, the data processing module 104 will send the initial data to be stored to the second type of data storage module.

[0088] The second type of data storage module is capable of receiving initial data to be stored sent by the data processing module 104 and storing the initial data to be stored.

[0089] In another embodiment, the data processing module 104 sends the initial data to be stored to the second type of data storage module. This can be understood as the data processing module 104 converting the initial data to be stored into log data and storing the log data in the log file of the second type of data storage module.

[0090] The second type of data storage module is able to obtain the log data newly written by the data processing module 104 from the log file and store the log data locally.

[0091] The following is in conjunction with the appendix Figure 2 Taking the application of the data processing system provided in this specification in a data storage scenario as an example, the data processing system will be further described. Among other things, Figure 2This specification illustrates a flowchart of the processing procedure of a data processing system according to an embodiment, which specifically includes the following steps.

[0092] It should be noted that the data processing system provided in this specification can be a distributed file storage system. By establishing independent Optane resource pools and QLC resource pools within the distributed file storage system, small-block writes in the foreground and large-block writes in the background are written to Optane and QLC media respectively. The small-block write data stream in the foreground can be understood as random write requests sent by the virtual machine, each carrying small blocks of data (i.e., relatively small data chunks), which are mostly small, unaligned, random write data. The large-block write data stream in the background can be understood as large-block write requests sent by the server, each carrying large blocks of data.

[0093] Meanwhile, this QLC SSD is positioned as a low-cost, high-density SSD for storing relatively cold data. To reduce costs, the FTL mapping table IU is increased from 4KB to 64KB, thus reducing the DRAM memory capacity requirement by 16 times.

[0094] Based on this, considering that if less than 64KB of data is written to a QLC SSD, or if it cannot be aligned to 64KB, many unaligned overlap writes will be generated, which will cause its lifespan to fall short of the officially specified 0.41 DWPD.

[0095] Therefore, to better utilize QLC SSDs, this solution pre-designs an IU data format (such as a 64KB IU data format) that is more conducive to improving QLC SSD performance. The QLC SSD stores data using this IU data format, thereby solving the problems of poor write performance of QLC media that is not aligned to 64KB and the short lifespan of QLC hardware itself. After using this IU data format, in cloud storage scenarios, QLC media (a type of solid-state drive) with better cost performance can be used on cloud disks to replace TLC media, thereby reducing the cost of cloud disks while meeting performance requirements.

[0096] It should be noted again that the Optane and QLC media in this specification are not integrated onto a single disk (i.e., hard drive), but rather used as two independent media resource pools within a distributed storage system. These two resource pools are the Optane resource pool and the QLC resource pool. Based on this, the advantage of flexibly configuring the space ratio of the two media according to actual needs, and also flexibly specifying which medium to write data to, is achieved.

[0097] The specific execution steps of this data processing system are as follows.

[0098] Step 202: The virtual machine or server sends an IO request (i.e., IO REQ) to the block storage service client;

[0099] Step 204: The block storage service determines the source of the IO request. If the IO request is sent by the virtual machine, proceed to step 206. If the IO request is sent by the server, proceed to step 210.

[0100] Specifically, the data carried in the IO requests issued by the virtual machine are mostly small blocks of unaligned random write data. Therefore, it is more appropriate to use Optane media, which has better support for small blocks of random writes.

[0101] The data carried in the IO requests issued by the server are mostly large blocks of data and do not require a fast response. Based on this, the data is stored using QLC media.

[0102] Step 206: The block storage service side writes the data sent by the virtual machine to the Optane media resource pool in the form of WAL.

[0103] Specifically, the block storage service client obtains small blocks of unaligned random write data carried in the IO request sent by the virtual machine, and sends this data as log data to the log file in the Optane resource pool.

[0104] Step 208: The Optane resource pool retrieves the newly added log data from the log file and stores the log data locally, thereby enabling the storage of small, unaligned random write data.

[0105] Step 210: The block storage service caches the large blocks of write data sent by the server in memory, and sends the accumulated large blocks of write data to the QLC resource pool when the size of the accumulated large blocks of write data exceeds a preset threshold.

[0106] Specifically, the block storage service client obtains large blocks of write data (also known as seg) from the IO requests sent by the server and caches and accumulates these large blocks of write data in memory (i.e., Merge in Memory). When the accumulated data reaches a sufficient size, such as the size of the accumulated large block of write data (i.e., 2GB), exceeding a preset threshold (1GB), the block storage service client will perform data processing on the accumulated data according to the needs of the actual scenario. This data processing operation includes, but is not limited to, data splitting, data EC calculation, and data compression. This results in multiple 64KB blocks of data, each containing the data to be stored and the corresponding erasure coding, where EC stands for Erasure Coding.

[0107] Then, the block storage service client dumps (sends) the block data to the QLC resource pool in 64KB alignment.

[0108] Step 212: QLC resource pool, stores multiple block data according to the preset IU data structure, stores block data in each IU data structure, and determines the corresponding metadata.

[0109] It should be noted that the size of the IU data structure in QLC SSD is 64KB. When the QLC resource pool stores block data by stripes (chunks) composed of multiple IU data structures, the QLC resource pool will use one IU data structure as the header area to store metadata (chunk meta). The size of the metadata is generally 4K, which cannot completely fill the header area. Therefore, in order to ensure that the data is stored in a 64KB aligned write manner, the area in the header area that cannot be filled by the metadata will be filled with the value "0", thereby ensuring that the data can be stored in a 64KB aligned write manner.

[0110] Specifically, because the IU Size (IU data structure size) in a QLC SSD is 64KB, in actual performance tests, if non-64K aligned writes are used on QLC, the write bandwidth it can provide will be significantly reduced. The FTL mapping unit in this QLC SSD is 64KB, which, although it can support 4K access by users, leads to a large number of RMW (Read-Modify-Write) operations, resulting in a sharp deterioration in write performance and write amplification.

[0111] Furthermore, while using Optane media and QLC media as separate resource pools allows for flexible use of QLC media based on specific scenarios, it also makes QLC more reliant on 64KB aligned writes. In some cases, the file system's own metadata updates do not meet 64KB alignment, meaning the chunk metadata cannot completely fill the header area. This results in some valid data being written to the header area, leading to unaligned writes. This unaligned data causes performance and lifespan degradation issues for QLC.

[0112] In the data processing system provided in this specification, a different data storage format than that previously stored on TLC is used to reduce the proportion of non-64KB writes. See [link / reference] Figure 3 , Figure 3This is a schematic diagram of a TLC data storage format provided in one embodiment of this specification. In previous TLC media, the Chunk data format had a special 4KB header area for each data chunk. This header area contained metadata related to the data chunk. Following the 4KB Chunk Meta was the user's data portion, which resided in physical sectors. Each physical sector was considered a data storage unit, consisting of two parts: a 4048B space for storing the payload data and a 48B space as the footer, used to store the erasure code corresponding to the payload data.

[0113] On QLC media, the Chunk Meta portion corresponding to a Chunk has increased to 64KB. See also Figure 4 , Figure 4 This is a schematic diagram of the data storage format of QLC in a data processing system provided in one embodiment of this specification. This is done to reduce the proportion of non-64KB aligned writes caused by metadata updates. The metadata of each chunk needs to be aligned to 64KB, and the chunk metadata and chunk data can be aligned by padding with special all-zero data. The advantage of this is that when the chunk metadata needs to be updated, it will not cause unnecessary RMW operations. In QLC, the metadata of the data chunk is placed in the header of the chunk itself (i.e., the header area). Chunk data is located after the chunk metadata, and the portion less than 64KB between the chunk metadata and chunk data needs to be padded with all-zero padding data. Each 64KB data portion in the chunk data consists of 16 data format units of 4KB each. Each unit contains 4048B of user-valid data and 48B of data footer. The data verification granularity is maintained at 4KB, which is more beneficial for small block reads and avoids unnecessary amplification caused by small-granularity reads.

[0114] Based on this, the data processing system provided in this manual combines Optane resource pool separation for small-block writes with a data format more suitable for large-block writes designed for 64KB IU QLC SSDs. This allows WAL data to be written to Optane media first, and then, when the data in memory accumulates to a certain scale, the merged data in memory is further processed (such as data splitting, EC calculation, and data compression) before being rewritten to QLC media. This significantly reduces the proportion of non-64KB aligned writes, thereby extending the lifespan of QLC media and achieving higher and more stable read / write throughput bandwidth. WAL (Write-Ahead Logging) is a write-ahead logging system, a highly efficient logging algorithm in databases.

[0115] Furthermore, by establishing independent Optane resource pools and independent QLC resource pools within the distributed system, the Optane resource pools handle small blocks of random write data, while the QLC resource pools handle large blocks of 64KB aligned write data. This approach provides the cluster with higher and more stable bandwidth throughput, while also reducing write amplification traffic within the QLC and extending the lifespan of the QLC media.

[0116] By using the Chunk data format specifically designed for QLC media, the proportion of unaligned 64KB writes caused by Chunk metadata updates can be eliminated, further improving the steady-state write performance achievable in actual QLC use and extending the actual lifespan of the media.

[0117] Furthermore, regarding the aforementioned solution of integrating Optane and QLC onto a single drive, since WAL (Write-Ahead Log) data does not need to be written to the QLC media, it can be merged in memory for a period of time before being directly dumped from memory to the QLC media. If Optane media is integrated into the QLC SSD, and WAL data is also written to the QLC media, even with optimizations for in-disk Optane, it will still cause unnecessary write amplification, leading to performance degradation and lifespan reduction of the QLC media.

[0118] The data processing system provided in this manual, by utilizing a scheme of establishing two independent resource pools in a distributed storage system, allows for the separate writing of WAL data to Optane, while the merged data in memory is then written to QLC media. This solution also features a redesigned data storage format more suitable for QLC media. Testing has shown that in typical cloud disk usage scenarios, a single QLC disk can achieve the bandwidth throughput required for practical applications, making it suitable as an alternative to TLC SSD media in specific cloud disk scenarios.

[0119] Figure 5A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0120] Step 502: Receive a data storage request, which carries initial data to be stored.

[0121] The data processing method provided in this specification can be applied to the data processing module of the above-mentioned data processing system. Based on this, the explanation of the step of receiving data storage request can be found in the corresponding or relevant content of the above-mentioned data processing system, and will not be elaborated further here.

[0122] Step 504: Determine the data type of the initial data to be stored, and cache the initial data to be stored if the data type meets the preset caching conditions.

[0123] Specifically, the data type includes the data source;

[0124] Accordingly, determining that the data type satisfies the preset caching conditions includes:

[0125] If the data source is consistent with the preset data source, the data type is determined to meet the preset caching conditions, wherein the initial data to be stored from the preset data source needs to be cached.

[0126] Specifically, the data processing module can determine the data source of the initial data to be stored and match the data source with the preset data source. If the matching result is consistent, it determines that the data type meets the preset storage conditions, thereby ensuring that the data sent by the preset data source is stored in the first type of data storage module, thus avoiding frequent data write operations on the first type of data storage module.

[0127] It should be noted that the data processing module will cache the initial data to be stored sent by the preset data source. The preset data source can be set according to the actual application scenario. For example, the preset data source can be set to the client source, virtual machine source, etc. This manual does not make specific restrictions on this.

[0128] In one embodiment provided in this specification, caching the initial data to be stored includes:

[0129] The initial data to be stored is stored in memory.

[0130] Specifically, the data processing module caches the initial data to be stored that meets the preset caching conditions in memory, and then writes the cached data to the first type of data storage module, thereby avoiding frequent data write operations to the first type of data storage module.

[0131] Step 506: If it is determined that the cached current data to be stored meets the preset storage conditions, the current data to be stored is converted into target data to be stored, and the target data to be stored is stored in the first type of data storage module, wherein the current data to be stored includes the initial data to be stored.

[0132] In one embodiment provided in this specification, determining that the currently cached data to be stored meets preset storage conditions includes:

[0133] Determine the size of the currently cached data to be stored, and if the data size is greater than a preset cache threshold, determine that the currently cached data to be stored meets a preset storage condition; or

[0134] The cache time of the currently cached data to be stored is determined, and if the cache time is greater than a preset time threshold, the cached data to be stored is determined to meet the preset storage conditions.

[0135] Here, the data size can be understood as the size of the data to be stored currently cached by the data processing module; for example, the data size could be 2GB. The preset cache threshold can be found in the description of the data processing system mentioned above, and will not be elaborated upon here.

[0136] Specifically, the data processing module can determine the size of the cached data to be stored, and if the size of the data is greater than a preset cache threshold, it determines that the cached data to be stored meets the preset storage conditions. For example, the block storage service can determine the size of the cached data to be stored, such as 2GB. Based on this, the block storage service compares the size of the data with the preset cache threshold (1GB). Since the size of the data is greater than the preset cache threshold, the block storage service determines that the cached data to be stored can be stored in the QLC resource pool.

[0137] Alternatively, the data processing module can determine the cache time of the currently cached data to be stored, and if the cache time is greater than a preset time threshold, determine that the currently cached data to be stored meets the preset storage conditions.

[0138] The cache time can be understood as the time it takes for the data processing module to receive the currently stored data. If the currently stored data is data obtained by caching and accumulating multiple initial data to be stored received by the data processing module, then the cache time can be understood as the reception time of the earliest received initial data to be stored among the multiple received initial data to be stored. This preset time threshold can be set according to the time-sensitive application scenario; for example, the preset time threshold can be 1 minute.

[0139] Using the previous example, the storage service client can determine the caching time of the currently accumulated data to be stored, such as 2 minutes. Based on this, in order to avoid data loss due to excessive caching time, the storage service client will determine in real time whether the data time is greater than the preset time threshold (1 minute). If so, the storage service client determines that the currently accumulated data to be stored can be stored in the QLC resource pool.

[0140] In one embodiment provided in this specification, when the data processing module stores the initial data to be stored in memory, the step of determining that the cached current data to be stored meets the preset storage conditions includes:

[0141] Determine the memory occupancy rate of the current data to be stored in the memory, and if the memory occupancy rate is greater than a preset ratio threshold, determine that the current data to be stored in the memory meets the preset storage conditions.

[0142] The memory utilization rate can be understood as the ratio of the storage space occupied by the currently stored data in memory. This preset ratio threshold can also be set according to the actual application scenario; this manual does not impose specific limitations on it. For example, the preset ratio threshold could be 50%.

[0143] Specifically, the data processing module can determine the memory occupancy rate of the current data to be stored in memory, and if the memory occupancy rate is greater than a preset ratio threshold, it determines that the current data to be stored in memory meets the preset storage conditions. For example, the block storage service client can determine the memory occupancy rate of the currently cached data to be stored, such as 70%. Based on this, the block storage service client compares the memory occupancy rate with the preset ratio threshold (50%), determines that the memory occupancy rate is greater than the preset ratio threshold, and therefore determines that the currently cached data to be stored can be stored in the QLC resource pool.

[0144] In one embodiment provided in this specification, converting the current data to be stored into target data to be stored includes:

[0145] Determine the data format corresponding to the target data to be stored;

[0146] Based on the data format, the current data to be stored is divided into at least two target data to be stored.

[0147] This data format can be understood as the size of the target data to be stored, for example, 64KB.

[0148] Specifically, the data processing module can determine the data format corresponding to the pre-set target data to be stored, and split the current data to be stored based on the data format to obtain at least two target data to be stored, so that the first type of data storage module can store the target data to be stored in an aligned write manner.

[0149] Furthermore, the step of dividing the current data to be stored into at least two target data to be stored based on the data format includes:

[0150] Based on the data format, the current data to be stored is segmented to obtain at least two candidate data to be stored.

[0151] Calculate the corresponding data security code for each candidate data to be stored;

[0152] Based on the at least two candidate data to be stored and the corresponding data security codes, at least two target data to be stored are constituted.

[0153] The data security encoding can be understood as an encoding that ensures data security and prevents data errors or loss during data transmission. Examples include erasure codes and parity check codes. Correspondingly, when the target data to be stored includes both the data itself and the data security encoding, the size of the candidate data to be stored plus the size of the data security encoding equals the size of the target data to be stored.

[0154] Specifically, the data processing module can segment the current data to be stored based on the data format to obtain at least two candidate data to be stored; and calculate the corresponding data security code for each candidate data to be stored. The data security code can be calculated using any erasure coding method, and this specification does not specify a particular method for calculating the data security code for the candidate data to be stored.

[0155] Subsequently, the data processing module, based on the at least two candidate data to be stored and their corresponding data security codes, constructs at least two target data to be stored. This facilitates the subsequent storage of the target data to be stored by the first type of data storage module in an aligned write manner.

[0156] In one embodiment provided in this specification, to avoid frequent data write operations to the QLC SSD, the block storage service will write the cached data to the QLC SSD when there is enough data accumulated in the current cache. However, if the block storage service determines that the data accumulated in the current cache is insufficient, it will continue to cache data from a preset data source until there is enough data accumulated in the current cache. Specifically, after caching the initial data to be stored, the process further includes:

[0157] If it is determined that the current data to be stored in the cache does not meet the preset storage conditions, the step of receiving data storage requests continues until it is determined that the current data to be stored in the cache meets the preset storage conditions.

[0158] The current data to be stored is converted into target data to be stored, and the target data to be stored is stored in the first type of data storage module.

[0159] Specifically, if the data processing module determines that the current data to be stored in the cache does not meet the preset storage conditions, it will continue to execute the step of receiving data storage requests until it determines that the current data to be stored in the cache meets the preset storage conditions.

[0160] Then, the current data to be stored is converted into the target data to be stored, and the target data to be stored is stored in the first type of data storage module.

[0161] For example, the block storage service client can determine the memory usage rate of the currently cached data to be stored, such as 30%. Based on this, the block storage service client compares the memory usage rate with a preset ratio threshold (50%). If it determines that the memory usage rate is less than the preset ratio threshold, the block storage service client will continue to cache and accumulate the data sent by the server in memory until it determines that the memory usage rate of the currently cached data to be stored is greater than the preset ratio threshold. Then, it will process the currently cached data to be stored and store the processed data to be stored in the QLC resource pool.

[0162] Furthermore, in the embodiments provided in this specification, after determining the data type of the initial data to be stored, the method further includes:

[0163] If it is determined that the data type does not meet the preset caching conditions, the initial data to be stored is stored in the second type of data storage module.

[0164] In practical applications, since the data carried in the IO requests issued by the virtual machine is mostly small, unaligned random write data, it is more appropriate to use Optane media, which has better support for small random writes.

[0165] Based on this, after receiving the data, the block storage service client will determine the source of the IO request. If the IO request is sent by the virtual machine, the block storage service client will write the data sent by the virtual machine to the Optane media resource pool.

[0166] Furthermore, storing the initial data to be stored in the second type of data storage module includes:

[0167] The initial data to be stored is converted into log data, and the log data is stored in the log file of the second type of data storage module, so that the second type of data storage module stores the initial data to be stored based on the log data in the log file.

[0168] Continuing with the previous example, the block storage service client retrieves small blocks of unaligned random write data carried in the IO requests sent by the virtual machine and sends this data as log data to the log file in the Optane resource pool. The Optane resource pool then retrieves the newly added log data from the log file and stores this log data locally, thereby achieving the storage of small blocks of unaligned random write data.

[0169] This specification provides a data processing method that, upon receiving initial data to be stored, does not immediately write the initial data to the first type of data storage module if the data type of the initial data meets preset caching conditions. Instead, it caches the initial data, thus avoiding the problem of a short lifespan for the first type of data storage module caused by frequent data writes, thereby extending its lifespan. Furthermore, if the current data to be stored meets the preset storage conditions, the cached data is written to the first type of data storage module, ensuring that the first type of data storage module performs its data storage function. This achieves the goal of ensuring the normal operation of the first type of data storage module while reducing the rate of degradation of its lifespan.

[0170] It should also be noted that the data processing method in this manual provides a way to use QLC SSDs that combines Optane media resource pools and a 64KB aligned data format. By combining Optane resource pools to separate small block writes and a more suitable large block write data format designed for 64KB IU QLC SSDs, the proportion of non-64KB aligned writes can be greatly reduced, thereby improving the lifespan of QLC media and achieving higher and more stable read and write throughput bandwidth.

[0171] Figure 6A flowchart of another data processing method according to an embodiment of this specification is shown. This method is applied to a first type of data storage module. Since the size of the IU data structure in the QLC SSD is 64KB, when the QLC resource pool stores metadata (Chunk meta) as the header area of ​​an IU data structure, the size of the metadata is generally 4K, which cannot completely fill the header area, resulting in non-aligned writes. These non-aligned data will cause performance and lifespan degradation problems for the use of QLC. Based on this, the other data processing method provided in this specification can fill the area in the header area where the metadata cannot be filled with the value "0" through the QLC SSD, thereby ensuring that the data can be stored in a 64KB aligned write manner. The specific steps include the following steps.

[0172] Step 602: Receive the target data to be stored, wherein the target data to be stored is the target data to be stored in the above-described data processing method, and the first type of data storage module is the first type of data storage module in the above-described data processing method.

[0173] Step 604: Determine the corresponding initial metadata for the target data to be stored, and determine the data storage unit for the target data to be stored and the corresponding metadata based on the data format of the target data to be stored.

[0174] Step 606: Based on the space size of the data storage unit, convert the initial metadata into target metadata.

[0175] Step 608: Store the target data to be stored and the corresponding target metadata into the data storage unit.

[0176] The initial metadata refers to the metadata corresponding to the target data to be stored, used for indexing the target data. A data storage unit can be understood as a unit in the first type of data storage module used for storing data, for example... Figure 4 The data storage unit is a storage unit within the system. It should be noted that the size of this data storage unit can be fixed; therefore, the target data to be stored and its corresponding initial metadata can be allocated to one or more data storage units. The space size of a data storage unit can be understood as its storage space, which can be set according to the actual application scenario. For example, 4KB, 64KB, 125KB, etc.

[0177] For a detailed explanation of this alternative data processing method, please refer to the explanations in the above-mentioned data processing and the above-mentioned data processing method; further details will not be provided here.

[0178] In one embodiment provided in this specification, converting the initial metadata into target metadata based on the space size of the data storage unit includes:

[0179] Determine the metadata storage unit corresponding to the initial metadata from the data storage unit;

[0180] Based on the data size of the initial metadata, the remaining storage space is determined from the metadata storage unit;

[0181] Based on the remaining storage space size, determine the corresponding fill data for the initial metadata;

[0182] The target metadata is constructed based on the initial metadata and the corresponding fill data.

[0183] In this context, the remaining storage space in the metadata storage unit can be understood as the space within that unit where the initial metadata cannot be filled or written. The filling data can be set according to the actual application scenario; this specification does not impose specific limitations on it. For example, the filling data can be 0. A metadata storage unit can be understood as a data storage unit that stores the metadata. For example, Figure 4 The head region of the middle.

[0184] Specifically, after allocating a corresponding data storage unit for the target data to be stored, the first type of data storage module can determine a metadata storage unit for storing the initial metadata from the data storage unit, and based on the space size of the metadata storage unit, determine the remaining storage space that the initial metadata cannot fill from the metadata storage unit, and based on the space size of the remaining storage space, determine the filling data corresponding to the space size, and construct the target metadata based on the filling data and the initial metadata.

[0185] In another data processing method provided in this specification, when the first type of data storage module receives the target data to be stored, it determines the corresponding initial metadata for the target data to be stored, and converts the initial metadata into target metadata based on the space size of the data storage unit; thus, based on the target data to be stored and the corresponding target metadata, it can be completely stored in the data storage unit, avoiding the problem of misaligned writing of the target data to be stored due to excessively small initial metadata, and avoiding performance and lifespan degradation problems caused by misaligned data in the first type of data storage module.

[0186] Corresponding to the above method embodiments, this specification also provides a data processing apparatus embodiment, the apparatus comprising:

[0187] The receiving module is configured to receive a data storage request, the data storage request carrying initial data to be stored.

[0188] The caching module is configured to determine the data type of the initial data to be stored, and cache the initial data to be stored if the data type meets a preset caching condition.

[0189] The storage module is configured to, when it is determined that the cached current data to be stored meets the preset storage conditions, convert the current data to be stored into target data to be stored, and store the target data to be stored in the first type of data storage module, wherein the current data to be stored includes the initial data to be stored.

[0190] Optionally, the data processing apparatus further includes a duplicate caching module, configured to:

[0191] If it is determined that the current data to be stored in the cache does not meet the preset storage conditions, the step of receiving data storage requests continues until it is determined that the current data to be stored in the cache meets the preset storage conditions.

[0192] The current data to be stored is converted into target data to be stored, and the target data to be stored is stored in the first type of data storage module.

[0193] Optionally, the data processing device further includes an initial data storage module, configured to:

[0194] If it is determined that the data type does not meet the preset caching conditions, the initial data to be stored is stored in the second type of data storage module.

[0195] Optionally, the initial data storage module is further configured to:

[0196] The initial data to be stored is converted into log data, and the log data is stored in the log file of the second type of data storage module, so that the second type of data storage module stores the initial data to be stored based on the log data in the log file.

[0197] Optionally, the data type includes the data source;

[0198] Accordingly, the cache module is also configured as follows:

[0199] If the data source is consistent with the preset data source, the data type is determined to meet the preset caching conditions, wherein the initial data to be stored from the preset data source needs to be cached.

[0200] Optionally, the storage module is further configured to:

[0201] The determination that the current data to be stored in the cache meets the preset storage conditions includes:

[0202] Determine the size of the currently cached data to be stored, and if the data size is greater than a preset cache threshold, determine that the currently cached data to be stored meets a preset storage condition; or

[0203] The cache time of the currently cached data to be stored is determined, and if the cache time is greater than a preset time threshold, the cached data to be stored is determined to meet the preset storage conditions.

[0204] Optionally, the cache module is further configured to:

[0205] The initial data to be stored is stored in memory.

[0206] Accordingly, determining that the currently cached data to be stored meets the preset storage conditions includes:

[0207] Determine the memory occupancy rate of the current data to be stored in the memory, and if the memory occupancy rate is greater than a preset ratio threshold, determine that the current data to be stored in the memory meets the preset storage conditions.

[0208] Optionally, the storage module is further configured to:

[0209] Determine the data format corresponding to the target data to be stored;

[0210] Based on the data format, the current data to be stored is divided into at least two target data to be stored.

[0211] Optionally, the storage module is further configured to:

[0212] Based on the data format, the current data to be stored is segmented to obtain at least two candidate data to be stored.

[0213] Calculate the corresponding data security code for each candidate data to be stored;

[0214] Based on the at least two candidate data to be stored and the corresponding data security codes, at least two target data to be stored are constituted.

[0215] This specification provides a data processing device that, upon receiving initial data to be stored, does not immediately write the initial data to the first type of data storage module if the data type of the initial data to be stored meets preset caching conditions. Instead, it caches the initial data to be stored, thus avoiding the problem of a short lifespan caused by frequent data writing to the first type of data storage module, thereby improving its lifespan. Furthermore, if the current data to be stored meets the preset storage conditions, the cached current data to be stored is written to the first type of data storage module, ensuring that the first type of data storage module performs its data storage function. This achieves the goal of ensuring the normal operation of the first type of data storage module while reducing the rate of degradation of its lifespan.

[0216] The above is an illustrative scheme of a data processing device according to this embodiment. It should be noted that the technical solution of this data processing device belongs to the same concept as the technical solutions of the two data processing methods described above. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solutions of the two data processing methods described above.

[0217] Corresponding to the above method embodiments, this specification also provides another data processing apparatus embodiment, applied to a first type of data storage module, including:

[0218] The receiving module is configured to receive the target data to be stored, wherein the target data to be stored is the target data to be stored in the data processing device described above, and the first type of data storage module is the first type of data storage module in the data processing device described above.

[0219] The determination module is configured to determine the corresponding initial metadata for the target data to be stored, and to determine the data storage unit for the target data to be stored and the corresponding metadata based on the data format of the target data to be stored;

[0220] The conversion module is configured to convert the initial metadata into target metadata based on the space size of the data storage unit;

[0221] The storage module is configured to store the target data to be stored and the corresponding target metadata into the data storage unit.

[0222] Optionally, the conversion module is further configured to:

[0223] Determine the metadata storage unit corresponding to the initial metadata from the data storage unit;

[0224] Based on the data size of the initial metadata, the remaining storage space is determined from the metadata storage unit;

[0225] Based on the remaining storage space size, determine the corresponding fill data for the initial metadata;

[0226] The target metadata is constructed based on the initial metadata and the corresponding fill data.

[0227] Another data processing device is provided in this specification.

[0228] Receive the target data to be stored, wherein the target data to be stored is the target data to be stored in the above data processing method, and the first type of data storage module is the first type of data storage module in the above data processing method;

[0229] Determine the corresponding initial metadata for the target data to be stored, and determine data storage units for the target data to be stored and the corresponding metadata based on the data format of the target data to be stored;

[0230] Based on the space size of the data storage unit, the initial metadata is converted into target metadata;

[0231] The target data to be stored and the corresponding target metadata are stored in the data storage unit.

[0232] The other data processing device provided in this specification, the first type of data storage module, upon receiving target data to be stored, determines the corresponding initial metadata for the target data to be stored, and converts the initial metadata into target metadata based on the space size of the data storage unit; thereby, based on the target data to be stored and the corresponding target metadata, it can be completely stored in the data storage unit, avoiding the problem of misaligned writing of the target data to be stored due to excessively small initial metadata, and avoiding performance and lifespan degradation problems caused by misaligned data in the first type of data storage module.

[0233] The above is an illustrative scheme of another data processing device in this embodiment. It should be noted that the technical solution of this other data processing device belongs to the same concept as the technical solutions of the two data processing methods described above. For details not described in detail in the technical solution of the other data processing device, please refer to the description of the technical solutions of the two data processing methods described above.

[0234] Figure 7A structural block diagram of a computing device 700 according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0235] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0236] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0237] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 700 can also be a mobile or stationary server.

[0238] The processor 720 is used to execute the following computer-executable instructions, which, when executed by the processor 720, implement the steps of the above-described data processing method.

[0239] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0240] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0241] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0242] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0243] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.

[0244] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0245] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0246] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0247] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0248] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Receive a data storage request, the data storage request carrying initial data to be stored; Determine the data type of the initial data to be stored, and cache the initial data to be stored if the data type meets the preset caching conditions; If the cached current data to be stored meets the preset storage conditions, the current data to be stored is converted into target data to be stored, and the target data to be stored is stored in the first type of data storage module. The current data to be stored includes the initial data to be stored. Converting the current data to be stored into target data to be stored includes: determining the data format corresponding to the target data to be stored; segmenting the current data to be stored based on the data format to obtain at least two candidate data to be stored; calculating the corresponding data security code for each candidate data to be stored; and constructing at least two target data to be stored based on the at least two candidate data to be stored and the corresponding data security code. The data format refers to the size of the target data to be stored.

2. The data processing method according to claim 1, after caching the initial data to be stored, further includes: If it is determined that the current data to be stored in the cache does not meet the preset storage conditions, the step of receiving data storage requests continues until it is determined that the current data to be stored in the cache meets the preset storage conditions. The current data to be stored is converted into target data to be stored, and the target data to be stored is stored in the first type of data storage module.

3. The data processing method according to claim 1, further comprising, after determining the data type of the initial data to be stored: If it is determined that the data type does not meet the preset caching conditions, the initial data to be stored is stored in the second type of data storage module.

4. The data processing method according to claim 3, wherein storing the initial data to be stored in the second type of data storage module comprises: The initial data to be stored is converted into log data, and the log data is stored in the log file of the second type of data storage module, so that the second type of data storage module stores the initial data to be stored based on the log data in the log file.

5. The data processing method according to claim 1, wherein the data type includes data source; Accordingly, determining that the data type satisfies the preset caching conditions includes: If the data source is consistent with the preset data source, the data type is determined to meet the preset caching conditions, wherein the initial data to be stored from the preset data source needs to be cached.

6. The data processing method according to claim 1, wherein determining that the currently cached data to be stored meets preset storage conditions includes: Determine the size of the current data to be stored in the cache, and if the size of the data is greater than a preset cache threshold, determine that the current data to be stored in the cache meets the preset storage conditions; or The cache time of the currently cached data to be stored is determined, and if the cache time is greater than a preset time threshold, the cached data to be stored is determined to meet the preset storage conditions.

7. The data processing method according to claim 1, wherein caching the initial data to be stored comprises: The initial data to be stored is stored in memory; Accordingly, determining that the currently cached data to be stored meets the preset storage conditions includes: Determine the memory occupancy rate of the current data to be stored in the memory, and if the memory occupancy rate is greater than a preset ratio threshold, determine that the current data to be stored in the memory meets the preset storage conditions.

8. A data processing method, applied to a first type of data storage module, comprising: Receive the target data to be stored, wherein the target data to be stored is the target data to be stored in claim 1, and the first type of data storage module is the first type of data storage module in claim 1; Determine the corresponding initial metadata for the target data to be stored, and determine data storage units for the target data to be stored and the corresponding metadata based on the data format of the target data to be stored; Based on the space size of the data storage unit, the initial metadata is converted into target metadata; The target data to be stored and the corresponding target metadata are stored in the data storage unit.

9. The data processing method according to claim 8, wherein converting the initial metadata into target metadata based on the space size of the data storage unit includes: Determine the metadata storage unit corresponding to the initial metadata from the data storage unit; Based on the data size of the initial metadata, the remaining storage space is determined from the metadata storage unit; Based on the remaining storage space size, determine the corresponding fill data for the initial metadata; The target metadata is constructed based on the initial metadata and the corresponding fill data.

10. A data processing system, comprising a data processing module and a first type of data storage module, wherein, The data processing module is configured to receive a data storage request, which carries initial data to be stored. The data type of the initial data to be stored is determined, and if the data type satisfies a preset caching condition, the initial data to be stored is cached; if the cached current data to be stored satisfies a preset storage condition, the current data to be stored is converted into target data to be stored, and the target data to be stored is sent to a first type of data storage module. The current data to be stored includes the initial data to be stored. Converting the current data to be stored into target data to be stored includes: determining the data format corresponding to the target data to be stored; segmenting the current data to be stored based on the data format to obtain at least two candidate data to be stored; calculating a corresponding data security code for each candidate data to be stored; and constructing at least two target data to be stored based on the at least two candidate data to be stored and the corresponding data security code. The data format refers to the size of the target data to be stored. The first type of data storage module is configured to receive the target data to be stored sent by the data processing module and store the target data to be stored.

11. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the data processing method according to any one of claims 1 to 7, or the data processing method according to any one of claims 8 to 9.

12. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method of any one of claims 1 to 7, or the data processing method of any one of claims 8 to 9.

Images