Method, device, storage medium and system for writing data

Abstract

A method, device, storage medium and system for writing data. The method comprises: obtaining target data to be written; determining a data type to which the target data belongs from three preset data types according to a size of the target data, wherein the three data types correspond to three ranges of data thresholds respectively, and the ranges of the data thresholds in the three data types have a relationship that a first data type is smaller than a second data type, and the second data type is smaller than a third data type; and writing the target data into a storage unit using a set writing mode according to the data type to which the target data belongs. The method can perform fine-grained shunt processing on input data and execute different data writing schemes, so as to reduce the writing flow of the input data and improve the data writing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, and in particular to a data writing method, device, storage medium and system. BACKGROUND

[0002] With the development of storage technology, storage systems begin to show more and more characteristics of scale and diversity, and the task scenarios they face are also more and more diversified.

[0003] Due to the unpredictability of task behavior, a single point of storage is prone to become a bottleneck problem. Therefore, based on the requirements of performance and distributed transaction atomicity, when writing data in a storage system, a log recording method is used, for example, a commonly used data writing method in the industry is as follows: a write copy mechanism is used in the foreground to write input data into a log and a buffer, and when the accumulated data in the buffer reaches a certain threshold, an erasure coding (EC) method is used to write the input data into a data file in the background.

[0004] The disadvantage of this storage method is that it causes additional amplification of data write traffic. Specifically, since the input data in the foreground has a higher requirement for latency and its size is not controllable, and the erasure coding has a stripe alignment requirement during encoding and decoding, the input data is difficult to adapt, so the write copy mechanism is used in the foreground, that is, the input data is written into the log through multiple copies. Since the redundancy effect of the write copy mechanism is poor, this leads to huge traffic consumption of the data of the write request, thereby increasing the traffic consumption of the data writing. SUMMARY

[0005] The present application provides a data writing method, device, storage medium and system to solve the problem of excessive data writing traffic consumption in a storage system.

[0006] In a first aspect, a data writing method is provided, comprising: obtaining target data to be written; determining a data type to which the target data belongs from three preset data types according to a size of the target data, wherein the three data types correspond to three ranges of data thresholds respectively, and the ranges of data thresholds in the three data types have the following relationship: the first data type is smaller than the second data type, and the second data type is smaller than the third data type; and writing the target data into a storage unit using a set writing method according to the data type to which the target data belongs.

[0007] In one possible implementation, the step of writing the target data into the storage unit using a set writing method according to the data type of the target data includes: if the target data belongs to the first data type, writing the target data into a first log in the storage unit and writing it into a buffer using a write copy mechanism; if the target data belongs to the second data type, writing the target data into a second log in the storage unit and writing it into the buffer using erasure coding; and if the target data belongs to the third data type, writing the target data into a data file in the storage unit using erasure coding.

[0008] In one possible implementation, if the target data belongs to the third data type, the target data is written to a data file in the storage unit using erasure coding, including: merging the target data and the data in the buffer to obtain merged data; and writing the merged data to the data file using erasure coding.

[0009] In one possible implementation, writing the merged data into the data file using erasure coding further includes: writing data in the merged data that meets the stripe alignment principle into the data file according to the stripe alignment principle of erasure coding; determining the remaining data in the merged data that does not meet the stripe alignment principle; and writing the remaining data into a log and the buffer, wherein the log includes the first log or the second log.

[0010] In one possible implementation, writing the remaining data into the log and the buffer, wherein the log includes the first log or the second log, comprises: determining, based on the size of the remaining data, whether the remaining data belongs to the first data type or the second data type; if the remaining data belongs to the first data type, writing the remaining data into the first log and the buffer; and if the remaining data belongs to the second data type, writing the remaining data into the second log and the buffer.

[0011] In one possible implementation, the target data is written to the storage unit using a set writing method according to the data type of the target data, including: when the amount of data in the buffer reaches a preset threshold, erasure coding is used to write the data in the buffer to the data file in the storage unit.

[0012] Secondly, an apparatus for writing data is provided, characterized in that it includes: an acquisition module for acquiring target data to be written; a determination module for determining the data type of the target data from three preset data types according to the size of the target data, wherein the three data types correspond to three data threshold ranges respectively, and the relationship between the data threshold ranges of the three data types is: the first data type is smaller than the second data type, and the second data type is smaller than the third data type; and a writing module for writing the target data into a storage unit according to the data type of the target data using a set writing method.

[0013] In one possible implementation, the writing module is specifically used to: write the target data into a first log in the storage unit and into a buffer using a write copy mechanism when the target data belongs to the first data type; write the target data into a second log in the storage unit and into the buffer using erasure coding when the target data belongs to the second data type; and write the target data into a data file in the storage unit using erasure coding when the target data belongs to the third data type.

[0014] In one possible implementation, the writing module is specifically used to: merge the target data and the data in the buffer to obtain merged data; and write the merged data into the data file using erasure coding.

[0015] In one possible implementation, the writing module is specifically used to: write data that meets the stripe alignment requirement in the merged data into the data file according to the stripe alignment principle of erasure coding; determine the remaining data in the merged data that does not meet the stripe alignment requirement; and write the remaining data into a log and the buffer, wherein the log includes the first log or the second log.

[0016] In one possible implementation, the writing module is specifically used to: determine whether the remaining data belongs to the first data type or the second data type based on the size of the remaining data; if the remaining data belongs to the first data type, write the remaining data into the first log and the buffer; if the remaining data belongs to the second data type, write the remaining data into the second log and the buffer.

[0017] In one possible implementation, the writing module is specifically used for: writing the target data into the storage unit according to the data type to which the target data belongs using a set writing method, including: when the amount of data in the buffer reaches a preset threshold, writing the data in the buffer into the data file in the storage unit using erasure coding.

[0018] Thirdly, a computer device is provided, including a processor for calling a computer program from memory, wherein when the computer program is executed, the processor is configured to perform the methods of the first aspect or any possible implementation thereof.

[0019] Fourthly, a computer-readable storage medium is provided for storing a computer program including code for performing the methods described in the first aspect or any possible implementation thereof.

[0020] Fifthly, a computer program product is provided, comprising a computer program including code for performing the methods described in the first aspect or any possible implementation thereof.

[0021] This application proposes a method for writing data. This method performs fine-grained splitting of input data, dividing the input data into three data streams according to the size of the input data, and executing different data writing schemes for each stream. This fine-grained writing process can reduce the write flow of input data and improve the efficiency of writing data. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 This is a schematic diagram of the architecture of a storage system according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the functional modules of a storage system implementing the data writing process according to an embodiment of this application;

[0025] Figure 3 This is a flowchart illustrating a method for writing data according to an embodiment of this application;

[0026] Figure 4 This is a schematic flowchart illustrating a method for writing data according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the structure of a device 500 according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of a device 600 according to an embodiment of this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0030] First, let me explain the terms used in this application:

[0031] Distributed storage systems refer to storage architectures that distribute data across multiple independent storage servers. The storage load is shared among multiple storage servers, and location servers are used to locate stored information. Distributed storage systems offer high reliability and are easily scalable.

[0032] Based on the size of the storage unit, storage blocks can be divided into two types: blocks and chunks. A block refers to the basic unit of data, i.e., a single data unit. A chunk is a larger unit of data. For example, the size of a block can range from 4 kilobytes (KB) to 1 megabyte (MB). The size of a chunk can range from 64 MB to 1 gigabyte (GB).

[0033] Erasure coding (EC) is a fault-tolerant coding technique. Its basic principle is to divide the stored data into fragments, and then use a certain checksum calculation method to generate m fragments from k original data fragments. The original data can be restored from any k fragments out of the k+m fragments. This way, even if some data is lost, the system can still recover the original data. k and m are both integers greater than or equal to 1.

[0034] Encoding group: The basic unit for erasure coding, which contains k data blocks and m parity blocks.

[0035] Stripe: a group of codes.

[0036] Write-ahead logging (WAL) refers to writing data in advance as a log to ensure data atomicity and durability. Modifications are not directly written to the data files, but rather to a WAL file. If a transaction fails, the records in the WAL are ignored, and the modifications are rolled back. If the transaction succeeds, the changes will be written back to the data files at a later time, committing the changes. WAL can be applied to scenarios such as key-value databases.

[0037] Rollback journal: This is a data recording method where, before modifying data, the data in the page to be modified is backed up in the rollback journal, and then the changes are written to the data file. If the transaction fails, the backup data is copied back, and the changes are undone; if the transaction succeeds, the backup data is deleted, and the changes are committed.

[0038] Redo log: A method of recording data. For example, before a transaction is committed, data or modifications to that data can be recorded in the redo log and written to disk. After the transaction is committed, the data in the redo log is deleted. If the transaction fails to commit, the data can be recovered using the redo log.

[0039] Figure 1 This is a schematic diagram of the architecture of a storage system according to an embodiment of this application. Figure 1 As shown, a storage system typically includes middleware service units and storage units. The middleware service units are used to perform storage-related tasks that are perceptible to the user; they can also be referred to as the front-end or foreground.

[0040] The middleware service unit can be used to communicate with clients over a network, which can be the Internet or a local area network. The middleware service unit receives and analyzes client requests for read and write data, and manages and distributes task traffic.

[0041] Storage units are used to provide storage services. Users are unaware of the tasks performed by the storage unit, hence they are also called backend or back-end systems. A storage unit may include multiple distributed storage nodes.

[0042] The middleware service unit and storage unit are divided by function. In practice, the middleware service unit and storage unit can be implemented by a server or a server cluster.

[0043] It should be understood that Figure 1 The application scenarios described are merely examples and not limitations. In practice, appropriate modifications and additions or subtractions can be made to the above scenarios, and the solutions of the embodiments of this application will still be applicable.

[0044] Figure 2 This is a schematic diagram of the functional modules implementing the data writing process in a storage system according to an embodiment of this application. Figure 2 As shown, the middleware service unit includes a buffer, and the storage unit includes a log and a data file. After receiving input data from the client, the middleware service unit can write the input data to the log using a write-copy mechanism, and simultaneously write the input data to the buffer. When the data accumulates to a certain size in the buffer, it can be written to the data file in the storage unit using erasure coding. After the input data is successfully written to the data file, its corresponding record in the log can be deleted. The log can be used to record data or modification operations on the data, so that the original data can be recovered using the log in case of a failed write operation. This application embodiment does not limit the type of log to be written; for example, the above-mentioned log types may include, but are not limited to, WAL, rollback journal, redo log, etc.

[0045] Logs are typically foreground-facing and require timely processing of tasks; therefore, they are usually recorded using faster, more expensive media, such as solid-state drives (SSDs). Data files, on the other hand, have lower timeliness requirements and can be stored using slower, less expensive media, such as hard disk drives (HDDs).

[0046] The logs mentioned above may include data and metadata, where metadata refers to data describing data attributes. Data files can be plain data files or files with a specific format.

[0047] The aforementioned buffer can be part of memory, used to buffer input or output data. Read and write operations are performed only after the data in the buffer has accumulated to a certain size. Since directly reading and writing data from the disk in the storage unit takes a long time and can easily damage the disk, setting up a buffer can reduce the number of data read and write operations, thus protecting the disk and improving the timeliness of the front end.

[0048] It should be understood that Figure 2 The description is merely an example and not a limitation. In practice, appropriate modifications and additions can be made to the above scenarios, and the solutions of the embodiments of this application will still be applicable.

[0049] In storage systems, to ensure data reliability, write replication or erasure coding are typically used for backup storage. Write replication, due to its copying mechanism, offers less redundancy than erasure coding. However, erasure coding requires stripe alignment during encoding and decoding, meaning data must be encoded and decoded in stripe units. Since the size of the input data at the front end is unpredictable, padding small amounts of input data to fill stripes can amplify data throughput. Waiting for the input data to accumulate to a certain size before erasure coding increases front-end latency and negatively impacts user experience. Therefore, a common industry practice is to use write replication at the front end for write operations, and then switch to erasure coding storage in the back end.

[0050] However, this approach still has drawbacks: when the input data is large, using a write replication mechanism will amplify the write traffic, resulting in significant system-level bandwidth consumption. For example, when writing data, the system writes the write request data to the log and simultaneously copies the data into a buffer. Once the data in the buffer reaches a certain size, it is then written to the backend storage unit. Assuming the write replication mechanism has 3 replicas, and the erasure coding uses an encoding / decoding scheme with k=4 and m=2, the total bandwidth consumption is 4.5 copies. Of these, the write replication mechanism uses 3 copies, and the erasure coding uses 1.5 copies.

[0051] To address the aforementioned issues, this application proposes a data writing method. This method performs fine-grained data splitting, categorizing the input data into three data types based on its size and executing different data writing schemes for each, thereby reducing the data writing flow and improving data writing efficiency.

[0052] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0053] Figure 3 This is a flowchart illustrating a method for writing data according to an embodiment of this application. Optionally, this method can be performed by... Figure 1 or Figure 2 The middleware service unit executes within it. For example... Figure 3 As shown, the method includes the following.

[0054] S301. Obtain the target data to be written.

[0055] As an example, a client can send a write data request to the storage system, which may include the target data mentioned above. A middleware service unit within the storage system can then receive this write data request.

[0056] S302. Based on the size of the target data, determine the data type of the target data from three preset data types, wherein the three data types correspond to three data threshold ranges, and the relationship between the data threshold ranges of the three data types is: the first data type is less than the second data type, and the second data type is less than the third data type.

[0057] Optionally, the data can be divided into three data types based on the size of the data being written. The threshold for dividing the data into these three types is not fixed and can be adjusted based on practical experience.

[0058] As a specific example, the above three data types can be divided in the following way:

[0059] First data type: Data size < 4KB;

[0060] Second data type: 4KB ≤ data size < 8MB;

[0061] Third data type: Data size ≥ 8MB.

[0062] As an example, the first to third data types mentioned above can also be referred to as: small input data, medium input data, and large input data, respectively.

[0063] S303. Based on the data type of the target data, write the target data into the storage unit using the set writing method.

[0064] As a specific example, the middleware service unit can divide the target data into different data types according to the size of the target data, and determine the way to write data into the storage unit according to the different data types.

[0065] Specifically, when the target data belongs to the first data type, a write-copy mechanism is used to write the target data into the first log and simultaneously into the buffer. Since the first data type is relatively small, even with the write-copy mechanism, it will not consume excessive storage space or bandwidth. Therefore, the write-copy mechanism can be used to write it into the first log and simultaneously into the buffer. Once the data in the buffer accumulates to a preset threshold, it can be written to the data file in the storage unit. The first log is used to record data copies of the first data type or modification operations related to the first data type. After the data of the first data type is successfully written to the storage unit, the relevant records in the first log can be deleted.

[0066] It should be noted that, due to the stripe alignment requirement of erasure coding, for data of the first data type with a small data size, the use of erasure coding for encoding and decoding requires filling the missing data space, which causes an increase in data traffic. Therefore, the embodiments of this application use fine-grained partitioning and a write copy mechanism to record the log of the first data type, thereby achieving the purpose of saving traffic.

[0067] Optionally, this application embodiment does not limit the type of the first log, as long as it can achieve the function of recording data. For example, the first log can be a WAL, a rollback log, or a redo log.

[0068] If the target data belongs to the second data type, erasure coding is used to write the target data into the second journal (EC) and simultaneously into the buffer. The second data type is larger than the first data type, but still does not meet the requirement of direct writing to the data file. Therefore, to save bandwidth, erasure coding can be used to write the second data type into the second journal and simultaneously into the buffer. Once the data in the buffer accumulates to a preset threshold, it can be written to the data file in the storage unit. The second journal records data of the second data type or modification operations related to the second data type. After the second data type is successfully written to the data file, the relevant records in the first journal can be deleted.

[0069] Optionally, this application embodiment does not limit the type of the second log, as long as it can achieve the function of recording data. For example, the second log can be a WAL, a rollback log, or a redo log.

[0070] Optionally, if the amount of data in the buffer reaches a preset threshold, erasure coding is used to write the data in the buffer into a data file.

[0071] If the target data belongs to the third data type, erasure coding is used to write the target data to the data file. Since the third data type is relatively large, it can be written directly to the data file without needing to write to a log or buffer.

[0072] Optionally, the third data type can be merged with the data in the buffer before being written to the data file, and then written to the data file after merging. However, since erasure coding requires stripe alignment, there may be residual data after stripe alignment. In this case, the residual data after stripe alignment can be written to the log and the buffer, and then written to the data file once the data in the buffer reaches a preset threshold.

[0073] As an example, when the target data belongs to the third data type, writing the target data into a data file using erasure coding includes: merging the target data and the data in the buffer to obtain merged data; and writing the merged data into a data file using erasure coding.

[0074] Optionally, different approaches can be used when writing target data, particularly for stripe alignment operations. For example, in one scenario, if stripe alignment is used, there might be residual data that doesn't meet the alignment requirements. This residual data can be written to a log and buffer, and then written to the data file only after the data in the buffer has accumulated to a certain threshold. Alternatively, in another scenario, empty data padding can be used to align the data, ensuring all target data is written to the data file promptly, thus eliminating residual data. Or, in yet another scenario, the target data can be provided as encapsulated data blocks that meet stripe alignment requirements, also eliminating residual data.

[0075] The step of writing the merged data to a data file using erasure coding further includes: writing data from the merged data that meets the stripe alignment principle into the data file according to the stripe alignment principle of erasure coding. If there is still residual data after stripe alignment, then the residual data in the merged data that does not meet the stripe alignment is identified; and the residual data is written into a log and a buffer.

[0076] Optionally, the remaining data can be determined to belong to either a first data type or a second data type based on its size. If the remaining data belongs to the first data type, it is written to both the first log and the buffer. If the remaining data belongs to the second data type, it is written to both the second log and the buffer. After the remaining data in the buffer is written to the storage unit, the corresponding record in the log can be deleted.

[0077] This application proposes a data writing method that performs fine-grained splitting of input data. Based on the size of the input data, it divides the input data into three data streams and executes different data writing schemes for each. Smaller data can be written to a first log and simultaneously to a buffer using a write copy method. Medium-sized data can be written to a second log and simultaneously to a buffer using erasure coding. Larger data can be directly written to a data file in the storage unit. This fine-grained write processing method can reduce the input data write flow and improve the efficiency of data writing.

[0078] For example, as an example, for related technologies, assuming the number of copies in the write replication mechanism is 3, and the erasure coding used when writing to the storage unit has k=8 and m=3, then the total traffic consumed is: 3+1.375=4.375; that is, the traffic is amplified to 4.375 times the data itself.

[0079] For the scheme in this embodiment, assuming the number of replicas in the write replication mechanism is 3, the erasure coding used when writing the second log has k=4 and m=2, and the erasure coding used when writing the data file has k=8 and m=3, and the data of the first data type accounts for 0.1% of the total written data, then the total traffic consumed is: 0.1×3+0.9×1.5+(0.1+0.9)×1.375=3.025; that is, the traffic is amplified to 3.025 times the data itself. It can be seen that by performing fine-grained traffic splitting on the input data, the write traffic of the input data can be reduced, and the efficiency of writing data can be improved.

[0080] In this embodiment of the application, when designing the data flow for writing logs, it can be divided into multiple data flows according to the data size, while the data for writing data files is aggregated into one data flow. This allows for flexible configuration of the data writing method based on the data type, minimizing the flow consumption of input data and improving the throughput of the storage system.

[0081] Figure 4 This is a schematic flowchart illustrating a method for writing data according to an embodiment of this application. Figure 4 The method can be derived from Figure 1 or Figure 2 The middleware service unit executes within it. For example... Figure 4 As shown, the method includes the following.

[0082] S401. Based on the size of the data to be written, determine that the data to be written is of the first data type.

[0083] S402. Use a write-copy mechanism to write data of the first data type into the first log.

[0084] S403. Write the data of the first data type into the buffer.

[0085] S404. Based on the size of the data to be written, determine that the data to be written is a second data type.

[0086] S405. Use erasure coding to write the data of the second data type into the second log.

[0087] S406. Write the data of the second data type into the buffer.

[0088] S407. Based on the size of the data to be written, determine that the data to be written is a third data type.

[0089] S408. Merge the data of the third data type with the data stored in the buffer to obtain merged data.

[0090] S409. Write the merged data to the data file using erasure coding.

[0091] If, during the erasure coding process, there is residual data after stripe alignment, the data type of the residual data is determined. If the residual data belongs to the first type of data, S410 is executed; if the residual data belongs to the second type of data, S411 is executed.

[0092] S410. Write the remaining data to the first log using a write-copy mechanism and write it to the buffer.

[0093] After the remaining data in the buffer is written to the data file, the corresponding record in the first log can be deleted.

[0094] S411. Write the remaining data to the second log in erasure coding mode and write it to the buffer.

[0095] After the remaining data in the buffer is written to the data file, the corresponding record in the second log can be deleted.

[0096] In this embodiment, the input data is processed in a fine-grained manner, divided into three data streams according to the size of the input data, and different writing schemes are executed for each stream. Smaller data can be written to the log and buffer simultaneously using a copy writing method, medium-sized data can be written to the log and buffer simultaneously using erasure coding, and larger data can be written directly to the data file. This fine-grained writing process reduces the write traffic of the input data and improves the efficiency of the storage system in writing data.

[0097] Figure 5 This is a schematic diagram of the structure of a device 500 according to an embodiment of this application. The device 500 is used to perform... Figure 3 or Figure 4 The method in the middle.

[0098] The device 500 includes:

[0099] The acquisition module 510 is used to acquire the target data to be written;

[0100] The determining module 520 is used to determine the data type of the target data from three preset data types based on the size of the target data. The three data types correspond to three data threshold ranges, and the relationship between the data threshold ranges of the three data types is: the first data type is smaller than the second data type, and the second data type is smaller than the third data type.

[0101] The writing module 530 is used to write the target data into the storage unit according to the data type of the target data and using a set writing method.

[0102] Figure 6 This is a schematic diagram of the structure of a device 600 according to an embodiment of this application. The device 600 is used to perform... Figure 3 or Figure 4 The method in the middle.

[0103] The device 600 includes a processor 610, which is configured to execute computer programs or instructions stored in the memory 620, or to read data stored in the memory 620, to perform the methods described in the above method embodiments. Optionally, there may be one or more processors 610.

[0104] Optionally, such as Figure 6 As shown, the device 600 also includes a memory 620 for storing computer programs or instructions and / or data. The memory 620 may be integrated with the processor 610 or may be disposed separately. Optionally, there may be one or more memories 620.

[0105] Optionally, such as Figure 6 As shown, the device 600 also includes a communication interface 630, which is used for receiving and / or transmitting signals. For example, the processor 610 is used to control the communication interface 630 to receive and / or transmit signals.

[0106] Optionally, the device 600 is used to implement the operations performed by the middleware service unit in the various method embodiments described above.

[0107] For example, processor 610 is used to execute computer programs or instructions stored in memory 620 to implement the relevant operations of the middleware service units in the various method embodiments described above. For example, processor 610 is used to execute... Figure 3 or Figure 4 The relevant methods in [the text].

[0108] It should be pointed out that, Figure 6 The device 600 in the above embodiment may be the middleware service unit, or it may be a component of the middleware service unit (such as a chip), and there is no limitation here.

[0109] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU (which can be understood as a type of microprocessor), or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0110] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0111] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0112] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement... Figures 1 to 4 The steps in the method executed by the middleware service unit.

[0113] Accordingly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, cause the processor to implement... Figures 1 to 4 The steps in the method executed by the middleware service unit.

[0114] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0119] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0120] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0122] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for writing data, comprising: Get the target data to be written; Based on the size of the target data, the data type to which the target data belongs is determined from three preset data types, wherein the three data types correspond to three data threshold ranges, and the relationship between the data threshold ranges of the three data types is: the first data type is smaller than the second data type, and the second data type is smaller than the third data type; According to the data type of the target data, the target data is written into the storage unit using the set writing method; The step of writing the target data into the storage unit according to the data type of the target data and using a set writing method includes: If the target data belongs to the first data type, the target data is written to the first log in the storage unit and written to the buffer using a write copy mechanism. If the target data belongs to the second data type, the target data is written to the second log in the storage unit and then written to the buffer using erasure coding. If the target data belongs to the third data type, erasure coding is used to write the target data into the data file in the storage unit.

2. The method as described in claim 1, wherein, when the target data belongs to the third data type, erasure coding is used to write the target data into a data file in the storage unit, comprising: The target data and the data in the buffer are merged to obtain merged data; The merged data is written to the data file using erasure coding.

3. The method as described in claim 2, wherein writing the merged data into the data file using erasure coding further includes: According to the stripe alignment principle of erasure coding, the data in the merged data that meets the stripe alignment is written into the data file; Identify the remaining data in the merged data that does not meet the stripe alignment requirement; The remaining data is written to the log and the buffer, the log including either the first log or the second log.

4. The method of claim 3, wherein writing the remaining data into the log and the buffer, the log including the first log or the second log, includes: Based on the size of the remaining data, it is determined whether the remaining data belongs to the first data type or the second data type; If the remaining data belongs to the first data type, the remaining data is written to the first log and the buffer; If the remaining data belongs to the second data type, the remaining data is written to the second log and the buffer.

5. The method as described in claim 1 or 2, wherein writing the target data into the storage unit using a set writing method according to the data type of the target data includes: When the amount of data in the buffer reaches a preset threshold, erasure coding is used to write the data in the buffer into the data file in the storage unit.

6. A device for writing data, comprising: The acquisition module is used to acquire the target data to be written. The determination module is used to determine the data type of the target data from three preset data types based on the size of the target data. The three data types correspond to three data threshold ranges, and the relationship between the data threshold ranges of the three data types is: the first data type is smaller than the second data type, and the second data type is smaller than the third data type. The writing module is used to write the target data into the storage unit according to the data type of the target data and using a set writing method; The writing module is specifically used for: If the target data belongs to the first data type, the target data is written to the first log in the storage unit and written to the buffer using a write copy mechanism. If the target data belongs to the second data type, the target data is written to the second log in the storage unit and then written to the buffer using erasure coding. If the target data belongs to the third data type, erasure coding is used to write the target data into the data file in the storage unit.

7. An electronic device comprising a processor configured to invoke a computer program from a memory, wherein when the computer program is executed, the processor is configured to perform the method as claimed in any one of claims 1 to 5.

8. A computer-readable storage medium for storing a computer program for performing the method as claimed in any one of claims 1 to 5.

9. A storage system, the storage system comprising a middleware service unit and a storage unit, the middleware service unit including a buffer, the storage unit including a data file, a first log, and a second log, the middleware service unit being used for: Get the target data to be written; Based on the size of the target data, the data type to which the target data belongs is determined from three preset data types, wherein, The three data types correspond to three data threshold ranges, and the relationship between the data threshold ranges of the three data types is: the first data type is less than the second data type, and the second data type is less than the third data type. According to the data type of the target data, the target data is written into the storage unit using the set writing method; Wherein, if the target data belongs to the first data type, the target data is written to the first log and then to the buffer using a write copy mechanism; If the target data belongs to the second data type, the target data is written to the second log and the buffer using erasure coding. If the target data belongs to the third data type, erasure coding is used to write the target data into the data file.

Images