Data writing method, device, computer equipment and storage medium

Abstract

The present application relates to a data writing method, apparatus, computer device, storage medium, and computer program product. The method includes: obtaining multiple writing processes corresponding to a data writing request; determining a target data block of the target writing process for a target writing process; dividing the target data block according to a preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and writing the data in the first data block corresponding to the target writing process; if there is a next writing process of the target writing process, transmitting the remaining data block to the next writing process of the target writing process via a preset data transmission network, so that the next writing process determines the remaining data block and the original data block of the next writing process as the target data block, and as a new target writing process, executes the step of dividing the target data block according to the preset stripe size.

Description

Technical Field

[0001] The present application relates to the field of storage, and in particular to a data writing method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] To ensure data consistency in a distributed file storage system, distributed interval locks are required for protection. Specifically, data in a distributed file storage system can be divided into stripes, and each stripe needs to be protected by a distributed interval lock.

[0003] In related technologies, when cross-process data needs to be written to the same stripe, a distributed interval lock is used to prevent different writing processes from processing the storage space within the same stripe. This lock needs to be called when each stripe is read. When reading multiple stripes, contention for the distributed interval lock will occur, resulting in low data writing efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide a data writing method, device, computer equipment, computer-readable storage medium and computer program product that can avoid distributed lock contention in response to the above technical problems.

[0005] In a first aspect, the present application provides a data writing method. The method comprises:

[0006] In response to a data write request, obtaining a plurality of write processes corresponding to the data write request;

[0007] For a target writing process among the multiple writing processes, determining a target data block of the target writing process;

[0008] Dividing the target data block according to a preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and writing data in the first data block corresponding to the target write process through the target write process;

[0009] In the case that there is a next write process of the target write process, the remaining data blocks are transmitted to the next write process of the target write process through a preset data transmission network, so that the next write process determines the remaining data blocks and the original data blocks of the next write process as target data blocks, and as a new target write process, executes the step of dividing the target data blocks according to the size of the preset stripe.

[0010] Based on the above solution, in the initial stage of data writing, the data to be written can be divided into full stripes, and the data that is not fully striped and thus written across processes can be exchanged to other writing processes for writing, avoiding the use of distributed interval locks. In addition, high-speed data transmission is performed through the preset data transmission network, which allows high-speed communication, exchange, and splicing of written data, ensuring strict stripe alignment, and further ensuring data writing efficiency.

[0011] In one embodiment, after the step of dividing the target data block according to the preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, the method further includes:

[0012] In the case that the target writing process does not have a next writing process, the remaining data blocks corresponding to the target writing process are cached in a preset cache space.

[0013] Based on the above solution, by caching the remaining data blocks, the integrity of the data and the timeliness of data writing can be guaranteed.

[0014] In one embodiment, the method further comprises:

[0015] After detecting a close instruction of the file to be written corresponding to the data write request, the remaining data blocks in the preset cache space are acquired, and the remaining data blocks are spliced ​​and written into the file to be written.

[0016] Based on the above solution, the remaining data blocks can be written in time to avoid data loss and improve the efficiency of data writing.

[0017] In one embodiment, determining, for each target writing process in the plurality of writing processes, a target data block of the target writing process includes:

[0018] For the target writing process, if there is a previous writing process of the target writing process in the target writing process, obtaining the remaining data blocks of the previous writing process and obtaining the original data blocks of the target writing process;

[0019] The remaining data blocks of the previous writing process and the original data blocks of the target writing process are spliced ​​to obtain the target data blocks of the target writing process.

[0020] Based on the above solution, the data block of the previous writing process can be exchanged to the target writing process, so that data writing can be achieved without distributed lock contention, thereby improving data writing efficiency.

[0021] In one embodiment, the method further comprises:

[0022] For the target writing process, if there is no previous writing process of the target writing process, the original data block of the target writing process is determined as the target data block of the target writing process.

[0023] Based on the above solution, the original data block of the writing process can be used as the target data block to achieve strict stripe-aligned data writing.

[0024] In one embodiment, the method further comprises:

[0025] For the target writing process, if there is a previous writing process of the target writing process and the previous writing process does not have the remaining data block, the original data block of the target writing process is determined as the target data block of the target writing process.

[0026] Based on the above solution, the original data block of the writing process can be used as the target data block to achieve strict stripe-aligned data writing.

[0027] In a second aspect, the present application further provides a data writing device. The device comprises:

[0028] A first acquisition module is configured to acquire, in response to a data write request, a plurality of write processes corresponding to the data write request;

[0029] A first determining module is configured to determine, for a target writing process among the multiple writing processes, a target data block of the target writing process;

[0030] a writing module, configured to divide the target data block according to a preset stripe size, obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and write data in the first data block corresponding to the target writing process through the target writing process;

[0031] A transmission module is used to transmit the remaining data blocks to the next writing process of the target writing process through a preset data transmission network when there is a next writing process of the target writing process, so that the next writing process determines the remaining data blocks and the original data blocks of the next writing process as target data blocks, and as a new target writing process, executes the step of dividing the target data blocks according to the size of the preset stripes.

[0032] In one embodiment, the data writing device further includes:

[0033] The cache module is used to cache the remaining data blocks corresponding to the target writing process into a preset cache space when there is no next writing process for the target writing process.

[0034] In one embodiment, the data writing device further includes:

[0035] The second acquisition module is configured to acquire the remaining data blocks in the preset cache space after detecting a close instruction of the file to be written corresponding to the data write request, and to splice the remaining data blocks into the file to be written.

[0036] In one embodiment, the first determining module is specifically configured to:

[0037] For the target writing process, if there is a previous writing process of the target writing process in the target writing process, obtaining the remaining data blocks of the previous writing process and obtaining the original data blocks of the target writing process;

[0038] The remaining data blocks of the previous writing process and the original data blocks of the target writing process are spliced ​​to obtain the target data blocks of the target writing process.

[0039] In one embodiment, the data writing device further includes:

[0040] The second determining module is configured to determine, for the target writing process, if there is no previous writing process of the target writing process, that the original data block of the target writing process is the target data block of the target writing process.

[0041] In one embodiment, the data writing device further includes:

[0042] The third determination module is used to determine, for the target writing process, if there is a previous writing process of the target writing process and the previous writing process does not have the remaining data block, that the original data block of the target writing process is the target data block of the target writing process.

[0043] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0044] In response to a data write request, obtaining a plurality of write processes corresponding to the data write request;

[0045] For a target writing process among the multiple writing processes, determining a target data block of the target writing process;

[0046] Dividing the target data block according to a preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and writing data in the first data block corresponding to the target write process through the target write process;

[0047] In the case that there is a next write process of the target write process, the remaining data blocks are transmitted to the next write process of the target write process through a preset data transmission network, so that the next write process determines the remaining data blocks and the original data blocks of the next write process as target data blocks, and as a new target write process, executes the step of dividing the target data blocks according to the size of the preset stripe.

[0048] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0049] In response to a data write request, obtaining a plurality of write processes corresponding to the data write request;

[0050] For a target writing process among the multiple writing processes, determining a target data block of the target writing process;

[0051] Dividing the target data block according to a preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and writing data in the first data block corresponding to the target write process through the target write process;

[0052] In the case that there is a next write process of the target write process, the remaining data blocks are transmitted to the next write process of the target write process through a preset data transmission network, so that the next write process determines the remaining data blocks and the original data blocks of the next write process as target data blocks, and as a new target write process, executes the step of dividing the target data blocks according to the size of the preset stripe.

[0053] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0054] In response to a data write request, obtaining a plurality of write processes corresponding to the data write request;

[0055] For a target writing process among the multiple writing processes, determining a target data block of the target writing process;

[0056] Dividing the target data block according to a preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and writing data in the first data block corresponding to the target write process through the target write process;

[0057] In the case that there is a next write process of the target write process, the remaining data blocks are transmitted to the next write process of the target write process through a preset data transmission network, so that the next write process determines the remaining data blocks and the original data blocks of the next write process as target data blocks, and as a new target write process, executes the step of dividing the target data blocks according to the size of the preset stripe.

[0058] The above-mentioned data writing method, device, computer equipment, storage medium and computer program product, wherein the method includes: in response to a data write request, obtaining multiple write processes corresponding to the data write request; for a target write process among the multiple write processes, determining the target data block of the target write process; dividing the target data block according to the size of a preset stripe to obtain a first data block that meets the size of the preset stripe and the remaining data blocks that do not meet the preset stripe size, and writing the data in the first data block corresponding to the target write process through the target write process; in the case where there is a next write process of the target write process, transmitting the remaining data blocks to the next write process of the target write process through a preset data transmission network, so that the next write process determines the remaining data blocks and the original data blocks of the next write process as target data blocks, and as a new target write process, executes the step of dividing the target data blocks according to the size of the preset stripe. By adopting this method, in the initial stage of data writing, the data to be written can be divided into full stripes, and the data that is not fully striped and thus written across processes can be exchanged to other writing processes for writing, avoiding the use of distributed interval locks, and high-speed data transmission through a preset data transmission network. By exchanging and splicing written data, strict stripe alignment is guaranteed, further improving data writing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 1 is a flow chart of a data writing method according to an embodiment;

[0060] Figure 2a A schematic diagram of dividing data blocks corresponding to a writing process in a data writing method in one embodiment;

[0061] Figure 2b A schematic diagram of dividing data blocks corresponding to a writing process in a data writing method in one embodiment;

[0062] Figure 3 Schematic diagram of a flow chart of the step of determining a target data block in one embodiment;

[0063] Figure 4 A schematic diagram of distributed lock contention in one embodiment;

[0064] Figure 5 is a structural block diagram of a data writing device in one embodiment;

[0065] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0067] In one embodiment, Figure 1 As shown, a data writing method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The above-mentioned terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server can be implemented as an independent server or a server cluster composed of multiple servers. In this embodiment, the data writing method includes the following steps:

[0068] Step 102: In response to a data write request, obtain multiple write processes corresponding to the data write request.

[0069] Among them, the data write request can be a write request received by the terminal, and the write request can be a request for instructing the terminal to perform a data write operation on a distributed file system. The distributed file system can be a physical storage resource managed by the file system, which can be connected to each node through a computer network, or can be a file system composed of multiple different logical disk partitions. The data write request can include at least one write process.

[0070] Optionally, the terminal may also obtain multiple writing processes that perform data writing operations on the distributed file system within a preset time period.

[0071] In practice, other terminal devices can generate multiple write processes based on the needs of actual application scenarios, and generate data write requests based on these write processes. In this way, the data write requests can be sent to the terminal corresponding to the distributed file system. After receiving the data write request, the terminal can parse the data write request to obtain the write process or processes included in the data write request. The terminal can also receive multiple data write requests within a preset time period and obtain the write processes included in each data write request.

[0072] Step 104 : for a target writing process among the multiple writing processes, determine a target data block of the target writing process.

[0073] The target writing process may be any one of the multiple writing processes. The target data block of the target writing process may be the original data block carried by the target writing process, or a data block generated by combining the original data block with data blocks of other writing processes.

[0074] In implementation, after obtaining multiple writing processes corresponding to the distributed file system, the terminal can sort the writing processes according to their corresponding time sequence and process each writing process in sequence. In this way, the terminal can specifically process a target writing process among the multiple writing processes as follows: after determining the target writing process, the terminal needs to determine the target data block corresponding to the target writing process.

[0075] Step 106: divide the target data block according to the preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and write the data in the first data block corresponding to the target write process through the target write process.

[0076] The preset stripe size may be the length of data contained in a preset stripe in the distributed file system, for example, A. The data length of the first data block meeting the preset stripe size may be the same as the length of data that can be contained in the pre-configured preset stripe, and the remaining data blocks that do not meet the preset stripe size may be data blocks whose data length is less than the length of data that can be contained in the pre-configured preset stripe.

[0077] In implementation, after obtaining the target data block corresponding to the target write process, the terminal can split the target data block according to the data length of the preset stripe to obtain one or more first data blocks (stripe-aligned data) and remaining data blocks. After obtaining the first data block and remaining data blocks corresponding to the target write process, the terminal can write the data in the first data block to the distributed file system to ensure the timeliness of data writing.

[0078] Step 108, in the case where there is a next write process of the target write process, the remaining data blocks are transmitted to the next write process of the target write process through a preset data transmission network, so that the next write process determines the remaining data blocks and the original data blocks of the next write process as target data blocks, and as a new target write process, executes the step of dividing the target data blocks according to the size of the preset stripe.

[0079] The preset data transmission network may be an OpenMPI network (High Performance Message Passing Library), which is used for data transmission between processes.

[0080] In implementation, after the terminal determines the first data block and the remaining data blocks corresponding to the target write process, the terminal can write the first data block to the distributed file system corresponding to the data write request. At the same time, the terminal can determine whether there is a next write process after the target write process.

[0081] After the terminal determines that the target writing process still has a next writing process, the terminal can exchange the remaining data blocks corresponding to the target writing process from the target writing process to the next writing process of the target writing process through a preset data transmission network, that is, transmit the remaining data blocks from the target writing process to the next writing process of the target writing process. In this way, after the terminal determines that the next writing process of the target writing process has received the remaining data blocks of the target writing process, the terminal can determine the remaining data blocks and the original data blocks of the next writing process of the target writing process as the target data blocks, and use the next writing process of the target writing process as the new target writing process, execute the steps of the above embodiment to divide the target data blocks according to the size of the preset stripe, obtain the first data blocks that meet the size of the preset stripe and the remaining data blocks that do not meet the preset stripe size, and write the data in the first data blocks corresponding to the target writing process through the target writing process.

[0082] In one example, if Figure 2a As shown, the target write process may be rank 0, and the target data block corresponding to the rank may be data block ①; the terminal divides data block ① according to the size of the preset stripe, and the obtained first data block that meets the preset stripe size and the remaining data blocks that do not meet the preset stripe size may be as shown below. Figure 2b As shown, the first data block may be data block ③ and data block ④, and the remaining data blocks may be data block ⑤, that is, the terminal may divide data block ① into data block ③, data block ④, and data block ⑤. The terminal may write data block ③ and data block ④ into the distributed file system.

[0083] like Figure 2bAs shown, the terminal can determine that there is a next write process after the target write process (rank0), that is, rank1. Based on this, the terminal can exchange the remaining data blocks (data block ⑤) of rank0 from rank0 to rank1 through a preset data transmission network (local high-speed network). In other words, the next write process of the target write process can be rank1, the original data block corresponding to rank1 can be data block ②, and the target data block of rank1 can be a data block after the remaining data blocks (data block ⑤) and the original data blocks (data block ②) are merged. After determining the target data block of rank1, the terminal can use rank1 as the new target write process to execute the steps of the above embodiment, that is, divide the target data block of rank1 according to the size of the preset stripe to obtain a new first data block and a new remaining data block. The new first data block can be a data block composed of data block ⑤ + data block ⑥, data block ⑦ and data block ⑧, and the new remaining data block can be data block ⑨.

[0084] In the above-mentioned data writing method, in response to a data write request, multiple write processes corresponding to the data write request are obtained. For a target write process among the multiple write processes, a target data block for the target write process is determined. The target data block is divided according to a preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size. The target write process then writes the data in the first data block corresponding to the target write process. If a next write process of the target write process exists, the remaining data block is transmitted to the next write process of the target write process via a preset data transmission network, so that the next write process determines the remaining data block and the original data block of the next write process as the target data block and, as the new target write process, executes the step of dividing the target data block according to the preset stripe size. By adopting this method, data to be written can be divided into full stripes at the initial stage of data writing, and data that is not full stripes and thus caused to be written across processes can be exchanged to other write processes for writing, avoiding the use of distributed interval locking. High-speed data transmission via the preset data transmission network allows for high-speed communication, exchange, and splicing of written data, ensuring strict stripe alignment, and further improving data writing efficiency.

[0085] In one embodiment, after the step of dividing the target data block according to the preset stripe size to obtain the first data block that meets the preset stripe size and the remaining data blocks that do not meet the preset stripe size, the data writing method further includes:

[0086] When the target writing process does not have a next writing process, the remaining data blocks corresponding to the target writing process are cached in a preset cache space.

[0087] In implementation, when the terminal determines that there is no next writing process after the target writing process, the terminal may store the remaining data blocks in a preset remaining cache space, that is, cache the remaining data blocks.

[0088] In this embodiment, by caching the remaining data blocks, the integrity of the data and the timeliness of data writing can be guaranteed.

[0089] In one embodiment, the data writing method further includes:

[0090] After detecting a close instruction of the file to be written corresponding to the data write request, the remaining data blocks in the preset cache space are obtained, and the remaining data blocks are spliced ​​and written into the file to be written.

[0091] Among them, the file to be written corresponding to the data write request can be a file located in a distributed file system, the address of the file to be written can be the destination storage address of the data carried by the writing process, and the closing instruction of the file to be written can be an operation executed after the file to be written has completed writing, used to instruct the terminal to close the file to be written.

[0092] In implementation, if the terminal detects that there is a close instruction for the file to be written corresponding to the data write request, the terminal can splice and write the remaining blocks cached in the preset cache space, that is, splice and write them to the file to be written in the distributed file system.

[0093] In one example, after the terminal stores the remaining data blocks in the preset cache space, if it determines that there is no other writing process within the preset time period, the remaining data blocks in the preset cache space can be spliced ​​and written into the file to be written, that is, after the current write and before the next write, the remaining data blocks in the preset cache space can be spliced ​​and written into the file to be written.

[0094] In this embodiment, the remaining data blocks can be written in time, thereby avoiding data loss and improving data writing efficiency.

[0095] In one embodiment, Figure 3 As shown, the specific processing process of the step of "determining the target data block of each target writing process in the multiple writing processes" includes:

[0096] Step 302 : for the target writing process, if there is a previous writing process of the target writing process, obtain the remaining data blocks of the previous writing process and obtain the original data blocks of the target writing process.

[0097] Step 304: splice the remaining data blocks of the previous writing process and the original data blocks of the target writing process to obtain the target data blocks of the target writing process.

[0098] The previous writing process of the target writing process may be a process in the migration order of the target writing process according to the arrangement order of the processes.

[0099] In implementation, for a target writing process, the terminal can determine whether there is another process preceding the target writing process, that is, determine whether the target writing process is the initial writing process. The initial writing process can be the writing process that is ranked first according to the process arrangement order. If the terminal determines that the target writing process has a previous writing process, the terminal can obtain the remaining data blocks of the previous writing process through a preset data transmission network. In this way, the terminal can also obtain the original data blocks carried by the target writing process itself, and splice the remaining data blocks of the previous writing process of the target writing process with the original data blocks of the target writing process to obtain the target data blocks of the target writing process.

[0100] In this embodiment, the data blocks of the previous writing process can be exchanged to the target writing process, so as to achieve data writing without distributed lock contention and improve data writing efficiency.

[0101] In one embodiment, the data writing method further includes:

[0102] For the target writing process, if there is no previous writing process of the target writing process, the original data block of the target writing process is determined as the target data block of the target writing process.

[0103] In implementation, for a target writing process, the terminal may determine whether any other process exists before the target writing process, that is, determine whether the target writing process is the initial writing process. The initial writing process may be the writing process that is determined to be the first writing process according to the process arrangement order. If the terminal determines that the target writing process does not have a previous writing process, that is, if the terminal determines that the target writing process is the initial writing process, the terminal may obtain the original data block carried by the target writing process itself and use the original data block as the target data block of the target writing process.

[0104] In this embodiment, the original data block of the write process may be used as the target data block to implement strict stripe-aligned data writing.

[0105] In one embodiment, the data writing method further includes:

[0106] For the target writing process, if there is a previous writing process for the target writing process and no remaining data blocks exist for the previous writing process, the original data block of the target writing process is determined as the target data block of the target writing process.

[0107] In implementation, when the terminal determines that the target writing process has a previous writing process of the target writing process, but the previous writing process of the target writing process does not have any remaining data blocks, the terminal may not need to exchange data through the preset data transmission network in this case. In this way, the terminal can determine that the original data block of the target writing process is the target data block of the target writing process.

[0108] In this embodiment, the original data block of the write process may be used as the target data block to implement strict stripe-aligned data writing.

[0109] In this embodiment, to ensure the consistency of data information in the distributed system, it must be protected by a distributed lock (distributed interval lock). Distributed applications operating on the same file based on multiple access points simultaneously involve distributed lock contention, and the resulting distributed lock requests and recalls. Therefore, distributed lock contention is a performance bottleneck for OpenMPI applications writing files. In other words, OpenMPI is a multi-process application, and the timing diagram of its interaction with a distributed file system with multiple access points is as follows: Figure 4 As shown in the figure, when the write process (rank0 and rank1) requests a distributed lock, if the lock is used at another access point, a recall operation is performed first to clear the local cache of the original access point, and then the distributed lock server caches the distributed lock locally to the requesting node.

[0110] The data writing method provided in this application can strictly ensure the alignment of written data stripes through data exchange and splicing, fundamentally avoiding distributed lock competition and the occurrence of re-reading, greatly improving writing efficiency. For OpenMPI application scenarios with only writing and no reading, the ioctl interface provided by the distributed file system can be used to disable the distributed interval lock, completely skipping the distributed interval lock logic, which can further improve writing efficiency. The data writing method provided in this application can utilize the high-speed communication of OpenMPI processes to exchange and splice written data, ensuring strict stripe alignment and avoiding the overhead of distributed locks in the distributed file system. For non-OpenMPI applications, the ioctl interface will not be called during the writing process, and the writing logic will not change, ensuring compatibility.

[0111] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0112] Based on the same inventive concept, embodiments of the present application further provide a data writing device for implementing the aforementioned data writing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more data writing device embodiments provided below can be found in the above-mentioned limitations of the data writing method and will not be further elaborated here.

[0113] In one embodiment, Figure 5 As shown, a data writing device 500 is provided, comprising:

[0114] A first acquisition module 502 is configured to, in response to a data write request, acquire multiple write processes corresponding to the data write request;

[0115] A first determining module 504 is configured to determine a target data block of a target writing process among the multiple writing processes;

[0116] A writing module 506 is configured to divide the target data block according to a preset stripe size, obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and write the data in the first data block corresponding to the target writing process through a target writing process;

[0117] The transmission module 508 is used to transmit the remaining data blocks to the next writing process of the target writing process through a preset data transmission network when there is a next writing process of the target writing process, so that the next writing process determines the remaining data blocks and the original data blocks of the next writing process as target data blocks, and as a new target writing process, executes the step of dividing the target data blocks according to the size of the preset stripe.

[0118] In one embodiment, the data writing device 500 further includes:

[0119] The cache module is used to cache the remaining data blocks corresponding to the target writing process into a preset cache space when there is no next writing process for the target writing process.

[0120] In one embodiment, the data writing device 500 further includes:

[0121] The second acquisition module is configured to acquire the remaining data blocks in the preset cache space after detecting a close instruction of the file to be written corresponding to the data write request, and to splice the remaining data blocks into the file to be written.

[0122] In one embodiment, the first determining module 504 is specifically configured to:

[0123] For the target writing process, if there is a previous writing process of the target writing process in the target writing process, obtaining the remaining data blocks of the previous writing process and obtaining the original data blocks of the target writing process;

[0124] The remaining data blocks of the previous writing process and the original data blocks of the target writing process are spliced ​​to obtain the target data blocks of the target writing process.

[0125] In one embodiment, the data writing device 500 further includes:

[0126] The second determining module is configured to determine, for the target writing process, if there is no previous writing process of the target writing process, that the original data block of the target writing process is the target data block of the target writing process.

[0127] In one embodiment, the data writing device 500 further includes:

[0128] The third determination module is used to determine, for the target writing process, if there is a previous writing process of the target writing process and the previous writing process does not have the remaining data block, that the original data block of the target writing process is the target data block of the target writing process.

[0129] Each module in the data writing device 500 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0130] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store write data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data writing method is implemented.

[0131] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0134] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0135] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0136] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data writing method, characterized in that: The method comprises: In response to a data write request for performing a data write operation to a distributed file system, obtaining a plurality of write processes corresponding to the data write request; For a target writing process among the multiple writing processes, determining a target data block of the target writing process; Dividing the target data block according to a preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and writing data in the first data block corresponding to the target write process through the target write process; In the case that there is a next write process of the target write process, the remaining data blocks are transmitted to the next write process of the target write process through a preset data transmission network, so that the next write process determines the remaining data blocks and the original data blocks of the next write process as target data blocks, and as a new target write process, executes the step of dividing the target data blocks according to the size of the preset stripe.

2. The method according to claim 1, characterized in that After the step of dividing the target data block according to the preset stripe size to obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, the method further includes: In the case that the target writing process does not have a next writing process, the remaining data blocks corresponding to the target writing process are cached in a preset cache space.

3. The method according to claim 2, characterized in that The method further comprises: After detecting a close instruction of the file to be written corresponding to the data write request, the remaining data blocks in the preset cache space are acquired, and the remaining data blocks are spliced ​​and written into the file to be written.

4. The method according to claim 1, wherein The step of determining, for each target writing process in the plurality of writing processes, a target data block of the target writing process comprises: For the target writing process, if there is a previous writing process of the target writing process in the target writing process, obtaining the remaining data blocks of the previous writing process and obtaining the original data blocks of the target writing process; The remaining data blocks of the previous writing process and the original data blocks of the target writing process are spliced ​​to obtain the target data blocks of the target writing process.

5. The method according to claim 4, characterized in that The method further comprises: For the target writing process, if there is no previous writing process of the target writing process, the original data block of the target writing process is determined as the target data block of the target writing process.

6. The method according to claim 4, characterized in that The method further comprises: For the target writing process, if there is a previous writing process of the target writing process and the previous writing process does not have the remaining data block, the original data block of the target writing process is determined as the target data block of the target writing process.

7. A data writing device, characterized in that: The device comprises: A first acquisition module is configured to, in response to a data write request for performing a data write operation on a distributed file system, acquire a plurality of write processes corresponding to the data write request; A first determining module is configured to determine, for a target writing process among the multiple writing processes, a target data block of the target writing process; a writing module, configured to divide the target data block according to a preset stripe size, obtain a first data block that meets the preset stripe size and remaining data blocks that do not meet the preset stripe size, and write data in the first data block corresponding to the target writing process through the target writing process; A transmission module is used to transmit the remaining data blocks to the next writing process of the target writing process through a preset data transmission network when there is a next writing process of the target writing process, so that the next writing process determines the remaining data blocks and the original data blocks of the next writing process as target data blocks, and as a new target writing process, executes the step of dividing the target data blocks according to the size of the preset stripes.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images