Data writing method and device in distributed system, and distributed system

Abstract

The embodiments of the present specification provide a data writing method and device, and a distributed system in a distributed system, wherein the method is applied to a client, and includes: sending a data block allocation request to a metadata node in the distributed system according to a received write request, and receiving index information of a target data block and index information of at least one backup data block returned by the metadata node; writing unit data in the data to be written to the target data block, and monitoring the response time of executing the unit data writing operation, wherein the target data block is determined according to the index information of the target data block; when the response time is greater than a preset threshold, stopping the writing operation, determining a new target data block from the backup data block according to the data block identifier, and continuing to execute the writing operation of writing the unit data in the data to be written to the target data block.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and more particularly to a method for writing data in a distributed system. One or more embodiments of this specification also relate to another method for writing data in a distributed system, two data writing devices in distributed systems, a distributed system, two computing devices, and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer technology, distributed systems have been widely used. The file reading function in distributed systems has become more and more frequent. With the development of new media and the improvement of hardware performance, the average request delay of distributed systems has gradually decreased. The request delay during normal system operation is already at the microsecond level.

[0003] Since the entire distributed system is often composed of multiple roles and a large number of computing nodes, any abnormality in any role or computing node may cause a surge in user request latency, causing the user's request latency to jump directly from the microsecond level to the millisecond or even second level. These requests with abnormal latency are called request glitches, which seriously affect the normal operation of the distributed system and the user experience.

[0004] Therefore, how to reduce the impact of request glitches and enhance the stability of user requests has become an urgent problem for technical personnel to solve. Summary of the invention

[0005] In view of this, the present specification provides a method for writing data in a distributed system. One or more embodiments of the present specification also relate to another method for writing data in a distributed system, two data writing devices in distributed systems, a distributed system, two computing devices, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a method for writing data in a distributed system is provided, which is applied to a client and includes:

[0007] Sending a data block allocation request to a metadata node in the distributed system according to the received write request, and receiving index information of a target data block and index information of at least one backup data block returned by the metadata node;

[0008] Writing unit data in the data to be written into a target data block, and monitoring the response time of executing the unit data writing operation, wherein the target data block is determined according to the index information of the target data block;

[0009] When the response time is greater than a preset threshold, the write operation is stopped, a new target data block is determined from the at least one backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued, wherein the backup data block is determined according to the index information of the backup data block.

[0010] Optionally, before determining a new target data block from the at least one backup data block according to the data block identifier, the method further includes:

[0011] Generate data block information of the target data block according to the index information of the target data block and the size of the data written in the target data block;

[0012] The data block information and the data identifier of the data to be written are uploaded to the metadata node.

[0013] Optionally, the method further includes:

[0014] receiving a write completion instruction, and ending the write operation in response to the write completion instruction;

[0015] Generate data block information of the target data block according to the index information of the target data block and the size of the data written in the target data block;

[0016] The data block information and the data identifier of the data to be written are uploaded to the metadata node.

[0017] Optionally, before stopping the write operation, the method further includes:

[0018] Acquire a flow control state of a new target data block determined in the distributed system, wherein the flow control state is executing flow control or releasing flow control;

[0019] When the flow control state is the execution flow control state, entering the waiting state;

[0020] When the flow control state is to release the flow control, the write stop operation is performed.

[0021] Optionally, after entering the waiting state, the method further includes:

[0022] Receive response information of unit data and obtain flow control status;

[0023] When receiving the response information of the unit data, executing a write operation of writing the unit data in the data to be written into the target data block;

[0024] If no response information of the unit data is received and the flow control state is to release the flow control, the write operation is stopped.

[0025] Optionally, the method further includes:

[0026] When the number of backup data blocks is less than a preset threshold, sending a request for adding a backup data block to the metadata node;

[0027] Receive the index information of the newly added backup data block returned by the metadata node.

[0028] Optionally, the method further includes:

[0029] Sending a data block acquisition request to the metadata node according to the received read request of the file to be read and receiving data block information and a data block identifier of at least one read data block corresponding to the file to be read returned by the metadata node, wherein the data block information of the read data block includes index information of the data block and the size of the data written in the data block;

[0030] Determine the position of each read data block according to the index information of each read data block;

[0031] The to-be-read file is read according to the data block identifier of each read data block and the size of the data written in the data block.

[0032] According to a second aspect of an embodiment of this specification, a method for writing data in a distributed system is provided, which is applied to a metadata node and includes:

[0033] receiving a data block allocation request;

[0034] Allocating a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier;

[0035] The index information of the target data block and the index information of the at least one backup data block are returned.

[0036] Optionally, the method further includes:

[0037] Receive data block information and data identifier of the target data block, and generate a data block information table according to the data block information and data identifier, wherein the data block information table includes the data identifier, data block identifier and data block information of the target data block, and the data block information includes index information of the data block and the size of the data written in the data block.

[0038] Optionally, the method further includes:

[0039] Receive a request to add a backup data block;

[0040] Allocating a new backup data block in response to the backup data block addition request, wherein the new backup data block includes a data block identifier;

[0041] Returns the index information of the newly added backup data block.

[0042] Optionally, the method further includes:

[0043] Receiving a data block acquisition request, wherein the data block acquisition request includes a file identifier to be read;

[0044] Determine, according to the data block information table, at least one piece of data block information corresponding to the to-be-read file identifier as read data block information;

[0045] Return each piece of read data block information and a data block identifier corresponding to each piece of read data block information.

[0046] According to a third aspect of an embodiment of this specification, a data writing device in a distributed system is provided, which is applied to a client and includes:

[0047] A sending module, configured to send a data block allocation request to a metadata node in a distributed system according to a received write request for data to be written and receive index information of a target data block and index information of at least one backup data block returned by the metadata node;

[0048] a writing module configured to execute a writing operation of writing unit data in the data to be written into a target data block, and monitor a response time of executing the writing operation of the unit data, wherein the target data block is determined according to index information of the target data block;

[0049] The switching module is configured to stop the write operation, determine a new target data block from the at least one backup data block according to the data block identifier, and continue to perform the write operation of writing the unit data in the data to be written to the target data block, wherein the backup data block is determined according to the index information of the backup data block.

[0050] According to a fourth aspect of an embodiment of this specification, a data writing device in a distributed system is provided, which is applied to a metadata node, including:

[0051] A receiving module, configured to receive a data block allocation request;

[0052] an allocation module, configured to allocate a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier;

[0053] The return module is configured to return the index information of the target data block and the index information of at least one backup data block.

[0054] According to a fifth aspect of the embodiments of this specification, a distributed system is provided, including:

[0055] The client is configured to send a data block allocation request to a metadata node in the distributed system according to the received write request;

[0056] The metadata node is configured to receive the data block allocation request, allocate a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier, and return index information of the target data block and index information of the at least one backup data block;

[0057] The client is further configured to write unit data in the data to be written to the target data block, and monitor the response time of executing the unit data write operation, wherein the target data block is determined according to the index information of the target data block, and when the response time is greater than a preset threshold, stop the write operation, determine a new target data block from the at least one backup data block according to the data block identifier, and continue to execute the write operation of writing the unit data in the data to be written to the target data block, wherein the backup data block is determined according to the index information of the backup data block.

[0058] According to a sixth aspect of an embodiment of this specification, a computing device is provided, including:

[0059] Memory and processor;

[0060] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions:

[0061] Sending a data block allocation request to a metadata node in the distributed system according to the received write request, and receiving index information of a target data block and index information of at least one backup data block returned by the metadata node;

[0062] Writing unit data in the data to be written into a target data block, and monitoring the response time of executing the unit data writing operation, wherein the target data block is determined according to the index information of the target data block;

[0063] When the response time is greater than a preset threshold, the write operation is stopped, a new target data block is determined from the at least one backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued, wherein the backup data block is determined according to the index information of the backup data block.

[0064] According to a seventh aspect of an embodiment of this specification, another computing device is provided, including:

[0065] Memory and processor;

[0066] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions:

[0067] receiving a data block allocation request;

[0068] Allocating a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier;

[0069] The index information of the target data block and the index information of the at least one backup data block are returned.

[0070] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the data writing method in the distributed system are implemented.

[0071] A method for writing data in a distributed system provided by an embodiment of the present specification is applied to a client, and sends a data block allocation request to a metadata node in the distributed system according to a received write request for data to be written, so that the metadata node allocates a target data block and at least one backup data block according to the data block allocation request, and executes an operation of writing the file to be written to the target data block. In the event that a request glitch occurs during the writing process, it indicates that an abnormality occurs in the data node corresponding to the current target data block. At this time, a new target data block is determined in the backup data block according to the data block identifier, and the write operation of writing the data of the unfinished write operation to the target data block is continued. Since data blocks are usually located in different data nodes, the write operation of abnormal nodes can be avoided, reducing request delays, thereby solving the problem of request glitches in write requests, maintaining the normal operation of the distributed system, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a processing flow chart of a data writing method applied to a client in a distributed system provided by an embodiment of this specification;

[0073] Figure 2 is a processing flow chart of another method for writing data to a client in a distributed system provided by an embodiment of the present specification;

[0074] Figure 3 It is a processing flow chart of a method for writing data applied to a metadata node in a distributed system provided by an embodiment of the present specification;

[0075] Figure 4 It is a schematic diagram of the structure of a distributed system provided by an embodiment of this specification;

[0076] Figure 5 is an interaction sequence diagram between a client and a metadata node provided by an embodiment of this specification;

[0077] Figure 6 It is a structural diagram of a data writing device in a distributed system configured in a client provided by an embodiment of the present specification;

[0078] Figure 7 It is a structural schematic diagram of a data writing device configured in a distributed system of a metadata node provided by an embodiment of the present specification;

[0079] Figure 8 It is a distributed system provided by an embodiment of this specification;

[0080] Fig. 9 is a structural block diagram of a computing device provided by one embodiment of this specification;

[0081] Fig.10 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0082] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0083] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0084] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0085] First, the terms involved in one or more embodiments of this specification are explained.

[0086] Data block: Chunk. The distributed system divides the files stored on it into a series of data blocks for storage. Each data block is called a Chunk.

[0087] Replica: A distributed system generates identical copies of each Chunk on different nodes. Each copy is called a Replica.

[0088] Metadata node: A node that stores the metadata of files in a distributed system. It is usually used to store file status information and chunk index information.

[0089] Data node: A node in a distributed system where data blocks are stored. It is usually responsible for operations such as writing, storing, reading, and deleting data blocks.

[0090] Request glitches: When a user sends a request to the system, the request latency is much greater than the average latency of the system. These requests are called request glitches.

[0091] In this specification, a method for writing data in a distributed system is provided. One or more embodiments of this specification also involve another method for writing data in a distributed system, two data writing devices in distributed systems, a distributed system, two computing devices, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0092] Figure 1 A processing flow chart of a method for writing data in a distributed system provided according to an embodiment of the present specification is shown. The method is applied to a client and includes steps 102 to 106.

[0093] Step 102: Send a data block allocation request to a metadata node in the distributed system according to the received write request, and receive index information of a target data block and index information of at least one backup data block returned by the metadata node.

[0094] The processing object corresponding to the write request is the data to be written. The data to be written is the data that the user needs to write into the distributed system. It can be files in various formats such as documents, pictures, audio, and video.

[0095] The write request for the data to be written includes information such as the data name and write instructions of the data to be written. For example, when the data to be written is an audio file with the file name A, the write request for the data to be written includes an instruction to write A.mp3 to the distributed system; when the data to be written is a video file with the file name B, the write request for the data to be written includes an instruction to write B.avi to the distributed system; when the data to be written is a photo file with the file name C, the write request for the data to be written includes an instruction to write C.jpg to the distributed system.

[0096] A data block allocation request is sent to a metadata node in a distributed system based on a received write request for the data to be written. The metadata node allocates at least two data blocks based on the data block allocation request. The data block on which the write operation of the data to be written is performed is called a target data block, and the remaining data blocks are backup data blocks. A data block identifier is set for each data block.

[0097] In a specific implementation provided in this specification, the metadata node allocates a target data block C1 and three backup data blocks C2, C3, and C4 according to a data block allocation request.

[0098] Receive index information of a target data block and index information of at least one backup data block returned by a metadata node, wherein the index information of the data block includes a location of the data block in the distributed system.

[0099] In a specific implementation provided in the present specification, the index information of the target data block C1 is "data node 11, data block C1", the index information of the backup data block C2 is "data node 12, data block C2", the index information of the backup data block C3 is "data node 13, data block C3", and the index information of the backup data block C4 is "data node 14, data block C4".

[0100] Step 104: writing the unit data in the data to be written into the target data block, and monitoring the response time of executing the unit data writing operation, wherein the target data block is determined according to the index information of the target data block.

[0101] The current target data block of the write request can be determined based on the index information of the data block returned by the metadata node. In response to the write request for the data to be written, an operation of writing the data to be written to the target data block is performed. During the write operation, the data to be written is written to the target data block in the form of a data stream, and the unit data in the data to be written is written to the target data block in sequence. After the unit data write is successfully executed, response information will be returned.

[0102] In a specific implementation provided in the present specification, the data to be written is an audio file A.mp3, the size of each unit data is 16K, and in response to a data streaming write request for the audio file A.mp3, the unit data in A.mp3 are written to the target data block in sequence, and the response time of executing the unit data write operation is monitored.

[0103] In another specific implementation manner provided in the present specification, the data to be written is a video file B.avi, the size of each unit data is 32K, and in response to a data streaming write request of the video file B.avi, the unit data in B.avi are written to the target data block in sequence, and the response time of executing the unit data write operation is monitored.

[0104] The response time of executing a unit data write operation is the time interval between the time when the unit data write operation is executed and the time when the response information returned after the unit data is successfully written is received. The time starts from the time when the unit data starts to be written and ends when the response information of the successful writing of the unit data is received. The time counted is the response time of the unit data. If no response information is received, the response time is the difference between the current time and the time when the unit data starts to be written.

[0105] It should be noted that the response information of successful writing of unit data is the response information of successfully writing the unit data into the target data block and the replica corresponding to the target data block, that is, the response information is returned only after the unit data is written into the target data block and the replica corresponding to the target data block. For example, the 628th unit data is successfully written into the target data block C1 and the replica C1 corresponding to the target data block C1. 11 、Copy C 12 and copy C 13 After that, a response message indicating that the 628th unit of data has been written successfully is returned. If any target data block or replica has not been written, no response message can be returned.

[0106] In a specific implementation provided in this specification, for ease of understanding, the time when the 99th unit data starts to be written is recorded as 0 microseconds, and the time when the response information of the successful writing of the 99th unit data is received is 10 microseconds, then the response time of the 99th unit data is 10 microseconds.

[0107] In a specific implementation provided in this specification, the time when the 7th unit data starts to be written is recorded as 0 microseconds. If no response information indicating successful writing is received, the write operation of the 7th unit data cannot be responded to, and the response time continues to increase with time.

[0108] Step 106: When the response time is greater than a preset threshold, stop the write operation, determine a new target data block from the at least one backup data block according to the data block identifier, and continue to execute the write operation of writing the unit data in the data to be written to the target data block, wherein the backup data block is determined according to the index information of the backup data block.

[0109] When the response time is greater than the preset threshold, it indicates that the write operation of writing the unit data to the target data block encounters an exception and the corresponding response information cannot be obtained.

[0110] In a specific implementation provided in this specification, the preset threshold is 20 milliseconds, and the 79th unit data does not receive a response message within 20 milliseconds. Therefore, it can be determined that the response time of the 79th unit data is greater than the preset threshold, and an abnormality occurs in the write operation of the 79th unit data, resulting in a request glitch.

[0111] In the event of a request glitch, the write operation is stopped, a new target data block is determined from the at least one backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued, wherein the backup data block is determined according to the index information of the backup data block.

[0112] In a specific implementation provided in the present specification, taking the 169th unit data as an example, when executing the writing of the 169th unit data to the target data block C1, if the response time of the 169th unit data is greater than a preset threshold, the writing operation is stopped, and among the three backup data blocks C2, C3, and C4, according to the order of the data block identifiers, the backup data block C2 with the data block identifier 2 is determined as the new target data block, and the writing operation of writing the unit data in the data to be written to the target data block is continued from the 169th unit data.

[0113] In another specific implementation manner provided in the present specification, taking the 870th unit data as an example, when the 870th unit data is written to the target data block C1, if the response time of the 870th unit data is greater than a preset threshold, the write operation is stopped, and the backup data block C2 with a data block identifier of 2 is determined as a new target data block according to the order of the data block identifiers among the three backup data blocks C2, C3, and C4, and the write operation of writing the unit data in the data to be written to the target data block C2 is continued from the 870th unit data; when the 1957th unit data is written to the target data block C2, if the response time of the 1957th unit data is greater than a preset threshold, the write operation is stopped, and the backup data block C3 with a data block identifier of 3 is determined as a new target data block according to the order of the data block identifiers among the two backup data blocks C3 and C4, and the write operation of writing the unit data in the data to be written to the target data block C3 is continued from the 1957th unit data.

[0114] The data writing method in a distributed system provided in this embodiment is applied to a client, and sends a data block allocation request to a metadata node in the distributed system according to a received write request for the data to be written, and receives index information of a target data block and index information of at least one backup data block returned by the metadata node; executes a write operation of writing unit data in the data to be written to the target data block, and monitors the response time of executing the unit data writing operation. In the case where a request glitch occurs during the write execution, it indicates that an abnormality occurs in the data node corresponding to the current target data block. At this time, a new target data block is determined in the backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued. Since data blocks are usually located in different data nodes, the write operation of abnormal nodes can be avoided, reducing request delays, thereby solving the problem of request glitches in write requests, maintaining the normal operation of the distributed system, and improving the user experience.

[0115] The following combination Figure 2 Taking the application of the data writing method in the distributed system provided in this specification in writing the video file B into the distributed system as an example, the data writing method in the distributed system applied to the client is further described. Figure 2 A processing flow chart of a method for writing data in a distributed system provided in an embodiment of the present specification is shown, and the specific steps include step 202 to step 212.

[0116] Step 202: Send a data block allocation request to a metadata node in a distributed system according to the received write request, and receive index information of a target data block and index information of at least one backup data block returned by the metadata node.

[0117] In the embodiment provided in the present specification, the data to be written is a video file B, a data streaming write request to write the video file B to a distributed system is received, and a data block allocation request is sent to a metadata node in the distributed system according to the write request, so that the metadata node allocates a target data block C1 and backup data blocks C2, C3, C4, C5, and C6 according to the data block allocation request, and returns index information of the target data block C1 and index information of backup data blocks C2, C3, C4, C5, and C6, and receives the index information of the target data block and the index information of the backup data block returned by the metadata node.

[0118] Step 204: Write the unit data in the data to be written into the target data block, and monitor the response time of executing the unit data writing operation, wherein the target data block is determined according to the index information of the target data block.

[0119] In the embodiment provided in the present specification, the position of the target data block C1 in the distributed system is determined according to the index information of the target data block C1, and in response to a write request, a write operation is performed to write the unit data in the video file B to the target data block C1, the size of each unit data is 32K, and the response time of executing the unit data write operation is monitored.

[0120] Step 206: Determine whether the response time is greater than a preset threshold, if so, execute step 208, if not, execute step 204.

[0121] In the embodiment provided in the present specification, the preset threshold is 10 microseconds, and the first unit data is written to the target data block C1. The response time of the first unit data is 8 microseconds, which is not greater than the preset threshold. Then, step 204 is executed to continue the operation of writing the second unit data to the target data block C1, and monitor the response time of the second unit data write operation.

[0122] Taking the 1524th unit data as an example, the 1524th unit data is written to the target data block C1. If the time the client waits for receiving the response information exceeds 10 microseconds, that is, greater than the preset threshold, step 208 is executed.

[0123] Step 208: Determine whether the flow control state in the distributed system is to execute flow control, if so, execute step 210, otherwise execute step 212.

[0124] The flow control state is the flow control for the number of times new target data blocks are determined in the distributed system. In order to prevent a large number of small-capacity data blocks from appearing in the distributed system due to frequent determination of new target data blocks, flow control for determining the number of new target data blocks is set. For example, in a distributed system, only 10 new target data blocks can be determined per second. If the number of times new target data blocks are determined in the distributed system exceeds 10 times within 1 second, flow control is executed. If the number of times new target data blocks are determined in the distributed system is less than 10 times within 1 second, flow control is released.

[0125] Determine whether the flow control state in the distributed system is to execute flow control. If the flow control state in the current distributed system is to execute flow control, execute step 210; if the flow control state in the current distributed system is to release flow control, execute step 212.

[0126] Step 210: Wait for the flow control to end or for response information to be returned. If response information is returned, execute step 204; if flow control ends, execute step 212.

[0127] When the flow control state is to execute flow control, the waiting state is entered, and the flow control is waited for to end, that is, the waiting state is to release the flow control or receive the response information of the unit data of the write operation.

[0128] When the response information of the unit data of the write operation is received, step 204 is executed.

[0129] When no response information of the unit data for executing the write operation is received and the flow control state is to release the flow control, step 212 is executed.

[0130] Step 212: stop the write operation, determine a new target data block from the at least one backup data block according to the data block identifier, and execute step 204.

[0131] In the embodiments provided in the present specification, taking the 1524th unit data as an example, if the response time when executing the write operation of writing the 1524th unit data to the target data block C1 is greater than a preset threshold, the write operation is stopped, and the backup data block C2 is determined as the new target data block from the backup data blocks C2, C3, C4, C5, and C6 according to the order of the data block identifiers, and the position of the target data block C2 in the distributed system is determined according to the index information of the target data block C2, and the write operation of writing the 1524th unit data to the target data block C2 is continued.

[0132] Optionally, before determining a new target data block from the at least one backup data block based on the data block identifier, the method further includes: generating data block information of the target data block based on index information of the target data block and the size of the data written in the target data block; and uploading the data block information and the data identifier of the data to be written to the metadata node.

[0133] In the embodiment provided in the present specification, taking the 1524th unit data as an example, before determining a new target data block from the at least one backup data block according to the data block identifier, there are 1523 units of data written to the target data block C1, and the size of each unit of data is 32K. Then the size of the written data in the target data block C1 is 48736K. The data block information of the target data block C1 is generated according to the data block index information of the target data block C1 and the size of the written data, 48736K, and the data block information is uploaded to the metadata node.

[0134] Optionally, the method further includes:

[0135] When the number of backup data blocks is less than a preset threshold, a request for adding a backup data block is sent to the metadata node; and index information of the newly added backup data block is received from the metadata node.

[0136] After each time a new target data block is determined from the at least one backup data block based on the data block identifier, the number of backup data blocks will decrease. When the number of backup data blocks is less than a preset threshold, it is necessary to send a backup data block addition request to the metadata node to prevent the situation where there are no available backup data blocks when a new target data block needs to be determined. The index information of the newly added backup data blocks is returned by the receiving metadata node in response to the backup data block addition request.

[0137] In a specific implementation provided in the present specification, taking the preset threshold of 3 as an example, when the number of backup data blocks is less than the preset threshold of 3, a backup data block addition request is sent to the metadata node, and index information of the newly added backup data blocks returned by the metadata node in response to the backup data block addition request is received.

[0138] Optionally, the method further includes:

[0139] A data block acquisition request is sent to the metadata node according to a received read request of the file to be read, and data block information and a data block identifier of at least one read data block corresponding to the file to be read returned by the metadata node are received, wherein the data block information of the read data block includes index information of the data block and the size of the data written in the data block; the position of each read data block is determined according to the index information of each read data block; and the file to be read is read according to the data block identifier of each read data block and the size of the data written in the data block.

[0140] In the embodiment provided in the present specification, a read request for video file B is obtained, and a data block acquisition request is sent to the metadata node based on the received read request, and the receiving metadata node determines and returns data block information of read data block C1 and data block information of read data block C2 based on the video file name B in the read request.

[0141] The read data block C1 and the read data block C2 are determined according to the index information of the read data block C1 and the index information of the read data block C2.

[0142] The reading order of the read data blocks is determined according to the data block identification sequence C1 to C2 of each read data block, and the data written in each read data block is read in sequence according to the size of the data written in each read data block to complete the operation of reading video file B.

[0143] The data writing method in a distributed system provided in this embodiment is applied to a client, and sends a data block allocation request to a metadata node in the distributed system according to a received write request for the data to be written, and receives index information of a target data block and index information of at least one backup data block returned by the metadata node; executes a write operation of writing unit data in the data to be written to the target data block, and monitors the response time of executing the unit data writing operation. In the case where a request glitch occurs during the write execution, it indicates that an abnormality occurs in the data node corresponding to the current target data block. At this time, a new target data block is determined in the backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued. Since data blocks are usually located in different data nodes, the write operation of abnormal nodes can be avoided, reducing request delays, thereby solving the problem of request glitches in write requests, maintaining the normal operation of the distributed system, and improving the user experience.

[0144] Secondly, setting flow control for the number of times new target data blocks are determined in the distributed system of this method can avoid the problem of a surge in the overall latency of the distributed system caused by the operation of large-scale determination of new target data blocks in the distributed system under abnormal circumstances.

[0145] Finally, when reading a file, the corresponding reading order and the size of the written data in each data block are determined according to the data block information and data block identifiers stored in the metadata node, thereby implementing the reading operation of the file to be read.

[0146] Figure 3 A processing flow chart of a method for writing data in a distributed system provided according to an embodiment of the present specification is shown. The method is applied to a metadata node and includes steps 302 to 306.

[0147] Step 302: Receive a data block allocation request.

[0148] Receive the data block allocation request sent by the client.

[0149] Step 304: Allocate a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier.

[0150] In response to the data block allocation request, a target data block and at least one backup data block are allocated, and the number of backup data blocks is determined by a preset threshold. The target data block is a data block for writing the data to be written, and the backup data block is configured by the metadata node for the target data block according to the data block allocation request, and a data block identifier is set for each data block, and the data block identifier is used to record the order of the data blocks.

[0151] In a specific implementation provided in this specification, a target data block C1 and three backup data blocks C2, C3, and C4 are allocated according to a data block allocation request, where 1, 2, 3, and 4 are data block identifiers of each data block.

[0152] Step 306: Return the index information of the target data block and the index information of the at least one backup data block.

[0153] The index information of the target data block and the index information of the at least one backup data block are returned to the client.

[0154] In a specific implementation provided in this specification, the above example is used to return the index information of the target data block C1 and the index information of the three backup data blocks C2, C3, and C4 to the client.

[0155] Optionally, the method further includes: receiving data block information and a data identifier of the target data block, and generating a data block information table based on the data block information and the data identifier, wherein the data block information table includes the data identifier, the data block identifier and the data block information of the target data block, and the data block information includes the index information of the data block and the size of the data written in the data block.

[0156] The data block information table is used to store the data identifier, data block identifier and data block information corresponding to the completed written data. The data block information includes the index information of the data block and the size of the data written in the data block, which is used to facilitate the query of the data block information of each data block during subsequent data reading.

[0157] In a specific implementation provided in this specification, data block information of data block C1 sent by the client is received and saved, and the data block information includes index information of data block C1 and the size of data written in data block C1 is 48736K.

[0158] Optionally, the method further includes: receiving a request for adding a backup data block; allocating a new backup data block in response to the request for adding a backup data block, wherein the new backup data block includes a data block identifier; and returning index information of the new backup data block.

[0159] A backup data block addition request sent by a client is received, a new data block is allocated as a newly added backup data block in response to the backup data block addition request, and index information of the newly added backup data block is returned to the client.

[0160] In a specific implementation manner provided in the present specification, a request for adding a backup data block is received from a client, and a metadata node allocates a new backup data block in response to the request for adding a backup data block. The number of new backup data blocks may be a preset number. For example, if the preset value is 3, 3 data blocks are allocated as new backup data blocks in response to the request for adding a backup data block. The number of new backup data blocks may also be determined based on the number of backup data blocks allocated by the data block allocation request. For example, if 1 write data block and 6 backup data blocks are allocated according to the data block allocation request, the number of backup data blocks is increased to 6 in response to the request for adding a backup data block. In the present specification, the method of allocating new data blocks as backup data blocks is not limited.

[0161] Optionally, the method further includes: receiving a data block acquisition request, wherein the data block acquisition request includes a file identifier to be read; determining, according to the data block information table, at least one data block information corresponding to the file identifier to be read as read data block information; and returning each of the read data block information and the data block identifier corresponding to each of the read data block information.

[0162] The data block acquisition request includes a file identifier to be read. According to the file identifier to be read, the data block information table is traversed to determine at least one data block information corresponding to the file identifier to be read as the read data block information, and each read data block information and the data block identifier corresponding to each read data block information are returned to the client.

[0163] In a specific implementation manner provided in the present specification, a data block acquisition request for a file B to be read is received, a data block information table is traversed according to the file name B of the file to be read in the data block acquisition request, data block C1 information and data block C2 information are determined as read data block information, and the read data block C1 information, the read data block C2 information and the data block identifiers corresponding to the two read data block information are returned to the client.

[0164] The data writing method in a distributed system provided in this embodiment is applied to a metadata node. The metadata node allocates multiple data blocks according to a data block allocation request, uses one of the data blocks as a target data block, uses the remaining data blocks as backup data blocks, sets a data block identifier for each data block, determines the order in which the data blocks are used, and returns the index information of the target data block and the index information of each backup data block to the client, thereby saving the time for allocating data blocks, improving the efficiency of allocating data blocks, and solving the problem of request glitches.

[0165] Secondly, when a request for adding a backup data block is received, a new data block is allocated as a backup data block in case no backup data block is available.

[0166] Thirdly, the data block information and data identifier of the received target data block are saved to facilitate querying the size of the written data of each data block during subsequent data reading.

[0167] Combine the following Figure 4 and Figure 5 , taking the application of writing file M into the distributed system by the method for writing data in the distributed system provided in this specification as an example, the method provided in this specification is further explained. Figure 4 The structure diagram of the distributed system provided in an embodiment of the present specification is shown. The distributed system includes 1 client, 1 metadata node and 4 data nodes. Figure 5 This is an interaction sequence diagram between a client and a metadata node provided in an embodiment of this specification, and the method includes steps 502 to 538.

[0168] Step 502: The client receives a write request.

[0169] In the embodiment provided in this specification, the client receives a write request for a file M to be written.

[0170] Step 504: The client sends a data block allocation request to the metadata node.

[0171] In the embodiment provided in this specification, the client sends a data block allocation request to the metadata node according to the write request.

[0172] Step 506: The metadata node allocates a target data block and at least one backup data block.

[0173] In the embodiment provided in this specification, the metadata node allocates data blocks 1 to 4 in data nodes 1 to 4 respectively according to the data block allocation request, where data block 1 is the target data block, and data blocks 2, 3, and 4 are backup data blocks.

[0174] Step 508: The metadata node returns the index information of the target data block and the index information of at least one backup data block to the client.

[0175] In the embodiment provided in this specification, the metadata node returns the index information of the target data block 1 and the index information of the backup data blocks 2, 3, and 4 to the client.

[0176] Step 510: The client performs a write operation and monitors the response time.

[0177] In the embodiment provided in this specification, an operation of writing unit data to be written in file M to target data block 1 is performed, the size of each unit data is 16K, and the response time of executing the unit data writing operation is monitored.

[0178] Step 512: The client determines whether the response time is greater than a preset threshold, and if so, executes step 514.

[0179] Step 514: The client determines whether the current flow control state in the distributed system is to execute flow control. If not, execute step 518.

[0180] Step 518 : The client stops the write operation, generates target data block information, and executes step 520 .

[0181] Step 520: The client sends the target data block information to the metadata node.

[0182] Step 522: The metadata node saves the target data block information.

[0183] Step 524: The metadata node returns the save success information to the client.

[0184] Step 526 : The client determines a new target data block and executes step 510 .

[0185] In the embodiments provided in this specification, the 1653rd unit data is taken as an example to explain in detail the processing details when the response time is greater than the preset threshold and the current flow control state is to release the flow control. When executing the write operation of writing the 1653th unit data to the target data block 1, the response time is greater than the preset threshold, and the flow control state of the new target data block determined in the distributed system at this time is to release the flow control, stop the write operation, and according to the total number of successfully written unit data of 1652, the size of each unit data is 16K, then the size of the target data block 1 is 16432K, generate the data block information corresponding to the target data block 1, and send the data block information to the metadata node.

[0186] The metadata node saves the data block information corresponding to data block 1 and returns the saving success information to the client.

[0187] The client determines backup data block 2 as the new target data block from backup data blocks 2, 3, and 4 according to the data block identification sequence, and continues to execute the write operation of writing the 1653th unit data of the file to be written to target data block 2.

[0188] Step 512: The client determines whether the response time is greater than a preset threshold, and if so, executes step 514.

[0189] Step 514: The client determines whether the flow control state in the distributed system is to execute flow control. If so, execute step 516.

[0190] Step 516: The client enters a waiting state, waiting for the flow control to be released or receiving a response message. If a response message is received, step 510 is executed.

[0191] In the embodiment provided in this specification, the processing details of receiving the response information when the client enters the waiting state are explained in detail by taking the 2598th unit data as an example. When the 2598th unit data is written to the target data block 2, the response time is greater than the preset threshold, and the flow control state of the number of times the new target data block is determined in the distributed system at this time is to execute flow control, and the new target data block cannot be determined, then the waiting state is entered, and when the flow control state is still to execute flow control, the response information of the 2598th unit data is received, and the write operation of writing the unit data in the data to be written to the target data block 2 is continued.

[0192] Step 512: The client determines whether the response time is greater than a preset threshold, and if so, executes step 514.

[0193] Step 514: The client determines whether the flow control state in the distributed system is to execute flow control. If so, execute step 516.

[0194] Step 516: The client enters a waiting state, waiting for the flow control to be released or receiving a response message. If the flow control is released, step 518 is executed.

[0195] Step 518 : The client stops the write operation, generates target data block information, and executes step 520 .

[0196] Step 520: The client sends the target data block information to the metadata node.

[0197] Step 522: The metadata node saves the target data block information.

[0198] Step 524: The metadata node returns the save success information to the client.

[0199] Step 526 : The client determines a new target data block and executes step 510 .

[0200] Step 534: The client sends a request to add a backup data block.

[0201] Step 536: The metadata node allocates a new data block as a backup data block according to the backup data block addition request.

[0202] Step 538: The metadata node returns the index information of the newly added backup data block to the client.

[0203] In the embodiment provided in this specification, the processing details of releasing the flow control when the client enters the waiting state are explained in detail by taking the 3652th unit data as an example. When the 3652th unit data is written to the target data block 2, the response time is greater than the preset threshold, and the flow control state of the number of times the new target data block is determined in the distributed system at this time is to execute flow control, and the new target data block cannot be determined, then the waiting state is entered, and when the flow control state is to release the flow control, the response information of the 3652th unit data is still not received, then the write operation of writing the 3652th unit data to the target data block 2 is stopped, and the backup data block 3 is determined as the new target data block from the backup data blocks 3 and 4 according to the order of the data block identifiers, and the write operation of writing the file to be written from the 3652th unit data to the target data block 3 is continued.

[0204] At this time, the number of backup data blocks is 4, totaling 1 backup data block, which is less than the preset minimum number of backup data blocks 2. The client asynchronously sends a backup data block addition request to the metadata node. The metadata node allocates data blocks 5, 6, and 7 as new backup data blocks in response to the backup data block addition request, and returns the index information of the new backup data blocks 5, 6, and 7 to the client. At this time, the backup data blocks are 4, 5, 6, and 7, totaling 4 backup data blocks.

[0205] Optionally, when the capacity of the target data block is full, step 518 is executed.

[0206] In the embodiment provided in this specification, the processing details of the target data block capacity being full are explained in detail by taking the 7748th unit data as an example. When writing to the 7748th unit data, the capacity of the target data block 3 is full, and the client cannot return the response information when writing the 7749th unit data, resulting in the response time of the 7749th unit data being greater than the preset threshold. The client stops the write operation of the 7749th unit data, and the size of the data written in the target data block 3 is 65536K. The data block information corresponding to the target data block 3 is generated and sent to the metadata node.

[0207] The metadata node saves the data block information corresponding to data block 3, and returns the information that data block 3 has been successfully saved to the client.

[0208] The client determines backup data block 4 as the new target data block from backup data blocks 4, 5, 6, and 7 according to the order of the data block identifiers, and continues to execute the write operation of writing the file to be written from the 7749th unit data to target data block 4.

[0209] Step 528: The client receives the write completion instruction.

[0210] Step 530: The client sends the target data block information to the metadata node.

[0211] Step 532: The metadata node saves the target data block information.

[0212] In the embodiment provided in this specification, the client receives a write completion instruction, at which time the size of the data written in the target data block 4 is 18480K, generates data block information corresponding to the target data block 4, and sends the data block information to the metadata node.

[0213] The metadata node stores the data block information corresponding to the data block 4. At this point, the operation of writing the file M to be written into the distributed system has been completed. At this time, the metadata node stores 4 data block information corresponding to the file M to be written.

[0214] The data writing method in a distributed system provided in the present embodiment sends a data block allocation request to a metadata node in the distributed system according to a received write request for data to be written, and receives index information of a target data block and index information of at least one backup data block returned by the metadata node; executes a write operation of writing unit data in the data to be written to the target data block, and monitors the response time of executing the unit data writing operation. In the event that a request glitch occurs during the writing process, a new target data block is determined in the backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued. Since data blocks are usually located in different data nodes, the write operation of abnormal nodes can be avoided, reducing request delays, thereby solving the problem of request glitches in write requests, maintaining the normal operation of the distributed system, and improving the user experience.

[0215] Secondly, setting flow control for the number of times new target data blocks are determined in the distributed system of this method can avoid the problem of a surge in the overall latency of the distributed system caused by the operation of large-scale determination of new target data blocks in the distributed system under abnormal circumstances.

[0216] Thirdly, when the number of backup data blocks is less than a preset threshold, a new data block is applied for as a backup data block, and when the backup data block is determined to be a new target data block, no backup data block is available.

[0217] Finally, the metadata node saves the data block information and data identifier of the received target data block, so as to facilitate the query of the size of the written data of each data block during the subsequent data reading.

[0218] Corresponding to the above-mentioned data writing method embodiment in the distributed system applied to the client, this specification also provides a data writing device embodiment in the distributed system, which is configured on the client. Figure 6 FIG. 1 shows a schematic diagram of a data writing device configured in a distributed system of a client provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:

[0219] The sending module 602 is configured to send a data block allocation request to a metadata node in the distributed system according to a received write request, and receive index information of a target data block and index information of at least one backup data block returned by the metadata node.

[0220] The writing module 604 is configured to execute a writing operation of writing unit data in the data to be written into a target data block, and monitor a response time of executing the writing operation of the unit data, wherein the target data block is determined according to the index information of the target data block.

[0221] The switching module 606 is configured to stop the write operation, determine a new target data block from the at least one backup data block according to the data block identifier, and continue to perform the write operation of writing the unit data in the data to be written to the target data block, wherein the backup data block is determined according to the index information of the backup data block.

[0222] Optionally, the device further comprises:

[0223] The first generation upload module generates data block information of the target data block according to the index information of the target data block and the size of the data written in the target data block; and uploads the data block information and the data identifier of the data to be written to the metadata node.

[0224] Optionally, the device further comprises:

[0225] The second generation upload module is configured to receive a write completion instruction and end the write operation in response to the write completion instruction; generate data block information of the target data block based on the index information of the target data block and the size of the data written in the target data block; and upload the data block information and the data identifier of the data to be written to the metadata node.

[0226] Optionally, the switching module 606 is further configured to obtain the flow control status of the new target data block determined in the distributed system, wherein the flow control status is to execute flow control or to release flow control; when the flow control status is to execute flow control, enter a waiting state; when the flow control status is to release flow control, perform a stop write operation.

[0227] Optionally, the switching module 606 is further configured to receive response information of unit data and obtain flow control status; when the response information of unit data is received, execute a write operation to write the unit data in the data to be written to the target data block; when no response information of unit data is received and the flow control status is to release the flow control, stop the write operation.

[0228] Optionally, the device further comprises:

[0229] The new request module is configured to send a backup data block addition request to the metadata node when the number of backup data blocks is less than a preset threshold; and receive index information of the newly added backup data block returned by the metadata node.

[0230] Optionally, the device further comprises:

[0231] A reading module is configured to send a data block acquisition request to the metadata node according to a received read request of a file to be read and receive data block information and a data block identifier of at least one read data block corresponding to the file to be read returned by the metadata node, wherein the data block information of the read data block includes index information of the data block and the size of data written in the data block; determine the position of each read data block according to the index information of each read data block; and read the file to be read according to the data block identifier of each read data block and the size of data written in the data block.

[0232] The data writing device in the distributed system provided by the present embodiment is configured on the client, sends a data block allocation request to the metadata node in the distributed system according to the received write request of the data to be written, and receives the index information of the target data block and the index information of at least one backup data block returned by the metadata node; executes the write operation of writing the unit data in the data to be written to the target data block, and monitors the response time of executing the unit data writing operation. In the case that a request glitch occurs during the writing process, it indicates that the data node corresponding to the current target data block is abnormal. At this time, a new target data block is determined in the backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued. Since the data blocks are usually located in different data nodes, the write operation of the abnormal node can be avoided, reducing the request delay, thereby solving the problem of request glitches in the write request, maintaining the normal operation of the distributed system, and improving the user experience.

[0233] Secondly, setting flow control can avoid the problem of a surge in the overall latency of the distributed system caused by the operation of large-scale determination of new target data blocks in the distributed system under abnormal circumstances.

[0234] Finally, a reading module is set up. When reading a file, the corresponding reading order and the size of the written data in each data block are determined according to the data block information and data block identifier stored in the metadata node, so as to implement the reading operation of the file to be read.

[0235] The above is a schematic scheme of a data writing device configured in a distributed system of a client in this embodiment. It should be noted that the technical scheme of the data writing device in the distributed system and the technical scheme of the data writing method in the distributed system applied to the client belong to the same concept, and the details of the technical scheme of the data writing device configured in the distributed system of the client that are not described in detail can all be referred to the description of the technical scheme of the data writing method in the distributed system applied to the client.

[0236] Corresponding to the above-mentioned data writing method embodiment in a distributed system applied to a metadata node, this specification also provides an embodiment of a data writing device in a distributed system, which is configured in a metadata node. Figure 7 FIG. 1 shows a schematic diagram of a data writing device configured in a distributed system of a metadata node provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0237] The receiving module 702 is configured to receive a data block allocation request.

[0238] The allocation module 704 is configured to allocate a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier.

[0239] The returning module 706 is configured to return the index information of the target data block and the index information of the at least one backup data block.

[0240] Optionally, the device further comprises:

[0241] The data block information receiving module is configured to receive the data block information and data identifier of the target data block, and generate a data block information table according to the data block information and data identifier, wherein the data block information table includes the data identifier, data block identifier and data block information of the target data block, and the data block information includes the index information of the data block and the size of the data written in the data block.

[0242] Optionally, the device further comprises:

[0243] The reallocation module is configured to receive a request for adding a backup data block; allocate a new backup data block in response to the request for adding a backup data block, wherein the new backup data block includes a data block identifier; and return index information of the new backup data block.

[0244] Optionally, the device further comprises:

[0245] The read data block determination module is configured to receive a data block acquisition request, wherein the data block acquisition request includes a file identifier to be read; determine at least one data block information corresponding to the file identifier to be read as read data block information according to the data block information table; and return each of the read data block information and the data block identifier corresponding to each of the read data block information.

[0246] The data writing device in the distributed system provided by the present embodiment is configured in the metadata node. The metadata node allocates multiple data blocks according to the data block allocation request, uses one of the data blocks as the target data block, uses the remaining data blocks as backup data blocks, sets a data block identifier for each data block, determines the order in which the data blocks are used, and returns the index information of the target data block and the index information of each backup data block to the client, thereby saving the time for allocating data blocks, improving the efficiency of allocating data blocks, and solving the problem of request glitches.

[0247] Secondly, when a request for adding a backup data block is received, a new data block is allocated as a backup data block in case no backup data block is available.

[0248] Thirdly, the data block information and data identifier of the received target data block are saved to facilitate querying the size of the written data of each data block during subsequent data reading.

[0249] The above is a schematic scheme of a data writing device configured in a distributed system of a metadata node in this embodiment. It should be noted that the technical scheme of the data writing device in the distributed system and the technical scheme of the data writing method in the distributed system applied to the metadata node belong to the same concept, and the details of the technical scheme of the data writing device in the distributed system configured in the metadata node that are not described in detail can all be referred to the description of the technical scheme of the data writing method in the distributed system applied to the metadata node.

[0250] Figure 8 A distributed system provided by an embodiment of this specification is shown, and the distributed system includes:

[0251] The client 802 is configured to send a data block allocation request to a metadata node in the distributed system according to the received write request.

[0252] The metadata node 804 is configured to receive the data block allocation request, allocate a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier, and return index information of the target data block and index information of the at least one backup data block.

[0253] The client 802 is further configured to execute a write operation of writing unit data in the data to be written to a target data block, and monitor a response time of executing the unit data write operation, wherein the target data block is determined based on index information of the target data block, and when the response time is greater than a preset threshold, stop the write operation, determine a new target data block from the at least one backup data block based on a data block identifier, and continue to execute the write operation of writing unit data in the data to be written to the target data block, wherein the backup data block is determined based on the index information of the backup data block.

[0254] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0255] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0256] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0257] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 900 may also be a mobile or stationary server.

[0258] The memory 910 is used to store computer executable instructions, and the processor 920 is used to execute the following computer executable instructions:

[0259] Sending a data block allocation request to a metadata node in the distributed system according to the received write request, and receiving index information of a target data block and index information of at least one backup data block returned by the metadata node;

[0260] Writing unit data in the data to be written into a target data block, and monitoring the response time of executing the unit data writing operation, wherein the target data block is determined according to the index information of the target data block;

[0261] When the response time is greater than a preset threshold, the write operation is stopped, a new target data block is determined from the at least one backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued, wherein the backup data block is determined according to the index information of the backup data block.

[0262] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of writing data in the distributed system applied to the client are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the method of writing data in the distributed system applied to the client.

[0263] Fig.10 The block diagram of the structure of a computing device 1000 according to an embodiment of the present application is shown. The components of the computing device 1000 include but are not limited to a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1050 is used to store data.

[0264] Computing device 1000 also includes an access device 1040 that enables computing device 1000 to communicate via one or more networks 1060 .

[0265] The access method of the computing device 1000 is the same as that of the computing device 900 described above, and will not be repeated here.

[0266] The processor 1020 is used to execute the following computer executable instructions:

[0267] receiving a data block allocation request;

[0268] Allocating a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier;

[0269] The index information of the target data block and the index information of the at least one backup data block are returned.

[0270] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of writing data in the distributed system applied to metadata nodes are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the method of writing data in the distributed system applied to metadata nodes.

[0271] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which are used to implement the steps of the data writing method in the distributed system when executed by a processor.

[0272] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the data writing method in the above-mentioned distributed system belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the data writing method in the above-mentioned distributed system.

[0273] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0274] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0275] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0276] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0277] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for writing data in a distributed system, applied to a client, comprising: Sending a data block allocation request to a metadata node in the distributed system according to the received write request, and receiving index information of a target data block and index information of at least one backup data block returned by the metadata node; Writing unit data in the data to be written into a target data block, and monitoring the response time of executing the unit data writing operation, wherein the target data block is determined according to the index information of the target data block; When the response time is greater than a preset threshold, obtaining a flow control state of determining a new target data block in the distributed system, wherein the flow control state is executing flow control or releasing flow control; When the flow control state is to execute flow control, entering a waiting state, wherein the waiting state includes waiting for the flow control to end or waiting for response information to be returned; When the flow control state is to release the flow control, the write operation is stopped, a new target data block is determined from the at least one backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued, wherein the backup data block is determined according to the index information of the backup data block.

2. The method for writing data in a distributed system according to claim 1, before determining a new target data block from the at least one backup data block according to the data block identifier, further comprising: Generate data block information of the target data block according to the index information of the target data block and the size of the data written in the target data block; The data block information and the data identifier of the data to be written are uploaded to the metadata node.

3. The method for writing data in a distributed system according to claim 1, further comprising: receiving a write completion instruction, and ending the write operation in response to the write completion instruction; Generate data block information of the target data block according to the index information of the target data block and the size of the data written in the target data block; The data block information and the data identifier of the data to be written are uploaded to the metadata node.

4. The method for writing data in a distributed system according to claim 1, after entering the waiting state, further comprising: Receive response information of unit data and obtain flow control status; When receiving the response information of the unit data, executing a write operation of writing the unit data in the data to be written into the target data block; If no response information of the unit data is received and the flow control state is to release the flow control, the write operation is stopped.

5. The method for writing data in a distributed system according to claim 1, further comprising: When the number of backup data blocks is less than a preset threshold, sending a request for adding a backup data block to the metadata node; Receive the index information of the newly added backup data block returned by the metadata node.

6. The method for writing data in a distributed system according to claim 1, further comprising: Sending a data block acquisition request to the metadata node according to the received read request of the file to be read and receiving data block information and a data block identifier of at least one read data block corresponding to the file to be read returned by the metadata node, wherein the data block information of the read data block includes index information of the data block and the size of the data written in the data block; Determine the position of each read data block according to the index information of each read data block; The to-be-read file is read according to the data block identifier of each read data block and the size of the data written in the data block.

7. A method for writing data in a distributed system, applied to a metadata node, comprising: receiving a data block allocation request; Allocating a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier; Returning index information of the target data block and index information of the at least one backup data block, wherein the backup data block is used to provide a new target data block for the client, and the new target data block is determined from the at least one backup data block according to a data block identifier when the client stops the write operation when the response time of the unit data write operation is greater than a preset threshold, wherein the response time of the unit data write operation is monitored by the client; Before stopping writing, it also includes: the client obtains the flow control status of the new target data block determined in the distributed system, wherein the flow control status is to execute flow control or to release flow control; when the flow control status is to execute flow control, enter a waiting state, and the waiting state includes waiting for the flow control to end or waiting for the response information to be returned; when the flow control status is to release flow control, stop the write operation.

8. The method for writing data in a distributed system according to claim 7, further comprising: Receive data block information and data identifier of the target data block, and generate a data block information table according to the data block information and data identifier, wherein the data block information table includes the data identifier, data block identifier and data block information of the target data block, and the data block information includes index information of the data block and the size of the data written in the data block.

9. The method for writing data in a distributed system according to claim 7, further comprising: Receive a request to add a backup data block; Allocating a new backup data block in response to the backup data block addition request, wherein the new backup data block includes a data block identifier; Returns the index information of the newly added backup data block.

10. The method for writing data in a distributed system according to claim 8, further comprising: Receiving a data block acquisition request, wherein the data block acquisition request includes a file identifier to be read; Determine, according to the data block information table, at least one piece of data block information corresponding to the to-be-read file identifier as read data block information; Return each piece of read data block information and a data block identifier corresponding to each piece of read data block information.

11. A data writing device in a distributed system, applied to a client, comprising: A sending module, configured to send a data block allocation request to a metadata node in a distributed system according to a received write request, and receive index information of a target data block and index information of at least one backup data block returned by the metadata node; A writing module is configured to write unit data in the data to be written into a target data block, and monitor a response time of executing the unit data writing operation, wherein the target data block is determined according to index information of the target data block; A switching module is configured to stop the write operation and obtain a flow control state of a new target data block in the distributed system, wherein the flow control state is to execute flow control or release flow control; When the flow control state is executing flow control, waiting for the flow control to end or waiting for response information to be returned; When the flow control state is to release the flow control, a new target data block is determined from the at least one backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued, wherein the backup data block is determined according to the index information of the backup data block.

12. A data writing device in a distributed system, applied to a metadata node, comprising: A receiving module, configured to receive a data block allocation request; an allocation module, configured to allocate a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier; a return module, configured to return index information of the target data block and index information of at least one backup data block, wherein the backup data block is used to provide a new target data block for the client, and the new target data block is determined from the at least one backup data block according to a data block identifier when the client stops the write operation when the response time of executing the unit data write operation is greater than a preset threshold, wherein the response time of the unit data write operation is monitored by the client; Before stopping writing, it also includes: the client obtains the flow control status of the new target data block determined in the distributed system, wherein the flow control status is to execute flow control or to release flow control; when the flow control status is to execute flow control, enter a waiting state, and the waiting state includes waiting for the flow control to end or waiting for the response information to be returned; when the flow control status is to release flow control, stop the write operation.

13. A distributed system comprising: The client is configured to send a data block allocation request to a metadata node in the distributed system according to the received write request; The metadata node is configured to receive the data block allocation request, allocate a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier, and return index information of the target data block and index information of the at least one backup data block; The client is further configured to write unit data in the data to be written to a target data block, and monitor a response time of executing the unit data write operation, wherein the target data block is determined according to index information of the target data block, and when the response time is greater than a preset threshold, obtain a flow control state of determining a new target data block in the distributed system, wherein the flow control state is executing flow control or releasing flow control; When the flow control state is to execute flow control, entering a waiting state, wherein the waiting state includes waiting for the flow control to end or waiting for response information to be returned; When the flow control state is to release the flow control, the write operation is stopped, a new target data block is determined from the at least one backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued, wherein the backup data block is determined according to the index information of the backup data block.

14. A computing device, applied to a client, comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: Sending a data block allocation request to a metadata node in the distributed system according to the received write request, and receiving index information of a target data block and index information of at least one backup data block returned by the metadata node; Writing unit data in the data to be written into a target data block, and monitoring the response time of executing the unit data writing operation, wherein the target data block is determined according to the index information of the target data block; When the response time is greater than a preset threshold, obtaining a flow control state of determining a new target data block in the distributed system, wherein the flow control state is executing flow control or releasing flow control; When the flow control state is executing flow control, waiting for the flow control to end or waiting for response information to be returned; When the flow control state is to release the flow control, the write operation is stopped, a new target data block is determined from the at least one backup data block according to the data block identifier, and the write operation of writing the unit data in the data to be written to the target data block is continued, wherein the backup data block is determined according to the index information of the backup data block.

15. A computing device comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: receiving a data block allocation request; Allocating a target data block and at least one backup data block in response to the data block allocation request, wherein the target data block and the at least one backup data block include a data block identifier; Returning index information of the target data block and index information of the at least one backup data block, wherein the backup data block is used to provide a new target data block for the client, and the new target data block is determined from the at least one backup data block according to a data block identifier when the client stops the write operation when the response time of the unit data write operation is greater than a preset threshold, wherein the response time of the unit data write operation is monitored by the client; Before stopping writing, it also includes: the client obtains the flow control status of the new target data block determined in the distributed system, wherein the flow control status is to execute flow control or to release flow control; when the flow control status is to execute flow control, enter a waiting state, and the waiting state includes waiting for the flow control to end or waiting for the response information to be returned; when the flow control status is to release flow control, stop the write operation.

16. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the data writing method in a distributed system as described in any one of claims 1 to 6 or 7 to 10.

17. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the data writing method according to any one of claims 1 to 6 or 7 to 10.

Images