Data processing methods, systems, devices, media, and products

Abstract

The embodiment of the application provides a data processing method, system, device, medium and product, and belongs to the field of data processing. The method comprises the following steps: a metadata server creates a user file corresponding to a file name and a plurality of internal files associated with the user file and having a set writing sequence according to a file creation request sent by a writing client. After the metadata server receives a first writing request corresponding to the user file sent by the writing client, the metadata server sends data block allocated to each of the plurality of internal files and data node information corresponding to the data block to the writing client. In this way, after the writing client successfully writes first data to be written into a first data node corresponding to a first data block in a first internal file, if the metadata server stops serving at this time, the writing client can continue to write second data to be written into a second data node corresponding to a second data block in a second internal file, thereby realizing high availability of file writing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, system, medium, and product. Background Technology

[0002] A distributed file system is a file system that allows clients to access files located in different physical locations over a network. It provides a transparent way for users to access remote files as if they were accessing a local file system, without having to care about the actual physical location of the files.

[0003] In a typical distributed file system, the main components usually include clients, a metadata server (Master), and data nodes (Chunk Servers). Clients are primarily used to initiate read and write requests for file data. The metadata server manages the metadata of the entire file system, such as filenames, directory structure, and the mapping between files and data blocks. It also processes client requests, assigns tasks to data nodes, and indicates the storage location of data blocks. Data nodes are primarily used to store the actual data blocks and execute read and write operations from clients or the metadata server.

[0004] It is evident that the metadata server is crucial in a distributed file system. A failure of the metadata server can lead to numerous failed read / write requests, ultimately causing the distributed file system to cease operation. Therefore, it is necessary to provide a highly available distributed file system. Summary of the Invention

[0005] This invention provides a data processing method, system, device, medium, and product to ensure high availability of file writing in a distributed file system without affecting file writing after the metadata server or data node stops working.

[0006] In a first aspect, embodiments of the present invention provide a data processing method applied to a writing client, the method comprising:

[0007] Send a file creation request, including the file name of the file to be created, to the metadata server so that the metadata server creates a user file corresponding to the file name and multiple internal files associated with the user file with a set write order;

[0008] Send the first write request corresponding to the user file to the metadata server;

[0009] Receive the first write response sent by the metadata server. The first write response includes the data blocks allocated by the metadata server in multiple internal files and the data node information corresponding to the data blocks.

[0010] Based on the data node information corresponding to the first data block in the first internal file, the first data to be written is written to the first data node corresponding to the first data block. The first internal file is the currently used internal file determined according to the writing order of multiple internal files.

[0011] In response to the successful writing of the first data, a second write request corresponding to the user file is sent to the metadata server;

[0012] If no second write response is received from the metadata server, the second data to be written is written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

[0013] Secondly, embodiments of the present invention provide a data processing method applied to a metadata server, the method comprising:

[0014] Receive a file creation request from the write client, which includes the file name of the file to be created;

[0015] Based on the file creation request, create a user file corresponding to the file name, as well as multiple internal files associated with the user file with a set write order;

[0016] Receive the first write request sent by the write client corresponding to the user file;

[0017] A first write response is sent to the write client. The first write response includes data blocks allocated by the metadata server in multiple internal files and the data node information corresponding to the data blocks. This enables the write client to write the first data to be written to the first data node corresponding to the first data block in the first internal file according to the data node information corresponding to the first data block. After the write client responds to the successful writing of the first data, it sends a second write request corresponding to the user file to the metadata server. If the second write response is not received from the metadata server, the second data to be written is written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

[0018] Thirdly, embodiments of the present invention provide a distributed file system, including a write client, a metadata server, and data nodes;

[0019] Write the client, which is used to send a file creation request, including the file name of the file to be created, to the metadata server;

[0020] The metadata server is used to create user files corresponding to file names and multiple internal files associated with user files with a set writing order based on file creation requests; and to send file creation responses to write clients.

[0021] The write client is also used to send the first write request corresponding to the user file to the metadata server after receiving the file creation response from the metadata server;

[0022] The metadata server is also used to receive the first write request sent by the write client corresponding to the user file; and to send the first write response to the write client, the first write response including the data blocks allocated by the metadata server in multiple internal files and the data node information corresponding to the data blocks;

[0023] The writing client is also used to receive a first write response sent by the metadata server; write the first data to be written to the first data node corresponding to the first data block according to the data node information corresponding to the first data block in the first internal file, wherein the first internal file is the currently used internal file determined according to the writing order of multiple internal files; in response to the successful writing of the first data, send a second write request corresponding to the user file to the metadata server; and if no second write response is received from the metadata server, write the second data to be written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

[0024] Fourthly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the data processing methods described above.

[0025] Fifthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the data processing methods described above.

[0026] Sixthly, embodiments of the present invention provide a computer program product containing a computer program that, when executed by a processor of an electronic device, enables the processor to at least implement the data processing methods described above.

[0027] In the solution provided by this embodiment of the invention, a write client sends a file creation request, including the file name of the file to be created, to a metadata server. This causes the metadata server to create a user file corresponding to the file name and multiple internal files associated with the user file, each with a predefined write order. After receiving the first write request corresponding to the user file, the metadata server opens the multiple internal files and allocates writable data blocks and corresponding data node information within each internal file. A data block can refer to a contiguous logical address space within an internal file. The metadata server sends the allocated data blocks and corresponding data node information from the multiple internal files to the write client via a first write response. The write client, based on the data node information corresponding to the first data block in the first internal file, writes the first data to be written into the first data node corresponding to the first data block. After the first data is successfully written, the write client continues to send a second write request corresponding to the user file to the metadata server, requesting the metadata server to allocate a second data node corresponding to the first data block in the first internal file. If the metadata server is in a stopped state at this time, the write client will not receive the second write response. In this case, the write client writes the second data to be written into the corresponding second data node based on the data node information corresponding to the second data block in the second internal file. In this way, since the metadata server creates multiple associated internal files for the user file, even after the metadata server stops providing services, the write client can still continue to write the second data to be written in the second internal file among the multiple internal files associated with the user file, thereby achieving high availability of file writing in the distributed file system. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 It is a signaling diagram of the file writing process in a distributed file system;

[0030] Figure 2 A schematic diagram of the structure of a distributed file system provided in an embodiment of the present invention;

[0031] Figure 3 A flowchart of a data processing method provided in an embodiment of the present invention;

[0032] Figure 4This is a schematic diagram of the user file created by the metadata server and its associated multiple internal files in the method provided in the embodiments of the present invention;

[0033] Figure 5 A flowchart of a method for synchronizing second data length information provided in an embodiment of the present invention;

[0034] Figure 6 A flowchart of another method for synchronizing second data length information provided in an embodiment of the present invention;

[0035] Figure 7 A flowchart of another data processing method provided in an embodiment of the present invention;

[0036] Figure 8 A flowchart of a data reading method provided in an embodiment of the present invention;

[0037] Figure 9 A flowchart illustrating another data reading method provided in an embodiment of the present invention;

[0038] Figure 10 A schematic diagram of a write client provided in an embodiment of the present invention;

[0039] Figure 11 This is a schematic diagram of the structure of a metadata server provided in an embodiment of the present invention;

[0040] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0043] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0044] First, some terms involved in the embodiments of this invention will be explained.

[0045] Distributed file system: A file system that allows clients to access files located in different physical locations over a network.

[0046] Write Client: A computing node in a distributed file system that responds to file write requests triggered by users.

[0047] Read Client: A computing node in a distributed file system that responds to file read requests triggered by users.

[0048] Data Node (Chunk Server, abbreviated as CS): A computing node in a distributed file system used to store the actual data blocks and perform read and write operations from clients or metadata servers.

[0049] Metadata server (Master): It manages the metadata of the entire file system, such as filenames, directory structure, and mapping relationship between files and data blocks. It is also responsible for handling client requests, assigning tasks to data nodes, and indicating the storage location of computing nodes for data blocks.

[0050] Remote Procedure Call (RPC) is a heartbeat detection mechanism that ensures a normal and continuous connection between a client and a server. It helps detect problems such as connection interruptions and network failures, and can take timely measures to restore the connection or clean up invalid connections.

[0051] High Availability (HA): refers to the ability of a distributed file system to continue providing services in the event of various failures.

[0052] A distributed file system is a special file system architecture that allows users to access files distributed across different physical locations over a network as if they were local files, without needing to know the actual physical location of the files. In a typical distributed file system, the main components usually include clients, a metadata server (Master), and data nodes (Chunk Servers). Clients are primarily used to initiate read and write requests for file data. The metadata server manages the metadata of the entire file system, such as filenames, directory structure, and the mapping between files and data blocks. It is also responsible for processing client requests, assigning tasks to data nodes, and indicating the storage location of data blocks. Data nodes are primarily used to store the actual data blocks and perform read and write operations from clients or the metadata server.

[0053] Figure 1 This is a signaling diagram of the file writing process in a distributed file system. For example... Figure 1 As shown, the file writing process in a distributed file system may include the following steps:

[0054] 101. Write the client to send a file creation request, including the file name of the file to be created, to the metadata server.

[0055] 102. The metadata server creates a user file corresponding to the file name based on the file creation request.

[0056] 103. The metadata server sends a file creation response to the write client.

[0057] 104. The write client sends a write request corresponding to the user file to the metadata server, and the write request carries the file name.

[0058] 105. The metadata server sends a write response to the write client based on the write request.

[0059] Specifically, after receiving a write request, the metadata server opens the user file corresponding to the file name carried in the write request, and allocates data blocks and corresponding data node information within the user file.

[0060] 106. The writing client writes the data to be written to the data node corresponding to the data block based on the write response.

[0061] After the first data is successfully written, the write client repeats the operation of sending a write request corresponding to the user file to the metadata server in step 104 above, in order to obtain the new data block allocated by the metadata server for the data to be written and the data node corresponding to the new data block, until all the data to be written to the user file is written to the data node.

[0062] If the metadata server malfunctions during the file writing process and cannot receive write requests from the write client, the write client will be unable to continue writing data because it cannot obtain the data blocks allocated by the metadata server and the corresponding point data nodes, resulting in data writing failure.

[0063] Based on this, the present invention proposes a method for file writing that does not affect file writing even when the metadata server is down. When the metadata server receives a file creation request from a write client, including the filename of the file to be created, the metadata server creates a user file corresponding to the filename and multiple internal files associated with the user file with a set writing order. When the metadata server receives a first write request from the write client corresponding to the user file, it allocates data blocks and corresponding data node information in the multiple internal files. Here, a data block refers to a contiguous logical address space within an internal file, and the corresponding data node information refers to the location information of this contiguous logical address space mapped to the data node. The core idea of ​​this data processing method is that the metadata server creates multiple associated internal files for the user file, and each internal file is allocated data blocks and corresponding data node information for those data blocks. In this way, after the write client successfully writes the first data to be written to the first data node corresponding to the first data block, the write client will send a second write request corresponding to the user file to the metadata server. If the metadata server does not respond at this time, the write client can write the first data length information and the second data to be written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file. This allows the write client to continue writing the first data length information and the second data to be written to the second internal file in each internal file associated with the user file even after the metadata server suspends service, thereby achieving high availability of file writing in the distributed file system.

[0064] The data processing method and distributed file system provided in the embodiments of the present invention are described below.

[0065] Figure 2 This is a schematic diagram of a distributed file system provided as an embodiment of the present invention. Figure 2 As shown, the distributed file system 20 includes a write client 21, a metadata server 22, and data nodes 23. Data nodes 23 include storage locations for multiple data blocks, such as data node CS1, data node CS2, ..., data node CSn.

[0066] by Figure 2Taking the distributed file system 20 shown as an example, assuming that the first data node corresponding to the first data block Chunk1_0 in the first internal file is CS1, and the second data node corresponding to the second data block Chunk2_0 in the second internal file is CS2, then the data writing process of the writing client 21 is as follows:

[0067] Write client 21, which is used to send a file creation request, including the file name of the file to be created, to metadata server 22.

[0068] Metadata server 22 is used to create a user file corresponding to the file name and multiple internal files associated with the user file with a set writing order according to the file creation request; and to send a file creation response to write client 21.

[0069] The writing client 21 is also used to send a first write request corresponding to the user file to the metadata server 22 after receiving the file creation response from the metadata server 22.

[0070] Metadata server 22 is also used to receive a first write request corresponding to a user file sent by write client 21; and to send a first write response to write client 21, the first write response including data blocks allocated by metadata server 22 in multiple internal files and data node information corresponding to the data blocks.

[0071] The client 21 is also configured to receive a first write response from the metadata server 22; and, based on the data node information corresponding to the first data block in the first internal file, write the first data to be written to the first data node corresponding to the first data block. The first internal file is the currently used internal file determined according to the write order of multiple internal files. In response to the successful writing of the first data, a second write request corresponding to the user file is sent to the metadata server 22. Furthermore, if no second write response is received from the metadata server 22, the second data to be written is written to the corresponding second data node based on the data node information corresponding to the second data block in the second internal file.

[0072] The writing client 21 is also used to write the first data length information to the second data node. The first data length information includes the length information of the data successfully written to the corresponding data node through the data blocks allocated in each of the previous internal files via the second internal file.

[0073] In the above scheme, when the metadata server 22 creates a user file corresponding to the file name according to the file creation request, it also simultaneously creates multiple internal files associated with the user file with a set write order. Upon receiving the first write request corresponding to the user file sent by the write client 21, it simultaneously opens and writes the multiple internal files, allocates data blocks and corresponding data node information in each of the internal files, and sends the first write response to the write client 21. Thus, the write client 21 can, according to the first write response and the set write order, write the first data to be written to the first data node CS1 corresponding to the first data block Chunk1_0. If the first data is successfully written, the write client 21 will send a second write request corresponding to the user file to the metadata server 22. If the metadata server 22 is working normally at this time, the write client 21 will receive the second write response from the metadata server 22. This second write response includes the next data block allocated by the metadata server 22 in the first internal file and the corresponding data node information. The write client 21 then writes the third data to be written to the corresponding third data node according to the data node information of the next data block.

[0074] If the metadata server 22 fails to respond to the second write request at this time, the write client 21 will successfully write the first data length information of the data in the corresponding data node of the data block allocated in each of the previous internal files of the second internal file, and write the second data to be written to the second data block Chunk2_0 corresponding to the second data node CS2 in the second internal file. If some data in the first data is not successfully written to the first data node CS1, the write client 21 will write the first data length information to the second data node CS2 according to the second data block Chunk2_0 corresponding to the second data node CS2 in the second internal file, and continue to write the part of the first data that was not successfully written to the first data node CS1 into the second data node CS2, thereby completing the writing process of the user file.

[0075] During the above-mentioned user file writing process, if the metadata server 22 is working normally, the writing operation on the user file will be performed according to the normal user file writing process. If the metadata server 22 is unable to respond due to a fault, the writing client can continue to write the first data length information and the second data to be written into the second internal file of each internal file associated with the user file to complete the writing of the user file.

[0076] The working process of the above-described distributed file system will be described in detail below with reference to the accompanying drawings and the following embodiments.

[0077] Figure 3This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. Figure 3 As shown, this method is applied to writing clients, and specifically, the method may include the following steps:

[0078] 301. Send a file creation request to the metadata server, including the file name of the file to be created, so that the metadata server creates a user file corresponding to the file name and multiple internal files associated with the user file with a set write order.

[0079] 302. Send the first write request corresponding to the user file to the metadata server.

[0080] 303. Receive the first write response sent by the metadata server. The first write response includes the data blocks allocated by the metadata server in multiple internal files and the data node information corresponding to the data blocks.

[0081] 304. Based on the data node information corresponding to the first data block in the first internal file, write the first data to be written to the first data node corresponding to the first data block. The first internal file is the currently used internal file determined according to the writing order of multiple internal files.

[0082] 305. In response to the successful writing of the first data, a second write request corresponding to the user file is sent to the metadata server.

[0083] 306. If no second write response is received from the metadata server, the second data to be written is written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

[0084] When a user wants to write to a user file, the write client triggers a file creation operation. Based on the user's trigger, the write client sends a file creation request, including the file name of the file to be created, to the metadata server. A user file refers to a file created and used by the user. User files can contain various types of content, such as text files, image files, audio files, video files, document files, database files, and source code files, etc.

[0085] When the metadata server receives a file creation request, it creates a user file corresponding to the file name and several internal files associated with the user file, each with a predefined write order. Specifically, the write order of these internal files is predetermined. When a write client writes data, it must first write the data to be written to the data node corresponding to the data block in the internal file whose write order is set first, according to the predefined write order.

[0086] like Figure 4 As shown, the writing process of a user file is explained by taking the example of the metadata server receiving a file creation request, creating a user file file corresponding to the file name, and creating four internal files file1, file2, file3 and file4 associated with the user file file, with the writing order of the four internal files being file1, file2, file3 and file4.

[0087] The write client sends a first write request corresponding to the user file to the metadata server, causing the metadata server to simultaneously open four internal files (file1, file2, file3, and file4) and allocate writable data blocks and corresponding data node information in each of the four internal files. For example, in internal file file1, a writable first data block (chunk1_0) is allocated, along with its corresponding data node (CS1); in internal file file2, a writable second data block (chunk2_0) is allocated, along with its corresponding data node (CS2); in internal file file3, a writable third data block (chunk3_0) is allocated, along with its corresponding data node (CS3); and in internal file file4, a writable fourth data block (chunk4_0) is allocated, along with its corresponding data node (CS4).

[0088] Subsequently, the metadata server sends a first write response to the write client. This first write response includes the metadata server allocating a writable first data block chunk1_0 and a corresponding data node CS1 in internal file file 1; allocating a writable second data block chunk2_0 and a corresponding data node CS2 in internal file file 2; allocating a writable third data block chunk3_0 and a corresponding data node CS3 in internal file file 3; and allocating a writable fourth data block chunk4_0 and a corresponding data node CS4 in internal file file 4.

[0089] According to the set write order, the write client needs to write the first data to be written to the internal file file1. At this time, the write client writes the first data to be written to the first data node CS1 corresponding to the first data block chunk1_0.

[0090] If the first data write is successful, the write client sends a second write request to the metadata server corresponding to the user file, so that the metadata server allocates the next data block chunk1_1 and the data node CS5 corresponding to the next data block chunk1_1 in the internal file file1.

[0091] If the metadata server is able to provide services normally at this time, the write client will receive a second write response from the metadata server. This second write response includes the next data block chunk1_1 allocated by the metadata server in the internal file file1, and the data node CS5 corresponding to the next data block chunk1_1. Based on the next data block chunk1_1 and the data node CS5 corresponding to the next data block chunk1_1, the write client writes the third data to be written into the corresponding third data node CS5.

[0092] It is understandable that if the third data is successfully written and the metadata server continues to provide normal service, the write client will send a write request to the metadata server corresponding to the user file after each piece of data to be written, so that the metadata server can allocate the next data block and the data node information corresponding to the next data block in the internal file file1. This process will continue until all the data to be written to the user file is successfully written, and the writing process of the user file will end.

[0093] After the write client sends a second write request corresponding to the user file to the metadata server, causing the metadata server to allocate the next data block chunk1_1 and the corresponding data node CS5 in the internal file file 1, if the metadata server fails or a network failure causes the service to be temporarily suspended, the write client will not receive the second write response sent by the metadata server. At this time, based on the writable second data block chunk2_0 allocated in the internal file file 2 and the corresponding data node CS2 in the previously received first write response, the write client continues to write the second data to be written to data node CS2. Simultaneously, to record the length information of the data written in the internal file file 1, the write client also writes the length information of the data already written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file 1 to the corresponding data node CS2. The length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file 1 is the first data length information in this embodiment.

[0094] In response to the successful second data write, the write client sends a third write request to the metadata server corresponding to the user file, so that the metadata server allocates the next data block chunk2_1 and the corresponding data node CS6 in the internal file file 2. If the metadata server is still in a temporarily suspended state at this time, the write client will not receive the third write response sent by the metadata server. At this time, the write client writes the third data to be written into the data node CS3 according to the writable third data block chunk3_0 allocated in the internal file file 3 and the corresponding data node CS3 in the first write response received earlier. At the same time, the write client also writes the length information of the data already written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file 1, and the length information of the data already written in the second data block chunk2_0 allocated in the internal file file 2 and the corresponding data node CS2 in the internal file file 2 into the corresponding data node CS3. In this embodiment, the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file le1, and the length information of the data written in the second data block chunk2_0 allocated in the internal file file le2 and the data node CS2 corresponding to the second data block chunk2_0 are the first data length information.

[0095] Similarly, in response to the successful third data write, the write client sends a fourth write request to the metadata server corresponding to the user file, so that the metadata server allocates the next data block chunk3_1 and the corresponding data node CS7 in the internal file file 3. If the metadata server is still in a temporarily suspended state at this time, the write client will not receive the fourth write response sent by the metadata server. At this time, the write client writes the fourth data to be written into the data node CS4 based on the writable fourth data block chunk4_0 allocated in the internal file file 4 and the corresponding data node CS4 in the first write response received earlier. Simultaneously, the writing client also writes the length information of the data already written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in internal file file 1, the length information of the data already written in the second data block chunk2_0 allocated in internal file file 2 and the data node CS2 corresponding to the second data block chunk2_0, and the length information of the data already written in the third data block chunk3_0 allocated in internal file file 3 and the data node CS3 corresponding to the third data block chunk3_0 into the corresponding data node CS4. The length information of the data already written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in internal file file 1, the length information of the data already written in the second data block chunk2_0 allocated in internal file file 2 and the data node CS2 corresponding to the second data block chunk2_0, and the length information of the data already written in the third data block chunk3_0 allocated in internal file file 3 and the data node CS3 corresponding to the third data block chunk3_0 constitutes the first data length information in this embodiment.

[0096] As can be seen, by creating multiple associated internal files for the user file and allocating data blocks and corresponding data node information in these internal files, the write client can continue writing to the user file in the data nodes unaffected even if the metadata server temporarily stops serving during the user file writing process, thus achieving high availability for user file writing. Simultaneously, the write client also writes the first data length information to the second data node to record the length of data successfully written to the corresponding data nodes from the data blocks allocated in the previous internal files. This allows the storage location of the data to be read to be determined based on the recorded first data length information when reading data from the user file.

[0097] In this embodiment, each of the multiple internal files, except for the internal file ranked first, includes a header block with a set capacity. That is, internal files file2, internal files file3, and internal files file4 all include a header block with a set capacity, and the first position corresponding to the header block is used to write the first data length information.

[0098] Specifically, when the writing client writes the first data length information and the second data to be written into the corresponding data node CS2, it writes the first data length information at the first position corresponding to the header block in the data node CS2 based on the header block in the internal file file 1, the second data block chunk2_0, and the corresponding data node CS2. That is, the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file 1, and writes the second data at the second position after the first position.

[0099] Similarly, when the writing client writes the first data length information and the third data to be written into the corresponding data node CS3, based on the header block in the internal file file 3, the third data block chunk3_0, and the corresponding data node CS3, the first data length information is written at the first position in the data node CS3 corresponding to the header block. That is, the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file 1, and the length information of the data already written in the second data block chunk2_0 allocated in the internal file file 2 and the data node CS2 corresponding to the second data block chunk2_0, and the third data is written at the second position after the first position.

[0100] Similarly, when the writing client writes the first data length information and the fourth data to be written into the corresponding data node CS4, based on the header block in the internal file file 4, the fourth data block chunk4_0, and the corresponding data node CS4, it writes the first data length information at the first position in data node CS4 corresponding to the header block. That is, the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file 1, the length information of the data already written in the second data block chunk2_0 allocated in the internal file file 2 and the data node CS2 corresponding to the second data block chunk2_0, and the length information of the data already written in the third data block chunk3_0 allocated in the internal file file 3 and the data node CS3 corresponding to the third data block chunk3_0, and then writes the fourth data at the second position after the first position.

[0101] By writing the first data length information to the first position in the second node, the length of the successfully written data is recorded. Then, the second data to be written is written to the second position after the first position, enabling the continued writing of the user file's data to the second data node, thus completing the writing of the user file in the data node.

[0102] In reality, during the writing process of user files, besides the temporary shutdown of the metadata server, data node failures, such as network failures or power outages, can also lead to data write failures. For example, if the first data node CS1 fails while the writing client is writing the first data to the first data block chunk1_0, then some data in the first data block will not be successfully written to the first data node CS1.

[0103] At this time, the write client will also write the length information of the first data that has been written to the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file 1, based on the writable second data block chunk2_0 allocated in the internal file file 1 and the data node CS2 corresponding to the second data block chunk2_0 in the first write response received earlier, and write the part of the first data that was not successfully written to the first data node CS1 to the second data node CS2.

[0104] Similarly, if, during the process of writing some data that was not successfully written to the first data node CS1 in the first data, to the second data node, assuming that the second data node CS2 fails, causing some data to fail to be written to the second data node CS2, the write client can, based on the writable third data block chunk3_0 allocated in the internal file file 3 and the data node CS3 corresponding to the third data block chunk3_0, write the length information of the first data already written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file 1 and the length information of the first data already written in the second data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file 2 to the third data node CS3, and continue to write some data to the third data node CS3.

[0105] If, during the process of writing some data that was not successfully written to the first data node CS1 in the first data, to the third data node, assuming that the third data node CS3 also fails, causing some data to fail to be written to the third data node CS2, the write client can, based on the writable fourth data block chunk4_0 allocated in the internal file file 4 and the data node CS4 corresponding to the fourth data block chunk4_0, write the length information of the first data already written in the first data block chunk1_0 allocated in the internal file file 1 to the first data node CS1, the length information of the first data already written in the second data block chunk2_0 allocated in the internal file file 2 to the second data node CS2, and the length information of the first data already written in the third data block chunk3_0 allocated in the internal file file 3 to the fourth data node CS4, and continue to write some data to the fourth data node CS4.

[0106] In this way, even if the first data node fails during the writing process of the user file, causing some data to fail to be written to the first data node CS1, it will not affect the writing client to continue to write the part of the data that was not successfully written to the first data node CS1. According to the set writing order, it will try to write to the second data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2, the third data node CS3 corresponding to the third data block chunk3_0 allocated in the internal file file3, and the fourth data node CS4 corresponding to the fourth data block chunk4_0 allocated in the internal file file4. As long as one data node can provide normal service, the data can be successfully written, thus still achieving high availability of user file writing.

[0107] Through the above process, the write client and the metadata server jointly complete the writing of the user file in the data node.

[0108] In order to enable the read client to read the most up-to-date data, in this embodiment of the invention, the write client can also record the second data length information successfully written in local memory at each recording time, and periodically synchronize the second data length information to the metadata server. When the read client initiates a read request for a user file, the metadata server can send the second data length information to the read client, so that the read client can determine which internal file's corresponding data node to read data from based on the second data length information.

[0109] Figure 5 A flowchart illustrating a method for synchronizing second data length information provided in an embodiment of the present invention. Figure 5 As shown, the method includes:

[0110] 501. During the process of writing target data to the target data node corresponding to the target data block through the data node information corresponding to the target data block in the target internal file, the second data length information of the successful write is recorded in the local memory at each recording time.

[0111] The second data length information includes the length of data successfully written to the corresponding data node through the target internal file and the data blocks allocated in each of the internal files before the target internal file at the recording time; the target internal file is any one of multiple internal files, the target data block is the data block allocated in the target internal file, the target data is the data currently being written, and a recording time is triggered whenever a set data slice granularity is written, and the data slice granularity is less than the capacity of a data block.

[0112] 502. Periodically send heartbeat packets to the metadata server. The heartbeat packets contain the second data length information corresponding to the sending time, so that the metadata server periodically stores and updates the second data length information of the local records. When a user file read request is received triggered by the read client, the storage location of the data to be read is determined according to the second data length information and fed back to the read client.

[0113] Assuming the target internal file is file1, the target data block is the first data block chunk1_0, the capacity of the first data block chunk1_0 is 512MB, and the data slice granularity of the target data is 8MB. During the process of writing target data to the target data node corresponding to the first data block chunk1_0 in the target internal file file1, the writing client triggers a recording moment every time 8MB of target data is successfully written, recording the second data length information in local memory.

[0114] For example, after the write client successfully writes 8MB of target data, a recording time is triggered. At this time, the second data length information recorded in local memory is 8MB of data written to the data node CS1 corresponding to the first data block chunk1_0 allocated in file1. The write client continues to write target data to the target data node corresponding to the target data block. After successful writing, another recording time is triggered, and the second data length information recorded in local memory is 16MB of data written to the data node CS1 corresponding to the first data block chunk1_0 allocated in file1. This continues until the second data length information recorded in local memory by the write client is updated to 512MB of data written to the data node CS1 corresponding to the first data block chunk1_0 allocated in file1, indicating that the first data block chunk1_0 is full. At this time, the write client will send a write request corresponding to the user file to the metadata server, so that the metadata server allocates the next data block chunk1_1 and the data node CS5 corresponding to the next data block chunk1_1 in the target internal file file1.

[0115] If the metadata server fails and cannot respond to the write request, the write client will continue writing the target data to the data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2. When the write client successfully writes target data to data node CS2, a recording event is triggered. The length information of the second data recorded in local memory is: 512MB of data was written to data node CS1 corresponding to the first data block chunk1_0 allocated in file1, and 8MB of data was written to data node CS2 corresponding to the second data block chunk2_0 allocated in file2.

[0116] Subsequently, the write client sends the second data length information to the metadata server by periodically sending heartbeat packets. The metadata server can periodically store and update the second data length information recorded locally based on the received heartbeat packets, and when it receives a read request for a user file triggered by a read client, it determines the storage location of the data to be read based on the second data length information and sends it back to the read client.

[0117] For example, continuing the above example, suppose that in the user file read request triggered by the read client, the data to be read is the data corresponding to 500M to 520M in the user file. When the metadata server receives the user file read request, it determines the storage location of the data to be read as the first data node CS1 corresponding to the internal file file1 and the second data node CS2 corresponding to the internal file file2 based on the second data length information, and then feeds back the determined storage location of the data to be read to the read client.

[0118] Based on the solution provided by the embodiments of the present invention, during the data writing process, the write client records the second data length information of successfully written data in its local memory at each recording moment, and periodically synchronizes the second data length information to the metadata server. When the read client initiates a read request for a user file, the metadata server can send the second data length information to the read client, allowing the read client to determine which internal file's corresponding data node to read data from based on the second data length information. This achieves the goal of enabling the read client to read the latest data whenever possible.

[0119] To further improve the real-time performance of data, such as Figure 6 As shown, another method for synchronizing second data length information provided by an embodiment of the present invention may include the following steps:

[0120] 601. During the process of writing target data to the target data node corresponding to the target data block through the data node information corresponding to the target data block in the target internal file, the second data length information of successful writing is recorded in local memory at each recording time.

[0121] The second data length information includes the length of data successfully written to the corresponding data node through the target internal file and the data blocks allocated in each of the internal files before the target internal file at the recording time; the target internal file is any one of multiple internal files, the target data block is the data block allocated in the target internal file, the target data is the data currently being written, and a recording time is triggered whenever a set data slice granularity is written, and the data slice granularity is less than the capacity of a data block.

[0122] 602. Send a first remote procedure call message to the metadata server. The first remote procedure call message includes the second data length information corresponding to the first sending time and a notification message. The notification message is used to instruct the metadata server to reply with a notification response when it receives a read request from the read client for reading the user file.

[0123] 603. If a notification response is received, a second remote procedure call information is sent to the metadata server. The second remote procedure call information includes the second data length information corresponding to the second sending time, so that the metadata server can determine the storage location of the data to be read based on the second data length information corresponding to the second sending time and provide feedback to the reading client.

[0124] The process by which the client records the length of the second data successfully written in local memory at each record time can be referred to the above. Figure 5 The relevant descriptions in the method embodiments shown will not be repeated here.

[0125] Specifically, the write client sends a notification message and the second data length information corresponding to the first sending time to the metadata server via the first remote procedure call information. The notification message is used to instruct the metadata server to reply with a notification response when it receives a read request from the read client for a user file.

[0126] In practical applications, the write client and the metadata server pre-agree on a timing mechanism to trigger the metadata server to send a notification response to the write client. This response timing could be a timeout limit. For example, suppose the timeout is 20 seconds. That is, starting from the first sending time included in the first remote procedure call information, if the metadata server does not receive a read request for the user file triggered by the read client within this period, the metadata server will send a notification response to the write client after 20 seconds. Afterward, the write client will resend the first remote procedure call information to the metadata server, repeating the above process.

[0127] If, after receiving the first remote procedure call (RPC) message, the metadata server receives a read request for a user file triggered by the read client within the timeout period (e.g., 5 seconds after the first send time), the metadata server immediately replies with a notification response to the write client. Upon receiving the notification response, the write client sends a second RPC message to the metadata server to synchronize the second data length information corresponding to the second send time to the metadata server. This allows the metadata server to determine the storage location of the data to be read based on the second data length information corresponding to the second send time and provide feedback to the read client.

[0128] In other words, when the metadata server receives a read request for a user file triggered by a read client, it obtains the latest second data length information from the write client. Based on this latest second data length information, it determines the storage location of the data to be read by the read client and sends this information back to the read client. This ensures, to the greatest extent possible, that the read client can read the latest data.

[0129] Figure 7 A flowchart illustrating another data processing method provided in an embodiment of the present invention. For example... Figure 7 As shown, this method is applied to a metadata server, and specifically, the method may include the following steps:

[0130] 701. Receive a file creation request sent by the write client, which includes the file name of the file to be created.

[0131] 702. Based on the file creation request, create a user file corresponding to the file name and multiple internal files associated with the user file with a set write order.

[0132] 703. Receive the first write request sent by the write client corresponding to the user file.

[0133] 704. Send a first write response to the write client. The first write response includes data blocks allocated by the metadata server in multiple internal files and the data node information corresponding to the data blocks, so that the write client writes the first data to be written to the first data node corresponding to the first data block according to the data node information corresponding to the first data block in the first internal file. After the write client responds to the successful writing of the first data, it sends a second write request corresponding to the user file to the metadata server. If it does not receive the second write response from the metadata server, it writes the second data to be written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

[0134] User files refer to files created and used by users. User files can contain various types of content, such as text files, image files, audio files, video files, document files, database files, and source code files, etc.

[0135] When the metadata server receives a file creation request, it creates a user file corresponding to the file name and several internal files associated with the user file, each with a predefined write order. The write order of these internal files is predetermined. When a write client writes data, it must first write the data to be written to the data node corresponding to the data block in the internal file with the predefined write order.

[0136] Afterwards, once the metadata server receives the first write request from the write client corresponding to the user file, the metadata server simultaneously opens multiple internal files associated with the user file and allocates writable data blocks and corresponding data node information in each of the multiple internal files.

[0137] The metadata server sends a first write response to the write client. This first write response includes writable data blocks allocated by the metadata server in multiple internal files, along with the corresponding data node information for each data block. This allows the write client to follow the configured write order and write the first data in the first internal file.

[0138] After the write client successfully writes the first piece of data, it sends a second write request to the metadata server corresponding to the user file. The metadata server receives this second write request and sends a second write response to the write client. This second write response includes the next data block allocated by the metadata server in the first internal file and the data node information corresponding to that next data block.

[0139] The above process assumes that the metadata server is functioning correctly and can provide normal service. If the metadata server malfunctions and cannot provide normal service, the write client, upon not receiving a second write response from the metadata server, can continue writing the first data length information and the second data to be written into the second internal file of each internal file associated with the user file to complete the writing of the user file. For a detailed description, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0140] Figure 8 This is a flowchart illustrating a data reading method provided in an embodiment of the present invention. Figure 8 As shown, the method may include the following steps:

[0141] 801. Receive heartbeat packets periodically sent by the write client. The heartbeat packets contain second data length information corresponding to the first sending time.

[0142] 802. Periodically store and update the second data length information of the local records.

[0143] 803. When a read request for a user file is received from the read client, the storage location of the data to be read is determined based on the second data length information.

[0144] 804. The storage location of the data to be read is fed back to the reading client.

[0145] The second data length information is the length of the successfully written data recorded in local memory at each recording moment during the process of the writing client writing target data to the target data node corresponding to the target data block through the data node information corresponding to the target data block in the target internal file.

[0146] The metadata server can periodically store and update the second data length information of the local records based on the heartbeat packets received periodically, and when it receives a user file read request triggered by the read client, it can determine the storage location of the data to be read based on the second data length information and feed it back to the read client.

[0147] For example, suppose that in a user file read request triggered by a read client, the data to be read is the data corresponding to 500M to 520M in the user file. When the metadata server receives the user file read request, it determines the storage location of the data to be read as the first data node CS1 corresponding to the internal file file1 and the second data node CS2 corresponding to the internal file file2 based on the second data length information. Then, it feeds back the determined storage location of the data to be read to the read client.

[0148] Based on the solution provided by the embodiments of the present invention, during the data writing process, the write client records the second data length information of successfully written data in its local memory at each recording moment, and periodically synchronizes the second data length information to the metadata server. When the read client initiates a read request for a user file, the metadata server can send the second data length information to the read client, allowing the read client to determine which internal file's corresponding data node to read data from based on the second data length information. This achieves the goal of enabling the read client to read the latest data whenever possible.

[0149] Figure 9 A flowchart illustrating another data reading method provided in an embodiment of the present invention. For example... Figure 9 As shown, the method may include the following steps:

[0150] 901. Receive the first remote procedure call information sent by the write client. The first remote procedure call information includes the second data length information corresponding to the first sending time and a notification message. The notification message is used to instruct the metadata server to reply with a notification response when it receives a read request from the read client for reading the user file.

[0151] 902. When receiving a read request from a read client for a user file, send a notification response to the write client.

[0152] 903. Receive the second remote procedure call information sent by the write client. The second remote procedure call information includes the second data length information corresponding to the second sending time.

[0153] 904. Determine the storage location of the data to be read based on the second data length information corresponding to the second transmission time.

[0154] 905. Send the storage location of the data to be read to the read client.

[0155] Specifically, the write client and the metadata server pre-agree on a timing mechanism to trigger the metadata server to send a notification response to the write client. This response timing could be a timeout limit. For example, suppose the timeout is 20 seconds. That is, starting from the first sending time included in the first remote procedure call information, if the metadata server does not receive a read request for the user file triggered by the read client within this period, then after 20 seconds, the metadata server sends a notification response to the write client. Afterward, the write client will send the first remote procedure call information to the metadata server again, repeating the above process.

[0156] If, after receiving the first remote procedure call information, the metadata server receives a read request for a user file triggered by the read client within the timeout period, for example, 5 seconds after the first sending time, the metadata server immediately replies with a notification response to the write client. This allows the write client, upon receiving the notification response, to synchronize the second data length information corresponding to the second sending time to the metadata server via the second remote procedure call information. Consequently, the metadata server determines the storage location of the data to be read based on the second data length information corresponding to the second sending time and provides feedback to the read client.

[0157] In other words, when the metadata server receives a read request for a user file triggered by a read client, it obtains the latest second data length information from the write client. Based on this latest second data length information, it determines the storage location of the data to be read by the read client and sends this information back to the read client. This ensures, to the greatest extent possible, that the read client can read the latest data.

[0158] Figure 10 This is a schematic diagram of a write client structure provided in an embodiment of the present invention, such as... Figure 10 As shown, the write client 1000 includes:

[0159] The sending module 1001 is used to send a file creation request, including the file name of the file to be created, to the metadata server so that the metadata server creates a user file corresponding to the file name and multiple internal files associated with the user file with a set writing order; and to send a first write request corresponding to the user file to the metadata server.

[0160] The receiving module 1002 is used to receive a first write response sent by the metadata server. The first write response includes data blocks allocated by the metadata server in multiple internal files and data node information corresponding to the data blocks.

[0161] The writing module 1003 is used to write the first data to be written to the first data node corresponding to the first data block according to the data node information corresponding to the first data block in the first internal file. The first internal file is the currently used internal file determined according to the writing order of multiple internal files.

[0162] The sending module 1001 is also used to send a second write request corresponding to the user file to the metadata server in response to the successful writing of the first data.

[0163] The writing module 1003 is further configured to, if the receiving module does not receive the second write response sent by the metadata server, write the first data length information and the second data to be written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file. The first data length information includes the length information of the data successfully written to the corresponding data node through the data blocks allocated in each of the previous internal files of the second internal file.

[0164] In one embodiment, the writing module 1003 is further configured to: if some data in the first data is not successfully written to the first data node, then write the first data length information to the second data node according to the data node information corresponding to the second data block in the second internal file.

[0165] In one embodiment, the receiving module 1002 is further configured to receive a second write response sent by the metadata server, the second write response including the next data block allocated by the metadata server in the first internal file and the data node information corresponding to the next data block. Correspondingly, the writing module 1003 is further configured to write the third data to be written to the corresponding third data node according to the data node information corresponding to the next data block.

[0166] In one embodiment, each of the multiple internal files, except for the internal file with the highest sort order, includes a header block of a set capacity. When the writing module 1003 writes the first data length information and the second data to be written to the corresponding second data node based on the data node information corresponding to the second data block in the second internal file, it specifically performs the following: based on the header block in the second internal file and the data node information corresponding to the second data block, it writes the first data length information at a first position in the second data node corresponding to the header block, and writes the second data at a second position after the first position.

[0167] In one embodiment, the write client 1000 may further include a recording module, configured to record second data length information successfully written at each recording moment in local memory during the process of the write module writing target data to the target data node corresponding to the target data block through the data node information corresponding to the target data block in the target internal file. The second data length information includes the length information of the data successfully written to the corresponding data node through the data blocks allocated in the target internal file and the previous internal files at the recording moment; the target internal file is any one of multiple internal files, the target data block is the data block allocated in the target internal file, and the target data is the data currently being written. A recording moment is triggered whenever a set data slice granularity is written, and the data slice granularity is less than the capacity of a data block.

[0168] In one embodiment, the sending module 1001 is further configured to periodically send heartbeat packets to the metadata server. The heartbeat packets contain second data length information corresponding to the sending time, so that the metadata server periodically stores and updates the second data length information of the local records, and when it receives a read request for a user file triggered by a read client, it determines the storage location of the data to be read based on the second data length information and feeds it back to the read client.

[0169] In one embodiment, the sending module 1001 is further configured to send first remote procedure call information to the metadata server. The first remote procedure call information includes second data length information corresponding to a first sending time and a notification message. The notification message is used to instruct the metadata server to reply with a notification response when it receives a read request from the read client for a user file. If the receiving module receives the notification response, the sending module 1001 is further configured to send second remote procedure call information to the metadata server. The second remote procedure call information includes second data length information corresponding to a second sending time, so that the metadata server determines the storage location of the data to be read based on the second data length information corresponding to the second sending time and provides feedback to the read client.

[0170] Figure 10 The client shown can execute the steps in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0171] Figure 11 This is a schematic diagram of the structure of a metadata server provided in an embodiment of the present invention, such as... Figure 11 As shown, the metadata server 1100 includes:

[0172] The receiving module 1101 is used to receive a file creation request sent by the writing client, which includes the file name of the file to be created.

[0173] The creation module 1102 is used to create a user file corresponding to the file name and multiple internal files associated with the user file with a set writing order, based on the file creation request.

[0174] The receiving module 1101 is also used to receive the first write request corresponding to the user file sent by the write client.

[0175] The sending module 1103 is used to send a first write response to the writing client. The first write response includes data blocks allocated by the metadata server in multiple internal files and the data node information corresponding to the data blocks. This enables the writing client to write the first data to be written to the first data node corresponding to the first data block according to the data node information corresponding to the first data block in the first internal file. In response to the successful writing of the first data, the writing client sends a second write request corresponding to the user file to the metadata server. If the second write response is not received from the metadata server, the writing client writes the second data to be written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

[0176] In one embodiment, the receiving module 1101 is further configured to receive a second write request corresponding to the first user file sent by the writing client, the second write request being sent by the writing client after the first data write is successful. Correspondingly, the sending module 1103 is further configured to send a second write response to the writing client, the second write response including the next data block allocated by the metadata server in the first internal file and the data node information corresponding to the next data block.

[0177] In one embodiment, the receiving module 1101 is further configured to receive heartbeat packets periodically sent by the write client, the heartbeat packets containing second data length information corresponding to the sending time. The metadata server 1100 may also include an update module, configured to periodically store and update the locally recorded second data length information. A determination module is configured to determine the storage location of the data to be read based on the second data length information when the receiving module receives a read request for a user file triggered by a read client; the sending module 1103 is further configured to feed back the storage location of the data to be read to the read client.

[0178] In one embodiment, the receiving module 1101 is further configured to receive first remote procedure call information sent by the writing client. The first remote procedure call information includes second data length information corresponding to a first sending time and a notification message. The notification message is used to instruct the metadata server to reply with a notification response when it receives a read request from the reading client for a user file. The sending module 1103 is further configured to reply with a notification response to the writing client when it receives a read request from the reading client for a user file. The receiving module 1101 is further configured to receive second remote procedure call information sent by the writing client. The second remote procedure call information includes second data length information corresponding to a second sending time. The determining module is configured to determine the storage location of the data to be read based on the second data length information corresponding to the second sending time. The sending module 1103 is further configured to send the storage location of the data to be read to the reading client.

[0179] Figure 11The metadata server shown can perform the steps in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0180] This invention also provides an electronic device, such as... Figure 12 As shown, the electronic device may include: a processor 1201, a memory 1202, and a communication interface 1203. The memory 1202 stores executable code, which, when executed by the processor 1201, enables the processor 1201 to at least implement the data processing method provided in the foregoing embodiments.

[0181] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is able to implement at least the data processing method provided in the foregoing embodiments.

[0182] This invention provides a computer program product, including a computer program that, when executed by a processor of an electronic device, enables the processor to implement the data processing method provided in the previous embodiments.

[0183] The device embodiments described above are merely illustrative, and the network elements described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0184] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that, Applied to writing clients, the method includes: Send a file creation request, including the file name of the file to be created, to the metadata server so that the metadata server creates a user file corresponding to the file name and multiple internal files associated with the user file with a set write order; Send a first write request corresponding to the user file to the metadata server; Receive a first write response sent by the metadata server, the first write response including data blocks allocated by the metadata server in multiple internal files and data node information corresponding to the data blocks; Based on the data node information corresponding to the first data block in the first internal file, the first data to be written is written to the first data node corresponding to the first data block. The first internal file is the currently used internal file determined according to the writing order of the plurality of internal files. In response to the successful writing of the first data, a second write request corresponding to the user file is sent to the metadata server; If the second write response sent by the metadata server is not received, the second data to be written is written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

2. The method according to claim 1, characterized in that, The method further includes: Write the first data length information into the second data node. The first data length information includes the length information of the data successfully written into the corresponding data node by the data blocks allocated in each of the previous internal files through the second internal file.

3. The method according to claim 2, characterized in that, The method further includes: If some data in the first data is not successfully written to the first data node, then the length information of the first data is written to the second data node according to the data node information corresponding to the second data block in the second internal file.

4. The method according to claim 1, characterized in that, The method further includes: The second write response sent by the metadata server is received, and the second write response includes the next data block allocated by the metadata server in the first internal file and the data node information corresponding to the next data block; Based on the data node information corresponding to the next data block, the third data to be written is written to the corresponding third data node.

5. The method according to claim 1, characterized in that, Each of the multiple internal files, except for the internal file that is ranked first, includes a header block with a set capacity. The step of writing the first data length information and the second data to be written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file includes: Based on the header block and the data node information corresponding to the second data block in the second internal file, the first data length information is written in the first position corresponding to the header block in the second data node, and the second data is written in the second position after the first position.

6. The method according to claim 1, characterized in that, The method further includes: During the process of writing target data to the target data node corresponding to the target data block through the data node information corresponding to the target data block in the target internal file, at each recording moment, the second data length information of the successfully written data is recorded in the local memory. The second data length information includes the length information of the data successfully written to the corresponding data node through the data blocks allocated in the target internal file and the previous internal files at the recording moment; the target internal file is any one of the multiple internal files, the target data block is the data block allocated in the target internal file, and the target data is the data currently being written. A recording moment is triggered whenever a set data slice granularity is written, and the data slice granularity is less than the capacity of a data block.

7. The method according to claim 6, characterized in that, The method further includes: The system periodically sends heartbeat packets to the metadata server. The heartbeat packets contain the second data length information corresponding to the sending time, so that the metadata server periodically stores and updates the second data length information recorded locally. When it receives a read request for the user file triggered by the read client, it determines the storage location of the data to be read based on the second data length information and feeds it back to the read client.

8. The method according to claim 6, characterized in that, The method further includes: Send a first remote procedure call information to the metadata server. The first remote procedure call information includes second data length information corresponding to the first sending time and a notification message. The notification message is used to instruct the metadata server to reply with a notification response when it receives a read request from a read client for the user file. If the notification response is received, a second remote procedure call information is sent to the metadata server. The second remote procedure call information includes second data length information corresponding to the second sending time, so that the metadata server can determine the storage location of the data to be read based on the second data length information corresponding to the second sending time and feed it back to the read client.

9. A data processing method, characterized in that, Applied to a metadata server, the method includes: Receive a file creation request from the write client, which includes the file name of the file to be created; Based on the file creation request, a user file corresponding to the file name and multiple internal files associated with the user file with a set writing order are created; Receive the first write request sent by the write client corresponding to the user file; A first write response is sent to the write client. The first write response includes data blocks allocated by the metadata server in multiple internal files and data node information corresponding to the data blocks. This enables the write client to write the first data to be written to the first data node corresponding to the first data block in the first internal file according to the data node information corresponding to the first data block. In response to the successful writing of the first data, the write client sends a second write request corresponding to the user file to the metadata server. If the write client does not receive the second write response from the metadata server, it writes the second data to be written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

10. The method according to claim 9, characterized in that, The method further includes: Receive a second write request sent by the write client corresponding to the first user file, wherein the second write request is sent by the write client after the first data is successfully written; A second write response is sent to the write client. The second write response includes the next data block allocated by the metadata server in the first internal file and the data node information corresponding to the next data block.

11. The method according to claim 9, characterized in that, The method further includes: Receive heartbeat packets periodically sent by the write client, wherein the heartbeat packets contain the second data length information corresponding to the sending time; The second data length information recorded locally is periodically stored and updated; When a read request for the user file triggered by the read client is received, the storage location of the data to be read is determined according to the second data length information; The storage location of the data to be read is fed back to the reading client.

12. The method according to claim 9, characterized in that, The method further includes: The system receives a first remote procedure call information sent by the write client. The first remote procedure call information includes second data length information corresponding to the first sending time and a notification message. The notification message is used to instruct the metadata server to reply with a notification response when it receives a read request from the read client for the user file. Upon receiving a read request from the read client for the user file, a notification response is sent to the write client. Receive the second remote procedure call information sent by the write client, wherein the second remote procedure call information includes the second data length information corresponding to the second sending time; The storage location of the data to be read is determined based on the second data length information corresponding to the second transmission time; The storage location of the data to be read is sent to the read client.

13. A distributed file system, characterized in that, This includes the write client, metadata server, and data nodes; The write client is used to send a file creation request, including the file name of the file to be created, to the metadata server; The metadata server is configured to create a user file corresponding to the file name and multiple internal files associated with the user file with a set writing order according to the file creation request; and send a file creation response to the write client. The write client is also configured to send a first write request corresponding to the user file to the metadata server after receiving the file creation response from the metadata server; The metadata server is also configured to receive a first write request corresponding to the user file sent by the write client; And send a first write response to the write client, the first write response including the data blocks allocated by the metadata server in multiple internal files and the data node information corresponding to the data blocks; The write client is further configured to receive a first write response sent by the metadata server; and to write the first data to be written to the first data node corresponding to the first data block according to the data node information corresponding to the first data block in the first internal file, wherein the first internal file is the currently used internal file determined according to the writing order of the plurality of internal files; In response to the successful writing of the first data, a second write request corresponding to the user file is sent to the metadata server; Furthermore, if no second write response is received from the metadata server, the second data to be written is written to the corresponding second data node according to the data node information corresponding to the second data block in the second internal file.

14. The system according to claim 13, characterized in that, The write client is further configured to: write first data length information into the second data node, wherein the first data length information includes the length information of data successfully written into the corresponding data node by the data blocks allocated in each of the previous internal files through the second internal file.

15. An electronic device, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 8; or causes the processor to perform the method as described in any one of claims 9 to 12.

16. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 8; or causes the processor to perform the method as described in any one of claims 9 to 12.

17. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 8; or causes the processor to perform the method as described in any one of claims 9 to 12.

Images