Data processing method, and system, device, medium and product

Abstract

The embodiments of the present disclosure belong to the field of data processing. Provided are a data processing method, and a system, a device, a medium and a product. The method comprises: on the basis of a file creation request sent by a write client, a master creating a user file corresponding to a file name, and a plurality of internal files which are associated with the user file and are in a set write order; and after receiving a first write request corresponding to the user file that is sent by the write client, the master sending, to the write client, data blocks respectively allocated to the plurality of internal files, and chunk server information corresponding to the data blocks. In this way, after the write client successfully writes first data to be written into a first chunk server corresponding to a first data block in a first internal file, if the master is out of service in this case, the write client can continue to write second data to be written into a second chunk server corresponding to a second data block in a second internal file, thereby realizing the high availability of file writing.

Description

Data processing method, system, device, medium and product

[0001] The present disclosure claims priority to Chinese Patent Application No. 202411294615.X, filed on September 13, 2024 with the Chinese Patent Office and entitled "Data processing method, system, device, medium and product", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of data processing, and in particular, to a data processing method, system, device, medium and product. BACKGROUND

[0003] A distributed file system is a file system that allows clients to access files located in different physical locations through a network. It provides a transparent way for users to access remote files as if they were accessing a local file system, without worrying about the actual physical location of the files.

[0004] In a typical distributed file system, the main components usually include a client, a metadata server and a data node. The client is mainly used to initiate read and write requests for file data. The metadata server is mainly used to manage the metadata of the entire file system, such as file names, directory structures, and file-to-data block mapping relationships. It is also responsible for handling client requests and assigning tasks to data nodes to indicate the storage location of data blocks. The data node is mainly used to store actual data blocks and perform read and write operations from the client or metadata server.

[0005] As can be seen, in a distributed file system, the metadata server is very important. Once the metadata server fails, a large number of file read and write requests will fail, and the distributed file system will stop serving. Therefore, it is necessary to provide a highly available distributed file system. SUMMARY

[0006] The embodiments of the present disclosure provide a data processing method, system, device, medium and product to realize high availability of file writing in a distributed file system without affecting file writing after the metadata server stops serving or the data node stops serving.

[0007] In a first aspect, the embodiments of the present disclosure provide a data processing method applied to a write client, the method comprising:

[0008] sending a file creation request including a file name of a file to be created to a metadata server, so that the metadata server creates a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order;

[0009] sending a first write request corresponding to the user file to the metadata server;

[0010] receiving a first write response sent by the metadata server, the first write response including data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks;

[0011] writing first data to be written into a first data node corresponding to a first data block in a first internal file according to the data node information corresponding to the first data block, the first internal file being a currently used internal file determined according to the write order of the plurality of internal files;

[0012] sending a second write request corresponding to the user file to the metadata server in response to successful writing of the first data;

[0013] if no second write response sent by the metadata server is received, writing second data to be written into a second data node corresponding to a second data block in a second internal file according to the data node information corresponding to the second data block.

[0014] In a second aspect, the embodiments of the present disclosure provide a data processing method applied to a metadata server, and the method comprises:

[0015] receiving a file creation request including a file name of a file to be created sent by a write client;

[0016] creating a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order according to the file creation request;

[0017] receiving a first write request corresponding to the user file sent by the write client;

[0018] sending a first write response to the write client, the first write response including data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks, so that the write client writes first data to be written into a first data node corresponding to a first data block in a first internal file according to the data node information corresponding to the first data block, and so that, after the write client sends a second write request corresponding to the user file to the metadata server in response to successful writing of the first data, if no second write response sent by the metadata server is received, the write client writes second data to be written into a second data node corresponding to a second data block in a second internal file according to the data node information corresponding to the second data block.

[0019] In a third aspect, the embodiments of the present disclosure provide a distributed file system, comprising a write client, a metadata server and a data node;

[0020] The write client is configured to send a file creation request comprising a file name of a to-be-created file to the metadata server;

[0021] The metadata server is configured to create, according to the file creation request, a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order, and send a file creation response to the write client;

[0022] The write client is further configured to, after receiving the file creation response from the metadata server, send a first write request corresponding to the user file to the metadata server;

[0023] The metadata server is further configured to receive the first write request corresponding to the user file sent by the write client, and send a first write response to the write client, wherein the first write response comprises data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks;

[0024] The write client is further configured to receive the first write response sent by the metadata server, write first data to be written into a first data node corresponding to a first data block in a first internal file according to data node information corresponding to the first data block, the first internal file being a currently used internal file determined according to the write order of the plurality of internal files, send a second write request corresponding to the user file to the metadata server in response to successful writing of the first data, and write second data to be written into a second data node corresponding to a second data block in a second internal file according to data node information corresponding to the second data block if a second write response sent by the metadata server is not received.

[0025] In a fourth aspect, the embodiments of the present disclosure provide an electronic device, comprising a memory, a processor and a communication interface, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the data processing method of the above aspects.

[0026] In a fifth aspect, the embodiments of the present disclosure provide a non-transitory machine readable storage medium, which stores executable code, and when the executable code is executed by a processor of an electronic device, the processor can at least implement the data processing method of the above aspects.

[0027] In a sixth aspect, the embodiments of the present disclosure provide a computer program product, which contains a computer program, and when the computer program is executed by a processor of an electronic device, the processor can at least implement the data processing method of the above aspects.

[0028] In the scheme provided by the embodiments of the present disclosure, a file creation request including a file name of a to-be-created file is sent to a metadata server by a write client, so that the metadata server creates a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write sequence. After the metadata server receives a first write request corresponding to the user file, the plurality of internal files are opened, and a writeable data block and data node information corresponding to the data block are respectively allocated in the plurality of internal files. The data block can refer to a continuous logical address space in the internal file. The metadata server sends the data block and the data node information corresponding to the data block respectively allocated in the plurality of internal files to the write client through a first write response. The write client writes first data to be written into a first data node corresponding to a first data block in a first internal file according to data node information corresponding to the first data block, and after the first data is written successfully, the write client continues to send a second write request corresponding to the user file to the metadata server to request the metadata server to allocate a second data node corresponding to the first data block in the first internal file. If the metadata server is in a service-stopping state at this time, the write client will not receive a second write response, and at this time, the write client writes second data to be written into a corresponding second data node according to data node information corresponding to a second data block in a second internal file. In this way, since the metadata server creates a plurality of associated internal files for the user file, the write client can continue to write the second data to be written into the second internal file in the plurality of internal files associated with the user file after the metadata server stops serving, so that high availability of file writing in a distributed file system is realized. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor.

[0030] FIG. 1 is a signaling diagram of a file writing process in a distributed file system;

[0031] FIG. 2 is a structural schematic diagram of a distributed file system provided by an embodiment of the present disclosure;

[0032] FIG. 3 is a flowchart of a data processing method provided by an embodiment of the present disclosure;

[0033] FIG. 4 is a schematic diagram of a user file and a plurality of internal files associated with the user file created by a metadata server in the method provided by an embodiment of the present disclosure;

[0034] FIG. 5 is a flowchart of a method for synchronizing second data length information according to an embodiment of the present disclosure;

[0035] FIG. 6 is a flowchart of another method for synchronizing second data length information according to an embodiment of the present disclosure;

[0036] FIG. 7 is a flowchart of another data processing method according to an embodiment of the present disclosure;

[0037] FIG. 8 is a flowchart of a data reading method according to an embodiment of the present disclosure;

[0038] FIG. 9 is a flowchart of another data reading method according to an embodiment of the present disclosure;

[0039] FIG. 10 is a structural diagram of a write client according to an embodiment of the present disclosure;

[0040] FIG. 11 is a structural diagram of a metadata server according to an embodiment of the present disclosure;

[0041] FIG. 12 is a structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and superiorities of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described below in a clear and complete manner with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are some but not all of the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.

[0043] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of the related data need to comply with the relevant laws, regulations, and standards of the country and region, and provide corresponding operation portals for the user to choose authorization or refusal.

[0044] Some embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features in the embodiments can be combined with each other without conflict between the embodiments. In addition, the time sequence of the steps in each method embodiment described below is only an example and is not strictly limited.

[0045] Some terms involved in the embodiments of the present disclosure will be described first.

[0046] Distributed File System: A file system that allows clients to access files located in different physical locations over a network.

[0047] Write Client: A computing node in a distributed file system that responds to user-triggered file write requests.

[0048] Read Client: A computing node in a distributed file system that responds to user-triggered file read requests.

[0049] Chunk Server (CS): A computing node in a distributed file system that stores actual data blocks and performs read and write operations from clients or metadata servers.

[0050] Master: A computing node that manages the metadata of the entire file system, such as file names, directory structures, and file-to-data block mapping relationships. It is also responsible for handling client requests and assigning tasks to data nodes and indicating the storage location of data blocks.

[0051] Remote Procedure Call (RPC): A heartbeat detection mechanism that ensures the connection between the client and the server is normal and continuously effective. It helps detect connection interruptions, network failures, and other issues and can take timely measures to restore the connection or clean up invalid connections.

[0052] High Availability (HA): The ability of a distributed file system to continue providing services in the event of various failures.

[0053] Distributed File System: A special file system architecture that allows users to access files distributed in different physical locations over a network without worrying about the actual physical location of the files. In a typical distributed file system, the main components usually include clients, metadata servers, and data nodes. Among them, the client is mainly used to initiate file data read and write requests, the metadata server is mainly used to manage the metadata of the entire file system, such as file names, directory structures, and file-to-data block mapping relationships, and it is also responsible for handling client requests and assigning tasks to data nodes and indicating the storage location of data blocks. The data node is mainly used to store actual data blocks and perform read and write operations from clients or metadata servers.

[0054] FIG. 1 is a signaling diagram of a file writing process in a distributed file system. As shown in FIG. 1, the file writing process in the distributed file system can include the following steps:

[0055] 101. The write client sends a file creation request including a file name of a file to be created to the metadata server.

[0056] 102. The metadata server creates a user file corresponding to the file name according to the file creation request.

[0057] 103. The metadata server sends a file creation response to the write client.

[0058] 104. The write client sends a write request corresponding to the user file to the metadata server, and the write request carries the file name.

[0059] 105. The metadata server sends a write response to the write client according to the write request.

[0060] Specifically, after receiving the write request, the metadata server opens the user file corresponding to the file name according to the file name carried in the write request, and allocates data blocks and data node information corresponding to the data blocks in the user file.

[0061] 106. The write client writes the data to be written into the data nodes corresponding to the data blocks according to the write response.

[0062] After the first data writing is successful, the write client repeats the operation of sending the write request corresponding to the user file to the metadata server in step 104 above to obtain new data blocks allocated by the metadata server for the data to be written and data nodes corresponding to the new data blocks, until all the data to be written of the user file is written into the data nodes.

[0063] In the above file writing process, if the metadata server fails to receive the write request from the write client, at this time, the write client cannot continue to write data due to the failure to obtain the data blocks allocated by the metadata server and the data nodes corresponding to the data blocks, thereby causing the data writing to fail.

[0064] Based on this, the application provides a method for file writing without influence even if the metadata server stops serving. When the metadata server receives a file creation request including a file name of a file to be created sent by a write client, the metadata server creates a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set writing sequence. When the metadata server receives a first writing request corresponding to the user file sent by the write client, data blocks and data node information corresponding to the data blocks in the plurality of internal files are respectively allocated. The data block refers to a continuous logical address space in the internal file, and the data node information corresponding to the data block refers to position information of the continuous logical address space in the data node. The core idea of the data processing method is that the metadata server creates a plurality of internal files associated with the user file, and each internal file is allocated a data block and data node information corresponding to the data block of the internal file. In this way, after the write client successfully writes the first data to be written into the first data node corresponding to the first data block, the write client sends a second writing request corresponding to the user file to the metadata server. If the metadata server does not respond at this time, the write client can write the first data length information and the second data to be written into the corresponding second data node according to the data node information corresponding to the second data block in the second internal file, so that the write client can continue to write the first data length information and the second data to be written in the second internal file associated with the user file after the metadata server stops serving, thereby realizing high availability of file writing in a distributed file system.

[0065] The data processing method and the distributed file system provided by the embodiments of the present disclosure are introduced below.

[0066] FIG. 2 is a structural schematic diagram of a distributed file system provided by an embodiment of the present disclosure. As shown in FIG. 2, the distributed file system 20 includes a write client 21, a metadata server 22, and a data node 23. The data node 23 includes storage positions of a plurality of data blocks, such as a data node CS1, a data node CS2, …, and a data node CSn.

[0067] Taking the distributed file system 20 shown in FIG. 2 as an example, assuming that the first data node corresponding to the first data block Chunk1_0 in the first internal file is CS1, and the second data node corresponding to the second data block Chunk2_0 in the second internal file is CS2, the data writing process of the write client 21 is as follows:

[0068] The write client 21 is configured to send a file creation request including a file name of a file to be created to the metadata server 22.

[0069] The metadata server 22 is configured to create, according to a file creation request, a user file corresponding to a file name and a plurality of internal files associated with the user file and having a set write sequence, and send a file creation response to the write client 21.

[0070] The write client 21 is further configured to send, after receiving the file creation response from the metadata server 22, a first write request corresponding to the user file to the metadata server 22.

[0071] The metadata server 22 is further configured to receive the first write request corresponding to the user file sent by the write client 21, and send a first write response to the write client 21, wherein the first write response comprises data block information and data node information corresponding to the data block allocated by the metadata server 22 in the plurality of internal files.

[0072] The write client 21 is further configured to receive the first write response sent by the metadata server 22, write first data to be written into a first data node corresponding to a first data block in a first internal file according to data node information corresponding to the first data block, wherein the first internal file is an internal file currently used according to the write sequence of the plurality of internal files, and send a second write request corresponding to the user file to the metadata server 22 in response to successful writing of the first data. In addition, if a second write response sent by the metadata server 22 is not received, second data to be written is written into a second data node according to data node information corresponding to a second data block in a second internal file.

[0073] The write client 21 is further configured to write first data length information into the second data node, wherein the first data length information comprises length information of data successfully written into corresponding data nodes through data blocks allocated in each internal file before the second internal file.

[0074] In the above scheme, when the metadata server 22 creates the user file corresponding to the file name according to the file creation request, it also creates a plurality of internal files associated with the user file and having a set write order, and when receiving the first write request corresponding to the user file sent by the write client 21, it simultaneously opens the plurality of internal files, respectively allocates data blocks and data node information corresponding to the data blocks in the plurality of internal files, and sends the first write response to the write client 21. In this way, the write client 21 can write the first data to be written into the first data node CS1 corresponding to the first data block Chunk1_0 according to the first write response in the set write order. If the first data is successfully written, the write client 21 will send a second write request corresponding to the user file to the metadata server 22. If the metadata server 22 is working normally at this time, the write client 21 will receive the second write response sent by the metadata server 22, which includes the next data block allocated by the metadata server 22 in the first internal file and the data node information corresponding to the next data block. The write client 21 writes the third data to be written into the corresponding third data node according to the data node information corresponding to the next data block.

[0075] If the metadata server 222 fails to respond to the second write request at this time due to a fault, the write client 212 writes the first data length information of the data successfully written into the corresponding data node in the data block allocated in each internal file before the second internal file and the second data to be written into the second data node CS2 corresponding to the second data block Chunk2_0 in the second internal file. If part of the first data is not successfully written into the first data node CS1, the write client 21 writes the first data length information into the second data node CS2 according to the second data node CS2 corresponding to the second data block Chunk2_0 in the second internal file, and continues to write the part of the first data that is not successfully written into the first data node CS1 in the second data node CS2, thereby completing the write process of the user file.

[0076] In the above write process of the user file, if the metadata server 22 can work normally, the write operation of the user file is performed according to the normal write process of the user file. If the metadata server 22 fails to respond due to a fault, the write client can continue to write the first data length information and the second data to be written in the second internal file in each internal file associated with the user file, thereby completing the write of the user file.

[0077] The working process of the above distributed file system will be described in detail below in combination with the accompanying drawings and the following embodiments.

[0078] FIG. 3 is a flowchart of a data processing method according to an embodiment of the present disclosure. As shown in FIG. 3, the method is applied to a write client. Specifically, the method can include the following steps.

[0079] 301. The write client sends a file creation request including a file name of a file to be created to a metadata server, so that the metadata server creates a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order.

[0080] 302. The write client sends a first write request corresponding to the user file to the metadata server.

[0081] 303. The write client receives a first write response sent by the metadata server, wherein the first write response includes data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks.

[0082] 304. According to the data node information corresponding to a first data block in a first internal file, the write client writes first data to be written to a first data node corresponding to the first data block, wherein the first internal file is a currently used internal file determined according to the write order of the plurality of internal files.

[0083] 305. In response to the success of the writing of the first data, the write client sends a second write request corresponding to the user file to the metadata server.

[0084] 306. If no second write response sent by the metadata server is received, the write client writes second data to be written to a corresponding second data node according to the data node information corresponding to a second data block in a second internal file.

[0085] When a user wants to write a user file, a file creation operation is triggered at the write client. At this time, the write client sends a file creation request including a file name of a file to be created to the metadata server based on the user's trigger. The user file refers to a file created and used by the user. The user file can contain various types of content, such as a text file, an image file, an audio file, a video file, a document file, a database file, and a source code file, etc.

[0086] When the metadata server receives the file creation request, the metadata server creates a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order. Specifically, the write order of the plurality of internal files having a set write order is set. When the write client writes data, the write client needs to write the data to be written to a data node corresponding to a data block in an internal file having a first write order in the plurality of internal files according to the set write order.

[0087] As shown in FIG. 4, when the metadata server receives the file creation request, a user file file corresponding to the file name is created, and four internal files filel, file2, file3 and file4 associated with the user file file are created. The write process of the user file is described by taking the set write order of filel, file2, file3 and file4 as an example.

[0088] The write client sends a first write request corresponding to the user file to the metadata server, so that the metadata server opens the four internal files filel, file2, file3 and file4 at the same time, and allocates writable data blocks and data node information corresponding to the data blocks in the four internal files respectively. For example, a first data block chunkl_0 is allocated in the internal file filel, and a data node CS1 corresponding to the first data block chunkl_0 is allocated; a second data block chunk2_0 is allocated in the internal file file2, and a data node CS2 corresponding to the second data block chunk2_0 is allocated; a third data block chunk3_0 is allocated in the internal file file3, and a data node CS3 corresponding to the third data block chunk3_0 is allocated; and a fourth data block chunk4_0 is allocated in the internal file file4, and a data node CS4 corresponding to the fourth data block chunk4_0 is allocated.

[0089] Then, the metadata server sends a first write response to the write client, which includes that the metadata server allocates the first data block chunkl_0 and the data node CS1 corresponding to the first data block chunkl_0 in the internal file filel, allocates the second data block chunk2_0 and the data node CS2 corresponding to the second data block chunk2_0 in the internal file file2, allocates the third data block chunk3_0 and the data node CS3 corresponding to the third data block chunk3_0 in the internal file file3, and allocates the fourth data block chunk4_0 and the data node CS4 corresponding to the fourth data block chunk4_0 in the internal file file4.

[0090] According to the set write order, the write client needs to write the first data to be written in the internal file filel. At this time, the write client writes the first data to be written in the first data node CS1 corresponding to the first data block chunkl_0.

[0091] If the first data is successfully written, the write client sends a second write request corresponding to the user file to the metadata server, so that the metadata server allocates the next data block chunkl_l and the data node CS5 corresponding to the next data block chunkl_l in the internal file filel.

[0092] If the metadata server can normally provide services at this time, the write client receives the second write response sent by the metadata server, which includes the next data block chunkl_l and the data node CS5 corresponding to the next data block chunkl_l allocated by the metadata server in the internal file filel. The write client writes the third data to be written into the corresponding third data node CS5 according to the next data block chunkl_l and the data node CS5 corresponding to the next data block chunkl_l.

[0093] It can be understood that if the third data is successfully written and the metadata server always remains in the state of normally providing services, the write client sends a write request corresponding to the user file to the metadata server every time after writing the data to be written, so that the metadata server allocates the next data block and the data node information corresponding to the next data block in the internal file filel, and the cycle continues until the data to be written of the user file is successfully written, and the writing process of the user file ends.

[0094] After the write client sends the second write request corresponding to the user file to the metadata server, so that the metadata server allocates the next data block chunkl_l and the data node CS5 corresponding to the next data block chunkl_l in the internal file filel, if the metadata server fails or the network fails at this time, the service is temporarily stopped, and the write client cannot receive the second write response sent by the metadata server. At this time, the write client continues to write the second data to be written into the data node CS2 according to the second data block chunk2_0 and the data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2 in the first write response received before. At the same time, in order to record the length information of the data written in the internal file filel, the write client also writes the length information of the data written in the first data node CS1 corresponding to the first data block chunkl_0 allocated in the internal file filel into the corresponding data node CS2. The length information of the data written in the first data node CS1 corresponding to the first data block chunkl_0 allocated in the internal file filel is the first data length information of the embodiment.

[0095] In response to the success of the second data write, the write client sends a third write request corresponding to the user file to the metadata server, so that the metadata server allocates the next data block chunk2_1 and the data node CS6 corresponding to the next data block chunk2_1 in the internal file file2. If the metadata server is still in the temporarily stopped state at this time, the write client cannot receive the third write response sent by the metadata server. At this time, the write client writes the third data to be written in the data node CS3 according to the third data block chunk3_0 allocated in the internal file file3 and the data node CS3 corresponding to the third data block chunk3_0 in the first write response received previously. At the same time, the write client also writes the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1 and the length information of the data written in the second data block chunk2_0 allocated in the internal file file2 and the data node CS2 corresponding to the second data block chunk2_0 into the corresponding data node CS3. The length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1 and the length information of the data written in the second data block chunk2_0 allocated in the internal file file2 and the data node CS2 corresponding to the second data block chunk2_0 are the first data length information of the embodiment.

[0096] Similarly, in response to the success of the third data write, the write client sends a fourth write request corresponding to the user file to the metadata server, so that the metadata server allocates the next data block chunk3_1 and the data node CS7 corresponding to the next data block chunk3_1 in the internal file file3. If the metadata server is still in the temporarily stopped state at this time, the write client cannot receive the fourth write response sent by the metadata server. At this time, the write client writes the fourth data to be written in the data node CS4 according to the fourth data block chunk4_0 allocated in the internal file file4 and the data node CS4 corresponding to the fourth data block chunk4_0 in the first write response received before. At the same time, the write client also writes the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1, the length information of the data written in the second data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2, and the length information of the data written in the third data node CS3 corresponding to the third data block chunk3_0 allocated in the internal file file3 into the corresponding data node CS4. The length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1, the length information of the data written in the second data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2, and the length information of the data written in the third data node CS3 corresponding to the third data block chunk3_0 allocated in the internal file file3 are the first data length information of the embodiment.

[0097] It can be seen that by creating multiple internal files associated with the user file, and by allocating data blocks and data node information corresponding to the data blocks in multiple internal files, the write client can continue to write the user file in the data node without being affected even if the metadata server is in a temporarily stopped state during the user file write process, thereby realizing high availability of user file write. At the same time, the write client also writes the first data length information into the second data node to record the length information of the data successfully written in the corresponding data node through the data blocks allocated in each internal file before the second internal file. In this way, when reading the data in the user file, the storage location of the data to be read can be determined according to the recorded first data length information.

[0098] In the embodiment, the head block of a set capacity is included in each of the internal files except the internal file ranked first, that is, the head block of a set capacity is included in the internal file file2, the internal file file3 and the internal file file4, and the first position corresponding to the head block is used to write the first data length information.

[0099] Specifically, when the write client writes the first data length information and the second data to be written into the corresponding data node CS2, the write client writes the first data length information into the first position corresponding to the head block in the data node CS2 based on the head block in the internal file file2 and the second data block chunk2_0 and the corresponding data node CS2, that is, the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1, and writes the second data after the second position after the first position.

[0100] Similarly, when the write client writes the first data length information and the third data to be written into the corresponding data node CS3, the write client writes the first data length information into the first position corresponding to the head block in the data node CS3 based on the head block in the internal file file3 and the third data block chunk3_0 and the corresponding data node CS3, that is, the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1 and the length information of the data written in the second data block chunk2_0 and the corresponding data node CS2 allocated in the internal file file2, and writes the third data after the second position after the first position.

[0101] Similarly, when the write client writes the first data length information and the fourth data to be written into the corresponding data node CS4, the write client writes the first data length information into the first position corresponding to the head block in the data node CS4 based on the head block in the internal file file4 and the fourth data block chunk4_0 and the corresponding data node CS4, that is, the length information of the data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1, the length information of the data written in the second data block chunk2_0 and the corresponding data node CS2 allocated in the internal file file2, the length information of the data written in the third data block chunk3_0 and the corresponding data node CS3 allocated in the internal file file3, and writes the fourth data after the second position after the first position.

[0102] The data length of the data successfully written is recorded by writing the first data length information into the first position in the second node. The second data to be written is written into the second position after the first position, and the writing of the data to be written in the second data node is continued to complete the writing of the user file in the data node.

[0103] In fact, in the writing process of the user file, in addition to the case that the metadata server temporarily stops serving, the data node may also fail, such as network failure, power failure, etc., resulting in failure of data writing. For example, in the process of writing the first data in the first data node CS1 corresponding to the first data block chunk1_0 by the write client, the first data node CS1 fails, and then part of the first data is not successfully written into the first data node CS1.

[0104] At this time, the write client also writes the length information of the first data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1 into the second data node CS2 according to the second data block chunk2_0 allocated in the internal file file2 and the data node CS2 corresponding to the second data block chunk2_0 in the first write response received previously, and writes the part of the first data not successfully written into the first data node CS1 into the second data node CS2.

[0105] Similarly, if part of the first data not successfully written into the first data node CS1 is written into the second data node, and the second data node CS2 fails, resulting in failure of writing part of the data in the second data node CS2, the write client can write the length information of the first data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1 and the length information of the first data written in the second data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2 into the third data node CS3 according to the third data block chunk3_0 allocated in the internal file file3 and the data node CS3 corresponding to the third data block chunk3_0 in the first write response received previously, and continue to write the part of the data into the third data node CS3.

[0106] If the partial data in the first data node CS1 fails to be written in the third data node in the process of writing the first data in the first data, and the third data node CS3 also fails, resulting in the partial data failing to be written in the third data node CS2, the write client can write the length information of the first data written in the first data node CS1 corresponding to the first data block chunk1_0 allocated in the internal file file1, the length information of the first data written in the second data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2, and the length information of the first data written in the third data block chunk3_0 allocated in the internal file file3 into the fourth data node CS4 according to the fourth data block chunk4_0 allocated in the internal file file4 and the data node CS4 corresponding to the fourth data block chunk4_0 in the first write response received previously, and continue to write the partial data into the fourth data node CS4.

[0107] In this way, even if the first data node fails during the process of writing the user file, resulting in the partial data failing to be written in the first data node, the write client can continue to write the partial data which fails to be written in the first data node CS1 according to the set writing sequence, and sequentially attempt to write into the second data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2, the third data node CS3 corresponding to the third data block chunk3_0 allocated in the internal file file3, and the fourth data node CS4 corresponding to the fourth data block chunk4_0 allocated in the internal file file4, and as long as one data node can normally provide services, the data can be successfully written, and thus the high availability of the user file writing can still be achieved.

[0108] Through the above process, the write client and the metadata server jointly complete the writing of the user file in the data node.

[0109] In order to enable the read client to read as new data as possible, in the embodiment of the present disclosure, the write client can also record the second data length information of the successfully written data in the local memory at each recording moment, and periodically synchronize the second data length information to the metadata server, so that when the read client initiates a read request of the user file, the metadata server can send the second data length information to the read client, so that the read client determines in which data node corresponding to the internal file to read the data according to the second data length information.

[0110] FIG. 5 is a flowchart of a method for synchronizing the second data length information provided by an embodiment of the present disclosure. As shown in FIG. 5, the method comprises:

[0111] 501、In the process of writing the target data to the target data node corresponding to the data node information of the target data block in the target internal file, the second data length information of the successfully written data is recorded in the local memory at each recording time.

[0112] The second data length information includes the length information of the data successfully written into the corresponding data node from the target internal file and the data blocks allocated in each internal file before the target internal file; the target internal file is any one of the multiple internal files, the target data block is the allocated data block in the target internal file, the target data is the data being written, and a recording time is triggered every time a set data slice granularity is written. The data slice granularity is smaller than the capacity of a data block.

[0113] 502、Periodically send a heartbeat packet to the metadata server, the heartbeat packet containing the second data length information corresponding to the sending time, so that the metadata server periodically stores and updates the locally recorded second data length information, and when receiving a read request triggered by a read client, determines the storage location of the data to be read according to the second data length information to feed back to the read client.

[0114] Suppose the target internal file is file1, the target data block is the first data block chunk1_0, the capacity corresponding to the first data block chunk1_0 is 512M, and the data slice granularity of the target data is 8M. In the process of writing the target data to the target data node corresponding to the first data block chunk1_0 in the target internal file file1, a recording time is triggered every time 8M of the target data is successfully written, and the second data length information is recorded in the local memory.

[0115] For example, after the write client successfully writes a target data of 8M, a record time is triggered, at which the second data length information recorded in the local memory is 8M of the data written in the data node CS1 corresponding to the first data block chunkl_0 allocated in filel. The write client continues to write target data in the target data node corresponding to the target data block, and after successful writing, another record time is triggered. Thus, the second data length information recorded in the local memory is 16M of the data written in the data node CS1 corresponding to the first data block chunkl_0 allocated in filel. In this way, the second data length information recorded in the local memory is updated to 512M of the data written in the data node CS1 corresponding to the first data block chunkl_0 allocated in filel, indicating that the first data block chunkl_0 is full. At this time, the write client sends a write request corresponding to the user file to the metadata server, so that the metadata server allocates the next data block chunkl_l and the data node CS5 corresponding to the next data block chunkl_l for the target internal file filel.

[0116] If the metadata server fails to reply to the write response at this time, the write client continues to write target data in the data node CS2 corresponding to the second data block chunk2_0 allocated in the internal file file2. When the write client successfully writes a target data in the data node CS2, a record time is triggered, and the second data length information recorded in the local memory is 512M of the data written in the data node CS1 corresponding to the first data block chunkl_0 allocated in filel and 8M of the data written in the data node CS2 corresponding to the second data block chunk2_0 allocated in file2.

[0117] Subsequently, the write client sends the second data length information to the metadata server by periodically sending a heartbeat packet to the metadata server. The metadata server can periodically store and update the second data length information recorded locally according to the heartbeat packet received periodically, and when receiving a read request of the user file triggered by the read client, determine the storage location of the data to be read according to the second data length information to feed back to the read client.

[0118] For example, based on the above example, assuming that the data to be read in the read request of the user file triggered by the read client is the data corresponding to 500M-520M in the user file, the metadata server determines the storage location of the data to be read as the first data node CS1 corresponding to the internal file filel and the second data node CS2 corresponding to the internal file file2 according to the second data length information when receiving the read request of the user file, and feeds back the determined storage location of the data to be read to the read client.

[0119] Based on the above scheme provided by the embodiments of the present disclosure, by the write client, the second data length information of the successfully written data is recorded in the local memory at each recording time in the process of writing the data, and the second data length information is periodically synchronized to the metadata server, so that when the read client initiates a read request of the user file, the metadata server can send the second data length information to the read client, so that the read client determines in which data node corresponding to the internal file to read the data according to the second data length information. Thus, the purpose of making the read client read the latest data as much as possible is achieved.

[0120] In order to further improve the real-time performance of the data, as shown in FIG. 6, another method for synchronizing the second data length information provided by the embodiments of the present disclosure can include the following steps:

[0121] 601. In the process of writing target data to the target data node corresponding to the target data block through the data node information corresponding to the target data block in the target internal file, the second data length information of the successfully written data is recorded in the local memory at each recording time.

[0122] Among them, the second data length information includes the length information of the data successfully written into the corresponding data node through the data blocks allocated in the target internal file and each internal file before the target internal file at the recording time; the target internal file is any one of the plurality of internal files, the target data block is the allocated data block in the target internal file, the target data is the current data being written, a recording time is triggered every time a set data slice granularity is written, and the data slice granularity is smaller than the capacity of a data block.

[0123] 602. The first remote procedure call information is sent to the metadata server, and the first remote procedure call information includes the second data length information corresponding to the corresponding first sending time and a notification message, and the notification message is used to instruct the metadata server to reply to the notification response when receiving the read request of the user file from the read client.

[0124] 603. If the notification response is received, the second remote procedure call information is sent to the metadata server, and the second remote procedure call information includes the second data length information corresponding to the corresponding second sending time, so that the metadata server determines the storage location of the data to be read according to the second data length information corresponding to the second sending time to feed back to the read client.

[0125] The process of the write client recording the second data length information of the successfully written data in the local memory at each recording time can refer to the related description in the method embodiment shown in FIG. 5 above, and will not be repeated here.

[0126] Specifically, the write client sends the notification message and the second data length information corresponding to the first sending time to the metadata server through the first remote procedure call information. The notification message is used to instruct the metadata server to reply to the notification response when the read client sends the read request for the user file.

[0127] In actual application, the write client and the metadata server agree on a reply time for the metadata server to reply to the notification response. For example, the reply time can be a timeout limit. For example, assuming that the timeout limit is 20 seconds, if the metadata server does not receive the read request for the user file triggered by the read client within the 20 seconds from the first sending time included in the first remote procedure call information, the metadata server sends the notification response to the write client after 20 seconds. Then, the write client sends the first remote procedure call information to the metadata server again, and repeats the above process.

[0128] If the metadata server receives the read request for the user file triggered by the read client within the timeout limit, for example, 5 seconds after the first sending time, the metadata server immediately replies to the notification response to the write client. After receiving the notification response, the write client sends the second remote procedure call information to the metadata server to synchronize the second data length information corresponding to the second sending time to the metadata server, so that the metadata server determines the storage location of the data to be read according to the second data length information corresponding to the second sending time to feed back to the read client.

[0129] That is, when the metadata server receives the read request for the user file triggered by the read client, the metadata server obtains the latest second data length information of the data written by the write client, and then determines the storage location of the data to be read by the read client based on the latest second data length information to feed back to the read client. Thus, the read client can read the latest data to the greatest extent.

[0130] FIG. 7 is a flowchart of another data processing method provided by an embodiment of the present disclosure. As shown in FIG. 7, the method is applied to a metadata server, and specifically, the method can include the following steps:

[0131] 701. Receive a file creation request sent by a write client, the file creation request including a file name of a file to be created.

[0132] 702. Create a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set writing sequence according to the file creation request.

[0133] 703. Receive a first writing request corresponding to the user file sent by the write client.

[0134] 704、send a first write response to the write client, the first write response including data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks, so that the write client writes first data to be written into a first data node corresponding to a first data block in the first internal file according to the data node information corresponding to the first data block, and so that, in response to successful writing of the first data, the write client sends a second write request corresponding to the user file to the metadata server, and, if no second write response sent by the metadata server is received, writes second data to be written into a second data node corresponding to a second data block in the second internal file according to data node information corresponding to the second data block.

[0135] The user file refers to a file created and used by a user. The user file can include various types of content, for example, the user file can be a text file, an image file, an audio file, a video file, a document file, a database file, and a source code file, etc.

[0136] When the metadata server receives the file creation request, the metadata server creates a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order. The write order of the plurality of internal files having the set write order is set, and the write client needs to write data to be written into a data node corresponding to a data block in an internal file having a first write order in the plurality of internal files according to the set write order.

[0137] Subsequently, after the metadata server receives a first write request corresponding to the user file sent by the write client, the metadata server simultaneously opens the plurality of internal files associated with the user file, and allocates writable data blocks respectively in the plurality of internal files and data node information corresponding to the data blocks.

[0138] The metadata server sends a first write response to the write client, the first write response including writable data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks. So that the write client follows the set write order and writes first data in the first internal file.

[0139] When the write client successfully writes the first data, the write client sends a second write request corresponding to the user file to the metadata server. Correspondingly, the metadata server receives the second write request and sends a second write response to the write client, the second write response including a next data block allocated by the metadata server in the first internal file and data node information corresponding to the next data block.

[0140] The above process is based on the premise that the metadata server does not fail and can normally provide services. For the case that the metadata server fails and cannot normally provide services, when the write client does not receive the second write response sent by the metadata server, the write client can continue to write the first data length information and the second data to be written in the second internal file among the internal files associated with the user file to complete the writing of the user file. The specific description can refer to the related description in the above embodiments, which will not be repeated here.

[0141] FIG. 8 is a flowchart of a data reading method provided by an embodiment of the present disclosure. As shown in FIG. 8, the method can include the following steps:

[0142] 801. Receive a heartbeat packet periodically sent by the write client, wherein the heartbeat packet contains second data length information corresponding to a first sending time.

[0143] 802. Periodically store and update the locally recorded second data length information.

[0144] 803. When receiving a read request of the user file triggered by the read client, determine the storage location of the data to be read according to the second data length information.

[0145] 804. Feed back the storage location of the data to be read to the read client.

[0146] The second data length information is the length information of the successfully written data recorded in the local memory by the write client at each recording time in the process of writing the target data to the target data node corresponding to the target data block through the data node information corresponding to the target data block in the target internal file.

[0147] The metadata server can periodically store and update the locally recorded second data length information according to the periodically received heartbeat packet, and when receiving a read request of the user file triggered by the read client, determine the storage location of the data to be read according to the second data length information to feed back to the read client.

[0148] For example, assuming that the data to be read in the read request of the user file triggered by the read client is the data corresponding to 500M-520M in the user file, when the metadata server receives the read request of the user file, according to the second data length information, the storage location of the data to be read is determined as the first data node CS1 corresponding to the internal file file1 and the second data node CS2 corresponding to the internal file file2, and the determined storage location of the data to be read is fed back to the read client.

[0149] Based on the scheme provided in the embodiments of the present disclosure, the second data length information is recorded in the local memory at each recording time point in the process of writing data by the write client, and the second data length information is synchronized to the metadata server regularly, so that when the read client initiates a read request of the user file, the metadata server can send the second data length information to the read client, so that the read client determines in which internal file corresponding to the data node to read data according to the second data length information. Thus, the purpose of making the read client read the latest data as much as possible is achieved.

[0150] FIG. 9 is a flowchart of another data reading method provided by the embodiments of the present disclosure. As shown in FIG. 9, the method can include the following steps:

[0151] 901, receiving the first remote procedure call information sent by the write client, the first remote procedure call information including the second data length information corresponding to the corresponding first sending time and a notification message, the notification message being used to indicate that the metadata server replies to the notification response when receiving the read request of the user file sent by the read client.

[0152] 902, replying to the notification response to the write client when receiving the read request of the user file sent by the read client.

[0153] 903, receiving the second remote procedure call information sent by the write client, the second remote procedure call information including the second data length information corresponding to the corresponding second sending time.

[0154] 904, determining the storage position of the to-be-read data according to the second data length information corresponding to the second sending time.

[0155] 905, sending the storage position of the to-be-read data to the read client.

[0156] Specifically, the write client and the metadata server pre-agree a reply time of the metadata server replying to the notification response to the write client, for example, the reply time can be a timeout time limit. For example, assuming that the timeout time is 20 seconds, that is, from the first sending time contained in the first remote procedure call information, if the metadata server does not receive the read request of the user file triggered by the read client during this period, the metadata server sends the notification response to the write client after 20 seconds. Then, the write client sends the first remote procedure call information to the metadata server again, and repeats the above process.

[0157] If the metadata server receives the first remote procedure call information, within a timeout time, for example, 5 seconds after the first sending time, receives the read request triggered by the read client, the metadata server immediately replies to the write client with a notification response. In this way, the write client, after receiving the notification response, synchronizes the second data length information corresponding to the second sending time to the metadata server through the second remote procedure call information, so that the metadata server determines the storage location of the to-be-read data according to the second data length information corresponding to the second sending time to feed back to the read client.

[0158] That is, when the metadata server receives the read request triggered by the read client, the metadata server obtains the second data length information of the latest data write from the write client, and then determines the storage location of the to-be-read data of the read client based on the latest second data length information to feed back to the read client. Thus, the read client can read the latest data to the greatest extent.

[0159] FIG. 10 is a structural schematic diagram of a write client provided by an embodiment of the present disclosure, as shown in FIG. 10, the write client 1000 includes:

[0160] The sending module 1001 is configured to send a file creation request including a file name of a to-be-created file to the metadata server, so that the metadata server creates a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order; and send a first write request corresponding to the user file to the metadata server.

[0161] The receiving module 1002 is configured to receive the first write response sent by the metadata server, and the first write response includes data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks.

[0162] The writing module 1003 is configured to write first data to be written to a first data node corresponding to a first data block in a first internal file according to data node information corresponding to the first data block, and the first internal file is a currently used internal file determined according to the write order of the plurality of internal files.

[0163] The sending module 1001 is further configured to send a second write request corresponding to the user file to the metadata server in response to the success of the first data writing.

[0164] The writing module 1003 is further configured to write, if the receiving module does not receive the second write response sent by the metadata server, the first data length information and the second data to be written into a second data node corresponding to the second data block in the second internal file according to the data node information corresponding to the second data block in the second internal file.

[0165] In one embodiment, the writing module 1003 is further configured to write, if part of the first data is not successfully written into the first data node, the first data length information into a second data node according to the data node information corresponding to the second data block in the second internal file.

[0166] In one embodiment, the receiving module 1002 is further configured to receive a second write response sent by the metadata server, the second write response including a next data block allocated by the metadata server in the first internal file and data node information corresponding to the next data block. Correspondingly, the writing module 1003 is further configured to write third data to be written into a third data node corresponding to the next data block according to the data node information corresponding to the next data block.

[0167] In one embodiment, each internal file except the first-sequenced internal file among the plurality of internal files includes a head block with a set capacity. When the writing module 1003 writes the first data length information and the second data to be written into a second data node corresponding to a second data block in a second internal file according to the data node information corresponding to the second data block in the second internal file, the writing module 1003 is specifically configured to write the first data length information into a first position corresponding to the head block in the second data node and write the second data into a second position after the first position based on the head block and the data node information corresponding to the second data block in the second internal file.

[0168] In one embodiment, the writing client 1000 can further include a recording module configured to record, in a local memory, second data length information successfully written at each recording time during a process in which the writing module writes target data into a target data node corresponding to a target data block in a target internal file according to data node information corresponding to the target data block in the target internal file, the second data length information including length information of data successfully written into corresponding data nodes through data blocks allocated in the target internal file and each internal file before the target internal file at the recording time; the target internal file is any one of the plurality of internal files, the target data block is a data block allocated in the target internal file, and the target data is data being written at present. A recording time is triggered every time a set data slice granularity is written, and the data slice granularity is smaller than a capacity of a data block.

[0169] In an embodiment, the sending module 1001 is further configured to periodically send a heartbeat packet to the metadata server, the heartbeat packet comprising second data length information corresponding to a sending time, so that the metadata server periodically stores and updates the locally recorded second data length information, and when receiving a read request triggered by the read client for the user file, determines the storage location of the data to be read according to the second data length information to feed back to the read client.

[0170] In an embodiment, the sending module 1001 is further configured to send first remote procedure call information to the metadata server, the first remote procedure call information comprising second data length information corresponding to a corresponding first sending time and a notification message, the notification message being used to instruct the metadata server to reply to a notification response when receiving a read request for the user file from the read client. If the receiving module receives the notification response, the sending module 1001 is further configured to send second remote procedure call information to the metadata server, the second remote procedure call information comprising second data length information corresponding to a corresponding second sending time, so that the metadata server determines the storage location of the data to be read according to the second data length information corresponding to the second sending time to feed back to the read client.

[0171] The write client shown in FIG. 10 can perform the steps in the foregoing embodiments, and the detailed execution process and technical effects can be referred to the descriptions in the foregoing embodiments, which will not be repeated here.

[0172] FIG. 11 is a structural schematic diagram of a metadata server provided by an embodiment of the present disclosure. As shown in FIG. 11, the metadata server 1100 comprises:

[0173] The receiving module 1101 is configured to receive a file creation request sent by the write client, the file creation request comprising a file name of a file to be created.

[0174] The creating module 1102 is configured to create, according to the file creation request, a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order.

[0175] The receiving module 1101 is further configured to receive a first write request corresponding to the user file sent by the write client.

[0176] The sending module 1103 is configured to send a first write response to the write client, the first write response comprising data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks, so that the write client writes the first data to be written into a first data node corresponding to a first data block in the first internal file according to the data node information corresponding to the first data block, and so that, after the write client writes the first data successfully and sends a second write request corresponding to the user file to the metadata server, if no second write response sent by the metadata server is received, the write client writes second data to be written into a second data node corresponding to a second data block in the second internal file according to the data node information corresponding to the second data block.

[0177] In an embodiment, the receiving module 1101 is further configured to receive a second write request corresponding to the first user file sent by the write client, the second write request being sent by the write client after the first data is written successfully. Correspondingly, the sending module 1103 is further configured to send a second write response to the write client, the second write response comprising a next data block allocated by the metadata server in the first internal file and data node information corresponding to the next data block.

[0178] In an embodiment, the receiving module 1101 is further configured to receive a heartbeat packet periodically sent by the write client, the heartbeat packet comprising second data length information corresponding to a sending time. The metadata server 1100 can further comprise an updating module configured to periodically store and update the locally recorded second data length information. The determining module is configured to determine a storage location of data to be read according to the second data length information when the receiving module receives a read request of the user file triggered by the read client. The sending module 1103 is further configured to feed back the storage location of the data to be read to the read client.

[0179] In an embodiment, the receiving module 1101 is further configured to receive first remote procedure call information sent by the write client, the first remote procedure call information comprising second data length information corresponding to a corresponding first sending time and a notification message, the notification message being used to indicate that the metadata server replies to a notification response when receiving a read request of the user file from the read client. The sending module 1103 is further configured to reply to the notification response to the write client when receiving the read request of the user file from the read client. The receiving module 1101 is further configured to receive second remote procedure call information sent by the write client, the second remote procedure call information comprising second data length information corresponding to a corresponding second sending time. The determining module is configured to determine a storage location of data to be read according to the second data length information corresponding to the second sending time. The sending module 1103 is further configured to send the storage location of the data to be read to the read client.

[0180] The metadata server shown in FIG. 11 can perform the steps in the foregoing embodiments, and the detailed execution process and technical effects are described in the foregoing embodiments, which are not repeated here.

[0181] The electronic device shown in FIG. 12 can include a processor 1201, a memory 1202, and a communication interface 1203. The memory 1202 stores executable code, and when the executable code is executed by the processor 1201, the processor 1201 can at least implement the data processing method provided in the foregoing embodiments.

[0182] In addition, the embodiments of the present disclosure provide a non-transitory machine readable storage medium, and the non-transitory machine readable storage medium stores executable code, and when the executable code is executed by the processor of the electronic device, the processor can at least implement the data processing method provided in the foregoing embodiments.

[0183] The embodiments of the present disclosure provide a computer program product, and the computer program is executed by the processor of the electronic device, and the processor can at least implement the data processing method provided in the foregoing embodiments.

[0184] The apparatus embodiments described above are only schematic, and the network elements described as separate components can or can not be physically separate. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0185] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of the general hardware platform, and of course, can also be implemented by means of hardware and software combination. Based on such understanding, the above technical solutions can be embodied in the form of computer program product, and the present disclosure can be embodied in the form of computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0186] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present disclosure, and not to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A data processing method, wherein, The method applied to a write client comprises: sending a file creation request including a file name of a file to be created to a metadata server, so that the metadata server creates a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write order; sending a first write request corresponding to the user file to the metadata server; receiving a first write response sent by the metadata server, the first write response including data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks; writing first data to be written into a first data node corresponding to a first data block in a first internal file according to data node information corresponding to the first data block, the first internal file being a currently used internal file determined according to the write order of the plurality of internal files; sending a second write request corresponding to the user file to the metadata server in response to successful writing of the first data; if no second write response sent by the metadata server is received, writing second data to be written into a second data node according to data node information corresponding to a second data block in a second internal file.

2. The method of claim 1, wherein, The method further comprises: writing first data length information into the second data node, the first data length information including length information of data successfully written into corresponding data nodes through data blocks respectively allocated in each internal file before the second internal file.

3. The method of claim 1 or 2, wherein, The method further comprises: if part of the first data is not successfully written into the first data node, writing the first data length information into the second data node according to data node information corresponding to the second data block in the second internal file.

4. The method of any one of claims 1 to 3, wherein, The method further comprises: receiving the second write response sent by the metadata server, the second write response including a next data block allocated by the metadata server in the first internal file and data node information corresponding to the next data block; writing third data to be written into a third data node according to data node information corresponding to the next data block.

5. The method of any one of claims 1 to 4, wherein, Each internal file in the plurality of internal files except an internal file with a first order includes a head block with a set capacity. The writing of the first data length information and the second data to be written into the second data node according to data node information corresponding to the second data block in the second internal file comprises: based on the head block in the second internal file and data node information corresponding to the second data block, writing the first data length information into a first position corresponding to the head block in the second data node, and writing the second data into a second position after the first position.

6. The method of any one of claims 1 to 5, wherein, The method further comprises: In the process of writing target data to a target data node corresponding to a target data block in a target internal file through data node information corresponding to the target data block, second data length information of successful writing at each recording time is recorded in local memory, and the second data length information includes length information of data successfully written into a corresponding data node at the recording time through the target internal file and data blocks allocated in each internal file before the target internal file; the target internal file is any one of the plurality of internal files, the target data block is a data block allocated in the target internal file, the target data is the data being written, and a recording time is triggered every time a set data slice granularity is written, and the data slice granularity is smaller than the capacity of a data block.

7. The method of any one of claims 1 to 6, wherein, The method further comprises: periodically sending a heartbeat packet to the metadata server, the heartbeat packet containing the second data length information corresponding to the sending time, so that the metadata server periodically stores and updates the locally recorded second data length information, and when receiving a read request triggered by a read client, determines the storage location of the data to be read according to the second data length information to feed back to the read client.

8. The method of any one of claims 1 to 7, wherein, The method further comprises: sending first remote procedure call information to the metadata server, the first remote procedure call information including second data length information corresponding to a corresponding first sending time and a notification message, the notification message being used to instruct the metadata server to reply to a notification response when receiving a read request of the user file from a read client; if the notification response is received, sending second remote procedure call information to the metadata server, the second remote procedure call information including second data length information corresponding to a corresponding second sending time, so that the metadata server determines the storage location of the data to be read according to the second data length information corresponding to the second sending time to feed back to the read client.

9. A data processing method, wherein, Applied to a metadata server, the method comprises: receiving a file creation request sent by a write client and including a file name of a file to be created; creating a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set writing sequence according to the file creation request; receiving a first writing request sent by the write client and corresponding to the user file; sending a first write response to the write client, the first write response including data blocks respectively allocated by the metadata server in the plurality of internal files and data node information corresponding to the data blocks, so that the write client writes first data to be written into a first data node corresponding to a first data block in the first internal file according to data node information corresponding to the first data block, and so that, in response to successful writing of the first data, the write client sends a second write request corresponding to the user file to the metadata server, and, if no second write response sent by the metadata server is received, writes second data to be written into a second data node corresponding to a second data block in the second internal file according to data node information corresponding to the second data block.

10. The method of claim 9, wherein, The method further comprises: receiving a second write request corresponding to the user file sent by the write client, the second write request being sent by the write client after successful writing of the first data; sending a second write response to the write client, the second write response including a next data block allocated by the metadata server in the first internal file and data node information corresponding to the next data block.

11. The method of claim 9 or 10, wherein, The method further comprises: receiving a heartbeat packet periodically sent by the write client, the heartbeat packet including second data length information corresponding to a sending time; periodically storing and updating the second data length information recorded locally; when receiving a read request for the user file triggered by a read client, determining a storage location of data to be read according to the second data length information; feeding back the storage location of the data to be read to the read client.

12. The method of any one of claims 9-11, wherein, The method further comprises: receiving first remote procedure call information sent by the write client, the first remote procedure call information including second data length information corresponding to a corresponding first sending time and a notification message, the notification message being used to indicate that the metadata server returns a notification response when receiving a read request for the user file sent by a read client; when receiving the read request for the user file sent by the read client, returning a notification response to the write client; receiving second remote procedure call information sent by the write client, the second remote procedure call information including second data length information corresponding to a corresponding second sending time; determining a storage location of data to be read according to the second data length information corresponding to the second sending time; sending the storage location of the data to be read to the read client.

13. A distributed file system, wherein, The method further comprises: a write client, a metadata server and data nodes; the write client is configured to send a file creation request including a file name of a file to be created to the metadata server; the metadata server is configured to create a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write sequence according to the file creation request, and send a file creation response to the write client; the metadata server is configured to create a user file corresponding to the file name and a plurality of internal files associated with the user file and having a set write sequence according to the file creation request, and send a file creation response to the write client; The write client is further configured to, after receiving a file creation response from the metadata server, send a first write request corresponding to the user file to the metadata server. The metadata server is further configured to receive the first write request corresponding to the user file sent by the write client. The metadata server is further configured to send a first write response to the write client, wherein the first write response comprises data blocks respectively allocated by the metadata server in a plurality of internal files and data node information corresponding to the data blocks. The write client is further configured to receive the first write response sent by the metadata server, and write first data to be written into a first data node corresponding to a first data block in a first internal file according to data node information corresponding to the first data block, wherein the first internal file is a currently used internal file determined according to a write order of the plurality of internal files. The write client is further configured to, in response to successful writing of the first data, send a second write request corresponding to the user file to the metadata server. If no second write response sent by the metadata server is received, the write client is further configured to write second data to be written into a second data node according to data node information corresponding to a second data block in a second internal file.

14. The system of claim 13, wherein, The write client is further configured to write first data length information into the second data node, wherein the first data length information comprises length information of data successfully written into corresponding data nodes through data blocks allocated in each internal file before the second internal file.

15. An electronic device, comprising: The electronic device comprises: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by the processor of the electronic device, the processor executes the method according to any one of claims 1 to 8, or the method according to any one of claims 9 to 12.

16. A non-transitory machine-readable storage medium, wherein, The computer program, when executed by the processor of the electronic device, causes the processor to execute the method according to any one of claims 1 to 8, or the method according to any one of claims 9 to 12.

17. A computer program product, wherein, ​ ​

Images