A method, apparatus and system for writing data

Abstract

The application discloses a data writing method, device and system, and relates to the technical field of computers. A specific implementation manner of the method comprises the following steps: obtaining a data writing request, wherein the data writing request indicates a target file to be written, a target position to be written and data to be written; generating an operation log corresponding to the data writing request according to a request identifier of the data writing request, an identifier of the target file, the target position and a writing state; writing the data to be written into the target position in the target file, and updating the writing state in the operation log. The implementation manner can avoid data writing amplification, improve the utilization rate of disk bandwidth, and ensure data consistency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method, device and system for writing data. BACKGROUND

[0002] In the field of distributed storage, in order to achieve high availability of data, a piece of data is generally stored on multiple physical machines at the same time.

[0003] The process of writing data into a physical machine generally includes two stages: in the first stage, the data is written into a log file; in the second stage, the data is written into a file on the physical machine according to the user data and metadata in the log file.

[0004] In the process of implementing the present application, the inventors have found that the prior art has at least the following problems:

[0005] In the prior art, data is written in both the stage of writing into a log file and the stage of writing into a real file, which causes data amplification and reduces the utilization rate of disk bandwidth. SUMMARY

[0006] Therefore, the embodiments of the present application provide a method, device and system for writing data, which can generate an operation log according to a request identifier, a target file identifier to be written, a target position to be written and a write state, without writing the data to be written into a log file. Thus, the data to be written can be written into a physical machine of a distributed cluster through only one write operation, so as to avoid data amplification and improve the utilization rate of disk bandwidth. In addition, the consistency of writing data on each node of the distributed cluster can be ensured according to the operation log on each node.

[0007] To achieve the above object, according to an aspect of the embodiments of the present application, a method for writing data is provided.

[0008] The method for writing data according to the embodiments of the present application can be applied to any node of a distributed cluster, and includes: obtaining a data write request, the data write request indicating a target file to be written, a target position to be written and data to be written;

[0009] generating an operation log corresponding to the data write request according to a request identifier of the data write request, an identifier of the target file, the target position and a write state;

[0010] writing the data to be written into the target position in the target file and updating the write state in the operation log.

[0011] Optionally, the generating the operation log corresponding to the data write request and the updating the write state in the operation log comprise:

[0012] generating a first operation log according to the write state before the data to be written is written, the request identifier, the identifier of the target file and the target position;

[0013] generating a second operation log according to the updated write state, the request identifier, the identifier of the target file and the target position.

[0014] Optionally, the method further comprises:

[0015] Optionally, the method further comprises:

[0016] receiving, by the master node, one or more operation logs respectively sent by the one or more slave nodes, the operation logs comprising the second operation log, or a combination of the first operation log and the second operation log;

[0017] comparing the received one or more operation logs respectively with one or more operation logs stored by the master node;

[0018] in the case of inconsistent comparison results, updating the data written on the slave node according to the inconsistent target operation log.

[0019] Optionally, the updating the data written on the slave node according to the inconsistent target operation log comprises:

[0020] reading target data on the master node according to the identifier of the target file and the target position included in the target operation log;

[0021] sending, by the master node, the target data to the slave node, so that the slave node updates the data in the target file stored by itself according to the target data.

[0022] Optionally, after the slave node generates the second operation log, the method further comprises:

[0023] sending, by the slave node, a response message of the data write request to the master node;

[0024] determining, by the master node, that the number of received response messages is not less than the number of slave nodes, and sending a write response message to the user.

[0025] To achieve the above object, according to a further aspect of the embodiments of the present application, a data writing device is provided.

[0026] The data writing device of the embodiments of the present application can be applied to any node of a distributed cluster, and comprises:

[0027] The request obtaining module is configured to obtain a data writing request, wherein the data writing request indicates a target file to be written, a target position to be written and data to be written;

[0028] The log generating module is configured to generate an operation log corresponding to the data writing request according to a request identifier of the data writing request, an identifier of the target file, the target position and a writing state;

[0029] The data writing module is configured to write the data to be written into the target position in the target file and update the writing state in the operation log.

[0030] To achieve the above object, according to a further aspect of the embodiments of the present application, a data writing system is provided.

[0031] The data writing system of the embodiments of the present application comprises a master node and one or more slave nodes, and the master node and the slave nodes are provided with the data writing device of the second aspect;

[0032] The master node is configured to receive operation logs sent by the one or more slave nodes, compare the received one or more operation logs with one or more operation logs stored by itself respectively, and update data written on the slave nodes according to inconsistent target operation logs in the case of inconsistent comparison results.

[0033] To achieve the above object, according to a further aspect of the embodiments of the present application, a data writing server is provided.

[0034] The data writing server of the embodiments of the present application comprises one or more processors and a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data writing method of the embodiments of the present application.

[0035] To achieve the above object, according to a further aspect of the embodiments of the present application, a computer readable storage medium is provided.

[0036] The computer readable storage medium of the embodiments of the present application has a computer program stored thereon, and the program is executed by a processor to implement the data writing method of the embodiments of the present application.

[0037] An embodiment of the above application has the following advantages or beneficial effects: in the process of writing data, an operation log is generated according to the request identifier, the target file identifier to be written, the target position to be written and the writing state, without writing the data to be written into the log file. Thus, the data to be written only needs one writing operation to be written into the physical machine of the distributed cluster, thereby avoiding the data writing amplification and improving the utilization rate of the disk bandwidth; and the consistency of the writing data on each node can be ensured according to the operation log on each node in the distributed cluster.

[0038] Further effects of the above non-conventional optional mode will be described in the following in combination with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings are used to better understand the present application, and do not constitute undue limitations on the present application. Among them:

[0040] Figure 1 is a schematic diagram of the main steps of a method for writing data according to an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of the main steps of another method for writing data according to an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of the main steps of yet another method for writing data according to an embodiment of the present application;

[0043] Figure 4 is a schematic diagram of the main modules of a device for writing data according to an embodiment of the present application;

[0044] Figure 5 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0045] Figure 6 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0046] The exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to help understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0047] It should be noted that the embodiments of the present application and the technical features in the embodiments can be combined with each other without conflict.

[0048] Figure 1 is a schematic diagram of main steps of a method for writing data according to an embodiment of the present application.

[0049] As shown in Figure 1 , the method for writing data according to an embodiment of the present application can be applied to any node of a distributed cluster, which mainly includes the following steps:

[0050] Step S101: obtaining a data write request, the data write request indicating a target file to be written, a target position to be written and data to be written.

[0051] When data update or data insertion or the like needs to be performed, a user can initiate a data write request to a master node in a distributed cluster, the data write request can include a target file (file ID) to be written and a target position (such as a write range size, an offset or the like in a file) to be written.

[0052] Step S102: generating an operation log corresponding to the data write request according to a request identifier of the data write request, an identifier of the target file, the target position and a write state.

[0053] After receiving the data write request, the master node in the distributed cluster can generate a corresponding request identifier (such as a sequence ID) for the data write request, the sequence ID is an incrementing and unique number, which can be used to uniquely identify the data write request. At the same time, the master node can also send the data write request to one or more slave nodes in the distributed cluster. Generally, in order to ensure high availability of data, the master node will send the data write request to multiple slave nodes so that multiple slave nodes can perform corresponding data write operations. Then, the slave nodes can also generate a request identifier of the data write request. It can be understood that since the request identifiers of the master node and the slave nodes are both incrementing and unique, the request identifiers generated by the slave nodes and the master node for the same data write request are the same. In addition, the master node can also send the data write request and the request identifier to the slave nodes after generating the request identifier of the data write request.

[0054] Then, the master node and the slave node can each generate an operation log corresponding to the data write request according to a sequence ID of the data write request, an ID of the target file, a target position to be written (including a write range size in the target file, i.e., from one position to another position in the target file, and an offset), and a write state (e.g., start when no data is written yet). The content of the operation log can be (sequence ID, file ID, size, offset, start).

[0055] Step S103: Write the data to be written into the target position in the target file and update the write state in the operation log.

[0056] After the operation log (sequence ID, file ID, size, offset, start) is generated, the master node and the slave node can start committing data to truly write the data into the target position in the target file. After the writing is completed, the write state in the operation log can be updated to update the operation log or generate a new operation log. For example, after the writing is completed, the write state is finish, the original operation log can be updated from (sequence ID, file ID, size, offset, start) to (sequence ID, file ID, size, offset, finish), or a new operation log can be directly generated as (sequence ID, file ID, size, offset, finish), where the write state finish indicates that the writing is completed.

[0057] In a preferred embodiment of the present application, the operation log is generated before and after writing to record the log to ensure the integrity of the log. That is, before the data is written, a first operation log (sequence ID, file ID, size, offset, start) is generated according to the write state before the data to be written is written, the sequence ID, the ID of the target file, and the target position. After the data is written, a second operation log (sequence ID, file ID, size, offset, finish) is generated according to the updated write state (finish), the sequence ID, the ID of the target file, and the target position.

[0058] In addition, in an embodiment of the present application, in order to ensure the consistency of data on the master node and the slave nodes, after completing the data write operation and generating the operation log, each slave node sends the operation log generated by itself to the master node, which can be the second operation log or the combination of the first operation log and the second operation log, that is, the slave node can send only the second operation log (sequenceID, fileID, size, offset, finish) to the master node, or send both the first operation log (sequenceID, fileID, size, offset, start) and the second operation log (sequenceID, fileID, size, offset, finish) to the master node. After receiving the operation log sent by one or more slave nodes, the master node can compare the received one or more operation logs with one or more operation logs stored by itself, and in the case of inconsistent comparison results, update the data written on the slave node according to the inconsistent target operation log. Alternatively, the master node can also compare the operation logs on the master node and the slave nodes by actively reading the operation logs on the slave nodes.

[0059] For example, the operation log stored on the master node is: [1000, fileA, 4096, 0, start], [1000, fileA, 4096, 0, finish], [1001, fileA, 4096, 8192, start], [1001, fileA, 4096, 8192, finish]. The operation log sent by the slave node A is: [1000, fileA, 4096, 0, start]. By comparing the operation logs, it can be found that the slave node A is missing 2 writes, and its data is inconsistent with the master node. Moreover, by comparing the operation logs, it can be determined that the 2 writes missing from the slave node A are the 2 writes corresponding to the request identifiers sequenceID 1000 and 1001.

[0060] After discovering the inconsistency of the operation logs, the data written on the slave node can be updated according to the inconsistent target operation log. Specifically, in an embodiment of the present application, according to the identifier and target position of the target file included in the target operation log, the target data on the master node is read, and the master node sends the target data to the slave node corresponding to the target operation log, so that the slave node updates the data in the target file stored by itself according to the target data.

[0061] For example, after determining that the operation logs are inconsistent, the slave node A can initiate a read request to the master node to obtain the target data of the sequence ID 1000 and the sequence ID 1001, and then the slave node can re-perform data writing according to the obtained target data, so as to ensure that the data of the slave node is consistent with that of the master node.

[0062] It is worth mentioning that at any time point or when any node fails, the consistency of the data on the nodes in the distributed cluster can be determined by comparing the operation logs. For example, when a slave node abnormally restarts, since the slave node can be in the process of incomplete data writing before the failure, after the slave node restarts, the master node can check the progress of the data writing of the slave node by comparing the operation logs, and make the slave node continue the data writing according to the progress before the failure. For another example, during the process of data writing, the master node and the slave node both restart, in which case the master node is re-elected, after the master node is re-elected, the master node can also determine the progress of the data writing of the slave node by comparing the operation logs, and make the data on the slave nodes consistent according to the data that the master node has written. That is, after all the nodes restart, the data on the slave nodes can be made consistent according to the operation logs on the re-elected master node.

[0063] In addition, after the data writing of each slave node is completed and the second operation log is generated, a response message of the data writing request can be sent to the master node, and the master node determines that the data writing of each slave node is completed when determining that the number of the received response messages is not less than the number of the slave nodes, at which time the write response message is sent to the user to inform the user that the data writing is completed, so as to ensure that the user is informed of the write result after the data is actually written successfully, thereby improving the user experience.

[0064] It can be understood that the slave node can send the response message to the master node together with the second operation log, or send the second operation log to the master node as the response message, or only send the response message to the master node, and the master node can determine that the data writing of the corresponding slave node is completed according to the response message, or actively read the operation log of the corresponding slave node according to the response message, and if the operation log on the slave node is the same as the operation log on the master node, the master node determines that the data writing of the slave node is completed.

[0065] The method for writing data provided by the embodiment of the application will be described in detail below by taking the master node and two slave nodes (a slave node A and a slave node B) in a distributed cluster performing the same data writing operation corresponding to a data writing request as an example. As shown in FIG. 1, the master node and the two slave nodes are connected to each other. Figure 2As shown, the data writing method provided by the embodiment of the application can include the following steps:

[0066] Step S201: The master node receives a data writing request sent by a user.

[0067] Step S202: The master node generates a request identifier of the data writing request, and sends the request identifier and the data writing request to the slave node A and the slave node B respectively.

[0068] For example, the request identifier is N.

[0069] Step S203: The master node, the slave node A and the slave node B respectively generate a first operation log according to the request identifier, an identifier of a target file to be written, a target position to be written and a writing state.

[0070] The first operation log is an operation log before the data is actually written, and can be recorded as Start Write N: (N, file ID, size, offset, start).

[0071] Step S204: The master node, the slave node A and the slave node B respectively write the data to be written into the target position in the target file, and generate a second operation log.

[0072] The second operation log can be recorded as Finish Write N: (N, file ID, size, offset, finish).

[0073] Step S205: The slave node A and the slave node B send a response message to the master node.

[0074] In this embodiment, the slave node sends the response message to the master node only after the writing is successful. If the slave node is down due to a fault during the writing, that is, the slave node is down before the writing is completed, the slave node will not send the response message to the master node.

[0075] Step S206: The master node determines that the number of the received response messages is not less than the number of the slave nodes, and thus determines that the data writing is successful, and sends a writing response message to the user.

[0076] In another embodiment of the application, the master node and / or the slave node can be restarted during the writing of the data, that is, the master node and / or the slave node is restarted before the writing of the data is completed. At this time, the master node can update the data by comparing the operation logs, so as to ensure the data consistency of the master node and the slave node. The data writing method provided by the embodiment of the application is further described below by taking the slave node A as an example. Figure 3 As shown, the data writing method provided by the embodiment of the application can include the following steps:

[0077] Step S301: The master node receives a data write request sent by a user.

[0078] Step S302: The master node generates a request identification of the data write request, and sends the request identification and the data write request to the slave node A and the slave node B respectively.

[0079] For example, the request identification is N.

[0080] Step S303: The master node, the slave node A and the slave node B respectively generate a first operation log according to the request identification, the identification of a target file to be written, a target position to be written and a write state.

[0081] The first operation log is an operation log before the data is actually written, and can be recorded as Start Write N: (N, file ID, size, offset, start).

[0082] Step S304: The master node and the slave node B respectively write the data to be written into the target position in the target file, and generate a second operation log.

[0083] The second operation log can be recorded as Finish Write N: (N, file ID, size, offset, finish).

[0084] Step S305: The slave node B sends a response message to the master node.

[0085] In this example, since the slave node A does not complete the data write, the second operation log cannot be generated, and the response message cannot be sent to the master node.

[0086] Step S306: The master node determines that the number of received response messages is less than the number of slave nodes.

[0087] It can be understood that if the master node does not receive the response messages sent by all the slave nodes within a preset time length, the operation logs can be compared.

[0088] Step S307: The master node determines the slave node A which does not send the response message, and compares the operation log of the master node with the operation log of the slave node A, and determines the target operation log which is inconsistent according to the comparison result.

[0089] For example, the operation log stored on the master node is: [1000, fileA, 4096, 0, start], [1000, fileA, 4096, 0, finish], [1001, fileA, 4096, 8192, start], [1001, fileA, 4096, 8192, finish]. The operation log on the slave node A is: [1000, fileA, 4096, 0, start]. By comparing the operation logs, it can be found that the slave node A is missing two write operations, and the data thereof is inconsistent with the master node. Moreover, by comparing the operation logs, it can be determined that the two write operations missing from the slave node A are the two write operations corresponding to the request identifiers N 1000 and N 1001.

[0090] Step S308: reading the target data on the master node according to the identifier of the target file and the target position included in the target operation log.

[0091] Step S309: the master node sends the target data to the slave node A, so that the slave node A updates the data in the target file according to the target data.

[0092] For example, after determining that the operation logs are inconsistent, the slave node A can initiate a read request to the master node to obtain the target data of N 1000 and N 1001, and then the slave node can re-perform data writing according to the obtained target data, thereby ensuring that the data thereof is consistent with that on the master node. It can be understood that after the slave node A updates the data in the target file, the data writing is completed, at which time the slave node A can send a response message to the master node again, and then the master node can send a write response message to the user after determining that the response messages of all the slave nodes are received.

[0093] According to the method for writing data provided in the embodiment of the present application, in the process of writing data, the operation log is generated according to the request identifier, the identifier of the target file to be written, the target position to be written, and the write state, and the to-be-written data itself is not written into the log file. Therefore, the to-be-written data only needs one write operation to be written into the physical machine of the distributed cluster, so that the write amplification of the data is avoided, and the utilization rate of the disk bandwidth is improved; and moreover, the consistency of the write data on each node can be ensured according to the operation logs on each node in the distributed cluster.

[0094] Figure 4 is a schematic diagram of main modules of a device for writing data according to the embodiment of the present application.

[0095] As shown in Figure 4 , the device 400 for writing data according to the embodiment of the present application comprises:

[0096] The request acquisition module 401 is configured to acquire a data write request, wherein the data write request indicates a target file to be written, a target position to be written, and data to be written.

[0097] The log generation module 402 is configured to generate an operation log corresponding to the data write request according to a request identifier of the data write request, an identifier of the target file, the target position, and a write state.

[0098] The data write module 403 is configured to write the data to be written into the target position in the target file and update the write state in the operation log.

[0099] In an embodiment of the present application, the log generation module 402 is configured to generate a first operation log according to a write state before the data to be written is written, the request identifier, the identifier of the target file, and the target position; and generate a second operation log according to an updated write state, the request identifier, the identifier of the target file, and the target position.

[0100] According to another embodiment of the present application, a system for writing data is also provided, comprising a master node and one or more slave nodes, wherein the master node and the slave nodes are provided with the data writing device according to any of the above embodiments.

[0101] The master node is configured to receive operation logs sent by the one or more slave nodes, compare the received one or more operation logs with one or more operation logs stored by the master node respectively, and update data written on the slave nodes according to inconsistent target operation logs in the case of inconsistent comparison results.

[0102] In an embodiment of the present application, the master node of the distributed cluster receives a data write request sent by a user and sends the data write request to one or more slave nodes of the distributed cluster.

[0103] In an embodiment of the present application, target data on the master node is read according to an identifier of a target file and a target position included in the target operation log, and the target data is sent to the slave node by the master node, so that the slave node updates data in the target file stored by the slave node according to the target data.

[0104] In an embodiment of the present application, after the second operation log is generated by the slave node, a response message of the data write request is also sent to the master node by the slave node, and the master node sends a write response message to the user in the case that the number of received response messages is not less than the number of slave nodes.

[0105] According to the data writing device and system provided by the embodiment of the present application, in the data writing process, the operation log is generated according to the request identifier, the target file identifier to be written, the target position to be written and the writing state, and the data to be written is not written into the log file. Thus, the data to be written is written into the physical machine of the distributed cluster only once, so that the data writing amplification is avoided, and the utilization of the disk bandwidth is improved. In addition, the consistency of the data writing on each node can be ensured according to the operation log on each node of the distributed cluster.

[0106] Figure 5 An exemplary system architecture 500 to which the data writing method or the data writing device provided by the embodiment of the present application can be applied is shown.

[0107] As shown in Figure 5 , the system architecture 500 can include terminal devices 501, 502, 503, a network 504 and a server 505. The network 504 is used to provide a communication link medium between the terminal devices 501, 502, 503 and the server 505. The network 504 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0108] The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0109] The terminal devices 501, 502, 503 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0110] The server 505 can be a server providing various services, such as a background management server providing data support for the request sent by the user using the terminal devices 501, 502, 503. The background management server can analyze and process the received request, and feed back the processing result (such as the data writing result) to the terminal device.

[0111] It should be noted that the data writing method provided by the embodiment of the present application is generally executed by the server 505, and correspondingly, the data writing device is generally arranged in the server 505.

[0112] It should be understood that Figure 5 the number of terminal devices, networks and servers in the above description is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

[0113] Reference will now be made to the drawings Figure 6 which show a schematic configuration of a computer system 600 suitable for use in implementing the terminal device of an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not impose any limitation on the functions and the range of use of an embodiment of the present application.

[0114] As shown in Figure 6 , the computer system 600 includes a central processing unit (CPU) 601 which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0115] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage section 608 as necessary.

[0116] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with the embodiments disclosed herein. For example, the embodiments disclosed herein include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 609 and / or installed from the removable media 611. When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the system of the present application are performed.

[0117] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0118] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0119] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The modules described can also be arranged in a processor, for example, a processor can be described as including a request obtaining module, a log generating module and a data writing module. In some cases, the names of the modules do not constitute a limitation on the modules themselves, for example, the data writing module can also be described as a module that writes the data to be written into a target file.

[0120] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, when the one or more programs are executed by the device, the device includes: obtaining a data writing request, the data writing request indicating a target file to be written, a target position to be written and data to be written; generating an operation log corresponding to the data writing request according to a request identifier of the data writing request, an identifier of the target file, the target position to be written and a writing state; writing the data to be written into the target position in the target file and updating the writing state in the operation log.

[0121] According to the technical solution of the embodiments of the present application, in the process of writing data, the corresponding operation log is generated according to the request identifier, the identifier of the target file to be written, the target position to be written and the writing state, and the data to be written is not written into the log file. Thus, the data to be written only needs one writing operation to be written into the physical machine of the distributed cluster, thereby avoiding the data writing amplification and improving the utilization rate of the disk bandwidth; and the consistency of the data written on each node can also be ensured according to the operation logs on each node in the distributed cluster.

[0122] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of writing data, characterized by, The method is applied to any node of a distributed cluster, and comprises the following steps: obtaining a data write request, the data write request indicating a target file to be written, a target position to be written, and data to be written; generating an operation log corresponding to the data write request according to a request identifier of the data write request, an identifier of the target file, the target position, and a write state; writing the data to be written into the target position in the target file and updating the write state in the operation log; the generating of the operation log corresponding to the data write request and the updating of the write state in the operation log comprise the following steps: generating a first operation log according to the write state before the data to be written is written, the request identifier, the identifier of the target file, and the target position; generating a second operation log according to the updated write state, the request identifier, the identifier of the target file, and the target position; receiving, by a master node of the distributed cluster, one or more operation logs respectively sent by one or more slave nodes of the distributed cluster, the operation logs comprising the second operation log, or a combination of the first operation log and the second operation log; comparing the received one or more operation logs respectively with one or more operation logs stored by the master node; in the case of inconsistent comparison results, updating the data written on the slave node according to the inconsistent target operation log.

2. The method of claim 1, wherein: the master node of the distributed cluster receives a data write request sent by a user and sends the data write request to one or more slave nodes of the distributed cluster.

3. The method of claim 1, wherein, the updating of the data written on the slave node according to the inconsistent target operation log comprises: reading target data on the master node according to the identifier of the target file and the target position included in the target operation log; sending the target data to the slave node by the master node, so that the slave node updates the data in the target file stored by itself according to the target data.

4. The method of claim 2, wherein, after the slave node generates the second operation log, the method further comprises: sending, by the slave node, a response message of the data write request to the master node; the master node sends a write response message to the user in the case that the number of received response messages is not less than the number of slave nodes.

5. An apparatus for writing data, characterized by The method is applied to any node of a distributed cluster, and comprises the following steps: a request obtaining module, configured to obtain a data write request, the data write request indicating a target file to be written, a target position to be written, and data to be written; a log generating module, configured to generate an operation log corresponding to the data write request according to a request identifier of the data write request, an identifier of the target file, the target position, and a write state; a data writing module, configured to write the data to be written into the target position in the target file and update the write state in the operation log; The log generation module is further configured to generate a first operation log according to a write state before the to-be-written data is written, the request identifier, an identifier of the target file, and the target position; and generate a second operation log according to the updated write state, the request identifier, the identifier of the target file, and the target position. The data writing module is further configured to receive one or more operation logs respectively sent by one or more slave nodes of the distributed cluster through a master node of the distributed cluster, the operation logs including the second operation log, or a combination of the first operation log and the second operation log; compare the received one or more operation logs respectively with one or more operation logs stored by the master node; and in a case where the comparison result is inconsistent, update data written on the slave nodes according to inconsistent target operation logs.

6. A system for writing data, characterized by The data writing apparatus comprises a master node and one or more slave nodes, and the master node and the slave nodes are provided with the data writing apparatus of claim 5. The master node is configured to receive operation logs sent by the one or more slave nodes; compare the received one or more operation logs respectively with one or more operation logs stored by the master node; and in a case where the comparison result is inconsistent, update data written on the slave nodes according to inconsistent target operation logs.

7. A server for writing data, characterized by The data writing apparatus comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-4.

8. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-4.

Images