Data writing method and device, server, and storage medium

Abstract

The application provides a data writing method and device, a server and a storage medium. The method comprises: obtaining multiple groups of to-be-written data, generating multiple cluster node leases based on the multiple groups of to-be-written data, and generating corresponding stream files in a cluster based on the cluster node leases; for any group of to-be-written data, when it is determined that the current to-be-written data is not successfully written into an allocation stream file corresponding to an allocation lease based on an allocation lease corresponding to the current to-be-written data within a preset time, any standby lease is selected, and the current to-be-written data is written into a standby stream file corresponding to the current standby lease based on the standby lease; receiving data writing information corresponding to the to-be-written data, and storing the data writing information, so that a client reads data based on the data writing information. The method of the application solves the technical problem that the data reliability is low when a distributed distribution node fails in the prior art.

Description

Technical Field

[0001] The present application relates to the field of distributed storage, and in particular to a data writing method, device, server and storage medium. Background Art

[0002] With the rise and widespread adoption of cloud computing technology, distributed storage systems are attracting increasing attention in the industry. Traditional network storage systems use centralized storage servers to store all data. These servers become a bottleneck for system performance and a focal point for reliability and security, failing to meet the needs of large-scale storage applications. Distributed network storage systems employ a scalable architecture, utilizing multiple storage servers to share the storage load. This not only improves system reliability, availability, and access efficiency, but also facilitates scalability. Cluster stability and reliability are crucial for distributed storage.

[0003] When a storage node in a distributed system fails, it often significantly impacts the stability and reliability of the cluster, and may even cause the storage system to become temporarily unavailable. In this case, client data reads and writes may be affected, reducing the data reliability of the distributed system. Summary of the Invention

[0004] The present application provides a data writing method, device, server and storage medium to solve the technical problem in the prior art that when a distributed drop node fails, data reliability is low.

[0005] In a first aspect, the present application provides a data writing method, including: being applied to a client, the client being in communication connection with a cluster, and being configured to write data to be written into the cluster, the method comprising:

[0006] Acquire multiple groups of data to be written, and multiple cluster node leases generated based on the multiple groups of data to be written, and generate corresponding stream files in the cluster based on each of the cluster node leases; the cluster node leases are used to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time; the multiple cluster node leases include allocation leases corresponding to the number of the multiple groups of data to be written and at least one standby lease;

[0007] For any set of data to be written, when it is determined that the current data to be written has not been successfully written into the allocated stream file corresponding to the allocated lease based on the allocated lease corresponding to the current data to be written, any backup lease is selected within a preset time, and the current data to be written is written into the backup stream file corresponding to the current backup lease based on the backup lease;

[0008] receive data write information corresponding to the to-be-written data, and store the data write information, so that the client reads data based on the data write information.

[0009] Optionally, the plurality of cluster node leases generated based on the plurality of groups of to-be-written data comprises:

[0010] generate n+m lease allocation requests based on the n groups of to-be-written data, and send each of the lease allocation requests to a preset management node in the cluster;

[0011] receive n+m cluster node leases sent by the management node; wherein the node leases comprise n allocation leases and m backup leases.

[0012] Optionally, the generating, based on each of the cluster node leases, a corresponding stream file in the cluster comprises:

[0013] For any cluster node lease, obtain node information of each data node included in the current cluster node lease;

[0014] generate, based on each of the node information, a stream file corresponding to the cluster node lease in the cluster, so that the client writes the to-be-written data to the cluster based on the current cluster node lease.

[0015] Optionally, the writing, by the client, a preset number of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time comprises:

[0016] For any group of to-be-written data, determine node information of a data node included in an allocation lease corresponding to the current group of to-be-written data, and perform sharding processing on the to-be-written data based on the node information of the data node to obtain a plurality of shard data;

[0017] write, by each of the data nodes, each of the shard data to an allocation stream file corresponding to the allocation lease within a preset time.

[0018] Optionally, the determining that the current to-be-written data is not successfully written into the allocation stream file corresponding to the allocation lease corresponding to the current to-be-written data comprises:

[0019] respectively receive write results of each of the shard data fed back by each of the data nodes; the write results comprise write success and write failure;

[0020] If the number of write successes in each of the write results is less than a preset number threshold, it is determined that the to-be-written data is not successfully written into the allocation stream file.

[0021] Optionally, the receiving the data write information corresponding to the to-be-written data and storing the data write information comprises:

[0022] For any successfully written shard data, receiving data write information fed back by a data node corresponding to the current shard data; the data write information comprises a stream file ID, a stream file position and shard data information;

[0023] The data write information is written into a preset metadata node in the cluster in a transaction manner.

[0024] Optionally, the method further comprises:

[0025] Obtaining a fault cluster node lease of each unsuccessfully written to-be-written data, and sending node information of a data node included in each fault cluster node lease to a preset management node in the cluster, so that the management node performs fault detection on each data node.

[0026] In a second aspect, the application provides a data write device,

[0027] The device is applied to a client, the client is in communication connection with a cluster, and the device is used for writing to-be-written data into the cluster, and the device comprises:

[0028] A cluster node lease obtaining module is configured to obtain a plurality of groups of to-be-written data, generate a plurality of cluster node leases based on the plurality of groups of to-be-written data, and generate corresponding stream files in the cluster based on each cluster node lease; the cluster node lease is used for enabling the client to write a preset number of data into a stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time; the plurality of cluster node leases comprise an allocation lease corresponding to the number of the plurality of groups of to-be-written data and at least one backup lease;

[0029] A data write module is configured to, for any group of to-be-written data, when it is determined that the current to-be-written data is not successfully written into an allocation stream file corresponding to an allocation lease corresponding to the current to-be-written data based on the allocation lease, select any backup lease within a preset time, and write the current to-be-written data into a backup stream file corresponding to the backup lease based on the backup lease;

[0030] An information storage module is configured to receive data write information corresponding to the to-be-written data, and store the data write information, so that the client performs data reading based on the data write information.

[0031] In a third aspect, the application provides a server, comprising a processor and a memory in communication connection with the processor.

[0032] The memory stores computer-executable instructions.

[0033] The processor executes the computer-executable instructions stored in the memory to implement the method of the first aspect.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method of the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0036] The technical solution provided by the present application is that the client simultaneously requests more than one lease allocation request of to-be-written data, and generates corresponding stream files based on the obtained allocation lease or backup lease for writing data; when the client finds that any data node included in any group of allocation leases cannot successfully write the to-be-written data into the cluster, the client immediately acquires the data node of the backup lease, writes the to-be-written data into the cluster through the data node of the backup lease, and stores the data write information of the to-be-written data, so as to improve the reliability of data writing, and further to fully utilize the cluster resources. Further, the client also reports the data node that fails to successfully write the to-be-written data into the cluster to the management node, so that the management node avoids the failure node when performing node allocation, and further improves the service performance of the cluster. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] Figure 1 An application scenario diagram of the data writing method provided by the present application;

[0039] Figure 2 A flowchart of the data writing method provided by the embodiment of the present application;

[0040] Figure 3 A structural diagram of a data writing device provided by the embodiment of the present application;

[0041] Figure 4 A structural diagram of a server provided by the embodiment of the present application.

[0042] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] In real-world applications, large distributed storage systems can experience hardware failures or be unable to provide services due to maintenance. This can affect client data reads and writes, reducing the reliability of the distributed system. Existing cloud storage services define clients, stream management nodes, and data nodes. However, these services focus on processing a single stream, and to ensure strong consistency, they require significant time for waiting and fault recovery when errors are detected. This can also lead to high latency in single-point failures, resulting in a poor user experience.

[0046] The data writing method provided in this application is intended to solve the above technical problems of the prior art. Specifically, the client can request to generate multiple stream files simultaneously for writing data; when a data writing error is detected, the client can quickly switch to other generated and normally working stream files in the cluster to write data, thereby avoiding writing data to the cluster through the faulty node. While ensuring data reliability, it reduces unnecessary write request failures and increased latency, and better utilizes cluster resources.

[0047] Figure 1 This is an application scenario diagram of the data writing method provided by this application. Figure 1 The application scenarios to which the embodiments of this application are applicable are described. Figure 1, including clients and clusters; specifically, the cluster includes management nodes, data nodes and metadata nodes. Among them, the client is used to write data to the cluster and read data; the management node is used to allocate data nodes to the client when requesting to write data, so that the client can successfully write data; the data node is used to write the data to be written in the client into the cluster; the metadata node is used to store the data write information written by the client into the cluster. Specifically, the client sends multiple lease acquisition requests to the management node to obtain the data nodes allocated by the management node, so as to write the data to be written into the cluster through the data nodes; further, when the client finds that any group of leased data nodes cannot successfully write the data to be written into the cluster, it immediately obtains the data nodes with a backup lease, writes the data to be written into the cluster through the data nodes with a backup lease, and stores the data write information of the data to be written, so as to improve the reliability of data writing and thereby fully utilize the cluster resources. Furthermore, the client will also report the data nodes that have failed to successfully write the data to be written into the cluster to the management node, so that the management node can avoid faulty nodes and make reasonable allocations when allocating nodes, thereby further improving data writing performance.

[0048] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0049] Figure 2 This is a flow chart of a data writing method provided in an embodiment of the present application. The method can be executed by a data writing device, which can be a server or a server. The following description takes the server as an example. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware, such as Figure 2 As shown, the method includes the following steps.

[0050] S210 , obtaining multiple groups of data to be written and multiple cluster node leases generated based on the multiple groups of data to be written, and generating corresponding stream files in the cluster based on each cluster node lease.

[0051] In an embodiment of the present application, the data to be written may include application data generated by the upper-layer application installed in the client during operation, and may also include operation data generated by the client during operation. This application does not limit the data content of the data to be written.

[0052] In order to better store the to-be-written data, the to-be-written data can be written into a distributed cluster connected with the client for data storage. In the present application, a plurality of data nodes are pre-configured in the cluster for writing the to-be-written data in the client into the cluster for data storage. Optionally, a management node is also provided in the cluster for managing the data nodes in the cluster and allocating different data nodes for different clients upon receiving a client request. Optionally, the management node can be any node selected from the data nodes or a pre-configured node, which is not limited in the present application.

[0053] In actual applications, in order to improve the data processing efficiency, the client can write a plurality of groups of to-be-written data into the cluster at the same time, and in order to ensure that the data is successfully written into the cluster, a cluster node lease acquisition request is sent to the cluster based on the plurality of groups of to-be-written data before data writing, so as to acquire a plurality of cluster node leases allocated by the management node in the cluster, and the to-be-written data in the client is written into the cluster through a stream file generated based on the cluster node leases.

[0054] In the embodiments of the present application, the cluster node lease includes an allocation lease corresponding to the number of the plurality of groups of to-be-written data and at least one backup lease; in order not to waste cluster resources and improve the utilization rate of the data nodes in the cluster, the allocation lease of any to-be-written data in the present application can be used as a backup lease for any other group of to-be-written data when it is idle. In other words, any allocation lease can be used as a backup lease for other to-be-written data when it is not processing the current group of to-be-written data, and the other group of to-be-written data is written into the cluster when the allocation lease of the other group of to-be-written data fails to successfully write the other group of to-be-written data into the cluster.

[0055] It should be explained that the cluster node lease is used to enable the client to write a preset number of data into the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time. In other words, when the client writes data into the cluster, the corresponding to-be-written data is written through the data node allocated by the management node, so as to improve the running efficiency of the cluster.

[0056] Optionally, the specific process of generating a plurality of cluster node leases based on a plurality of groups of to-be-written data in the embodiments of the present application can include: generating n+m lease allocation requests based on n groups of to-be-written data, and sending each lease allocation request to a pre-configured management node in the cluster; receiving n+m cluster node leases sent by the management node; wherein the node lease includes n allocation leases and m backup leases.

[0057] Since the data nodes included in the cluster node lease allocated by the cluster to the client can fail and temporary operation and maintenance can cause the data nodes to be unable to provide services, in order to ensure that the data can be successfully written into the cluster, the client can send a lease allocation request to the cluster, the number of which is greater than the number of groups of data to be written, to ensure that when a data node in the allocated cluster node lease fails, a standby lease can be enabled for data writing.

[0058] Specifically, the client generates a lease allocation request based on the obtained data to be written. For example, when it is determined that there are n groups of data to be written in the client that need to be written into the cluster for storage, n+m lease allocation requests are generated to obtain multiple cluster node leases. Further, the above lease allocation request is sent to a preset management node in the cluster. The management node generates n+m cluster node leases based on the received lease allocation request and the node state of each data node in the cluster, and feeds back the n+m cluster node leases to the client.

[0059] It should be noted that the n+m cluster node leases include n allocation leases and m standby leases. In other words, if the data nodes in the allocation lease fail to successfully write the data to be written into the cluster, the standby lease is enabled for data writing, thereby improving the data writing efficiency and further improving the utilization rate of cluster resources.

[0060] Further, when the client obtains the cluster node lease allocated by the cluster, the client also needs to generate a corresponding stream file in the cluster based on the data nodes included in the cluster node lease, so that the client writes the data to be written into the cluster. Optionally, the specific process of generating the stream file in the present application can include: for any cluster node lease, obtaining the node information of each data node included in the current cluster node lease; generating a stream file corresponding to the cluster node lease in the cluster based on the node information, so that the client writes the data to be written into the cluster based on the current cluster node lease.

[0061] In the embodiment of the present application, since the cluster node lease includes multiple data nodes, and the data to be written needs to be written into the cluster through each data node, the node information of each data node needs to be obtained, and then a corresponding stream file is requested to be generated in the cluster based on the node information, so as to facilitate subsequent writing processing of the data to be written. It should be noted that in the present application, the stream file generated based on the cluster node lease can only be written into the corresponding data to be written. Specifically, each data node can write the data to be written into the stream file in an appending manner, so as to enable the client to write the data to be written into the cluster based on the current cluster node lease.

[0062] S220. For any group of data to be written, when it is determined that the current data to be written has not been successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written, any backup lease is selected within a preset time, and the current data to be written is written into the backup stream file corresponding to the current backup lease based on the backup lease.

[0063] In an embodiment of the present application, the specific process of the client writing the data to be written into the stream file through each data node in the cluster node lease may include: for any group of data to be written, determining the node information of the data nodes contained in the allocation lease corresponding to the current group of data to be written, and sharding the data to be written based on the data node information to obtain multiple sharded data; based on each data node, writing each sharded data to the allocation stream file corresponding to the allocation lease within a preset time.

[0064] Specifically, to improve data writing efficiency and fully utilize data node resources, before writing the data to be written, the data to be written can be pre-sharded to obtain multiple shards. For example, the allocation lease corresponding to the data to be written can be determined, and then the data to be written can be sharded in multiple copies or erasure coding according to the number of data nodes included in the allocation lease to obtain multiple shards. Then, write requests are sent to the data nodes respectively to write each shard data to the allocation stream file corresponding to the allocation lease in an appended manner based on each data node, so as to write the data to the cluster.

[0065] In actual applications, in order to ensure that the allocated lease can provide services normally, that is, the data to be written in the client can be successfully written into the cluster, after the data to be written is processed by the data nodes in the cluster, the data writing results fed back by the data nodes can be received, and then based on the writing results, it can be determined whether the data to be written is successfully written into the cluster, further ensuring the reliability of data storage.

[0066] Optionally, in an embodiment of the present application, the specific process of determining whether the data to be written is successfully written into the cluster may include: receiving the write results of each shard data fed back by each data node respectively; the write results include write success and write failure; if the number of write successes in each write result is less than a preset number threshold, it is determined that the data to be written is not successfully written to the allocated stream file.

[0067] Specifically, when a client writes data to be written into the cluster via the data nodes in the assigned lease, it receives the write results of the sharded data from each data node and counts the number of successful writes received within a preset time. Furthermore, if the number of successful writes in each write result is greater than a preset threshold, the data to be written is determined to have been successfully written to the assigned stream file.

[0068] Optionally, if the number of successfully written results in each write result is less than a preset number threshold, it is determined that the data to be written is not successfully written to the allocated stream file.

[0069] Furthermore, in order to ensure that the data can be written successfully, it is necessary to obtain other standby leases allocated to the client within the preset time, and generate a standby stream file corresponding to the data to be written based on each data node in the standby lease, and then write the data to be written into the standby stream file based on the data nodes included in the standby lease to achieve writing into the cluster.

[0070] Optionally, when using a backup lease to write the data to be written, in order to ensure the success rate of data writing, data writing can be performed based on multiple backup leases at the same time, and when it is determined that any of the backup leases has successfully written the data to be written into the cluster, the writing processing of other backup leases is stopped.

[0071] S230: Receive data writing information corresponding to the data to be written, and store the data writing information so that the client can read the data based on the data writing information.

[0072] In an embodiment of the present application, when it is determined based on the above implementation that the data to be written is successfully written into the cluster through the allocated lease or the standby lease, the data write information feedback from the data node in the allocated lease or the standby lease is received and the data write information is stored so that subsequent clients can read the above data in the cluster.

[0073] Optionally, in an embodiment of the present application, the specific process of receiving data write information corresponding to the data to be written and storing the data write information may include: for any successfully written shard data, receiving data write information feedback from the data node corresponding to the current shard data; the data write information includes the stream file ID, stream file location and shard data information; and writing the data write information to a preset metadata node in the cluster in a transactional manner.

[0074] Specifically, after successfully writing shard data, the data node sends the write result information, including the stream file ID, stream file location, and corresponding shard data information, back to the client. Furthermore, the client stores this information in a pre-defined metadata node in the cluster, allowing subsequent read operations based on the data write information stored in the metadata node.

[0075] Based on the above implementation method, in the embodiment of the present application, the client will also perform information reporting processing on the management node in the cluster based on the data writing status of the data to be written, so that the management node in the cluster can know the service status of each data node in the cluster in real time, and then manage each data node.

[0076] Optionally, the specific process of the information reporting processing of the client can include: obtaining the lease of each failed cluster node for writing the to-be-written data, and sending the node information of the data node contained in each lease to a preset management node in the cluster, so that the management node detects the failure of each data node.

[0077] In the embodiments of the present application, the lease of the failed cluster node is an allocation lease or a backup lease. Specifically, if any lease is determined to be a lease of a failed cluster node during the data writing process, the client sends a request to the management node, truncates the last successful writing position of the stream file corresponding to the lease, and modifies the state of the stream file to an unwritable state, so as to avoid the data node continuing to write data into the stream file and waste cluster resources.

[0078] Optionally, if the client does not write data into the stream file through the data node within a preset time, the management node can also directly modify the state of the stream file to an unwritable state after the lease expires.

[0079] Specifically, the lease of the cluster node that has occurred errors during the read-write process, the client can actively report to the management node when changing the state of the stream file corresponding to the lease, or can periodically report to the management node based on a certain strategy. On this basis, the data node in the lease of the cluster node can actively report the errors and device states occurring in the node to the management node based on a certain strategy periodically, so as to facilitate the management node to handle the failure. It should be noted that the reported failure includes but is not limited to data node downtime, data node operation and maintenance upgrade, network error and disk read-write error number. Further, the management node detects the node device through the report of the client and the data node, and immediately records the device failure time and failure type when detecting the failure; then distributes the device failure information to other clients needed in the cluster according to a preset strategy. Further, the management node will actively avoid the data node with device failure in subsequent allocation of leases, and allocate other normal devices to replace the node, so as to avoid subsequent writing errors. After the failed device recovers to normal, continue to allocate.

[0080] In the technical solution, the client simultaneously requests more than the lease allocation request of the data to be written, and generates corresponding stream files based on the obtained allocation lease or backup lease for writing data; when the client finds that any data node included in any group of allocation leases cannot successfully write the data to be written into the cluster, the client immediately obtains the data node of the backup lease, writes the data to be written into the cluster through the data node of the backup lease, and stores the data write information of the data to be written, so as to improve the reliability of data writing, and further to fully utilize the cluster resources. Further, the client also reports the data node that fails to successfully write the data to be written into the cluster to the management node, so that the management node avoids the failure node when allocating the nodes, and further improves the service performance of the cluster.

[0081] Figure 3 A structural diagram of a data writing device provided by the embodiment of the application is shown in FIG. 3. Figure 3 The device comprises a cluster node lease obtaining module 310, a data writing module 320 and an information storage module 330, wherein

[0082] The cluster node lease obtaining module 310 is configured to obtain multiple groups of data to be written, generate multiple cluster node leases based on the multiple groups of data to be written, and generate corresponding stream files in the cluster based on each cluster node lease. The cluster node lease is used to enable the client to write a preset number of data into the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time. The multiple cluster node leases comprise allocation leases corresponding to the number of the multiple groups of data to be written and at least one backup lease.

[0083] The data writing module 320 is configured to, for any group of data to be written, select any backup lease within a preset time when it is determined that the current data to be written is not successfully written into the allocation stream file corresponding to the allocation lease based on the current data to be written, and write the current data to be written into the backup stream file corresponding to the backup lease based on the backup lease.

[0084] The information storage module 330 is configured to receive data write information corresponding to the data to be written, and store the data write information, so that the client reads data based on the data write information.

[0085] Optionally, the cluster node lease obtaining module 310 is specifically configured to:

[0086] Generate n+m lease allocation requests based on n groups of data to be written, and send each lease allocation request to a preset management node in the cluster;

[0087] Receive n+m cluster node leases sent by the management node; wherein the node leases comprise n allocation leases and m backup leases.

[0088] Optionally, the cluster node lease acquisition module 310 is specifically configured to:

[0089] For any cluster node lease, obtain the node information of each data node included in the current cluster node lease;

[0090] A stream file corresponding to the cluster node lease is generated in the cluster based on the information of each node, so that the client writes the data to be written to the cluster based on the current cluster node lease.

[0091] Optionally, the data writing module 320 is specifically configured to:

[0092] For any group of data to be written, determine the node information of the data nodes included in the allocation lease corresponding to the current group of data to be written, and shard the data to be written based on the data node information to obtain multiple shard data;

[0093] Based on the preset time, each data node writes each shard data to the allocation stream file corresponding to the allocation lease.

[0094] Optionally, the data writing module 310 is specifically configured to:

[0095] Receive the write results of each shard data fed back by each data node; the write results include write success and write failure;

[0096] If the number of successfully written results in each write result is less than a preset number threshold, it is determined that the to-be-written data is not successfully written to the allocated stream file.

[0097] Optionally, the data writing module 310 is specifically configured to:

[0098] For any successfully written shard data, receive the data write information fed back by the data node corresponding to the current shard data; the data write information includes the stream file ID, stream file location and shard data information;

[0099] Write data write information to the preset metadata node in the cluster in a transactional manner.

[0100] Optionally, the device further includes:

[0101] Obtain the lease of each faulty cluster node that has not successfully written the data to be written, and send the node information of the data nodes respectively contained in each faulty cluster node lease to a preset management node in the cluster, so that the management node performs fault detection on each data node.

[0102] Figure 4 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. Figure 4 As shown, the server of this embodiment may include:

[0103] at least one processor 401; and

[0104] a memory 402 communicatively coupled to at least one processor;

[0105] The memory 402 stores instructions that can be executed by at least one processor 401 , and the instructions are executed by at least one processor 401 to enable the server to execute a method as described in any of the above embodiments.

[0106] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0107] The implementation principle and technical effects of the server provided in this embodiment can be found in the aforementioned embodiments and will not be repeated here.

[0108] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0109] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is only a logical function division. In actual implementation, other division methods may be used. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not implemented.

[0111] The above-mentioned integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0112] It should be appreciated that the above-described processor can be a central processing unit (CPU), a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or the like. The general purpose processor can be a microprocessor or the processor can be any conventional processor. The steps of a method disclosed in conjunction with the application can be implemented directly in hardware, in a software module executed by a processor or in a combination of the two. The memory can include a RAM memory and can also include a non-volatile memory NVM such as at least one disk memory, a USB memory, a read only memory, a magnetic or optical disk, or the like.

[0113] The storage medium can be realized by any type of volatile or non-volatile storage devices or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read only memory (EEPROM), a programmable read only memory (EPROM), a programmable read only memory (PROM), a read only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0114] An exemplary storage medium is coupled to the processor so that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can be a part of the processor. The processor and the storage medium can be located in an application specific integrated circuits (ASIC). Of course, the processor and the storage medium can exist as separate components in a server or a master device.

[0115] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0116] It should be understood that the application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the appended claims.

Claims

1. A data writing method, characterized in that: Applied to a client, the client is in communication connection with a cluster, and is used to write data to be written into the cluster. The method includes: Acquire multiple groups of data to be written, and multiple cluster node leases generated based on the multiple groups of data to be written, and generate corresponding stream files in the cluster based on each of the cluster node leases; the cluster node leases are used to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time; the multiple cluster node leases include allocation leases corresponding to the number of the multiple groups of data to be written and at least one standby lease; For any set of data to be written, when it is determined that the current data to be written has not been successfully written into the allocated stream file corresponding to the allocated lease based on the allocated lease corresponding to the current data to be written, any backup lease is selected within a preset time, and the current data to be written is written into the backup stream file corresponding to the current backup lease based on the backup lease; Data writing information corresponding to the data to be written is received, and the data writing information is stored, so that the client reads the data based on the data writing information.

2. The method according to claim 1, characterized in that The multiple cluster node leases generated based on the multiple groups of data to be written include: Generate n+m lease allocation requests based on n groups of data to be written, and send each of the lease allocation requests to a preset management node in the cluster; Receive n+m cluster node leases sent by the management node; wherein the node leases include n allocation leases and m standby leases.

3. The method according to claim 1, characterized in that Generating corresponding stream files in the cluster based on the leases of the cluster nodes includes: For any cluster node lease, obtain the node information of each data node included in the current cluster node lease; A stream file corresponding to the cluster node lease is generated in the cluster based on information of each node, so that the client writes the to-be-written data to the cluster based on the current cluster node lease.

4. The method according to claim 1, wherein The client writes a preset amount of data to a stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time, including: For any group of data to be written, determine the node information of the data nodes included in the allocation lease corresponding to the current group of data to be written, and shard the data to be written based on the data node information to obtain multiple shard data; Based on each of the data nodes writing each of the shard data into the allocation stream file corresponding to the allocation lease within a preset time.

5. The method according to claim 4, characterized in that The determining that the current data to be written is not successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written includes: Receiving the write results of each shard data fed back by each data node respectively; the write results include write success and write failure; If the number of successfully written results in each of the writing results is less than a preset number threshold, it is determined that the to-be-written data is not successfully written into the allocated stream file.

6. The method according to claim 1, characterized in that The receiving data writing information corresponding to the data to be written and storing the data writing information includes: For any successfully written shard data, receive data write information fed back by the data node corresponding to the current shard data; the data write information includes the stream file ID, stream file location and shard data information; The data write information is written to a preset metadata node in the cluster in a transactional manner.

7. The method according to claim 1, characterized in that The method further comprises: Obtain the lease of each faulty cluster node that has not successfully written the data to be written, and send the node information of the data nodes respectively contained in each faulty cluster node lease to a preset management node in the cluster, so that the management node performs fault detection on each data node.

8. A data writing device, characterized in that: Applied to a client, the client is connected to a cluster for communication, and is used to write data to be written into the cluster. The device includes: A cluster node lease acquisition module, configured to acquire multiple groups of data to be written and multiple cluster node leases generated based on the multiple groups of data to be written, and to generate corresponding stream files in the cluster based on each of the cluster node leases; the cluster node leases are configured to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease via a preset number of data nodes in the cluster within a preset time; the multiple cluster node leases include allocation leases corresponding to the number of the multiple groups of data to be written and at least one standby lease; a data writing module configured to, for any set of data to be written, select any backup lease within a preset time period when determining that the data to be written to the allocated stream file corresponding to the allocated lease has not been successfully written to the allocated stream file corresponding to the allocated lease based on the allocated lease, and write the data to be written to the backup stream file corresponding to the current backup lease based on the backup lease; The information storage module is used to receive data writing information corresponding to the data to be written, and store the data writing information so that the client can read the data based on the data writing information.

9. A server, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images