Data writing method and device, server and storage medium

Abstract

The invention provides a data writing method and device, a server and a storage medium. The method comprises the steps of obtaining multiple groups of to-be-written data, generating multiple cluster node leases based on the multiple groups of to-be-written data, and generating a corresponding stream file in a cluster based on each cluster node lease; for any group of to-be-written data, when it is determined that the current to-be-written data is not successfully written into the distribution stream file corresponding to the distribution lease based on the distribution lease corresponding to the current to-be-written data, selecting any standby lease within preset time, writing the current to-be-written data into a standby stream file corresponding to the current standby lease based on the standby lease; and receiving data write-in information corresponding to the to-be-written data, and storing the data write-in information, so that the client performs data reading based on the data write-in information. According to the method provided by the invention, the technical problem of low data reliability caused by the fault of the distributed branch node in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage, and in particular to a data writing method, device, server and storage medium. Background Art

[0002] With the rise and popularization of cloud computing technology, distributed storage systems have begun to attract more and more attention from the industry. Traditional network storage systems use centralized storage servers to store all data. Storage servers become the bottleneck of system performance and the focus of reliability and security, and cannot meet the needs of large-scale storage applications. Distributed network storage systems use an extensible system structure and use multiple storage servers to share the storage load, which not only improves the reliability, availability and access efficiency of the system, but also makes it easy to expand. For distributed storage, the stability and reliability of the cluster are crucial.

[0003] When a storage node in a distributed system fails, it often has a significant impact on the stability and reliability parameters of the cluster, and may even cause the storage system to be temporarily unavailable. In this case, the client's data reading and writing may be affected, thereby reducing the data reliability of the distributed system. Summary of the invention

[0004] The present application provides a data writing method, device, server and storage medium, which are used to solve the technical problem in the prior art that when a distributed drop node fails, data reliability is low.

[0005] In a first aspect, the present application provides a data writing method, comprising: being applied to a client, the client being communicatively connected to a cluster, and being used to write data to be written into the cluster, the method comprising:

[0006] Acquire multiple groups of data to be written, and multiple cluster node leases generated based on the multiple groups of data to be written, and generate corresponding stream files in the cluster based on each of the cluster node leases; the cluster node lease is used to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time; the multiple cluster node leases include an allocation lease corresponding to the number of multiple groups of data to be written and at least one standby lease;

[0007] For any group of data to be written, when it is determined that the current data to be written has not been successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written, any standby lease is selected within a preset time, and the current data to be written is written into the standby stream file corresponding to the current standby lease based on the standby lease;

[0008] Data writing information corresponding to the data to be written is received, and the data writing information is stored, so that the client reads data based on the data writing information.

[0009] Optionally, the multiple cluster node leases generated based on the multiple groups of data to be written include:

[0010] Generate n+m lease allocation requests based on n groups of data to be written, and send each of the lease allocation requests to a preset management node in the cluster;

[0011] Receive n+m cluster node leases sent by the management node; wherein the node leases include n allocation leases and m standby leases.

[0012] Optionally, generating a corresponding stream file in the cluster based on the lease of each cluster node includes:

[0013] For any cluster node lease, obtain the node information of each data node included in the current cluster node lease;

[0014] A stream file corresponding to the cluster node lease is generated in the cluster based on information of each node, so that the client writes the data to be written to the cluster based on the current cluster node lease.

[0015] Optionally, the client writes a preset amount of data to a stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time, including:

[0016] For any group of data to be written, determine the node information of the data nodes included in the allocation lease corresponding to the current group of data to be written, and perform sharding processing on the data to be written based on the data node information to obtain multiple sharded data;

[0017] Based on each of the data nodes, each of the shard data is written to the allocation stream file corresponding to the allocation lease within a preset time.

[0018] Optionally, the determining that the current data to be written is not successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written includes:

[0019] Respectively receiving the writing results of each shard data fed back by each data node; the writing results include writing success and writing failure;

[0020] If the number of successfully written results in each of the writing results is less than a preset number threshold, it is determined that the to-be-written data is not successfully written into the allocated stream file.

[0021] Optionally, the receiving data writing information corresponding to the data to be written, and storing the data writing information includes:

[0022] For any successfully written shard data, receive the data writing information fed back by the data node corresponding to the current shard data; the data writing information includes the stream file ID, stream file location and shard data information;

[0023] The data write information is written to a preset metadata node in the cluster in a transactional manner.

[0024] Optionally, the method further includes:

[0025] The leases of the failed cluster nodes that have not successfully written the data to be written are obtained, and the node information of the data nodes respectively included in the leases of the failed cluster nodes are sent to a management node preset in the cluster, so that the management node performs fault detection on each of the data nodes.

[0026] In a second aspect, the present application provides a data writing device,

[0027] Applied to a client, the client is connected to a cluster for communication, and is used to write data to be written into the cluster. The device includes:

[0028] A cluster node lease acquisition module, used to acquire multiple groups of data to be written, and multiple cluster node leases generated based on the multiple groups of data to be written, and generate corresponding stream files in the cluster based on each of the cluster node leases; the cluster node lease is used to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time; the multiple cluster node leases include an allocation lease corresponding to the number of multiple groups of data to be written and at least one standby lease;

[0029] A data writing module, for any group of data to be written, when it is determined that the current data to be written has not been successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written, select any standby lease within a preset time, and write the current data to be written into the standby stream file corresponding to the current standby lease based on the standby lease;

[0030] The information storage module is used to receive data writing information corresponding to the data to be written, and store the data writing information so that the client can read data based on the data writing information.

[0031] In a third aspect, the present application provides a server, comprising: a processor, and a memory communicatively connected to the processor;

[0032] The memory stores computer-executable instructions;

[0033] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0036] The technical solution provided by the present application is that the client simultaneously requests lease allocation requests that are more than the data to be written, and generates corresponding stream files for writing data based on the obtained allocation lease or standby lease; when the client finds that any group of data nodes included in the allocation lease cannot successfully write the data to be written into the cluster, it immediately obtains the data node of the standby lease, writes the data to be written into the cluster through the data node of the standby lease, and stores the data writing information of the data to be written, so as to improve the reliability of data writing, and then make full use of cluster resources. Furthermore, the client will also report the data nodes that have failed to successfully write the data to be written into the cluster to the management node, so that the management node can avoid faulty nodes and make reasonable allocation when performing node allocation, thereby improving the service performance of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] Figure 1 An application scenario diagram of the data writing method provided in this application;

[0039] Figure 2 A schematic diagram of a data writing method provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the structure of a data writing device provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present application.

[0042] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] In actual applications, in large-scale distributed storage systems, some devices may have hardware failures or be unable to provide services due to maintenance. At this time, the client's data reading and writing will be affected, resulting in reduced data reliability in the distributed system. For the above situation, existing products in the prior art provide cloud storage services, which define clients, stream management nodes, and data nodes. However, this service focuses on the processing of a single stream, and in order to ensure strong consistency, when an error is detected, it has to spend more time on waiting and fault recovery. There may still be high latency in single-point failures, which brings a bad user experience.

[0046] The data writing method provided by the present application is intended to solve the above technical problems of the prior art. Specifically, the client can request to generate multiple stream files for writing data at the same time; when a data writing error is detected, the client can quickly switch to other generated and normally working stream files in the cluster to write data, so as to avoid writing data to the cluster through the faulty node, while ensuring data reliability, reducing unnecessary write request failures and increased latency, and making better use of cluster resources.

[0047] Figure 1 This is an application scenario diagram of the data writing method provided in this application. Figure 1 The application scenarios to which the embodiments of the present application are applicable are described. Figure 1, including a client and a cluster; specifically, the cluster includes a management node, a data node and a metadata node. Among them, the client is used to write data to the cluster and read data; the management node is used to allocate data nodes to the client when requesting to write data, so that the client can successfully write data; the data node is used to write the data to be written in the client into the cluster; the metadata node is used to store the data writing information written by the client into the cluster. Specifically, the client sends multiple lease acquisition requests to the management node to obtain the data nodes allocated by the management node, so as to write the data to be written into the cluster through the data nodes; further, when the client finds that any group of leased data nodes cannot successfully write the data to be written into the cluster, it immediately obtains the data nodes of the standby lease, writes the data to be written into the cluster through the data nodes of the standby lease, and stores the data writing information of the data to be written, so as to improve the reliability of data writing, and then make full use of cluster resources. Furthermore, the client will also report the data nodes that have failed to successfully write the data to be written into the cluster to the management node, so that the management node can avoid faulty nodes and make reasonable allocation when allocating nodes, and further improve the data writing performance.

[0048] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0049] Figure 2 The flowchart of a data writing method provided in an embodiment of the present application is shown in FIG. The method can be executed by a data writing device, which can be a server or a server. The following description takes the server as an example. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware, such as Figure 2 As shown, the method includes the following steps.

[0050] S210, obtaining multiple groups of data to be written and multiple cluster node leases generated based on the multiple groups of data to be written, and generating corresponding stream files in the cluster based on each cluster node lease.

[0051] In an embodiment of the present application, the data to be written may include application data generated by an upper-layer application installed in the client during operation, and may also include operation data generated by the client during operation. The present application does not limit the data content of the data to be written.

[0052] In order to better store the data to be written, the data to be written can be written into a distributed cluster that is connected to the client for data storage. In the present application, a plurality of data nodes are pre-set in the cluster for writing the data to be written in the client into the cluster for data storage. Optionally, a management node is also provided in the cluster to manage the data nodes in the cluster and to allocate different data nodes to different clients when receiving a client request. Optionally, the management node can be any node selected from the data nodes or a pre-set node, and the present application does not make specific restrictions on this.

[0053] In actual applications, in order to improve data processing efficiency, the client can write multiple sets of data to be written to the cluster at the same time. In order to ensure that the data is written to the cluster smoothly, before writing the data, it is necessary to send a cluster node lease acquisition request to the cluster based on the multiple sets of data to be written in advance, so as to obtain multiple cluster node leases allocated by the management node in the cluster, and write the data to be written in the client to the cluster through the streaming file generated based on the cluster node lease.

[0054] In an embodiment of the present application, the cluster node lease includes an allocation lease corresponding to the number of multiple groups of data to be written and at least one standby lease; in order not to waste cluster resources and improve the utilization rate of each data node in the cluster, any allocation lease of data to be written in the present application can be used as a standby lease for any other group of data to be written when idle. In other words, any allocation lease can be used as a standby lease for other data to be written when the current group of data to be written is not processed, and when the allocation lease of other groups of data to be written fails to successfully write other groups of data to be written into the cluster, other groups of data to be written are written into the cluster.

[0055] It should be explained that the cluster node lease is used to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time. In other words, when the client writes data to the cluster, the corresponding data to be written is written through the data node allocated by the management node to improve the cluster operation efficiency.

[0056] Optionally, in an embodiment of the present application, the specific process of the client generating multiple cluster node leases based on multiple groups of data to be written may include: generating n+m lease allocation requests based on n groups of data to be written, and sending each lease allocation request to a preset management node in the cluster; receiving n+m cluster node leases sent by the management node; wherein the node lease includes n allocation leases and m standby leases.

[0057] Since the data nodes included in the cluster node lease allocated by the cluster to the client may fail or be temporarily operated and maintained, resulting in the inability to provide services, in order to ensure that data can be written to the cluster smoothly, the client can send a lease allocation request to the cluster with a number greater than the number of data groups to be written, so as to ensure that when a data node in the allocated cluster node lease fails later, the backup lease can be enabled for data writing.

[0058] Specifically, the client generates a lease allocation request based on the acquired data to be written. For example, when it is determined that there are n groups of data to be written in the client that need to be written into the cluster for storage, n+m lease allocation requests are generated to obtain multiple cluster node leases. Further, the lease allocation request is sent to a preset management node in the cluster. The management node generates n+m cluster node leases based on the received lease allocation request and the node status of each data node in the cluster, and feeds back the n+m cluster node leases to the client.

[0059] It should be noted that the n+m cluster node leases include n allocation leases and m standby leases. In other words, if the data node included in the allocation lease fails to successfully write the data to be written into the cluster, the standby lease is enabled to write the data, thereby improving the data writing efficiency and further improving the cluster resource utilization.

[0060] Furthermore, when the client obtains the cluster node lease allocated by the cluster, it is also necessary to generate a corresponding stream file in the cluster based on the data nodes included in the cluster node lease, so that the client can write the data to be written into the cluster. Optionally, the specific process of generating a stream file in the present application may include: for any cluster node lease, obtaining the node information of each data node included in the current cluster node lease; generating a stream file corresponding to the cluster node lease in the cluster based on the information of each node, so that the client can write the data to be written into the cluster based on the current cluster node lease.

[0061] In an embodiment of the present application, since the cluster node lease includes multiple data nodes, and the data to be written needs to be written into the cluster through each data node, it is necessary to obtain the node information of each data node, and then request to generate the corresponding stream file in the cluster based on the node information, so as to facilitate the subsequent writing and processing of the data to be written. It should be noted that in the present application, the stream file generated based on the cluster node lease can only be written to the corresponding data to be written. Specifically, each data node can write the data to be written into the stream file by appending, so that the client can write the data to be written into the cluster based on the current cluster node lease.

[0062] S220. For any group of data to be written, when it is determined that the current data to be written has not been successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written, select any backup lease within a preset time, and write the current data to be written into the backup stream file corresponding to the current backup lease based on the backup lease.

[0063] In an embodiment of the present application, the specific process of the client writing the data to be written into the stream file through each data node in the cluster node lease may include: for any group of data to be written, determining the node information of the data nodes contained in the allocation lease corresponding to the current group of data to be written, and sharding the data to be written based on the data node information to obtain multiple shard data; based on each data node, writing each shard data to the allocation stream file corresponding to the allocation lease within a preset time.

[0064] Specifically, in order to improve the efficiency of data writing and make full use of data node resources, before writing the data to be written, the data to be written can be pre-sliced ​​to obtain multiple slicing data. Exemplarily, the allocation lease corresponding to the data to be written can be determined, and then the data to be written can be sliced ​​in multiple copies or erasure codes according to the number of data nodes included in the allocation lease to obtain multiple slicing data, and then write requests are sent to the data nodes respectively, so as to write each slicing data to the allocation stream file corresponding to the allocation lease in an appended manner based on each data node, so as to write the data to the cluster.

[0065] In actual applications, in order to ensure that the allocated lease can provide services normally, that is, the data to be written in the client can be successfully written into the cluster, after the data to be written is processed by the data nodes in the cluster, the data writing results fed back by the data nodes can be received, and then based on the writing results, it can be determined whether the data to be written is successfully written into the cluster, thereby further ensuring the reliability of data storage.

[0066] Optionally, in an embodiment of the present application, the specific process of determining whether the data to be written is successfully written into the cluster may include: receiving the write results of each shard data fed back by each data node respectively; the write results include write success and write failure; if the number of write successes in each write result is less than a preset number threshold, it is determined that the data to be written is not successfully written to the allocated stream file.

[0067] Specifically, when the client writes the data to be written into the cluster by allocating the data nodes in the lease, the write results of the shard data fed back by each data node are received, and the number of successful writes received within a preset time is counted. Further, if it is determined that the number of successful writes in each write result is greater than a preset number threshold, it is determined that the data to be written is successfully written into the allocated stream file.

[0068] Optionally, if the number of successfully written results in each writing result is less than a preset number threshold, it is determined that the data to be written is not successfully written to the allocated stream file.

[0069] Furthermore, in order to achieve successful data writing, it is necessary to obtain other standby leases allocated to the client within a preset time, and generate a standby stream file corresponding to the data to be written based on each data node in the standby lease, and then write the data to be written into the standby stream file based on the data nodes included in the standby lease to achieve writing into the cluster.

[0070] Optionally, when using a backup lease to write the data to be written, in order to ensure the success rate of data writing, the data to be written can be written based on multiple backup leases at the same time, and when it is determined that any of the backup leases has successfully written the data to be written into the cluster, the write processing of other backup leases is stopped.

[0071] S230: Receive data writing information corresponding to the data to be written, and store the data writing information so that the client can read the data based on the data writing information.

[0072] In an embodiment of the present application, when it is determined based on the above implementation that the data to be written is successfully written into the cluster by allocating a lease or a standby lease, the data write information fed back by the data node in the allocated lease or the standby lease is received and the data write information is stored so that subsequent clients can read the above data in the cluster.

[0073] Optionally, the specific process of receiving data write information corresponding to the data to be written and storing the data write information in an embodiment of the present application may include: for any successfully written shard data, receiving data write information fed back by the data node corresponding to the current shard data; the data write information includes the stream file ID, the stream file location and the shard data information; and writing the data write information to a preset metadata node in the cluster in a transactional manner.

[0074] Specifically, after successfully writing the shard data, the data node feeds back the stream file ID, stream file location and corresponding shard data information contained in the write result information to the client. Furthermore, the client stores the above information in a preset metadata node in the cluster, so that when data needs to be read later, data reading operations can be performed directly based on the data write information stored in the metadata node.

[0075] Based on the above implementation method, in the embodiment of the present application, the client will also perform information reporting processing on the management node in the cluster based on the data writing status of the data to be written, so that the management node in the cluster can know the service status of each data node in the cluster in real time, and then manage each data node.

[0076] Optionally, the specific process of the client performing information reporting processing may include: obtaining the lease of each faulty cluster node that failed to successfully write the data to be written, and sending the node information of the data nodes respectively contained in each faulty cluster node lease to a preset management node in the cluster, so that the management node performs fault detection on each data node.

[0077] In the embodiment of the present application, the failed cluster node lease is an allocation lease or a standby lease. Specifically, if any lease is determined to be a failed cluster node lease during the data writing process, the client sends a request to the management node to cut off the last successful write position of the stream file corresponding to the lease, and changes the state of the stream file to a non-writable state to prevent the data node from continuing to write data to the stream file and wasting cluster resources.

[0078] Optionally, if the client fails to write data to the stream file through the data node within a preset time, the management node may directly modify the state of the stream file to a non-writable state after the lease times out.

[0079] Specifically, for cluster node leases that have experienced errors during the reading and writing process, the client can actively report to the management node when changing the status of the stream file corresponding to the lease, or it can periodically report to the management node based on a certain strategy. On this basis, the data node in the cluster node lease can periodically and actively report to the management node the errors and device status that occurred in the node based on a certain strategy, so that the management node can handle the fault. It should be noted that the reported faults include but are not limited to data node downtime, data node operation and maintenance upgrades, network errors, and disk read and write errors. Further, the management node detects faults on the node device through reports from the client and data node, and immediately records the device failure time and failure type when a fault is detected; and then distributes the device failure information to other clients in the cluster according to the preset strategy. Further, the management node will actively avoid the data node with device failure in the subsequent allocation of leases, and allocate other normal devices to replace the node, thereby avoiding subsequent write errors. After the faulty device returns to normal, continue to allocate.

[0080] In the above technical solution, the client simultaneously requests more lease allocation requests than the data to be written, and generates corresponding stream files for writing data based on the obtained allocation lease or standby lease; when the client finds that any group of data nodes included in the allocation lease cannot successfully write the data to be written into the cluster, it immediately obtains the data node of the standby lease, writes the data to be written into the cluster through the data node of the standby lease, and stores the data writing information of the data to be written, so as to improve the reliability of data writing and make full use of cluster resources. Furthermore, the client will also report the data nodes that have failed to successfully write the data to be written into the cluster to the management node, so that the management node can avoid faulty nodes and make reasonable allocation when allocating nodes, thereby improving the service performance of the cluster.

[0081] Figure 3 This is a schematic diagram of the structure of a data writing device provided in an embodiment of the present application. Figure 3 The device includes: a cluster node lease acquisition module 310, a data writing module 320 and an information storage module 330; wherein,

[0082] The cluster node lease acquisition module 310 is used to acquire multiple groups of data to be written and multiple cluster node leases generated based on the multiple groups of data to be written, and generate corresponding stream files in the cluster based on each cluster node lease; the cluster node lease is used to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time; the multiple cluster node leases include an allocation lease corresponding to the number of the multiple groups of data to be written and at least one standby lease;

[0083] The data writing module 320 is used for selecting any standby lease within a preset time for any group of data to be written, and writing the data to be written to the standby stream file corresponding to the current standby lease based on the standby lease when it is determined that the data to be written to the current allocation stream file corresponding to the allocation lease is not successfully written based on the allocation lease corresponding to the data to be written to the current allocation stream file;

[0084] The information storage module 330 is used to receive data writing information corresponding to the data to be written, and store the data writing information so that the client can read the data based on the data writing information.

[0085] Optionally, the cluster node lease acquisition module 310 is specifically used to:

[0086] Generate n+m lease allocation requests based on n groups of data to be written, and send each lease allocation request to a preset management node in the cluster;

[0087] Receive n+m cluster node leases sent by the management node; wherein the node leases include n allocation leases and m standby leases.

[0088] Optionally, the cluster node lease acquisition module 310 is specifically used to:

[0089] For any cluster node lease, obtain the node information of each data node included in the current cluster node lease;

[0090] A stream file corresponding to the cluster node lease is generated in the cluster based on the information of each node, so that the client writes the data to be written to the cluster based on the current cluster node lease.

[0091] Optionally, the data writing module 320 is specifically used for:

[0092] For any group of data to be written, determine the node information of the data nodes included in the allocation lease corresponding to the current group of data to be written, and perform sharding processing on the data to be written based on the data node information to obtain multiple sharded data;

[0093] Based on each data node, each shard data is written to the allocation stream file corresponding to the allocation lease within a preset time.

[0094] Optionally, the data writing module 310 is specifically used for:

[0095] Receive the write results of each shard data fed back by each data node respectively; the write results include write success and write failure;

[0096] If the number of successfully written results in each writing result is less than a preset number threshold, it is determined that the data to be written is not successfully written to the allocated stream file.

[0097] Optionally, the data writing module 310 is specifically used for:

[0098] For any successfully written shard data, receive the data writing information fed back by the data node corresponding to the current shard data; the data writing information includes the stream file ID, stream file location and shard data information;

[0099] Write data write information to the preset metadata node in the cluster in a transactional manner.

[0100] Optionally, the device further comprises:

[0101] The leases of the failed cluster nodes that have not successfully written the data to be written are obtained, and the node information of the data nodes respectively included in the leases of the failed cluster nodes are sent to the management node preset in the cluster, so that the management node performs fault detection on each data node.

[0102] Figure 4 A schematic diagram of the structure of a server provided in an embodiment of the present application. Figure 4 As shown, the server of this embodiment may include:

[0103] at least one processor 401; and

[0104] A memory 402 communicatively coupled to at least one processor;

[0105] The memory 402 stores instructions that can be executed by at least one processor 401 , and the instructions are executed by at least one processor 401 to enable the server to execute a method as described in any of the above embodiments.

[0106] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0107] The implementation principle and technical effects of the server provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0108] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of any of the aforementioned embodiments is implemented.

[0109] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of any of the aforementioned embodiments when executed by a processor.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0111] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0112] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0113] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0114] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in a server or a main control device.

[0115] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0116] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data writing method, characterized in that: Applied to a client, the client is connected to a cluster for communication, and is used to write data to be written into the cluster. The method includes: Acquire multiple groups of data to be written, and multiple cluster node leases generated based on the multiple groups of data to be written, and generate corresponding stream files in the cluster based on each of the cluster node leases; the cluster node lease is used to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time; the multiple cluster node leases include an allocation lease corresponding to the number of multiple groups of data to be written and at least one standby lease; For any group of data to be written, when it is determined that the current data to be written has not been successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written, any standby lease is selected within a preset time, and the current data to be written is written into the standby stream file corresponding to the current standby lease based on the standby lease; Data writing information corresponding to the data to be written is received, and the data writing information is stored, so that the client reads data based on the data writing information.

2. The method according to claim 1, characterized in that The multiple cluster node leases generated based on the multiple groups of data to be written include: Generate n+m lease allocation requests based on n groups of data to be written, and send each of the lease allocation requests to a preset management node in the cluster; Receive n+m cluster node leases sent by the management node; wherein the node leases include n allocation leases and m standby leases.

3. The method according to claim 1, characterized in that The generating a corresponding stream file in the cluster based on the lease of each cluster node includes: For any cluster node lease, obtain the node information of each data node included in the current cluster node lease; A stream file corresponding to the cluster node lease is generated in the cluster based on information of each node, so that the client writes the data to be written to the cluster based on the current cluster node lease.

4. The method according to claim 1, characterized in that: The client writes a preset amount of data to a stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time, including: For any group of data to be written, determine the node information of the data nodes included in the allocation lease corresponding to the current group of data to be written, and perform sharding processing on the data to be written based on the data node information to obtain multiple sharded data; Based on each of the data nodes, each of the shard data is written to the allocation stream file corresponding to the allocation lease within a preset time.

5. The method according to claim 4, characterized in that The determining that the current data to be written is not successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written includes: Respectively receiving the writing results of each shard data fed back by each data node; the writing results include writing success and writing failure; If the number of successfully written results in each of the writing results is less than a preset number threshold, it is determined that the to-be-written data is not successfully written into the allocated stream file.

6. The method according to claim 1, characterized in that The receiving data writing information corresponding to the data to be written, and storing the data writing information, includes: For any successfully written shard data, receive the data writing information fed back by the data node corresponding to the current shard data; the data writing information includes the stream file ID, stream file location and shard data information; The data write information is written to a preset metadata node in the cluster in a transactional manner.

7. The method according to claim 1, characterized in that The method further comprises: The leases of the failed cluster nodes that have not successfully written the data to be written are obtained, and the node information of the data nodes respectively included in the leases of the failed cluster nodes are sent to a management node preset in the cluster, so that the management node performs fault detection on each of the data nodes.

8. A data writing device, characterized in that: Applied to a client, the client is connected to a cluster for communication, and is used to write data to be written into the cluster. The device includes: A cluster node lease acquisition module, used to acquire multiple groups of data to be written, and multiple cluster node leases generated based on the multiple groups of data to be written, and generate corresponding stream files in the cluster based on each of the cluster node leases; the cluster node lease is used to enable the client to write a preset amount of data to the stream file corresponding to the cluster node lease through a preset number of data nodes in the cluster within a preset time; the multiple cluster node leases include an allocation lease corresponding to the number of multiple groups of data to be written and at least one standby lease; A data writing module, for any group of data to be written, when it is determined that the current data to be written has not been successfully written into the allocation stream file corresponding to the allocation lease based on the allocation lease corresponding to the current data to be written, select any standby lease within a preset time, and write the current data to be written into the standby stream file corresponding to the current standby lease based on the standby lease; The information storage module is used to receive data writing information corresponding to the data to be written, and store the data writing information so that the client can read data based on the data writing information.

9. A server, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images