Distributed cluster data backup method and device, equipment and storage medium

Abstract

The invention provides a distributed cluster data backup method, which is applied to a distributed cluster data backup system, the distributed cluster data backup system comprises a distributed cluster, a client and a server, the client is a node in the distributed cluster, and the distributed cluster data backup method is characterized in that the client is a node in the distributed cluster. The method comprises the following steps: the client receives a backup instruction sent by the server; the client side inquires metadata of data to be backed up according to the backup instruction, the metadata is stored in the client side, and the metadata comprises position information of the data to be backed up; the client reads the data to be backed up from a distributed cluster according to the metadata; and the client sends the data to be backed up to the server through a high-speed serial bus, and the server receives and stores the data to be backed up.

Description

Technical Field

[0001] The present invention relates to the field of distributed cluster technology, and in particular to a distributed cluster data backup method, device, equipment and storage medium. Background Art

[0002] The current backup system usually uses the upper-layer file system interface, object interface or block interface to access through the upper-layer business network of distributed storage. This will cause two problems in the backup process: first, when backing up a large number of small files, the upper-layer file, object interface or block interface needs to interact with the distributed system through the network, and the distributed system needs to interact directly with the metadata server through the internal network, and then transmit data through the network. Such multiple transmissions of metadata and data will bring a large number of TCP (Transmission Control Protocol) or IP (Internet Protocol) small data packets, and introduce access delays, thereby lengthening the backup window and even causing the backup to be unable to be completed during non-business hours; second, even if large files are backed up, due to data transmission through the business network, a large amount of backup data will occupy a large amount of bandwidth of the business network, which will affect the business host that accesses the distributed system at the front end.

[0003] Therefore, in view of the shortcomings of the prior art solutions, the present invention provides a distributed cluster data backup method. Summary of the invention

[0004] Based on this, it is necessary to provide a distributed cluster data backup method, device, equipment and storage medium to address the above technical problems.

[0005] On the one hand, a distributed cluster data backup method is provided, which is applied to a distributed cluster data backup system, wherein the distributed cluster data backup system includes a distributed cluster, a client and a server, wherein the client is a node in the distributed cluster, and the method includes: the client receives a backup instruction sent by the server; the client queries metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, and the metadata includes location information of the data to be backed up; the client reads the data to be backed up from the distributed cluster according to the metadata; the client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0006] Optionally, the method also includes: determining a target node from multiple nodes of the distributed cluster and using the target node as a client; connecting the client and the server via a universal serial bus; establishing a network connection between the client and the server via a universal serial bus protocol; and the client and the server transmitting data via a universal serial bus and a universal serial bus protocol.

[0007] Optionally, determining the target node from the multiple nodes of the distributed cluster includes: detecting the multiple nodes in the distributed cluster in turn, and determining whether there is a metadata management unit in the multiple nodes; acquiring the node where the metadata management unit exists, and determining the target node based on the performance parameters of the node.

[0008] Optionally, before the client receives the backup instruction sent by the server, the method also includes: the server sends a metadata read instruction; the client receives the metadata read instruction and sends the metadata to the server via the high-speed serial bus; the server reads the metadata, detects metadata that has not been backed up, and generates the backup instruction.

[0009] Optionally, the method also includes: determining a first transmission speed and a second transmission speed of the client according to demand, wherein the second transmission speed is greater than the first transmission speed; the client receives data sent by the server according to the first transmission speed; and the client sends the data to be backed up to the server according to the second transmission speed.

[0010] Optionally, the distributed cluster data backup system also includes a processing server, which is connected to the client via the high-speed serial bus, and the client sends the data to be backed up to the server via the high-speed serial bus, and the server receives and stores the data to be backed up, including: the client sends the data to be backed up to the processing server via the high-speed serial bus; the processing server divides the data to be backed up into multiple data blocks, calculates the characteristic value of each data block, and compares the characteristic value of each data block with the characteristic value of the data block in the server in turn; when the characteristic value of the data block in the processing server is the same as the characteristic value of the data block in the server, the data block with the same characteristic value is marked; the marked data blocks are checked, and the data blocks that meet the deletion criteria are deleted; the processing server obtains the multiple data blocks after deletion, compresses the multiple data blocks, and obtains the compressed data to be backed up; the processing server sends the compressed data to be backed up to the server, and the server receives and stores the compressed data to be backed up.

[0011] Optionally, the method also includes: the server is connected to multiple clients respectively through multiple high-speed serial buses, wherein the multiple clients are nodes in multiple distributed clusters; the multiple clients send the data to be backed up of multiple distributed clusters to the server through multiple high-speed serial buses.

[0012] On the other hand, a distributed cluster data backup device is provided, which is arranged in a distributed cluster data backup system, and the distributed cluster data backup system includes a distributed cluster, a client and a server, the client is a node in the distributed cluster, and the device includes: a receiving module, used by the client to receive a backup instruction sent by the server; a query module, used by the client to query metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, and the metadata includes location information of the data to be backed up; a reading module, used by the client to read the data to be backed up from the distributed cluster according to the metadata; a backup module, used by the client to send the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0013] On the other hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: the client receives a backup instruction sent by the server; the client queries metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, and the metadata includes location information of the data to be backed up; the client reads the data to be backed up from a distributed cluster according to the metadata; the client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0014] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: the client receives a backup instruction sent by the server; the client queries metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, and the metadata includes location information of the data to be backed up; the client reads the data to be backed up from a distributed cluster according to the metadata; the client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0015] The above-mentioned distributed cluster data backup method, device, equipment and storage medium are applied to a distributed cluster data backup system, which includes a distributed cluster, a client and a server, wherein the client is a node in the distributed cluster, and the method includes: the client receives a backup instruction sent by the server; the client queries the metadata of the data to be backed up according to the backup instruction, wherein the metadata is stored in the client, and the metadata includes the location information of the data to be backed up; the client reads the data to be backed up from the distributed cluster according to the metadata; the client sends the data to be backed up to the server through a high-speed serial bus, and the server receives and stores the data to be backed up; in this way, the metadata is directly received and queried by the client, thereby avoiding the conversion of metadata representation and the overhead of obtaining metadata through the network, thereby speeding up the metadata query process and improving the query efficiency, and transmitting data through a high-speed serial bus directly can improve the transmission rate and more effectively reduce the cost, and can also decouple the data backup business from other businesses to improve the backup efficiency; at the same time, query services are performed in the client without performing processing operations, which can avoid the client being overloaded and affecting the performance and reliability of the entire cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a process of a distributed cluster data backup method in one embodiment;

[0017] Figure 2 A schematic diagram of the overall structure of a distributed cluster data backup method in one embodiment;

[0018] Figure 3 Another schematic diagram of the overall structure of a distributed cluster data backup method in one embodiment;

[0019] Figure 4 A structural block diagram of a distributed cluster data backup device in one embodiment;

[0020] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0023] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0024] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0025] In one embodiment, Figure 1 As shown, a distributed cluster data backup method is provided, which is applied to a distributed cluster data backup system. The distributed cluster data backup system includes a distributed cluster, a client and a server. The client is a node in the distributed cluster, and includes the following steps:

[0026] S101: The client receives a backup instruction sent by the server.

[0027] Here, the backup instruction includes metadata of the data to be backed up.

[0028] S102: The client queries metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata including location information of the data to be backed up.

[0029] Here, the data to be backed up can be files (such as NAS or gpfs), objects (such as S3), or blocks. Metadata is data that describes data, mainly information that describes data attributes, and is used to support functions such as indicating storage location, historical data, resource search, and file records. For example, metadata can include the physical location and offset of the data to be backed up on the disk.

[0030] S103: The client reads the data to be backed up from the distributed cluster according to the metadata.

[0031] Here, the data in the distributed cluster is distributed on different nodes of the distributed cluster to achieve redundant protection and distributed access of the data. The distributed cluster can be a storage cluster, a file system cluster, a virtualized hyper-converged cluster, a high-performance computing cluster, and an AI cluster, etc.

[0032] Specifically, the client determines the specific location where the data to be backed up is stored in the distributed cluster according to the metadata, finds the corresponding location of the corresponding node in the distributed cluster, and obtains the data to be backed up from the location.

[0033] S104: The client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0034] Here, the high-speed serial bus is a technology for quickly transmitting data between electronic devices, which may include PCIE, SATA (Serial Advanced Technology Attachment), USB (Universal Serial Bus), Ethernet, and IB (InfiniBand), etc.

[0035] Specifically, the client and the server perform data backup via a high-speed serial bus. This transmission channel is specifically used for transmission related to backup data. Other services between the client and the server are performed via other transmission channels.

[0036] In one embodiment, the client sends the metadata to the server via the high-speed serial bus.

[0037] It should be noted that the present application directly receives and queries metadata through the client, avoiding the conversion of metadata representation and the overhead of obtaining metadata through the network, thereby speeding up the metadata query process and improving query efficiency. By directly transmitting data through a high-speed serial bus, the transmission rate can be increased and costs can be reduced more effectively. The data backup business and other businesses can also be decoupled to improve backup efficiency. At the same time, query services are performed in the client without performing processing operations, which can avoid excessive client load affecting the performance and reliability of the entire cluster.

[0038] In some embodiments, the method further comprises:

[0039] Determine a target node from multiple nodes of the distributed cluster, and use the target node as a client;

[0040] Connecting the client and the server via a universal serial bus;

[0041] Establishing a network connection between the client and the server via a universal serial bus protocol;

[0042] The client and the server perform data transmission via a universal serial bus and a universal serial bus protocol.

[0043] Here, the high-speed serial bus may specifically be a universal serial bus, such as USB4, and correspondingly, the universal serial bus protocol may be an Ethernet over USB4 inter-domain protocol.

[0044] Specifically, the client and the server are connected via a USB4 cable, and a point-to-point network connection between the client and the server is established via the Ethernet over USB4 inter-domain protocol.

[0045] In this way, the connection between the client and the server is achieved through the universal serial bus, which improves the transmission speed. At the same time, compared with Ethernet and IB network, USB is more cost-effective while maintaining a high speed.

[0046] In some specific implementations, determining the target node from the multiple nodes of the distributed cluster includes:

[0047] sequentially detecting multiple nodes in the distributed cluster to determine whether a metadata management unit exists in the multiple nodes;

[0048] A node where a metadata management unit exists is obtained, and a target node is determined according to performance parameters of the node.

[0049] Here, the metadata management unit may serve the MDS.

[0050] Specifically, a node is selected from multiple nodes with metadata services in the distributed cluster, and the various performances of multiple nodes are compared to obtain the node with the best performance. The client software is installed on the node and the node is used as the client. The client software is responsible for sending data to the server. The client needs to be adapted to the software of the distributed cluster to directly retrieve metadata information from a specific node in the distributed cluster without using a general file system or protocol, and directly obtain the corresponding data.

[0051] In this way, the metadata of the data to be backed up can be queried directly by the client, and this query can be accelerated by the local memory. Compared with the traditional method of using NAS client or S3 client, it eliminates the conversion of metadata representation and the overhead of obtaining metadata through the network, which can bring an order of magnitude improvement in query efficiency.

[0052] In some specific implementations, before the client receives the backup instruction sent by the server, the method further includes:

[0053] The server sends a metadata reading instruction;

[0054] The client receives the metadata reading instruction and sends the metadata to the server via the high-speed serial bus;

[0055] The server reads the metadata, detects metadata that has not been backed up, and generates the backup instruction.

[0056] Here, by reading the metadata, it is determined whether the metadata in the distributed cluster has changed. If there is a change, it means that the data corresponding to the metadata has not been backed up.

[0057] Specifically, when the server detects that the data of the distributed cluster has not been backed up, it backs up all the data of the distributed cluster, and then reads the metadata of the distributed cluster at fixed intervals to determine whether there is any change in the metadata of the distributed cluster. If there is any change, a backup instruction is generated.

[0058] In this way, all data in the distributed cluster can be backed up.

[0059] In some embodiments, the method further comprises:

[0060] Determine a first transmission speed and a second transmission speed of the client according to demand, wherein the second transmission speed is greater than the first transmission speed;

[0061] The client receives the data sent by the server according to the first transmission speed;

[0062] The client sends the data to be backed up to the server according to the second transmission speed.

[0063] Here, transmission is carried out through USB4, which supports asymmetric transmission.

[0064] For example, when the client performs data backup, the upstream data transmission direction can be configured to be 120 Gbps, while the downstream data transmission speed can be configured to be 40 Gbps.

[0065] In one embodiment, asymmetric transmission can be applied to the data restoration process. For example, the client's downstream data reception direction can be configured to be 120Gbps, while the upstream speed is 40Gbps. This allows faster data restoration and reduces the downtime of the business system, thereby achieving a smaller recovery time target.

[0066] In this way, through asymmetric transmission, data can be transmitted from the distributed system faster, effectively reducing the backup time window and increasing the data backup rate.

[0067] In some specific implementations, the distributed cluster data backup system further includes a processing server, the processing server is connected to the client via the high-speed serial bus, the client sends the data to be backed up to the server via the high-speed serial bus, and the server receives and stores the data to be backed up, including:

[0068] The client sends the data to be backed up to the processing server via the high-speed serial bus;

[0069] The processing server divides the data to be backed up into a plurality of data blocks, calculates a characteristic value of each data block, and sequentially compares the characteristic value of each data block with the characteristic value of the data block in the server;

[0070] When the characteristic value of the data block in the processing server is the same as the characteristic value of the data block in the server, marking the data blocks with the same characteristic value;

[0071] Check the marked data blocks and delete the data blocks that meet the deletion criteria;

[0072] The processing server obtains the deleted data blocks, compresses the data blocks, and obtains compressed data to be backed up;

[0073] The processing server sends the compressed data to be backed up to the server, and the server receives and stores the compressed data to be backed up.

[0074] Here, the processing server itself has a certain storage space, and the storage space of the processing server can be used to provide a cache function for the data to be backed up, so as to store the data to be backed up that has not been transmitted in time.

[0075] The characteristic value of the data block may be a hash value or the like.

[0076] Specifically, the processing server receives the data to be backed up sent by the client through a high-speed serial bus, divides the original data sent by the client into data blocks, deduplicates the data blocks, compresses the deduplicated data blocks, and transmits the deduplicated and compressed data blocks to a backup server that is farther away.

[0077] In one embodiment, considering the data restoration process, the processing server may retain data blocks that appear more frequently to speed up the restoration process, and may even cache a complete set of backup data if the storage space is sufficient.

[0078] In this way, by connecting the processing server between the server and the client, the data to be backed up can be compressed, effectively reducing the amount of data sent to the server, so that the data to be backed up can be transmitted to a distant location more efficiently, realizing long-distance data backup. At the same time, the processing function requires a large amount of central processing unit and memory. By performing it in the processing server instead of in the client, it can avoid the client's overload affecting the performance and reliability of the entire cluster.

[0079] In some embodiments, the method further comprises:

[0080] The server is connected to a plurality of clients respectively through a plurality of high-speed serial buses, wherein the plurality of clients are nodes in a plurality of distributed clusters;

[0081] The multiple clients send the to-be-backed-up data of multiple distributed clusters to the server via multiple high-speed serial buses.

[0082] Specifically, a server is connected to multiple clients, wherein each client is a node selected from each distributed cluster.

[0083] In this way, a server can be connected to clients of different distributed clusters through multiple interfaces to achieve the purpose of backing up data for multiple distributed clusters.

[0084] In some embodiments, the method further comprises:

[0085] The server detects that the data transmission process is suspended or the data transmission rate is abnormal, and determines that the client fails;

[0086] The server obtains a node that is running and has a metadata management unit from the distributed cluster as a target client;

[0087] The server disconnects from the service end and establishes a connection with the target client.

[0088] In this way, failover can be achieved and the backup instructions can be transferred to the healthy node for continued execution.

[0089] In one embodiment, Figure 2 Schematic diagram of the overall structure of the system in the embodiment of the present application. Figure 2 As shown, the overall system architecture includes: backup server, cluster 1 and cluster 2. Cluster 1 includes MDS node 1 (ie, client), node 2, node 3, etc., and cluster 2 includes MDS node 1 (ie, client), node 2, node 3, etc.

[0090] Specifically, the execution steps include: the server issues a backup instruction, the client queries the metadata of the backup and reads data according to the metadata, the client sends the data to the server through a high-speed serial bus, and the server receives and stores the data.

[0091] Specifically, the backup server and the MDS node 1 are connected via USB4, and backup data is transmitted via USB4. Other services are transmitted between the backup server and the MDS node 1 via Ethernet.

[0092] In one embodiment, Figure 3 is another schematic diagram of the overall structure of the system in the embodiment of the present application, such as Figure 3 As shown in the figure, the overall system architecture includes: backup server, deduplication and storage forwarding server (i.e. processing server), cluster 1 and cluster 2. Cluster 1 includes MDS node 1 (i.e. client), node 2, node 3, etc., and cluster 2 includes MDS node 1 (i.e. client), node 2, node 3, etc.

[0093] Specifically, the execution steps include: the server issues a backup instruction, the client queries the backup metadata and reads data based on the metadata, the client sends the data to the deduplication and storage forwarding server through a high-speed serial bus, the deduplication and storage forwarding server deduplicates and compresses the data and sends it to the backup server, and the server receives and stores the data.

[0094] Specifically, the backup server and MDS node 1 are connected via USB4, backup data is transmitted via USB4, other services are transmitted between the backup server and MDS node 1 via Ethernet, and data is transmitted between the deduplication and store-and-forward server and the backup server via Ethernet.

[0095] It should be understood that although Figure 1-3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0096] In one embodiment, Figure 4As shown, a distributed cluster data backup device is provided, and the device is arranged in a distributed cluster data backup system, and the distributed cluster data backup system includes a distributed cluster, a client and a server, and the client is a node in the distributed cluster, and the device includes: a receiving module 401, used for the client to receive the backup instruction sent by the server; a query module 402, used for the client to query the metadata of the data to be backed up according to the backup instruction, wherein the metadata is stored in the client, and the metadata includes the location information of the data to be backed up; a reading module 403, used for the client to read the data to be backed up from the distributed cluster according to the metadata; a backup module 404, used for the client to send the data to be backed up to the server through a high-speed serial bus, and the server receives and stores the data to be backed up.

[0097] As a preferred implementation, in an embodiment of the present application, the device also includes: a first processing module, the first processing module is specifically used to: determine a target node from multiple nodes of the distributed cluster, and use the target node as a client; connect the client and the server through a universal serial bus; establish a network connection between the client and the server through a universal serial bus protocol; the client and the server transmit data through a universal serial bus and a universal serial bus protocol.

[0098] As a preferred implementation manner, in an embodiment of the present application, the first processing module is specifically used to: detect multiple nodes in the distributed cluster in turn, and determine whether there is a metadata management unit in the multiple nodes; obtain the node where the metadata management unit exists, and determine the target node based on the performance parameters of the node.

[0099] As a preferred implementation, in an embodiment of the present application, the device also includes: an instruction generation module, and the instruction generation module is specifically used for: the server sends a metadata reading instruction; the client receives the metadata reading instruction and sends the metadata to the server through the high-speed serial bus; the server reads the metadata, detects metadata that has not been backed up, and generates the backup instruction.

[0100] As a preferred implementation manner, in an embodiment of the present application, the device also includes: a speed configuration module, and the speed configuration module is specifically used to: determine the first transmission speed and the second transmission speed of the client according to demand, wherein the second transmission speed is greater than the first transmission speed; the client receives the data sent by the server according to the first transmission speed; and the client sends the data to be backed up to the server according to the second transmission speed.

[0101] As a preferred implementation mode, in an embodiment of the present application, the distributed cluster data backup system also includes a processing server, and the processing server is connected to the client via the high-speed serial bus, and the backup module 404 is specifically used for: the client sends the data to be backed up to the processing server via the high-speed serial bus; the processing server divides the data to be backed up into multiple data blocks, calculates the characteristic value of each data block, and compares the characteristic value of each data block with the characteristic value of the data block in the server in turn; when the characteristic value of the data block in the processing server is the same as the characteristic value of the data block in the server, the data block with the same characteristic value is marked; the marked data blocks are checked, and the data blocks that meet the deletion criteria are deleted; the processing server obtains the multiple data blocks after deletion, compresses the multiple data blocks, and obtains the compressed data to be backed up; the processing server sends the compressed data to be backed up to the server, and the server receives and stores the compressed data to be backed up.

[0102] As a preferred implementation, in an embodiment of the present application, the device also includes: a third processing module, and the third processing module is specifically used for: the server is connected to multiple clients through multiple high-speed serial buses respectively, wherein the multiple clients are nodes in multiple distributed clusters; the multiple clients send the data to be backed up of multiple distributed clusters to the server through multiple high-speed serial buses.

[0103] For the specific definition of the distributed cluster data backup device, please refer to the definition of the distributed cluster data backup method above, which will not be repeated here. Each module in the above-mentioned distributed cluster data backup device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0104] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a distributed cluster data backup method is implemented.

[0105] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the following steps are implemented when the processor executes the computer program: S1: the client receives a backup instruction sent by the server; S2: the client queries metadata of the data to be backed up according to the backup instruction, wherein the metadata is stored in the client, and the metadata includes location information of the data to be backed up; S3: the client reads the data to be backed up from a distributed cluster according to the metadata; S4: the client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0107] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining a target node from multiple nodes of the distributed cluster and using the target node as a client; connecting the client and the server via a universal serial bus; establishing a network connection between the client and the server via a universal serial bus protocol; the client and the server perform data transmission via a universal serial bus and a universal serial bus protocol.

[0108] In one embodiment, when the processor executes the computer program, the following steps are also implemented: detecting multiple nodes in the distributed cluster in turn, and determining whether there is a metadata management unit in the multiple nodes; obtaining the node where the metadata management unit exists, and determining the target node based on the performance parameters of the node.

[0109] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the server sends a metadata read instruction; the client receives the metadata read instruction and sends the metadata to the server via the high-speed serial bus; the server reads the metadata, detects metadata that has not been backed up, and generates the backup instruction.

[0110] In one embodiment, when the processor executes the computer program, the following steps are also implemented: according to demand, a first transmission speed and a second transmission speed of the client are determined, wherein the second transmission speed is greater than the first transmission speed; according to the first transmission speed, the client receives the data sent by the server; according to the second transmission speed, the client sends the data to be backed up to the server.

[0111] In one embodiment, the processor further implements the following steps when executing the computer program: the client sends the data to be backed up to the processing server through the high-speed serial bus; the processing server divides the data to be backed up into multiple data blocks, calculates the characteristic value of each data block, and compares the characteristic value of each data block with the characteristic value of the data block in the server in turn; when the characteristic value of the data block in the processing server is the same as the characteristic value of the data block in the server, the data block with the same characteristic value is marked; the marked data blocks are checked, and the data blocks that meet the deletion criteria are deleted; the processing server obtains the multiple data blocks after deletion, compresses the multiple data blocks, and obtains the compressed data to be backed up; the processing server sends the compressed data to be backed up to the server, and the server receives and stores the compressed data to be backed up.

[0112] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the server is connected to multiple clients through multiple high-speed serial buses, wherein the multiple clients are nodes in multiple distributed clusters; the multiple clients send the data to be backed up of the multiple distributed clusters to the server through multiple high-speed serial buses.

[0113] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: S1: the client receives a backup instruction sent by the server; S2: the client queries metadata of the data to be backed up according to the backup instruction, wherein the metadata is stored in the client, and the metadata includes location information of the data to be backed up; S3: the client reads the data to be backed up from a distributed cluster according to the metadata; S4: the client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0114] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining a target node from multiple nodes of the distributed cluster and using the target node as a client; connecting the client and the server via a universal serial bus; establishing a network connection between the client and the server via a universal serial bus protocol; the client and the server perform data transmission via a universal serial bus and a universal serial bus protocol.

[0115] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: detecting multiple nodes in the distributed cluster in turn, and determining whether there is a metadata management unit in the multiple nodes; obtaining the node where the metadata management unit exists, and determining the target node based on the performance parameters of the node.

[0116] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the server sends a metadata read instruction; the client receives the metadata read instruction and sends the metadata to the server via the high-speed serial bus; the server reads the metadata, detects metadata that has not been backed up, and generates the backup instruction.

[0117] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: according to demand, the first transmission speed and the second transmission speed of the client are determined, wherein the second transmission speed is greater than the first transmission speed; according to the first transmission speed, the client receives the data sent by the server; according to the second transmission speed, the client sends the data to be backed up to the server.

[0118] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the client sends the data to be backed up to the processing server through the high-speed serial bus; the processing server divides the data to be backed up into multiple data blocks, calculates the characteristic value of each data block, and compares the characteristic value of each data block with the characteristic value of the data block in the server in turn; when the characteristic value of the data block in the processing server is the same as the characteristic value of the data block in the server, the data block with the same characteristic value is marked; the marked data blocks are checked, and the data blocks that meet the deletion criteria are deleted; the processing server obtains the multiple data blocks after deletion, compresses the multiple data blocks, and obtains the compressed data to be backed up; the processing server sends the compressed data to be backed up to the server, and the server receives and stores the compressed data to be backed up.

[0119] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the server is connected to multiple clients through multiple high-speed serial buses respectively, wherein the multiple clients are nodes in multiple distributed clusters; the multiple clients send the data to be backed up of the multiple distributed clusters to the server through multiple high-speed serial buses.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A distributed cluster data backup method, applied to a distributed cluster data backup system, wherein the distributed cluster data backup system comprises a distributed cluster, a client and a server, wherein the client is a node in the distributed cluster, and wherein: The method comprises: The client receives a backup instruction sent by the server; The client queries metadata of the data to be backed up according to the backup instruction, wherein the metadata is stored in the client, and the metadata includes location information of the data to be backed up; The client reads the data to be backed up from the distributed cluster according to the metadata; The client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

2. The distributed cluster data backup method according to claim 1, characterized in that: The method further comprises: Determine a target node from multiple nodes of the distributed cluster, and use the target node as a client; Connecting the client and the server via a universal serial bus; Establishing a network connection between the client and the server via a universal serial bus protocol; The client and the server perform data transmission via a universal serial bus and a universal serial bus protocol.

3. The distributed cluster data backup method according to claim 2, characterized in that: The determining of the target node from the multiple nodes of the distributed cluster includes: sequentially detecting multiple nodes in the distributed cluster to determine whether a metadata management unit exists in the multiple nodes; A node where a metadata management unit exists is obtained, and a target node is determined according to performance parameters of the node.

4. The distributed cluster data backup method according to claim 1, characterized in that: Before the client receives the backup instruction sent by the server, the method further includes: The server sends a metadata reading instruction; The client receives the metadata reading instruction and sends the metadata to the server via the high-speed serial bus; The server reads the metadata, detects metadata that has not been backed up, and generates the backup instruction.

5. The distributed cluster data backup method according to claim 2, characterized in that: The method further comprises: Determine a first transmission speed and a second transmission speed of the client according to demand, wherein the second transmission speed is greater than the first transmission speed; The client receives the data sent by the server according to the first transmission speed; The client sends the data to be backed up to the server according to the second transmission speed.

6. The distributed cluster data backup method according to claim 1, wherein the distributed cluster data backup system further comprises a processing server, wherein the processing server is connected to the client via the high-speed serial bus, wherein: The client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up, including: The client sends the data to be backed up to the processing server via the high-speed serial bus; The processing server divides the data to be backed up into a plurality of data blocks, calculates a characteristic value of each data block, and sequentially compares the characteristic value of each data block with the characteristic value of the data block in the server; When the characteristic value of the data block in the processing server is the same as the characteristic value of the data block in the server, marking the data blocks with the same characteristic value; Check the marked data blocks and delete the data blocks that meet the deletion criteria; The processing server obtains the deleted data blocks, compresses the data blocks, and obtains compressed data to be backed up; The processing server sends the compressed data to be backed up to the server, and the server receives and stores the compressed data to be backed up.

7. The distributed cluster data backup method according to claim 1, characterized in that: The method further comprises: The server is connected to a plurality of clients respectively through a plurality of high-speed serial buses, wherein the plurality of clients are nodes in a plurality of distributed clusters; The multiple clients send the to-be-backed-up data of multiple distributed clusters to the server via multiple high-speed serial buses.

8. A distributed cluster data backup device, characterized in that: The device is arranged in a distributed cluster data backup system, the distributed cluster data backup system comprises a distributed cluster, a client and a server, the client is a node in the distributed cluster, and the device comprises: A receiving module, used for the client to receive the backup instruction sent by the server; A query module, used for the client to query metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, and the metadata includes location information of the data to be backed up; A reading module, used for the client to read the data to be backed up from the distributed cluster according to the metadata; The backup module is used for the client to send the data to be backed up to the server through a high-speed serial bus, and the server receives and stores the data to be backed up.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images