Distributed cluster data backup method and device, equipment and storage medium

Abstract

The application provides a distributed cluster data backup method applied to a distributed cluster data backup system, the distributed cluster data backup system comprising a distributed cluster, a client and a server, the client being a node in the distributed cluster, and the method comprises the following steps: the client receives a backup instruction sent by the server; the client queries metadata of to-be-backed-up data according to the backup instruction, wherein the metadata is stored in the client, and the metadata comprises position information of the to-be-backed-up data; the client reads the to-be-backed-up data from the distributed cluster according to the metadata; the client sends the to-be-backed-up data to the server through a high-speed serial bus, and the server receives and stores the to-be-backed-up data.

Description

Technical Field

[0001] This invention relates to the field of distributed cluster technology, and in particular to a distributed cluster data backup method, apparatus, device, and storage medium. Background Technology

[0002] Current backup systems typically use upper-layer file system interfaces, object interfaces, or block interfaces, accessed through the upper-layer business network of distributed storage. This creates two problems during the backup process: First, when backing up a large number of small files, the upper-layer file, object, or block interfaces need to interact with the distributed system over the network. The distributed system then needs to interact directly with the metadata server through its internal network before data transmission occurs. This multiple transmission of metadata and data results in a large number of small TCP (Transmission Control Protocol) or IP (Internet Protocol) packets, introducing access latency, lengthening the backup window, and even preventing backups from completing outside of business hours. Second, even when backing up large files, the large amount of backup data transmitted over the business network consumes significant bandwidth, impacting the business hosts accessing the distributed system from the front end.

[0003] Therefore, in view of the shortcomings of existing technical solutions, the present invention provides a distributed cluster data backup method. Summary of the Invention

[0004] Therefore, it is necessary to provide a distributed cluster data backup method, apparatus, device, and storage medium to address the aforementioned technical problems.

[0005] On one hand, a distributed cluster data backup method is provided, applied to a distributed cluster data backup system. The distributed cluster data backup system includes a distributed cluster, a client, and a server. The client is a node in the distributed cluster. The method includes: the client receiving a backup instruction sent by the server; the client querying metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, and the metadata includes the location information of the data to be backed up; the client reading the data to be backed up from the distributed cluster according to the metadata; and the client sending the data to be backed up to the server via a high-speed serial bus, and the server receiving and storing the data to be backed up.

[0006] Optionally, the method further includes: determining a target node from multiple nodes in the distributed cluster, and using the target node as a client; connecting the client and the server via a Universal Serial Bus (USB); establishing a network connection between the client and the server via a USB protocol; and transmitting data between the client and the server via USB and the USB protocol.

[0007] Optionally, determining the target node from multiple nodes in the distributed cluster includes: sequentially detecting multiple nodes in the distributed cluster, determining whether a metadata management unit exists among the multiple nodes; obtaining the node with the metadata management unit, and determining the target node based on the performance parameters of the node.

[0008] Optionally, before the client receives the backup instruction sent by the server, the method further includes: the server sending a metadata read instruction; the client receiving the metadata read instruction and sending the metadata to the server via the high-speed serial bus; the server reading the metadata, detecting unbacked-up metadata, and generating the backup instruction.

[0009] Optionally, the method further includes: determining a first transmission speed and a second transmission speed for the client according to requirements, wherein the second transmission speed is greater than the first transmission speed; the client receiving data sent by the server according to the first transmission speed; and the client sending the data to be backed up to the server according to the second transmission speed.

[0010] Optionally, the distributed cluster data backup system further includes a processing server. The processing server is connected to the client via the high-speed serial bus. The client sends the data to be backed up to the server via the high-speed serial bus. The server receives and stores the data to be backed up, including: the client sending the data to be backed up to the processing server via the high-speed serial bus; the processing server dividing the data to be backed up into multiple data blocks, calculating the feature value of each data block, and comparing the feature value of each data block with the feature value of the data blocks in the server; when the feature value of the data block in the processing server is the same as the feature value of the data block in the server, marking the data block with the same feature value; checking the marked data blocks and deleting the data blocks that meet the deletion criteria; the processing server obtaining the multiple deleted data blocks, compressing the multiple data blocks to obtain compressed data to be backed up; the processing server sending the compressed data to be backed up to the server, and the server receiving and storing the compressed data to be backed up.

[0011] Optionally, the method further includes: the server connecting to multiple clients via multiple high-speed serial buses, wherein the multiple clients are nodes in multiple distributed clusters; the multiple clients sending backup data of multiple distributed clusters to the server via the multiple high-speed serial buses.

[0012] On the other hand, a distributed cluster data backup device is provided. The device is installed in a distributed cluster data backup system, which includes a distributed cluster, a client, and a server. The client is a node in the distributed cluster. The device includes: a receiving module for receiving a backup instruction sent by the server; a query module for querying metadata of the data to be backed up based on the backup instruction, wherein the client stores metadata including location information of the data to be backed up; a reading module for reading the data to be backed up from the distributed cluster based on the metadata; and a backup module for sending the data to be backed up to the server via a high-speed serial bus, whereby the server receives and stores the data to be backed up.

[0013] In another aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: the client receives a backup instruction sent by the server; the client queries the metadata of the data to be backed up according to the backup instruction, wherein the client stores the metadata, and the metadata includes the location information of the data to be backed up; the client reads the data to be backed up from a distributed cluster according to the metadata; the client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0014] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the following steps: the client receives a backup instruction sent by the server; the client queries the metadata of the data to be backed up according to the backup instruction, wherein the client stores the metadata, the metadata including the location information of the data to be backed up; the client reads the data to be backed up from a distributed cluster according to the metadata; the client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0015] The aforementioned distributed cluster data backup method, apparatus, device, and storage medium are applied to a distributed cluster data backup system. The distributed cluster data backup system includes a distributed cluster, a client, and a server. The client is a node in the distributed cluster. The method includes: the client receiving a backup instruction sent by the server; the client querying metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, including the location information of the data to be backed up; the client reading the data to be backed up from the distributed cluster according to the metadata; and the client sending the data to be backed up to the server via a high-speed serial bus, with the server receiving and storing the data. Thus, by directly receiving and querying metadata through the client, the conversion of metadata representation and the overhead of obtaining metadata over the network are avoided, thereby accelerating the metadata query process and improving query efficiency. Furthermore, direct data transmission via a high-speed serial bus can improve transmission speed and more effectively reduce costs. It can also decouple data backup services from other services, improving backup efficiency. Simultaneously, performing the query service on the client without processing operations can prevent excessive client load from affecting the performance and reliability of the entire cluster. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a distributed cluster data backup method in one embodiment;

[0017] Figure 2 This is a schematic diagram of the overall system structure of a distributed cluster data backup method in one embodiment;

[0018] Figure 3 This is a schematic diagram of another system overall structure of the distributed cluster data backup method in one embodiment;

[0019] Figure 4 This is a structural block diagram of a distributed cluster data backup device in one embodiment;

[0020] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be understood that, in the description of this application, unless the context explicitly requires it, words such as "including" or "comprising" throughout the specification should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0023] It should also be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0024] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0025] In one embodiment, such as Figure 1 As shown, a distributed cluster data backup method is provided, applied to a distributed cluster data backup system. The distributed cluster data backup system includes a distributed cluster, a client, and a server. The client is a node in the distributed cluster. The method includes the following steps:

[0026] S101: The client receives the backup instruction sent by the server.

[0027] Here, the backup command includes the metadata of the data to be backed up.

[0028] S102: The client queries the metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, and the metadata includes the location information of the data to be backed up.

[0029] Here, the data to be backed up can be files (such as NAS or GPFS), objects (such as S3), or blocks. Metadata describes the data, mainly information about the data attributes, used to support functions such as indicating storage location, historical data, resource lookup, and file records. For example, metadata may include the physical location and offset of the data to be backed up on the disk.

[0030] S103: The client reads the data to be backed up from the distributed cluster based on the metadata.

[0031] Here, data in a distributed cluster is distributed across different nodes in the distributed cluster to achieve data redundancy protection and distributed access. Distributed clusters can be storage clusters, file system clusters, virtualized hyperconverged clusters, high-performance computing clusters, and AI clusters, etc.

[0032] Specifically, the client determines the exact location of the data to be backed up in the distributed cluster based on the metadata, finds the corresponding location of the corresponding node in the distributed cluster, and retrieves the data to be backed up from that location.

[0033] S104: The client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0034] Here, high-speed serial bus is a technology used to quickly transfer data between electronic devices, which may include PCIe, SATA (Serial Advanced Technology Attachment), USB (Universal Serial Bus), Ethernet, and IB (InfiniBand), etc.

[0035] Specifically, the client and server perform data backup via a high-speed serial bus. This transmission channel is dedicated to the transmission of backup data, while other business between the client and server is conducted through other transmission channels.

[0036] In one embodiment, the client sends the metadata to the server via the high-speed serial bus.

[0037] It should be noted that this application directly receives and queries metadata through the client, avoiding the conversion of metadata representation and the overhead of obtaining metadata over the network, thereby speeding up the metadata query process and improving query efficiency. Furthermore, data is transmitted via direct connection through a high-speed serial bus, which can improve the transmission rate and more effectively reduce costs. It can also decouple data backup services from other services, improving backup efficiency. At the same time, querying services on the client without performing processing operations can avoid excessive client load, which could affect the performance and reliability of the entire cluster.

[0038] In some specific embodiments, the method further includes:

[0039] The target node is determined from multiple nodes in the distributed cluster, and the target node is used as the client;

[0040] The client and the server are connected via a universal serial bus;

[0041] A network connection is established between the client and the server using the Universal Serial Bus protocol;

[0042] The client and the server transmit data via Universal Serial Bus (USB) and the USB protocol.

[0043] Here, the high-speed serial bus can specifically be a universal serial bus, such as USB4, and the corresponding universal serial bus protocol can be the Ethernet over USB4 inter-domain protocol.

[0044] Specifically, the client and server are connected via a USB4 cable, and a point-to-point network connection between the client and server is established via the Ethernet over USB4 inter-domain protocol.

[0045] In this way, the connection between the client and the server is realized through the Universal Serial Bus, which improves the transmission speed. At the same time, compared with Ethernet and IB networks, USB is more cost-effective while maintaining high speed.

[0046] In some specific implementations, determining the target node from multiple nodes of the distributed cluster includes:

[0047] The distributed cluster is sequentially inspected to determine whether a metadata management unit exists in the multiple nodes.

[0048] Obtain nodes that have metadata management units, and determine target nodes based on the performance parameters of the nodes.

[0049] Here, the metadata management unit can be the MDS service.

[0050] Specifically, from multiple nodes in the distributed cluster that provide metadata services, one node is selected. The performance of these nodes is compared across various aspects to determine the node with the best performance. Client software is then installed on this node, which acts as the client. The client software is responsible for sending data to the server. The client needs to be adapted to the software in the distributed cluster to retrieve metadata information directly from the specific node and obtain the corresponding data without needing to use a common file system or protocol.

[0051] In this way, the client can directly query the metadata of the data to be backed up. This query can be accelerated by the local memory. Compared with the traditional method of using NAS client or S3 client, it saves the overhead of converting the metadata representation and obtaining metadata over the network, which can bring an order of magnitude improvement in query efficiency.

[0052] In some specific embodiments, before the client receives the backup instruction sent by the server, the method further includes:

[0053] The server sends a metadata read command;

[0054] The client receives the metadata read instruction and sends the metadata to the server via the high-speed serial bus;

[0055] The server reads the metadata, detects unbacked-up metadata, and generates the backup instruction.

[0056] Here, by reading the metadata, it is determined whether there are any changes to the metadata in the distributed cluster. If there are changes, it means that the data corresponding to the metadata has not been backed up.

[0057] Specifically, when the server detects that the data in the distributed cluster has not been backed up, it backs up all the data in the distributed cluster. Then, at fixed intervals, it reads the metadata of the distributed cluster to determine whether there are any changes in the metadata. If there are changes, it generates a backup instruction.

[0058] In this way, a complete backup of the data in the distributed cluster is achieved.

[0059] In some specific embodiments, the method further includes:

[0060] Based on the requirements, a first transmission speed and a second transmission speed of the client are determined, wherein the second transmission speed is greater than the first transmission speed.

[0061] The client receives data sent by the server according to the first transmission speed;

[0062] The client sends the data to be backed up to the server according to the second transmission speed.

[0063] Here, data is transferred via USB4, which supports asymmetric data transfer.

[0064] For example, when the client performs data backup, the uplink data transmission direction can be configured to 120Gbps, while the downlink speed can be configured to 40Gbps.

[0065] In one embodiment, asymmetric transmission can be applied to the data restoration process. For example, the downlink data reception direction of the client can be configured to 120Gbps, while the uplink speed is 40Gbps. This allows for faster data restoration, reduces downtime of the business system, and achieves a shorter recovery time target.

[0066] In this way, asymmetric transmission can transmit data from the distributed system faster, effectively reducing the backup time window and improving the data backup rate.

[0067] In some specific embodiments, the distributed cluster data backup system further includes a processing server, which is connected to the client via the high-speed serial bus. The client sends the data to be backed up to the server via the high-speed serial bus, and the server receives and stores the data to be backed up, including:

[0068] The client sends the data to be backed up to the processing server via the high-speed serial bus;

[0069] The processing server divides the data to be backed up into multiple data blocks, calculates the feature value of each data block, and compares the feature value of each data block with the feature value of the data blocks in the server in turn.

[0070] When the feature value of a data block in the processing server is the same as the feature value of another data block in the server, the data block with the same feature value is marked.

[0071] The marked data blocks are checked, and those that meet the deletion criteria are deleted.

[0072] The processing server obtains multiple data blocks after deletion, compresses the multiple data blocks, and obtains compressed data to be backed up.

[0073] The processing server sends the compressed backup data to the server, and the server receives and stores the compressed backup data.

[0074] Here, the processing server itself has a certain amount of storage space, which can be used to provide a caching function for the data to be backed up, storing the data that has not been transmitted in time.

[0075] The characteristic value of a data block can be a hash value, etc.

[0076] Specifically, the processing server receives the backup data sent by the client through a high-speed serial bus, divides the raw data sent by the client into data blocks, deduplicates the data blocks, compresses the deduplicated data blocks, and transmits the deduplicated and compressed data blocks to a backup server located at a distance.

[0077] In one embodiment, considering the data restoration process, the processing server can retain frequently occurring data blocks to speed up the restoration process, and if there is enough storage space, it can even cache a complete set of backup data.

[0078] In this way, by connecting a processing server between the server and the client, the data to be backed up can be compressed, effectively reducing the amount of data sent to the server. This allows for more efficient transmission of data to distant locations, enabling long-distance data backup. Simultaneously, the processing function requires significant CPU and memory resources. By performing this on the processing server, rather than on the client, it avoids overloading the client and impacting the overall cluster performance and reliability.

[0079] In some specific embodiments, the method further includes:

[0080] The server is connected to multiple clients via multiple high-speed serial buses, wherein the multiple clients are nodes in multiple distributed clusters;

[0081] The multiple clients send backup data from multiple distributed clusters to the server via multiple high-speed serial buses.

[0082] Specifically, the server is connected to multiple clients, with each client being a node selected from each distributed cluster.

[0083] In this way, a single server can connect to clients of different distributed clusters through multiple interfaces to achieve the purpose of backing up data for multiple distributed clusters.

[0084] In some specific embodiments, the method further includes:

[0085] The server detects that the data transmission process is paused or the data transmission rate is abnormal, and determines that the client has malfunctioned.

[0086] The server obtains running nodes with metadata management units from the distributed cluster as target clients;

[0087] The server disconnects from the server-side and establishes a connection with the target client.

[0088] In this way, failover can be achieved, transferring backup commands to healthy nodes to continue execution.

[0089] In one embodiment, Figure 2 This is a schematic diagram of the overall system structure in the embodiments of this application, such as... Figure 2 As shown, the overall system architecture includes: a backup server, cluster 1, and cluster 2. Cluster 1 includes MDS node 1 (i.e., the client), node 2, node 3, etc., and cluster 2 includes MDS node 1 (i.e., the client), node 2, node 3, etc.

[0090] Specifically, the execution steps include: the server issues a backup command, the client queries the backup metadata and reads the data based on the metadata, the client sends the data to the server via a high-speed serial bus, and the server receives and stores the data.

[0091] Specifically, the backup server and MDS node 1 are connected via USB4 for backup data transmission, while other services are transmitted between them via Ethernet.

[0092] In one embodiment, Figure 3 This is a schematic diagram of another system overall structure in an embodiment of this application, such as... Figure 3 As shown, the overall system architecture includes: a backup server, a deduplication and store-and-forward server (i.e., a processing server), cluster 1, and cluster 2. Cluster 1 includes MDS node 1 (i.e., the client), node 2, node 3, etc., and cluster 2 includes MDS node 1 (i.e., the client), node 2, node 3, etc.

[0093] Specifically, the execution steps include: the server issues a backup command, the client queries the backup metadata and reads the data based on the metadata, the client sends the data to the deduplication and store-and-forward server via a high-speed serial bus, the deduplication and store-and-forward server deduplicates and compresses the data before sending it to the backup server, and the server receives and stores the data.

[0094] Specifically, the backup server and MDS node 1 are connected via USB4 for backup data transmission, while other services are transmitted between the backup server and MDS node 1 via Ethernet. The deduplication and store-and-forward server and the backup server also transmit data via Ethernet.

[0095] It should be understood that, although Figure 1-3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1-3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0096] In one embodiment, such as Figure 4As shown, a distributed cluster data backup device is provided. The device is installed in a distributed cluster data backup system, which includes a distributed cluster, a client, and a server. The client is a node in the distributed cluster. The device includes: a receiving module 401, used by the client to receive a backup instruction sent by the server; a query module 402, used by the client to query the metadata of the data to be backed up according to the backup instruction, wherein the client stores the metadata, which includes the location information of the data to be backed up; a reading module 403, used by the client to read the data to be backed up from the distributed cluster according to the metadata; and a backup module 404, used by the client to send the data to be backed up to the server via a high-speed serial bus, wherein the server receives and stores the data to be backed up.

[0097] In a preferred embodiment of this application, the apparatus further includes: a first processing module, which is specifically configured to: determine a target node from multiple nodes of the distributed cluster and designate the target node as a client; connect the client and the server via a Universal Serial Bus (USB); establish a network connection between the client and the server via a USB protocol; and transmit data between the client and the server via a USB protocol.

[0098] In a preferred embodiment of this application, the first processing module is further configured to: sequentially detect multiple nodes in the distributed cluster, determine whether a metadata management unit exists among the multiple nodes; obtain the nodes with metadata management units, and determine the target node based on the performance parameters of the nodes.

[0099] In a preferred embodiment of this application, the apparatus further includes an instruction generation module, which is specifically used for: the server sending a metadata read instruction; the client receiving the metadata read instruction and sending the metadata to the server via the high-speed serial bus; the server reading the metadata, detecting unbacked-up metadata, and generating the backup instruction.

[0100] In a preferred embodiment of this application, the device further includes a speed configuration module, which is specifically used to: determine a first transmission speed and a second transmission speed of the client according to requirements, wherein the second transmission speed is greater than the first transmission speed; the client receives data sent by the server according to the first transmission speed; and the client sends the data to be backed up to the server according to the second transmission speed.

[0101] In a preferred embodiment of this application, the distributed cluster data backup system further includes a processing server. The processing server is connected to the client via the high-speed serial bus. The backup module 404 is specifically used for: the client sending the data to be backed up to the processing server via the high-speed serial bus; the processing server dividing the data to be backed up into multiple data blocks, calculating the feature value of each data block, and comparing the feature value of each data block with the feature value of the data blocks in the server; when the feature value of the data block in the processing server is the same as the feature value of the data block in the server, marking the data block with the same feature value; checking the marked data blocks and deleting the data blocks that meet the deletion criteria; the processing server obtaining the multiple deleted data blocks, compressing the multiple data blocks to obtain compressed data to be backed up; the processing server sending the compressed data to be backed up to the server, and the server receiving and storing the compressed data to be backed up.

[0102] In a preferred embodiment of this application, the device further includes a third processing module, which is specifically used for: the server connecting to multiple clients via multiple high-speed serial buses, wherein the multiple clients are nodes in multiple distributed clusters; and the multiple clients sending backup data of multiple distributed clusters to the server via the multiple high-speed serial buses.

[0103] Specific limitations regarding the distributed cluster data backup device can be found in the limitations of the distributed cluster data backup method described above, and will not be repeated here. Each module in the aforementioned distributed cluster data backup device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0104] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a distributed cluster data backup method.

[0105] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0106] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: S1: The client receives a backup instruction sent by the server; S2: The client queries the metadata of the data to be backed up according to the backup instruction, wherein the client stores the metadata, and the metadata includes the location information of the data to be backed up; S3: The client reads the data to be backed up from a distributed cluster according to the metadata; S4: The client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0107] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a target node from multiple nodes of the distributed cluster, and designating the target node as a client; connecting the client and the server via a Universal Serial Bus (USB); establishing a network connection between the client and the server via a USB protocol; and transmitting data between the client and the server via USB and the USB protocol.

[0108] In one embodiment, when the processor executes the computer program, it further performs the following steps: sequentially detecting multiple nodes in the distributed cluster, determining whether a metadata management unit exists among the multiple nodes; obtaining the node with the metadata management unit, and determining the target node based on the performance parameters of the node.

[0109] In one embodiment, when the processor executes the computer program, it further performs the following steps: the server sends a metadata read instruction; the client receives the metadata read instruction and sends the metadata to the server via the high-speed serial bus; the server reads the metadata, detects unbacked-up metadata, and generates the backup instruction.

[0110] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a first transmission speed and a second transmission speed for the client according to requirements, wherein the second transmission speed is greater than the first transmission speed; the client receiving data sent by the server according to the first transmission speed; and the client sending the data to be backed up to the server according to the second transmission speed.

[0111] In one embodiment, when the processor executes the computer program, it further performs the following steps: the client sends the data to be backed up to the processing server via the high-speed serial bus; the processing server divides the data to be backed up into multiple data blocks, calculates the feature value of each data block, and compares the feature value of each data block with the feature value of the data blocks in the server; when the feature value of the data block in the processing server is the same as the feature value of the data block in the server, the data block with the same feature value is marked; the marked data blocks are checked, and the data blocks that meet the deletion criteria are deleted; the processing server obtains the multiple data blocks after deletion, compresses the multiple data blocks to obtain compressed data to be backed up; the processing server sends the compressed data to be backed up to the server, and the server receives and stores the compressed data to be backed up.

[0112] In one embodiment, when the processor executes the computer program, it further performs the following steps: the server connects to multiple clients respectively through multiple high-speed serial buses, wherein the multiple clients are nodes in multiple distributed clusters; the multiple clients send backup data of multiple distributed clusters to the server through multiple high-speed serial buses.

[0113] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: S1: The client receives a backup instruction sent by the server; S2: The client queries the metadata of the data to be backed up according to the backup instruction, wherein the client stores the metadata, and the metadata includes the location information of the data to be backed up; S3: The client reads the data to be backed up from a distributed cluster according to the metadata; S4: The client sends the data to be backed up to the server via a high-speed serial bus, and the server receives and stores the data to be backed up.

[0114] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining a target node from multiple nodes in the distributed cluster, and designating the target node as a client; connecting the client and the server via a Universal Serial Bus (USB); establishing a network connection between the client and the server via a USB protocol; and transmitting data between the client and the server via USB and the USB protocol.

[0115] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sequentially detecting multiple nodes in the distributed cluster, determining whether a metadata management unit exists among the multiple nodes; obtaining the node with the metadata management unit, and determining the target node based on the performance parameters of the node.

[0116] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: the server sends a metadata read instruction; the client receives the metadata read instruction and sends the metadata to the server via the high-speed serial bus; the server reads the metadata, detects unbacked-up metadata, and generates the backup instruction.

[0117] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: determining a first transmission speed and a second transmission speed for the client according to requirements, wherein the second transmission speed is greater than the first transmission speed; the client receiving data sent by the server according to the first transmission speed; and the client sending the data to be backed up to the server according to the second transmission speed.

[0118] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: the client sends the data to be backed up to the processing server via the high-speed serial bus; the processing server divides the data to be backed up into multiple data blocks, calculates the feature value of each data block, and compares the feature value of each data block with the feature value of the data blocks in the server in turn; when the feature value of the data block in the processing server is the same as the feature value of the data block in the server, the data block with the same feature value is marked; the marked data blocks are checked, and the data blocks that meet the deletion criteria are deleted; the processing server obtains the multiple data blocks after deletion, compresses the multiple data blocks to obtain compressed data to be backed up; the processing server sends the compressed data to be backed up to the server, and the server receives and stores the compressed data to be backed up.

[0119] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: the server connects to multiple clients respectively through multiple high-speed serial buses, wherein the multiple clients are nodes in multiple distributed clusters; the multiple clients send backup data of multiple distributed clusters to the server through multiple high-speed serial buses.

[0120] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A distributed cluster data backup method, applied to a distributed cluster data backup system, the distributed cluster data backup system comprising a distributed cluster, a client, a first server, and a second server, wherein the client is a node in the distributed cluster, and the second server is connected to the client via a high-speed serial bus, characterized in that, The method includes: The target node is determined from multiple nodes in the distributed cluster, and the target node is used as the client; The client receives a backup command sent by the first server; The client queries the metadata of the data to be backed up according to the backup instruction, wherein the client stores metadata, and the metadata includes the location information of the data to be backed up; The client reads the data to be backed up from the distributed cluster based on the metadata; The client sends the data to be backed up to the first server via a high-speed serial bus. The first server receives and stores the data to be backed up. The high-speed serial bus is dedicated to the transmission of data backup. Other services between the client and the first server are conducted through other transmission channels. The step of determining the target node from multiple nodes in the distributed cluster includes: The distributed cluster is sequentially inspected to determine whether a metadata management unit exists in the multiple nodes. Obtain nodes that have metadata management units, and determine target nodes based on the performance parameters of the nodes; The client sends the data to be backed up to the first server via a high-speed serial bus, and the first server receives and stores the data to be backed up, including: The client sends the data to be backed up to the second server via the high-speed serial bus; The second server divides the data to be backed up into multiple data blocks, calculates the feature value of each data block, and compares the feature value of each data block with the feature value of the data block in the first server in turn. When the feature value of a data block in the second server is the same as the feature value of a data block in the first server, the data block with the same feature value is marked. The marked data blocks are checked, and those that meet the deletion criteria are deleted. The second server retains data blocks whose occurrence frequency meets preset conditions, including occurrence frequency higher than a first threshold and / or data block size less than a second threshold; When the storage space meets the preset capacity condition, the second server caches a complete set of backup data. The second server obtains multiple data blocks after deletion, compresses the multiple data blocks, and obtains compressed data to be backed up; The second server sends the compressed backup data to the first server, and the first server receives and stores the compressed backup data. The method further includes: The first server is connected to multiple clients via multiple high-speed serial buses, wherein the multiple clients are nodes in multiple distributed clusters; The multiple clients send the backup data of multiple distributed clusters to the first server through multiple high-speed serial buses; The method further includes: The first server detects that the data transmission process is paused or the data transmission rate is abnormal, and determines that the client has malfunctioned. The first server obtains a running node with a metadata management unit from the distributed cluster as the target client; The first server disconnects from the faulty client and establishes a connection with the target client.

2. The distributed cluster data backup method according to claim 1, characterized in that, The method further includes: The client and the first server are connected via a universal serial bus; A network connection is established between the client and the first server using the Universal Serial Bus protocol; The client and the server transmit data via Universal Serial Bus (USB) and the USB protocol.

3. The distributed cluster data backup method according to claim 1, characterized in that, Before the client receives the backup instruction sent by the first server, the method further includes: The first server sends a metadata read command; The client receives the metadata read instruction and sends the metadata to the first server via the high-speed serial bus; The first server reads the metadata, detects the unbacked-up metadata, and generates the backup instruction.

4. The distributed cluster data backup method according to claim 2, characterized in that, The method further includes: Based on the requirements, a first transmission speed and a second transmission speed of the client are determined, wherein the second transmission speed is greater than the first transmission speed. The client receives data sent by the first server according to the first transmission speed; The client sends the data to be backed up to the first server according to the second transmission speed.

5. A distributed cluster data backup apparatus for implementing the distributed cluster data backup method as described in any one of claims 1-4, characterized in that, The device is installed in a distributed cluster data backup system, which includes a distributed cluster, a client, and a first server. The client is a node in the distributed cluster. The device includes: A receiving module is used for the client to receive backup instructions sent by the first server; The query module is used by the client to query the metadata of the data to be backed up according to the backup instruction, wherein the client stores the metadata, and the metadata includes the location information of the data to be backed up; A reading module is used by the client to read the data to be backed up from the distributed cluster based on the metadata; The backup module is used by the client to send the data to be backed up to the first server via a high-speed serial bus, and the first server receives and stores the data to be backed up.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images