Data backup method, device, computer equipment and storage medium

Abstract

The present application relates to a data backup method, device, computer equipment and storage medium. The method includes: determining multiple load nodes, and mounting the created shared directory to the multiple load nodes; obtaining the load information of the load node, and evaluating the performance index of the load node based on the load information to obtain an evaluation result, and determining one or more target load nodes according to the evaluation result; slicing the data to be backed up according to the slicing strategy corresponding to the target load node to obtain multiple data slicings; obtaining the load rate of the target load node, and determining the transmission mode according to the load rate and the preset transmission strategy, and allocating the data slicing to one or more target load nodes based on the transmission mode, so as to obtain the original data file of the data to be backed up and upload it to the shared directory. This method can dynamically select the optimal node and transmission mode based on the load information of the load node.

Description

Technical Field

[0001] The present application relates to the technical field of cloud computing, and in particular to a data backup method, apparatus, computer equipment and storage medium. Background Art

[0002] In the era of cloud computing, as the scale of the Internet market continues to expand, a large number of heterogeneous resources are managed uniformly through cloud platforms. When creating a virtual machine on a cloud platform, you need to upload the local image file to the image library on the cloud platform. Figure 1 As shown, the current implementation method of the image library is usually to create an FTP configuration file based on the virtualized node, create a local storage pool based on the virtualized node, configure the local storage pool as a location for storing image files, and then start the FTP service in the local storage pool based on the FTP configuration file.

[0003] This traditional image library design method meets the basic image file storage requirements to a certain extent, but there are some significant problems in practical applications. First, failover cannot be achieved. Since the image library is implemented based on the local storage pool in a single virtualization node, if the virtualization node fails, the entire image library will be unavailable, resulting in system unavailability. Secondly, high availability cannot be achieved. For example, in the case of three virtualization nodes, each node has a bandwidth of 100Mbps, but usually only one of the nodes (such as node 1) is selected as the image library storage node. All user uploads and downloads of images can only be performed on node 1, which will cause the bandwidth of node 1 to be fully occupied, while the bandwidth of nodes 2 and 3 is completely idle, which not only causes a waste of resources, but also causes excessive pressure on a certain node, affecting the operating efficiency and stability of the entire system. Summary of the invention

[0004] Based on this, it is necessary to provide a data backup method, device, computer equipment and storage medium that can dynamically select appropriate nodes to address the above technical problems.

[0005] In one aspect, a data backup method is provided, the method comprising:

[0006] Determine multiple load nodes, and mount the created shared directory on the multiple load nodes;

[0007] Acquire load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result;

[0008] The data to be backed up is sharded according to the sharding strategy corresponding to the target load node to obtain multiple data shards; the load rate of the target load node is obtained, and a transmission mode is determined according to the load rate and a preset transmission strategy, and the data shards are allocated to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0009] In one of the embodiments, before acquiring the load information of the load node, the method further includes:

[0010] Performing heartbeat detection on the load node to obtain the response time of the load node;

[0011] Checking the service availability of the load node to determine whether the load node is in a normal working state;

[0012] In response to the response time exceeding a preset time threshold, or the service status of the load node being unavailable, determining that the load node has sent a fault and removing the load node from the node list;

[0013] In response to the load node being in a normal working state, load information of the load node is acquired.

[0014] In one embodiment, the acquiring the load information of the load node, performing a performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determining one or more target load nodes according to the evaluation result, includes:

[0015] Obtaining a preset performance indicator, and obtaining load information of the load node based on the performance indicator, wherein the load information includes one or more of a load rate, a network bandwidth, a storage occupancy rate, a historical transmission success rate, and a physical distance;

[0016] The performance score of the load node is obtained by multiplying each performance indicator of the load node with the corresponding fusion weight;

[0017] The multiple load nodes are sorted according to the performance scores, and a target load node is determined according to the sorting result.

[0018] In one embodiment, the sharding of the data to be backed up according to the sharding strategy corresponding to the target load node to obtain multiple data shards includes:

[0019] Acquire the current network bandwidth of the target load node in real time, and determine whether the current network bandwidth reaches a preset bandwidth value;

[0020] In response to the current network bandwidth not reaching the preset bandwidth value, using the preset shard value as the target shard value of the data shard;

[0021] In response to the current network bandwidth reaching the preset bandwidth value, obtaining a correspondence between the current network bandwidth and the size of the data shard according to a preset data table, and confirming a target shard value corresponding to the current network bandwidth according to the correspondence;

[0022] The data to be backed up is sliced ​​according to the target slice value to obtain a plurality of data slices, and slice sequence identifiers are added to the data slices according to the slice processing order.

[0023] In one of the embodiments, obtaining the load rate of the target load node, determining a transmission mode according to the load rate and a preset transmission strategy, and allocating the data slices to one or more target load nodes based on the transmission mode includes:

[0024] In response to taking the first load node as the target load node, obtaining a load rate of the first load node, wherein the load rate includes one or more of a processor utilization rate, a network bandwidth utilization rate, and a memory utilization rate;

[0025] In response to a load rate of the first load node being less than a preset first threshold, setting the transmission mode to a serial transmission mode, and allocating the data slices to the first load node according to the slice sequence identifier;

[0026] In response to the load rate of the first load node being greater than or equal to the preset first threshold and less than a preset second threshold, adding one or more second load nodes as the target load nodes according to the sorting result;

[0027] Obtaining the load rate of each target load node, and setting the transmission mode to a parallel transmission mode, and distributing and processing the data slices according to the load rate of each target load node;

[0028] In response to the load rate of the first load node being greater than or equal to a preset second threshold, adding a preset number of second load nodes as target load nodes, and adding one or more third load nodes as standby nodes according to the sorting result;

[0029] The transmission mode is set as a standby transmission mode, and the data slices are distributed and processed according to the load rates of the target load nodes and the standby nodes.

[0030] In one embodiment, the obtaining of the original data file of the data to be backed up and uploading it to the shared directory includes:

[0031] In response to the target load node receiving the data slices, obtaining the slice sequence identifiers of the data slices, and performing integrity checks on the data slices according to the slice sequence identifiers;

[0032] In response to the data slice passing the integrity check, the multiple data slices are spliced ​​according to the slice sequence identifiers to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0033] In one embodiment, the method further comprises:

[0034] Monitoring the transmission status of the data slices, wherein the transmission status includes a completed status and an uncompleted status;

[0035] Obtaining a first number of data shards in the completed state and a second number of data shards in the uncompleted state;

[0036] In response to a failure of the load node, starting a failure recovery procedure;

[0037] In response to successful failure recovery, resuming transmission of the second number of data shards, and performing integrity checks on the first number of data shards;

[0038] In response to a failure of fault recovery, the first number of data shards are synchronized to a spare node, and the second number of data shards are allocated to the spare node.

[0039] In another aspect, a data backup device is provided, the device comprising:

[0040] A creation module is used to determine a plurality of load nodes and mount the created shared directory to the plurality of load nodes;

[0041] An evaluation module, used to obtain load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result;

[0042] A sharding module, used to shard the backup data according to the sharding strategy corresponding to the target load node to obtain multiple data shards;

[0043] A transmission module is used to obtain the load rate of the target load node, and determine the transmission mode according to the load rate and a preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0044] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0045] Determine multiple load nodes, and mount the created shared directory on the multiple load nodes;

[0046] Acquire load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result;

[0047] Slice the data to be backed up according to the slicing strategy corresponding to the target load node to obtain multiple data slicings;

[0048] Obtain the load rate of the target load node, and determine the transmission mode according to the load rate and the preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0049] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0050] Determine multiple load nodes, and mount the created shared directory on the multiple load nodes;

[0051] Acquire load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result;

[0052] Slice the data to be backed up according to the slicing strategy corresponding to the target load node to obtain multiple data slicings;

[0053] Obtain the load rate of the target load node, and determine the transmission mode according to the load rate and the preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0054] The above-mentioned data backup method, device, computer equipment and storage medium determine one or more target load nodes by evaluating the performance indicators through the load information of each load node, and determine the size of the data shard according to the sharding strategy corresponding to the target load node, and determine the transmission mode according to the load rate of the target load node and the preset transmission strategy, so that each data shard can be allocated to a suitable node, maximizing the resources of each node, improving data transmission efficiency, thereby increasing the image transmission bandwidth and the high availability and fault tolerance of the image library. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a structural block diagram of NFS shared storage image library in the prior art;

[0056] Figure 2 A schematic diagram of a data backup method in one embodiment;

[0057] Figure 3 A schematic diagram of a data backup process in one embodiment;

[0058] Figure 4 A structural block diagram of a data backup system in one embodiment;

[0059] Figure 5 is a structural block diagram of a data backup device in one embodiment;

[0060] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] NFS (Network File System) shared storage refers to the technology of sharing storage resources on the network using the NFS protocol. Through NFS shared storage, multiple computers (usually servers or virtualization nodes) can share their local storage resources with other computers over the network, allowing these computers to access and operate shared files and data just like accessing local storage.

[0063] In one embodiment, Figure 2-4 As shown, a data backup method is provided, comprising the following steps:

[0064] Step S1, determining a plurality of load nodes, and mounting the created shared directory on the plurality of load nodes.

[0065] When the system starts, it is necessary to perform initial configuration, load necessary services and components, and establish the basic environment required for system operation. Then mount the NFS shared storage to all virtualized nodes to ensure that all nodes can access the shared directory.

[0066] In an optional embodiment, before obtaining the load information of the load node, it also includes: performing heartbeat detection on the load node to obtain the response time of the load node; checking the service availability of the load node to determine whether the load node is in normal working condition; in response to the response time exceeding a preset time threshold, or the service status of the load node is unavailable, determining that the load node has sent a fault and removing the load node from the node list; in response to the load node being in normal working condition, obtaining the load information of the load node.

[0067] Among them, it is necessary to regularly perform health checks on each load node to ensure its normal operation. Once a faulty node is found, it will be immediately isolated and automatically restarted or rescheduled. Specifically, heartbeat requests are regularly sent to each load node to verify the node's response time, ensure that the node can respond to requests in a timely manner, and check whether the services on each load node are operating normally, such as databases, cache services, etc.

[0068] Step S2, obtaining load information of the load node, performing performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determining one or more target load nodes according to the evaluation result.

[0069] In an optional embodiment, the load information of the load node is obtained, a performance indicator evaluation is performed on the load node based on the load information to obtain an evaluation result, and one or more target load nodes are determined according to the evaluation result, including: obtaining preset performance indicators, and obtaining the load information of the load node based on the performance indicators, wherein the load information includes one or more of load rate, network bandwidth, storage occupancy, historical transmission success rate and physical distance; obtaining a performance score of the load node by multiplying each performance indicator of the load node with a corresponding fusion weight; and sorting the multiple load nodes according to the performance score, and determining the target load node according to the sorting result.

[0070] For example, a monitoring tool can be used to collect the CPU utilization, memory utilization, and network bandwidth utilization of each load node, and use them as a comprehensive load factor to calculate the load rate of the node, for example, LoadFactor = (CPU utilization + network bandwidth utilization + memory utilization) / 3. Then the performance score of each load node is calculated using the following formula:

[0071] Score=w1*(1-LoadFactor)+w2*(1 / Distance)+w3*(Bandwidth / MaxBandwidth)+w4*(StorageAvailable / TotalStorage)+w5*(SuccessRate);

[0072] Among them, LoadFactor is the comprehensive load factor, Distance is the weighted value of physical and network distances, SuccessRate is the historical transmission success rate, and w1-w5 are dynamic weight coefficients that can be adaptively adjusted based on real-time network conditions.

[0073] Based on the above performance score calculation formula, the performance score of each node can be calculated, and the node can be selected according to the score to ensure that data transmission is completed by the best load node, thereby improving the efficiency of the system, reducing the transmission time, and avoiding overload.

[0074] In an optional implementation, the acquiring the load information of the load node, performing a performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determining one or more target load nodes according to the evaluation result, further includes:

[0075] Collecting historical load data of the server, and determining the average load and peak load of the server within a preset time range according to the historical load data;

[0076] Calculate the total load requirement of the server according to the average load;

[0077] Performing performance tests on the multiple load nodes respectively, simulating concurrent transmission scenarios, and obtaining the load amounts of the load nodes;

[0078] Calculating the number of initial load nodes required to meet the average load and the number of standby nodes required to meet the peak load according to the total load demand and the load amount of the load nodes;

[0079] The target load node is determined according to the initial load node quantity and the sorting result.

[0080] Step S3, sharding the data to be backed up according to the sharding strategy corresponding to the target load node to obtain multiple data shards.

[0081] In an optional embodiment, the data to be backed up is sharded according to the sharding strategy corresponding to the target load node to obtain multiple data shards, including: acquiring the current network bandwidth of the target load node in real time, and determining whether the current network bandwidth reaches a preset bandwidth value; in response to the current network bandwidth not reaching the preset bandwidth value, using the preset sharding value as the target sharding value of the data shard; in response to the current network bandwidth reaching the preset bandwidth value, acquiring the correspondence between the current network bandwidth and the size of the data shard according to a preset data table, and confirming the target sharding value corresponding to the current network bandwidth according to the correspondence; sharding the data to be backed up according to the target sharding value to obtain multiple data shards, and adding slice sequence identifiers to the data shards according to the order of the sharding processing.

[0082] Specifically, by detecting the network bandwidth, the size of each data fragment can be dynamically adjusted. When the network condition is poor, reducing the fragment size can reduce the possibility of transmission failure, while when the network is good, increasing the fragment size can increase the transmission speed, thereby achieving the purpose of optimizing data transmission efficiency.

[0083] Step S4, obtain the load rate of the target load node, and determine the transmission mode according to the load rate and the preset transmission strategy, and distribute the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0084] In an optional embodiment, the method of obtaining the load rate of the target load node, determining a transmission mode according to the load rate and a preset transmission strategy, and allocating the data slices to one or more target load nodes based on the transmission mode includes: in response to taking a first load node as the target load node, obtaining the load rate of the first load node, wherein the load rate includes one or more of processor utilization, network bandwidth utilization, and memory utilization; in response to the load rate of the first load node being less than a preset first threshold, setting the transmission mode to a serial transmission mode, and allocating the data slices to the first load node according to the slice sequence identifier; in response to the load rate of the first load node being greater than a preset first threshold, is equal to the preset first threshold and less than the preset second threshold, adding one or more second load nodes as the target load nodes according to the sorting result; obtaining the load rate of each target load node, setting the transmission mode to the parallel transmission mode, and distributing and processing the data slices according to the load rate of each target load node; in response to the load rate of the first load node being greater than or equal to the preset second threshold, adding a preset number of second load nodes as target load nodes, and adding one or more third load nodes as backup nodes according to the sorting result; setting the transmission mode to the backup transmission mode, and distributing and processing the data slices according to the load rates of each target load node and the backup node.

[0085] Specifically, when the load rate is low, the serial transmission mode is selected to transmit data slices in sequence at one node to ensure the stability of transmission. When the load rate is medium, the number of load nodes can be increased, and the data slices can be transmitted simultaneously on multiple nodes through the parallel transmission mode to improve efficiency, until each data slice is allocated to each load node, the load rate of each load node is maintained within the preset range, and the data slices to be allocated can be executed. Among them, the data slices can be dynamically allocated according to the current load conditions of each node to ensure load balancing, and the size of the data slices can be adjusted according to the current network bandwidth of the node to dynamically adjust the transmission rate to prevent congestion. When the load rate is high, the backup node can be enabled to disperse the pressure and ensure that the data slices are successfully transmitted. The data of the main node and the backup node can be synchronized regularly to ensure that the backup node is available at any time, and when the main node fails, it can be switched to the backup node to ensure the continuity of the transmission task.

[0086] Based on the above data backup steps, when the target load node is determined, the appropriate transmission mode is selected according to the load situation of the target load node. When the load is high, distributed upload can be performed by adding load nodes to improve the transmission bandwidth and high availability of the image library, and the system fault tolerance can be improved by adding spare nodes.

[0087] In an optional embodiment, obtaining the original data file of the data to be backed up and uploading it to the shared directory includes: in response to the target load node receiving the data slices, obtaining the slice sequence identifiers of each of the data slices, and performing an integrity check on the data slices according to the slice sequence identifiers; in response to the data slices passing the integrity check, splicing the multiple data slices according to the slice sequence identifiers to obtain the original data file of the data to be backed up and uploading it to the shared directory.

[0088] Specifically, first verify whether all received data slices are complete, check the correctness of the data slices, and confirm whether the transmission quality meets the standards. Each data slice is provided with an identity identifier. By comparing the identity identifier of the data slice with the identity identifier of the received data slice, it is determined whether there is a missing data slice. If so, the missing data slice is sent to the target load node. After all data slices are transmitted, the slices are merged according to the slice order identifier to ensure that the generated target file is complete and consistent.

[0089] In an optional embodiment, the method further includes: monitoring the transmission status of the data shards, the transmission status including a completed state and an uncompleted state; obtaining a first number of data shards in the completed state, and a second number of data shards in the uncompleted state; in response to a failure of the load node, starting a fault recovery program; in response to successful fault recovery, resuming transmission of the second number of data shards, and performing an integrity check on the first number of data shards; in response to a failed fault recovery, synchronizing the first number of data shards to a backup node, and allocating the second number of data shards to the backup node.

[0090] Specifically, by real-time monitoring of the transmission status of data shards, transmission failures can be discovered and handled in a timely manner. When a transmission failure is detected, fault diagnosis is immediately performed and the fault recovery program is started to avoid interruption or data loss during data transmission. The fault recovery program includes node switching and breakpoint resumption to ensure the continuity and integrity of data transmission.

[0091] It should be understood that although Figure 2-3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-3At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0092] In one embodiment, Figure 5 As shown, a data backup device is provided, including: a creation module, an evaluation module, a slicing module and a transmission module, wherein:

[0093] A creation module is used to determine a plurality of load nodes and mount the created shared directory to the plurality of load nodes;

[0094] An evaluation module, used to obtain load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result;

[0095] A sharding module, used to shard the backup data according to the sharding strategy corresponding to the target load node to obtain multiple data shards;

[0096] A transmission module is used to obtain the load rate of the target load node, and determine the transmission mode according to the load rate and a preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0097] In one embodiment, the device also includes a detection module, which is used to perform heartbeat detection on the load node to obtain the response time of the load node; check the service availability of the load node to determine whether the load node is in normal working condition; in response to the response time exceeding a preset time threshold, or the service status of the load node is unavailable, determine that the load node has a fault and remove the load node from the node list; in response to the load node being in normal working condition, obtain the load information of the load node.

[0098] In one embodiment, the evaluation module is also used to obtain preset performance indicators, and obtain load information of the load node based on the performance indicators, wherein the load information includes one or more of load rate, network bandwidth, storage occupancy, historical transmission success rate and physical distance; the performance score of the load node is obtained by multiplying each performance indicator of the load node with the corresponding fusion weight; the load nodes are sorted according to the performance score, and the target load node is determined according to the sorting result.

[0099] In one embodiment, the sharding module is also used to obtain the current network bandwidth of the target load node in real time, and determine whether the current network bandwidth reaches a preset bandwidth value; in response to the current network bandwidth not reaching the preset bandwidth value, use the preset sharding value as the target sharding value of the data sharding; in response to the current network bandwidth reaching the preset bandwidth value, obtain the correspondence between the current network bandwidth and the size of the data shard according to a preset data table, and confirm the target sharding value corresponding to the current network bandwidth according to the correspondence; perform sharding processing on the data to be backed up according to the target sharding value to obtain multiple data shards, and add sharding sequence identifiers to the data shards according to the order of the sharding processing.

[0100] In one embodiment, the transmission module is also used to, in response to taking the first load node as the target load node, obtain the load rate of the first load node, wherein the load rate includes one or more of processor utilization, network bandwidth utilization and memory utilization; in response to the load rate being less than a preset first threshold, adopting a serial transmission mode, and assigning the data slices to the target load node according to the slice sequence identifier; in response to the load rate being greater than or equal to the preset first threshold and less than a preset second threshold, adding one or more load nodes as the target load nodes according to the sorting result; obtaining the load rate of each target load node, and adopting a parallel transmission mode, assigning the data slices to each target load node according to the slice sequence identifier according to the load rate of each target load node; in response to the load rate being greater than or equal to the preset second threshold, taking the multiple load nodes as the target load nodes, and adding a preset number of load nodes as spare nodes, and assigning the data slices to the target load node and the spare node according to the slice sequence identifier.

[0101] In one embodiment, the device also includes an integration module, which is used to obtain the slice sequence identifiers of each of the data slices in response to receiving the data slices, and perform integrity checks on the data slices according to the slice sequence identifiers; in response to the data slices passing the integrity check, the multiple data slices are spliced ​​according to the slice sequence identifiers to obtain the original data files of the data to be backed up.

[0102] In one embodiment, the device includes a recovery module, which is used to monitor the transmission status of the data shards, wherein the transmission status includes a completed state and an uncompleted state; obtain a first number of data shards in the completed state, and a second number of data shards in the uncompleted state; in response to a failure of the load node, start a fault recovery program; in response to successful fault recovery, resume transmission of the second number of data shards, and perform an integrity check on the first number of data shards; in response to a failed fault recovery, synchronize the first number of data shards to a backup node, and allocate the second number of data shards to the backup node.

[0103] For the specific definition of the data backup device, please refer to the definition of the data backup method above, which will not be repeated here. Each module in the above data backup device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0104] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store load data of each load node. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data backup method is implemented.

[0105] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0107] Determine multiple load nodes, and mount the created shared directory on the multiple load nodes;

[0108] Acquire load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result;

[0109] Slice the data to be backed up according to the slicing strategy corresponding to the target load node to obtain multiple data slicings;

[0110] Obtain the load rate of the target load node, and determine the transmission mode according to the load rate and the preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0111] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing heartbeat detection on the load node to obtain the response time of the load node; checking the service availability of the load node to determine whether the load node is in normal working condition; in response to the response time exceeding a preset time threshold, or the service status of the load node is unavailable, determining that the load node has sent a fault and removing the load node from the node list; in response to the load node being in normal working condition, obtaining the load information of the load node.

[0112] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining preset performance indicators, and obtaining load information of the load node based on the performance indicators, wherein the load information includes one or more of load rate, network bandwidth, storage occupancy, historical transmission success rate and physical distance; obtaining a performance score of the load node by multiplying each performance indicator of the load node with the corresponding fusion weight; sorting the multiple load nodes according to the performance score, and determining the target load node according to the sorting result.

[0113] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining the current network bandwidth of the target load node in real time, and determining whether the current network bandwidth reaches a preset bandwidth value; in response to the current network bandwidth not reaching the preset bandwidth value, using a preset shard value as a target shard value for the data shard; in response to the current network bandwidth reaching the preset bandwidth value, obtaining a correspondence between the current network bandwidth and the size of the data shard according to a preset data table, and confirming the target shard value corresponding to the current network bandwidth according to the correspondence; sharding the data to be backed up according to the target shard value to obtain multiple data shards, and adding a slice sequence identifier to the data shard according to the order of the sharding processing.

[0114] In one embodiment, when the processor executes the computer program, the following steps are also implemented: in response to taking the first load node as the target load node, obtaining the load rate of the first load node, wherein the load rate includes one or more of processor utilization, network bandwidth utilization and memory utilization; in response to the load rate being less than a preset first threshold, adopting a serial transmission mode, and allocating the data slices to the target load node according to the slice sequence identifier; in response to the load rate being greater than or equal to the preset first threshold and less than a preset second threshold, adding one or more load nodes as the target load nodes according to the sorting result; obtaining the load rate of each target load node, and adopting a parallel transmission mode, allocating the data slices to each target load node according to the slice sequence identifier according to the load rate of each target load node; in response to the load rate being greater than or equal to the preset second threshold, taking the multiple load nodes as the target load nodes, and adding a preset number of load nodes as spare nodes, and allocating the data slices to the target load nodes and the spare nodes according to the slice sequence identifier.

[0115] In one embodiment, when the processor executes the computer program, the following steps are also implemented: in response to receiving the data slices, obtaining the slice sequence identifiers of each of the data slices, and performing an integrity check on the data slices according to the slice sequence identifiers; in response to the data slices passing the integrity check, splicing the multiple data slices according to the slice sequence identifiers to obtain the original data files of the data to be backed up.

[0116] In one embodiment, when the processor executes the computer program, the following steps are also implemented: monitoring the transmission status of the data shards, the transmission status including a completed state and an uncompleted state; obtaining a first number of data shards in the completed state, and a second number of data shards in the uncompleted state; in response to a failure of the load node, starting a fault recovery program; in response to successful fault recovery, resuming transmission of the second number of data shards, and performing an integrity check on the first number of data shards; in response to a failed fault recovery, synchronizing the first number of data shards to a backup node, and allocating the second number of data shards to the backup node.

[0117] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0118] Determine multiple load nodes, and mount the created shared directory on the multiple load nodes;

[0119] Acquire load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result;

[0120] Slice the data to be backed up according to the slicing strategy corresponding to the target load node to obtain multiple data slicings;

[0121] Obtain the load rate of the target load node, and determine the transmission mode according to the load rate and the preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0122] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing heartbeat detection on the load node to obtain the response time of the load node; checking the service availability of the load node to determine whether the load node is in normal working condition; in response to the response time exceeding a preset time threshold, or the service status of the load node is unavailable, determining that the load node has sent a fault and removing the load node from the node list; in response to the load node being in normal working condition, obtaining the load information of the load node.

[0123] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining preset performance indicators, and obtaining load information of the load node based on the performance indicators, wherein the load information includes one or more of load rate, network bandwidth, storage occupancy, historical transmission success rate and physical distance; obtaining a performance score of the load node by multiplying each performance indicator of the load node with the corresponding fusion weight; sorting the multiple load nodes according to the performance score, and determining the target load node according to the sorting result.

[0124] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the current network bandwidth of the target load node in real time, and determining whether the current network bandwidth reaches a preset bandwidth value; in response to the current network bandwidth not reaching the preset bandwidth value, using a preset shard value as a target shard value for the data shard; in response to the current network bandwidth reaching the preset bandwidth value, obtaining a correspondence between the current network bandwidth and the size of the data shard according to a preset data table, and confirming a target shard value corresponding to the current network bandwidth according to the correspondence; sharding the data to be backed up according to the target shard value to obtain multiple data shards, and adding a slice sequence identifier to the data shard according to the order of the sharding processing.

[0125] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to taking the first load node as the target load node, obtaining the load rate of the first load node, wherein the load rate includes one or more of processor utilization, network bandwidth utilization and memory utilization; in response to the load rate being less than a preset first threshold, adopting a serial transmission mode, and allocating the data slices to the target load node according to the slice sequence identifier; in response to the load rate being greater than or equal to the preset first threshold and less than a preset second threshold, adding one or more load nodes as the target load nodes according to the sorting result; obtaining the load rate of each target load node, and adopting a parallel transmission mode, allocating the data slices to each target load node according to the slice sequence identifier according to the load rate of each target load node; in response to the load rate being greater than or equal to the preset second threshold, taking the multiple load nodes as the target load nodes, and adding a preset number of load nodes as spare nodes, and allocating the data slices to the target load nodes and the spare nodes according to the slice sequence identifier.

[0126] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to receiving the data slices, obtaining the slice sequence identifiers of each of the data slices, and performing an integrity check on the data slices according to the slice sequence identifiers; in response to the data slices passing the integrity check, splicing the multiple data slices according to the slice sequence identifiers to obtain the original data files of the data to be backed up.

[0127] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: monitoring the transmission status of the data shards, the transmission status including a completed state and an uncompleted state; obtaining a first number of data shards in the completed state, and a second number of data shards in the uncompleted state; in response to a failure of the load node, starting a fault recovery program; in response to a successful fault recovery, resuming the transmission of the second number of data shards, and performing an integrity check on the first number of data shards; in response to a failed fault recovery, synchronizing the first number of data shards to a backup node, and allocating the second number of data shards to the backup node.

[0128] In one embodiment, a computer product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0129] Determine multiple load nodes, and mount the created shared directory on the multiple load nodes;

[0130] Acquire load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result;

[0131] Slice the data to be backed up according to the slicing strategy corresponding to the target load node to obtain multiple data slicings;

[0132] Obtain the load rate of the target load node, and determine the transmission mode according to the load rate and the preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

[0133] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing heartbeat detection on the load node to obtain the response time of the load node; checking the service availability of the load node to determine whether the load node is in normal working condition; in response to the response time exceeding a preset time threshold, or the service status of the load node is unavailable, determining that the load node has sent a fault and removing the load node from the node list; in response to the load node being in normal working condition, obtaining the load information of the load node.

[0134] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining preset performance indicators, and obtaining load information of the load node based on the performance indicators, wherein the load information includes one or more of load rate, network bandwidth, storage occupancy, historical transmission success rate and physical distance; obtaining a performance score of the load node by multiplying each performance indicator of the load node with the corresponding fusion weight; sorting the multiple load nodes according to the performance score, and determining the target load node according to the sorting result.

[0135] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the current network bandwidth of the target load node in real time, and determining whether the current network bandwidth reaches a preset bandwidth value; in response to the current network bandwidth not reaching the preset bandwidth value, using a preset shard value as a target shard value for the data shard; in response to the current network bandwidth reaching the preset bandwidth value, obtaining a correspondence between the current network bandwidth and the size of the data shard according to a preset data table, and confirming a target shard value corresponding to the current network bandwidth according to the correspondence; sharding the data to be backed up according to the target shard value to obtain multiple data shards, and adding a slice sequence identifier to the data shard according to the order of the sharding processing.

[0136] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to taking the first load node as the target load node, obtaining the load rate of the first load node, wherein the load rate includes one or more of processor utilization, network bandwidth utilization and memory utilization; in response to the load rate being less than a preset first threshold, adopting a serial transmission mode, and allocating the data slices to the target load node according to the slice sequence identifier; in response to the load rate being greater than or equal to the preset first threshold and less than a preset second threshold, adding one or more load nodes as the target load nodes according to the sorting result; obtaining the load rate of each target load node, and adopting a parallel transmission mode, allocating the data slices to each target load node according to the slice sequence identifier according to the load rate of each target load node; in response to the load rate being greater than or equal to the preset second threshold, taking the multiple load nodes as the target load nodes, and adding a preset number of load nodes as spare nodes, and allocating the data slices to the target load nodes and the spare nodes according to the slice sequence identifier.

[0137] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to receiving the data slices, obtaining the slice sequence identifiers of each of the data slices, and performing an integrity check on the data slices according to the slice sequence identifiers; in response to the data slices passing the integrity check, splicing the multiple data slices according to the slice sequence identifiers to obtain the original data files of the data to be backed up.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: monitoring the transmission status of the data shards, the transmission status including a completed state and an uncompleted state; obtaining a first number of data shards in the completed state, and a second number of data shards in the uncompleted state; in response to a failure of the load node, starting a fault recovery program; in response to a successful fault recovery, resuming the transmission of the second number of data shards, and performing an integrity check on the first number of data shards; in response to a failed fault recovery, synchronizing the first number of data shards to a backup node, and allocating the second number of data shards to the backup node.

[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0140] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data backup method, characterized in that: include: Determine multiple load nodes, and mount the created shared directory on the multiple load nodes; Acquire load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result; Slice the data to be backed up according to the slicing strategy corresponding to the target load node to obtain multiple data slicings; Obtain the load rate of the target load node, and determine the transmission mode according to the load rate and the preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

2. The data backup method according to claim 1, characterized in that: The acquiring the load information of the load node includes: Performing heartbeat detection on the load node to obtain the response time of the load node; Checking the service availability of the load node to determine whether the load node is in a normal working state; In response to the response time exceeding a preset time threshold, or the service status of the load node being unavailable, determining that the load node has sent a fault and removing the load node from the node list; In response to the load node being in a normal working state, load information of the load node is acquired.

3. The data backup method according to claim 1, characterized in that: The acquiring the load information of the load node, performing a performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determining one or more target load nodes according to the evaluation result, includes: Obtaining a preset performance indicator, and obtaining load information of the load node based on the performance indicator, wherein the load information includes one or more of a load rate, a network bandwidth, a storage occupancy rate, a historical transmission success rate, and a physical distance; The performance score of the load node is obtained by multiplying each performance indicator of the load node with the corresponding fusion weight; The multiple load nodes are sorted according to the performance scores, and a target load node is determined according to the sorting result.

4. The data backup method according to claim 3, characterized in that: The sharding of the data to be backed up according to the sharding strategy corresponding to the target load node to obtain multiple data shards includes: Acquire the current network bandwidth of the target load node in real time, and determine whether the current network bandwidth reaches a preset bandwidth value; In response to the current network bandwidth not reaching the preset bandwidth value, using the preset shard value as the target shard value of the data shard; In response to the current network bandwidth reaching the preset bandwidth value, obtaining a correspondence between the current network bandwidth and the size of the data shard according to a preset data table, and confirming a target shard value corresponding to the current network bandwidth according to the correspondence; The data to be backed up is sliced ​​according to the target slice value to obtain a plurality of data slices, and slice sequence identifiers are added to the data slices according to the slice processing order.

5. The data backup method according to claim 4, characterized in that: The acquiring the load rate of the target load node, determining a transmission mode according to the load rate and a preset transmission strategy, and allocating the data slices to one or more target load nodes based on the transmission mode includes: In response to taking the first load node as the target load node, obtaining a load rate of the first load node, wherein the load rate includes one or more of a processor utilization rate, a network bandwidth utilization rate, and a memory utilization rate; In response to a load rate of the first load node being less than a preset first threshold, setting the transmission mode to a serial transmission mode, and allocating the data slices to the first load node according to the slice sequence identifier; In response to the load rate of the first load node being greater than or equal to the preset first threshold and less than a preset second threshold, adding one or more second load nodes as the target load nodes according to the sorting result; Obtaining the load rate of each target load node, and setting the transmission mode to a parallel transmission mode, and distributing and processing the data slices according to the load rate of each target load node; In response to the load rate of the first load node being greater than or equal to a preset second threshold, adding a preset number of second load nodes as target load nodes, and adding one or more third load nodes as standby nodes according to the sorting result; The transmission mode is set as a standby transmission mode, and the data slices are distributed and processed according to the load rates of the target load nodes and the standby nodes.

6. The data backup method according to claim 1, characterized in that: The obtaining of the original data file of the data to be backed up and uploading it to the shared directory includes: In response to the target load node receiving the data slices, obtaining the slice sequence identifiers of the data slices, and performing integrity checks on the data slices according to the slice sequence identifiers; In response to the data slice passing the integrity check, the multiple data slices are spliced ​​according to the slice sequence identifiers to obtain the original data files of the data to be backed up and upload them to the shared directory.

7. The data backup method according to claim 1 or 2, characterized in that: The method further comprises: Monitoring the transmission status of the data slices, wherein the transmission status includes a completed status and an uncompleted status; Obtaining a first number of data shards in the completed state and a second number of data shards in the uncompleted state; In response to a failure of the load node, starting a failure recovery procedure; In response to successful failure recovery, resuming transmission of the second number of data shards, and performing integrity checks on the first number of data shards; In response to a failure of fault recovery, the first number of data shards are synchronized to a spare node, and the second number of data shards are allocated to the spare node.

8. A data backup device, characterized in that: The device comprises: A creation module is used to determine a plurality of load nodes and mount the created shared directory to the plurality of load nodes; An evaluation module, used to obtain load information of the load node, perform performance indicator evaluation on the load node based on the load information to obtain an evaluation result, and determine one or more target load nodes according to the evaluation result; A sharding module, used to shard the backup data according to the sharding strategy corresponding to the target load node to obtain multiple data shards; A transmission module is used to obtain the load rate of the target load node, and determine the transmission mode according to the load rate and a preset transmission strategy, and allocate the data shards to one or more target load nodes based on the transmission mode to obtain the original data files of the data to be backed up and upload them to the shared directory.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images