Method and device for determining backup node in global deduplication storage scene

Abstract

The invention relates to a method and a device for determining backup nodes in a global deduplication storage scene. The method comprises the following steps: determining a plurality of backup tasks; the backup task comprises data to be backed up; each backup task corresponds to a fingerprint feature vector determined according to the data to be backed up; according to the feature distance between the fingerprint feature vectors corresponding to the backup tasks, merging the to-be-backed-up data of the backup tasks meeting the merging condition in the backup tasks, and generating a plurality of new backup tasks; according to the fingerprint feature vectors corresponding to the plurality of new backup tasks and the fingerprint feature vectors of the cache data in the plurality of backup nodes, determining the similarity between the plurality of new backup tasks and the plurality of backup nodes; and according to the similarity and the performance information of the backup nodes, sequentially determining target backup nodes for executing the backup task from the plurality of backup nodes. By adopting the method, the determination efficiency of the backup node in the global deduplication storage scene can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for determining a backup node in a global deduplication storage scenario. Background Art

[0002] With the rapid growth of data volume, enterprises may have dozens or even hundreds of backup nodes for data to be backed up, which makes it a huge challenge to manage backup nodes. Deduplication is a widely used technology. Deduplication can also be referred to as deduplication. Deduplication can refer to replacing the physical storage of redundant data with logical references, realizing single-instance storage of data, significantly saving storage space and reducing user costs.

[0003] Distributed global deduplication is a technology that performs data deduplication in multiple storage locations (backup nodes). In traditional technologies, users are usually required to select a reasonable backup node from multiple backup nodes for data backup. However, manually selecting the backup node for backup data will lead to low efficiency in determining the backup node in the global deduplication storage scenario. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for determining backup nodes in a global deduplication storage scenario, which can improve the efficiency of determining backup nodes in a global deduplication storage scenario, in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for determining a backup node in a global deduplication storage scenario, including:

[0006] Determine a plurality of backup tasks; the backup tasks include data to be backed up; each of the backup tasks corresponds to a fingerprint feature vector determined according to the data to be backed up;

[0007] According to the characteristic distances between the fingerprint characteristic vectors corresponding to the backup tasks, the to-be-backed-up data of the backup tasks that meet the merging conditions in the backup tasks are merged to generate multiple new backup tasks;

[0008] Determining similarities between the new multiple backup tasks and the multiple backup nodes according to the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes;

[0009] According to the similarity and the performance information of the backup node, a target backup node for executing the backup task is determined in sequence from the multiple backup nodes; the target backup node is used to store the data to be backed up of the backup task and delete duplicate data in the data to be backed up and the cached data.

[0010] In one embodiment, merging the to-be-backed-up data of the backup tasks that meet the merging condition in the backup tasks according to the feature distances between the fingerprint feature vectors corresponding to the backup tasks to generate multiple new backup tasks includes:

[0011] Determine two backup tasks with the smallest characteristic distances according to characteristic distances between fingerprint characteristic vectors corresponding to the backup tasks;

[0012] If the total amount of data to be backed up after merging the two backup tasks is less than the preset total amount of data, the two backup tasks are merged to obtain new multiple backup tasks, and the step of determining the two backup tasks with the smallest feature distance based on the feature distance between the fingerprint feature vectors corresponding to each of the backup tasks is returned until the number of tasks in the new multiple backup tasks is less than the preset number of categories.

[0013] In one of the embodiments, the preset total amount of data is determined based on the total amount of data of the multiple backup tasks and the preset number of categories; the preset number of categories is determined based on the performance information and the number of nodes of the multiple backup nodes.

[0014] In one embodiment, determining in sequence from the plurality of backup nodes a target backup node for executing the backup task according to the similarity and the performance information of the backup node includes:

[0015] For any backup task among the new multiple backup tasks, according to the similarity between the any backup task and the multiple backup nodes and the performance information of the backup nodes, determine a target backup node for executing the any backup task from the multiple backup nodes;

[0016] In the case that the target backup node completes executing any one of the backup tasks, the backup tasks other than the any one of the backup tasks are taken as the new multiple backup tasks, and the backup nodes other than the target backup node of the any one of the backup tasks are taken as the new multiple backup nodes, and the step of returning to any one of the new multiple backup tasks, and determining the efficiency parameters of the multiple backup nodes according to the similarity between the any one of the backup tasks and the multiple backup nodes and the performance information of the backup nodes.

[0017] In one embodiment, determining in sequence from the plurality of backup nodes a target backup node for executing the backup task according to the similarity and the performance information of the backup node includes:

[0018] Obtaining the storage cost of the backup node and the efficiency cost of the backup node; the storage cost is determined according to the deduplication saving amount of the backup node, the deduplication saving amount includes the amount of data that can be saved by the backup node after deleting duplicate data in the to-be-backed-up data and the cached data; the efficiency cost is determined according to the network performance and backup efficiency of the backup node;

[0019] Determining a transmission cost of the backup node according to the similarity and the data movement cost of the backup node;

[0020] Determining efficiency parameters of the plurality of backup nodes according to the storage cost, the efficiency cost and the transmission cost;

[0021] According to the efficiency parameters of the multiple backup nodes, a target backup node for executing the backup task is determined in sequence from the multiple backup nodes.

[0022] In one of the embodiments, before merging the to-be-backed-up data of the backup tasks that meet the merging condition among the backup tasks according to the feature distances between the fingerprint feature vectors corresponding to the backup tasks, the method further includes:

[0023] Determining a plurality of data blocks to be backed up from the data to be backed up in the backup task;

[0024] Determine multiple hash values ​​corresponding to any of the data blocks to be backed up;

[0025] Using the minimum hash value among the multiple hash values ​​as the characteristic value of any of the data blocks to be backed up;

[0026] According to the characteristic values ​​of each of the data blocks to be backed up, a fingerprint characteristic vector corresponding to the backup task is constructed.

[0027] In one of the embodiments, before determining the similarity between the new multiple backup tasks and the multiple backup nodes based on the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes, the method further includes:

[0028] Determine the cache data according to the hot data in the backup node; the hot data includes data in the backup node whose access frequency is higher than a frequency threshold;

[0029] Determining a plurality of node data blocks from the cache data;

[0030] A fingerprint feature vector of the cache data is determined according to the fingerprints of the multiple node data blocks.

[0031] In a second aspect, the present application further provides a device for determining a backup node in a global deduplication storage scenario, including:

[0032] A task module, used to determine a plurality of backup tasks; the backup tasks include data to be backed up; each of the backup tasks corresponds to a fingerprint feature vector determined according to the data to be backed up;

[0033] A merging module, configured to merge the to-be-backed-up data of the backup tasks that meet the merging conditions among the backup tasks according to the feature distances between the fingerprint feature vectors corresponding to the backup tasks, and generate multiple new backup tasks;

[0034] A determination module, used to determine the similarity between the new multiple backup tasks and the multiple backup nodes according to the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes; the backup nodes are used to store the data to be backed up and delete duplicate data in the data to be backed up and the cached data;

[0035] The backup module is used to determine the target backup node for executing the backup task from the multiple backup nodes in sequence according to the similarity and the performance information of the backup node.

[0036] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0038] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0039] The method, apparatus, computer device, computer-readable storage medium and computer program product for determining backup nodes in the above-mentioned global deduplication storage scenario determine multiple backup tasks, wherein the backup tasks include data to be backed up, and each backup task corresponds to a fingerprint feature vector determined according to the data to be backed up; based on the feature distance between the fingerprint feature vectors corresponding to each backup task, the data to be backed up of the backup tasks that meet the merging conditions in each backup task are merged to generate multiple new backup tasks; based on the fingerprint feature vectors corresponding to the multiple new backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes, the similarity between the multiple new backup tasks and the multiple backup nodes is determined; based on the similarity and the performance information of the backup nodes, the backup nodes for executing the backup tasks are sequentially determined from the multiple backup nodes. The target backup node is used to store the data to be backed up of the backup task, and delete duplicate data in the data to be backed up and the cached data. By analyzing the data to be backed up in the backup task and determining its fingerprint feature vector, the data to be backed up with high similarity is merged, so as to efficiently classify the backup task; by comparing the fingerprint feature vectors of the newly generated backup task with the cached data in the backup node, the similarity analysis between each backup task and each backup node is realized, and according to the similarity and the performance information of the backup node, the target backup node for executing the backup task is determined in turn, which can ensure that the data to be backed up is allocated to the backup node with high data similarity and excellent performance, simplify the complexity of backup task scheduling, and improve the efficiency of determining the backup node in the global deduplication storage scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] FIG1 is an application environment diagram of a method for determining a backup node in a global deduplication storage scenario in one embodiment;

[0042] FIG2 is a schematic flow chart of a method for determining a backup node in a global deduplication storage scenario in one embodiment;

[0043] FIG3 is a schematic diagram of the structure of a backup node in one embodiment;

[0044] FIG4 is a flowchart of a backup task merging process in one embodiment;

[0045] FIG5 is a schematic diagram of a result of merging backup tasks in one embodiment;

[0046] FIG6 is a flowchart of a method for selecting a backup node in one embodiment;

[0047] FIG7 is a structural block diagram of a device for determining a backup node in a global deduplication storage scenario in one embodiment;

[0048] FIG. 8 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The method for determining backup nodes in a global deduplication storage scenario provided in an embodiment of the present application can be applied to an application environment as shown in FIG1. ​​A global deduplication storage scenario may refer to a scenario in which data deduplication technology is applied in a distributed system spanning multiple storage nodes, devices, or storage pools to reduce redundant data storage. The global deduplication storage scenario may include a backup system deployed through a distributed architecture, which applies deduplication technology and can deduplicate data globally (across multiple storage nodes) to improve storage efficiency and optimize resource utilization.

[0051] As shown in Figure 1, the backup system may include a network device 100 and multiple servers. The network device 100 can be connected to one or more servers. These servers may include computing servers, virtualization servers and storage servers, providing various cloud services and business applications. These servers can achieve data management, data monitoring, data backup and data disaster recovery by connecting to the network device 100.

[0052] The multiple servers of the backup system may include a backup center server 102 and a backup node 104. The multiple backup nodes 104 may constitute a deduplication storage cluster, which may include multiple storage nodes working in coordination to manage and store data globally through deduplication technology.

[0053] The network device 100 may include but is not limited to various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, portable wearable devices and other terminals, and may also include workstations within the enterprise.

[0054] The backup node 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0055] The backup service center 102 can be used to establish and assign backup tasks to the backup node 104 based on the data to be backed up uploaded by the network device 100, and can monitor and manage the process and status of data backup, thereby providing data recovery and business continuity support when a disaster occurs.

[0056] The backup center server 102 communicates with the network device 100 and the backup node 104 through the network. Specifically, the network device 100 can be connected to the backup node 104 through a network connector and a cache service, and the network device 100 can also be connected to the backup center server 102 through a network connector.

[0057] The backup center server 102 determines multiple backup tasks; the backup tasks include data to be backed up; each backup task corresponds to a fingerprint feature vector determined based on the data to be backed up; the backup center server 102 merges the data to be backed up of the backup tasks that meet the merging conditions in each backup task according to the feature distance between the fingerprint feature vectors corresponding to each backup task, and generates new multiple backup tasks; the backup center server 102 determines the similarity between the new multiple backup tasks and the multiple backup nodes 104 according to the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes 104; the backup center server 102 determines the target backup node for executing the backup task from the multiple backup nodes 104 in turn according to the similarity and the performance information of the backup node 104; the target backup node is used to store the data to be backed up of the backup task, and delete duplicate data in the data to be backed up and the cached data.

[0058] In an exemplary embodiment, as shown in FIG. 2 , a method for determining a backup node in a global deduplication storage scenario is provided, and the method is described by taking the backup center server 102 in FIG. 1 as an example, including:

[0059] Step S202: determine multiple backup tasks.

[0060] The backup task includes data to be backed up; each backup task corresponds to a fingerprint feature vector determined according to the data to be backed up.

[0061] In a specific implementation, the data to be backed up included in the backup task may be a data set consisting of one or more files. Each backup task is responsible for backing up a specific data set. The backup center server may determine the fingerprint feature vector of the backup task according to the data to be backed up included in the backup task.

[0062] The fingerprint feature vector may be a unique identifier generated based on data content and used to describe data features in the backup task. The fingerprint feature vector may uniquely identify the data set of the backup task.

[0063] As an example, before the backup task starts, the backup center server can calculate the fingerprint set of the backup task, and construct a fingerprint feature vector through the fingerprint set. The feature value in the fingerprint feature vector can be the fingerprint in the fingerprint set, so as to identify the data to be backed up of the backup task through the fingerprint feature vector. Optionally, the fingerprint set can include multiple fingerprints. The fingerprint is a short, unique identifier or summary generated from the data, which is used to characterize the characteristics of the data.

[0064] As an example, the fingerprint in the fingerprint set can be a hash fingerprint determined by a hash algorithm. The hash algorithm is an algorithm that converts input of any length (such as text or data) into a fixed-length output through a specific algorithm. The result of the hash algorithm output can be called a hash value, which can be used to uniquely identify the data content of the data to be backed up.

[0065] In one embodiment, the fingerprint set of the backup task can be a set of fingerprints corresponding to each data block in the data to be backed up of the backup task, and the size of the data block of the data to be backed up can be determined according to the file structure and specifications of the data to be backed up. Exemplarily, if the data to be backed up includes large files such as virtual machines and databases, the data to be backed up can be divided into 64 kilobytes (KB) or 128kB data blocks, and a fingerprint is calculated for each data block; if the data to be backed up is small unstructured data, such as logs, documents, pictures and other data, the data to be backed up can be divided into 8KB or 16kB data blocks, and a fingerprint is calculated for each data block, so that the fingerprint set of the backup task is composed according to the fingerprints of each data block, and a fingerprint feature vector is constructed according to the fingerprint set, and the feature value in the fingerprint feature vector can be the fingerprint in the fingerprint set.

[0066] In one of the embodiments, when a fingerprint is calculated for each data block, the calculated fingerprint can be truncated to reduce the size of the fingerprint. For example, a 64KB data can calculate a 256-bit fingerprint value, but in order to save storage space and processing efficiency, while ensuring the data collision rate, N (N<256, N is a power of 2) bits can be used for representation. If the fingerprint value is a hash value, a compact hash sequence can be constructed based on the hash values ​​of each data block, and the fingerprint feature vector generated based on the hash sequence is used to represent the data features of the data to be backed up, so as to support data similarity comparison between backup nodes in cross-backup tasks and deduplication storage scenarios.

[0067] Step S204: merging the to-be-backed-up data of the backup tasks that meet the merging condition among the backup tasks according to the feature distances between the fingerprint feature vectors corresponding to the backup tasks, and generating multiple new backup tasks.

[0068] The characteristic distance between the fingerprint characteristic vectors can be used to represent the similarity between the backup tasks, and can be calculated based on the fingerprint characteristic vectors of the two backup tasks. The smaller the characteristic distance, the more similar the data contents of the to-be-backed-up data of the two backup tasks are.

[0069] In a specific implementation, the feature distance between fingerprint feature vectors may include but is not limited to Euclidean distance, Manhattan distance, Jaccard distance, etc.

[0070] In the specific implementation, if the characteristic distance of two or more backup tasks is within a certain threshold range (i.e., the similarity is high), the backup center server can merge the data to be backed up of these backup tasks into a new backup task. After the merge operation, a new set of backup tasks can be generated, and each new backup task may contain a combination of multiple data to be backed up, which can reduce the complexity of the backup operation, simplify the subsequent process of determining the backup node, improve the efficiency of determining the backup node, and enable the data to be backed up with high similarity to be in the same backup task and assigned to the same backup node for concurrent processing. Since the processed data to be backed up has a certain similarity, the pressure on network transmission is reduced, and the efficiency parameters of the backup node and the efficiency of deduplication are improved.

[0071] Step S206: Determine the similarity between the new multiple backup tasks and the multiple backup nodes according to the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes.

[0072] Among them, the backup node can refer to the server responsible for executing the backup task in the global deduplication storage scenario, and can deduplicate the backup data through the deduplication pool, thereby deleting duplicate data and retaining a unique, non-duplicate data copy.

[0073] To facilitate the understanding of those skilled in the art, FIG3 exemplarily provides a schematic diagram of the structure of a backup node. The backup node includes a file storage and reading system, which has data classification capabilities. In this system, the most similar file sets are grouped together through a clustering algorithm, and the deduplication storage cluster is iteratively expanded to include more file sets until the expected capacity requirements fill the deduplication node. At the same time, the input and output of data is accelerated, and the memory and computing overhead of this process are minimized to form a deduplication storage cluster that minimizes storage requirements. In one embodiment, the file storage and reading system analyzes files through a storage processor, partitions files and their metadata through storage areas, performs content-aware processing on files through a content-aware processor, timely releases cache space in hot storage areas through a space release processor, and reads files to be read through a read processor. Therefore, on the one hand, through the perception of the content of the file, the file storage and reading system can obtain metadata including the semantic features of the file, which is convenient for the subsequent merging of each file based on the file metadata, thereby reducing the space resources occupied by file storage and reducing the time resources spent on file reading; on the other hand, through the partitioning management of data in the cold storage area and the hot storage area, the respective characteristics of the cold storage area and the hot storage area can be fully utilized, reducing the time resources spent on file reading; based on the above-mentioned file storage and reading system, it is possible to reduce the space resources occupied by small file storage and the time resources spent on file reading, thereby improving the resource utilization rate during file storage and reading.

[0074] The cache data in the backup node is the data stored in the backup node. The cache data in the backup node may also correspond to a fingerprint feature vector for characterizing the data features of the cache data in the backup node.

[0075] In a specific implementation, the fingerprint feature vector of the cached data can be generated based on the fingerprint corresponding to each data block in the cached data. The cached data can be data randomly extracted from the backup node, or can be more important data selected from the backup node that meets the backup format requirements. Optionally, the fingerprint corresponding to each data block in the cached data can also be a hash value calculated by a hash function.

[0076] In a specific implementation, after determining the new multiple backup tasks, the fingerprint feature vector corresponding to the new backup tasks can be recalculated for the data to be backed up in the new multiple backup tasks, so as to determine the similarity between the new multiple backup tasks and the multiple backup nodes based on the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes. Optionally, the Euclidean distance similarity, cosine similarity, Jaccard similarity, etc. between the fingerprint feature vectors of the data to be backed up and the fingerprint feature vectors of the cached data can be calculated to determine the similarity between the new multiple backup tasks and the multiple backup nodes.

[0077] The higher the similarity between the backup task and the backup node, the more duplicate parts there are between the backup task's data to be backed up and the existing cached data on the backup node, so that similar data can be stored in one node, improving the deduplication efficiency. In the process of deduplication technology deleting duplicate data, there is no need to frequently transmit data between backup nodes, reducing the consumption of network bandwidth.

[0078] Step S208: determining a target backup node for executing the backup task from the plurality of backup nodes in sequence according to the similarity and the performance information of the backup nodes.

[0079] The target backup node is used to store the data to be backed up of the backup task and delete duplicate data in the data to be backed up and cached data.

[0080] As an example, the performance information of the backup node may include but is not limited to the storage speed of the backup node, the data storage density of the backup node, network characteristics, load conditions and other information, so that a suitable backup node for data storage can be planned for each backup task based on multiple measurement criteria such as similarity and performance information of the backup node.

[0081] In the specific implementation, for any backup task, the backup center server can prioritize the backup nodes according to the similarity and performance information of the backup nodes, select a backup node with high similarity to the backup task and excellent performance from multiple backup nodes, and use the backup node as the target backup node for executing the backup task. Since the similarity between the data to be backed up of the backup task and the cache data stored in the selected backup node is high, the complexity of backup task scheduling is simplified, the overall efficiency of the backup system is improved, and the purpose of storage network load balancing is achieved. Then, for the next backup task, the similarity and performance information of the backup node can be recalculated, and the target backup node most suitable for executing the next backup task can be selected from multiple backup nodes, so that the target backup node for executing the backup task can be determined in sequence for multiple backup tasks.

[0082] If similar data is stored in different backup nodes, the deduplication process needs to frequently transfer data between backup nodes, resulting in large network bandwidth consumption. In addition, if similar data is stored in a dispersed manner, the storage space of the backup node cannot be fully utilized, resulting in low storage space utilization. Therefore, by analyzing the data similarity of backup tasks and backup nodes, and combining performance information such as network characteristics, node storage density, and storage speed, the optimal backup node is automatically selected, which reduces the complexity and error rate of manual node selection by users, improves the efficiency of determining backup nodes in global deduplication storage scenarios, and reduces the amount of redundant data transmitted in the network by reasonably selecting backup nodes, thereby reducing network bandwidth occupancy and improving the efficiency of data backup and data recovery.

[0083] It can be seen that this solution describes a method for determining the backup node used to perform backup tasks in a global deduplication storage scenario. Through the fingerprint feature vector and similarity analysis of the data, the appropriate backup node is selected and data deduplication storage is performed, which not only optimizes the allocation of backup tasks, but also improves data storage and efficiency parameters.

[0084] The method, apparatus, computer device, computer-readable storage medium and computer program product for determining backup nodes in the above-mentioned global deduplication storage scenario determine multiple backup tasks, wherein the backup tasks include data to be backed up, and each backup task corresponds to a fingerprint feature vector determined according to the data to be backed up; based on the feature distance between the fingerprint feature vectors corresponding to each backup task, the data to be backed up of the backup tasks that meet the merging conditions in each backup task are merged to generate multiple new backup tasks; based on the fingerprint feature vectors corresponding to the multiple new backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes, the similarity between the multiple new backup tasks and the multiple backup nodes is determined; based on the similarity and the performance information of the backup nodes, the backup nodes for executing the backup tasks are sequentially determined from the multiple backup nodes. The target backup node is used to store the data to be backed up of the backup task, and delete duplicate data in the data to be backed up and the cached data. By analyzing the data to be backed up in the backup task and determining its fingerprint feature vector, the data to be backed up with high similarity is merged, so as to efficiently classify the backup task; by comparing the fingerprint feature vectors of the newly generated backup task with the cached data in the backup node, the similarity analysis between each backup task and each backup node is realized, and according to the similarity and the performance information of the backup node, the target backup node for executing the backup task is determined in turn, which can ensure that the data to be backed up is allocated to the backup node with high data similarity and excellent performance, simplify the complexity of backup task scheduling, and improve the efficiency of determining the backup node in the global deduplication storage scenario.

[0085] In another embodiment, before merging the data to be backed up of the backup tasks that meet the merging conditions in the backup tasks according to the characteristic distances between the fingerprint feature vectors corresponding to the various backup tasks, it also includes: determining multiple data blocks to be backed up from the data to be backed up in the backup tasks; determining multiple hash values ​​corresponding to any data block to be backed up; using the minimum hash value among the multiple hash values ​​as the characteristic value of any data block to be backed up; and constructing the fingerprint feature vector corresponding to the backup task according to the characteristic values ​​of each data block to be backed up.

[0086] The data blocks to be backed up are data blocks determined in the data to be backed up. Optionally, the data blocks to be backed up may be data blocks randomly selected from the data to be backed up, and the size of the data blocks to be backed up may be determined according to the file structure and specifications of the data to be backed up. Exemplarily, if the data to be backed up includes large files such as virtual machines and databases, the data blocks to be backed up may be data blocks of 64 kilobytes (KB) or 128kB in size; if the data to be backed up is small unstructured data, such as logs, documents, pictures and other data, the data to be backed up may be data blocks of 8KB or 16kB in size.

[0087] In a specific implementation, a data block to be backed up can calculate multiple hash values. The backup center server can calculate multiple hash values ​​corresponding to any data block to be backed up by constructing a hash function sequence, and the hash function sequence can be defined by two unrelated hash functions.

[0088] As an example, assuming that the two unrelated hash functions are defined as h1 and h2, and the two unrelated hash functions are a message-digest (MD5) hash function and a secure hash (SHA1) function, the hash function sequence hi can be defined as:

[0089] hi=h1+i*h2 mod p, 0≤i <k。

[0090] Wherein, i is an integer greater than or equal to 0 and less than k; p is a prime number greater than k; mod is a modulus operator; the larger k is, the more accurate the subsequent calculation of the similarity between feature vectors is. Optionally, k can be 1024.

[0091] As another example, assuming that the two unrelated hash functions are defined as h1 and h2, and the two unrelated hash functions are two multiplication and rotation hash (Murmur) functions with different initial seeds, the hash function sequence hi can be defined as:

[0092] hi=h1+i*h2+ i 2 , 0≤i <k。

[0093] Here, i is an integer greater than or equal to 0 and less than k. The larger k is, the more accurate the subsequent calculation of the similarity between feature vectors is. Optionally, k can be 1024.

[0094] It can be seen that by constructing the above hash function sequence, k hash values ​​corresponding to any data block to be backed up can be calculated. Compared with using k independent hash functions to calculate multiple hash values, the above method of constructing the hash function sequence is simpler and more convenient.

[0095] After determining the multiple hash values ​​corresponding to each data block to be backed up, the minimum hash value of each data block to be backed up can be taken, and the minimum hash value can be used as the feature value of the data block to be backed up. After sequentially obtaining the feature values ​​of all data blocks to be backed up in the backup task, the fingerprint feature vector of the backup task can be obtained.

[0096] In one embodiment, assuming that there are T backup tasks, the feature distances between the fingerprint feature vectors corresponding to each backup task are calculated, and a The similarity upper triangular matrix can also be called a distance matrix. Taking the Jaccard distance as an example, the The similarity upper triangular matrix (distance matrix) can be expressed as Table 1.

[0097] Table 1

[0098]

[0099] Among them, FP i The fingerprint feature vector that can be used to indicate the backup task, J(FP i , FP j ) is the Jaccard distance between the fingerprint feature vectors of the backup tasks.

[0100] It can be seen that by filtering and sampling the data to be backed up of the backup task, the data block used to calculate the fingerprint is obtained, and by calculating the characteristic distance between the backup tasks, the upper triangular matrix of similarity between the backup tasks is obtained, and the subsequent similarity evaluation of the backup tasks and the backup nodes is performed, the best backup node can be quickly evaluated, and the load balancing of the storage network can be automatically performed; in the processing process, through these characteristic vectors and characteristic matrices, the system's demand for resources such as memory and processors is significantly reduced, and through the hash function sequence and continuous iterative merging of backup tasks, the accuracy of the similarity evaluation of backup tasks and backup nodes can be further improved.

[0101] The technical solution of this embodiment uniquely identifies the data block to be backed up by a hash value, and uses the minimum hash value among multiple hash values ​​as the characteristic value of the data block to be backed up, thereby improving the accuracy and efficiency of constructing the fingerprint feature vector corresponding to the backup task, thereby improving the accuracy and efficiency of subsequent merging of backup tasks.

[0102] In another embodiment, the data to be backed up of the backup tasks that meet the merging conditions in each backup task are merged according to the characteristic distance between the fingerprint feature vectors corresponding to each backup task to generate new multiple backup tasks, including: determining the two backup tasks with the smallest characteristic distance according to the characteristic distance between the fingerprint feature vectors corresponding to each backup task; if the total amount of data to be backed up included in the two backup tasks after merging is less than the preset total amount of data, the two backup tasks are merged to obtain new multiple backup tasks, and the step of determining the two backup tasks with the smallest characteristic distance according to the characteristic distance between the fingerprint feature vectors corresponding to each backup task is returned until the number of tasks in the new multiple backup tasks is less than the preset number of categories.

[0103] In order to simplify the subsequent calculation of selecting backup nodes, similar backup tasks can be merged. These merged backup tasks can use the same backup node, so that the backup tasks of each backup node can be processed concurrently at the same time. Since the backup data to be processed by the same backup node has a certain similarity, the pressure on network transmission is reduced and the storage efficiency is improved.

[0104] In another embodiment, the preset total amount of data is determined based on the total amount of data of multiple backup tasks and the preset number of categories; the preset number of categories is determined based on the performance information and the number of nodes of multiple backup nodes.

[0105] The preset number of categories is used to indicate the number of new backup tasks generated after selectively merging multiple backup tasks. The preset number of categories can be determined based on performance information such as the number of backup nodes, network capabilities, and storage density of backup nodes in the current backup system, and the preset number of categories can be dynamically adjusted based on the performance information and number of nodes of multiple backup nodes.

[0106] The preset total amount of data can be used to indicate the upper limit of the total amount of data to be backed up included in a backup task. If each backup task is regarded as a class, the data size range of each class can be specified for uniformity of data storage, that is, the preset total amount of data to be backed up included in a backup task is specified. The preset total amount of data can be determined based on the total amount of data of multiple backup tasks and the number of preset classes.

[0107] In one embodiment, assuming that the number of preset categories is cn, there are T backup tasks, that is, there are T categories, the preset total amount of data can be expressed as: 1.2*total_size{T} / cn, where total_size{T} is the total amount of data of all backup tasks. If the total amount of data to be backed up of a single backup task exceeds 1.2*total_size{T} / cn, the backup task is a separate category and is no longer merged.

[0108] To facilitate understanding by those skilled in the art, FIG4 exemplarily provides a flowchart of a backup task merge. In a specific implementation, the backup center server may search for two backup tasks C with the smallest characteristic distance from the distance matrix shown in Table 1. i and C j If the total amount of data to be backed up after merging the data to be backed up in this backup task is less than the preset total amount of data, the two backup tasks C i and C j Merge into a new class C ij Otherwise, return to the step of finding the two backup tasks with the smallest feature distance from the distance matrix; after merging, the new class C is obtained. ij Afterwards, C can be calculated ij The characteristic distance between the two backup tasks and other backup tasks is calculated, and the distance matrix is ​​updated. The step of finding the two backup tasks with the smallest characteristic distance from the distance matrix is ​​returned, and the merging is continued until the number of new backup tasks is less than the preset number of categories. The merging can be stopped, so that a new T c backup tasks, and T c Less than or equal to T.

[0109] In order to facilitate the understanding of those skilled in the art, FIG5 exemplarily provides a schematic diagram of the result of backup task merging. In FIG5 , T1~T6 are used to refer to multiple backup tasks. It can be seen that T1 and T2 can be merged into a class C1. Although T3 and C1 have a high similarity (small feature distance), there is more data to be backed up in the T3 backup task (it has exceeded the preset total amount of data). Therefore, T3 can be independently a class C2. T4~T6 meet the requirements in terms of similarity (small feature distance) and backup task capacity (less than the preset total amount of data), so T4~T6 are merged into class C3. After the merging is completed, the classification result can be obtained.

[0110] In another embodiment, before determining the similarity between the new multiple backup tasks and the multiple backup nodes based on the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes, it also includes: determining the cached data based on the hot data in the backup nodes; determining multiple node data blocks from the cached data; and determining the fingerprint feature vector of the cached data based on the fingerprints of the multiple node data blocks.

[0111] The hot data includes data in the backup node whose access frequency is higher than a frequency threshold. In other words, the hot data may refer to data in the backup node that is frequently accessed or frequently used.

[0112] The node data block is a data block determined in the cache data.

[0113] In a specific implementation, the backup center server can obtain hot data from the storage area in the backup node, and extract multiple node data blocks from the hot data, which can be extracted randomly or obtained by sampling and filtering. For multiple node data blocks of any backup node, the hash value of each node data block can be calculated through a hash function sequence, and the feature value is determined according to the hash value of each node data block, thereby determining the fingerprint feature vector of the cached data.

[0114] In another embodiment, the backup center server can select F file sets from any backup node, and then extract M node data blocks from each file set, and calculate the fingerprint feature vector of the cached data based on the feature values ​​of the M node data blocks extracted from the F file sets. The larger the value of F, the more file sets are selected, the more accurate the estimated similarity between the backup task and the backup node, and the more computational effort will be.

[0115] In a specific implementation, the backup center server determines the similarity between the new multiple backup tasks and the multiple backup nodes based on the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes. The fingerprint set corresponding to each backup task and the fingerprint set corresponding to the cached data in each backup node can be determined based on the characteristic values ​​in the fingerprint feature vector. Then, for any backup task, the Jaccard similarity between the fingerprint set of any backup task and the fingerprint set corresponding to the cached data in each backup node can be calculated. The Jaccard similarity can be determined based on the ratio of the intersection and the union of the elements in the two sets. Finally, based on the Jaccard similarity between the fingerprint set of each backup task and the fingerprint set corresponding to the cached data in each backup node, a similarity matrix can be calculated. The values ​​in the similarity matrix include the Jaccard similarity between the fingerprint set of each backup task and the fingerprint set corresponding to the cached data in each backup node.

[0116] In the technical solution of this embodiment, since the fingerprint set stored in the backup node is massive, the backup node preferentially uses the fingerprint of the hot data in the hot storage area to compare with the fingerprint of the data to be backed up of the backup task, which can ensure to the greatest extent that the intersection between the fingerprint set of the backup task and the fingerprint set of the backup node is not empty, thereby improving the accuracy of calculating the similarity between the backup task and the backup node, thereby improving the accuracy of determining the backup node in the global deduplication storage scenario.

[0117] In another embodiment, a target backup node for performing a backup task is determined in sequence from a plurality of backup nodes based on similarity and performance information of the backup node, including: obtaining the storage cost of the backup node and the efficiency cost of the backup node; the storage cost is determined based on the deduplication savings of the backup node, the deduplication savings including the amount of data that can be saved by the backup node after deleting duplicate data in the data to be backed up and cached data; the efficiency cost is determined based on the network performance and backup efficiency of the backup node; the transmission cost of the backup node is determined based on the similarity and the data movement cost of the backup node; the efficiency parameters of the plurality of backup nodes are determined based on the storage cost, the efficiency cost and the transmission cost; and the target backup node for performing the backup task is determined in sequence from the plurality of backup nodes based on the efficiency parameters of the plurality of backup nodes.

[0118] Among them, storage cost refers to the resource consumption or cost incurred by the backup node when storing data, which is usually related to factors such as storage capacity, storage media, and deduplication efficiency. A node with lower storage cost means that it has greater deduplication savings, that is, less physical space is required to store data. Storage cost is an important consideration when selecting a backup node.

[0119] Deduplication savings refers to the amount of storage space that can be saved by the backup node after deduplication technology (deleting duplicate data), which includes the savings after deleting duplicate data in the data to be backed up and cached data. The greater the deduplication savings, the more efficiently the backup node can store data and reduce the physical storage occupied.

[0120] The efficiency cost is calculated based on the backup node's network performance and backup efficiency. The backup node's network performance determines the speed of data transmission, and the backup efficiency determines the speed of data processing and storage. The lower the efficiency cost, the higher the data transmission and storage processing capabilities of the backup node, and the faster the backup task can be completed.

[0121] The data movement cost can be the cost incurred when transferring data from the source node to the backup node during the backup process. The cost may include network bandwidth consumption, data transfer time, and system resource usage. The lower the data movement cost, the higher the efficiency of transferring data from the source node to the backup node.

[0122] The transmission cost is determined based on the similarity of the backup nodes and the data movement cost. The transmission cost represents the overall cost of transmitting data from the source node to the backup node. The lower the transmission cost, the more efficient the data transmission to the node is. It is one of the important factors in selecting the target backup node.

[0123] The efficiency parameter is an indicator that takes into account storage cost, efficiency cost and transmission cost, and measures the overall efficiency of the backup node when performing backup tasks. The higher the efficiency parameter, the faster and more economical the backup node can complete the backup task. Therefore, among multiple backup nodes, the system will give priority to nodes with higher efficiency parameters.

[0124] The target backup node refers to the node that is ultimately selected from multiple backup nodes to perform backup tasks based on factors such as the storage cost, efficiency cost, and transmission cost of each node. This ensures that the backup task is performed on the optimal node, which can maximize the utilization efficiency of storage resources and optimize backup performance.

[0125] In one embodiment, the efficiency parameter may be equal to the sum of the storage cost and the efficiency cost divided by the transmission cost. The efficiency parameter may be expressed as:

[0126] ;

[0127] Among them, efficiency is the efficiency parameter, cost(saving) is the storage cost, cost(speed) is the efficiency cost, and cost(movment) is the transmission cost.

[0128] The technical solution of this embodiment automatically selects the optimal backup node by analyzing the data similarity between backup tasks and backup nodes, and the efficiency parameters determined by combining factors such as storage cost, efficiency cost and transmission cost, thereby improving the accuracy and efficiency of determining the backup node. The backup node can complete the backup task more quickly and economically, thereby improving the backup efficiency in the global deduplication storage scenario, and at the same time, reducing the complexity and error rate of users' manual node selection.

[0129] In another embodiment, a target backup node for executing a backup task is determined in sequence from a plurality of backup nodes based on similarity and performance information of the backup nodes, including: for any backup task in the new plurality of backup tasks, based on similarity between any backup task and the plurality of backup nodes and performance information of the backup nodes, a target backup node for executing any backup task is determined from the plurality of backup nodes; when the target backup node completes executing any backup task, other backup tasks except for any backup task are used as new plurality of backup tasks, and other backup nodes except for the target backup node of any backup task are used as new plurality of backup nodes, and a step of determining efficiency parameters of the plurality of backup nodes for any backup task in the new plurality of backup tasks based on similarity between any backup task and the plurality of backup nodes and performance information of the backup nodes is returned.

[0130] In order to facilitate the understanding of those skilled in the art, FIG6 exemplarily provides a flowchart of a method for selecting a backup node. In a specific implementation, the backup center server can calculate the efficiency parameters of each backup node for any backup task based on the similarity between any backup task and multiple backup nodes and the performance information of the backup nodes, sort the backup nodes from high to low according to the efficiency parameters, and then start to select the backup node corresponding to the backup task, which may include the following steps: a. Determine the backup node with the highest efficiency parameter. b. Check whether the backup node meets all constraints, such as whether it meets the constraints of the minimum savings, the maximum movement, and the maximum node capacity. The minimum savings is the minimum amount of storage space that the backup node can save when storing data through data deduplication technology, the maximum movement is the maximum amount of data that the backup node needs to migrate in the process of data deduplication in order to achieve data balance or system optimization goals, and the maximum node capacity is the maximum amount of valid data that the backup node can store after applying the data deduplication technology. c. If the constraints are met, select the backup node as the target backup node for executing the backup task, and record its processing results. d. Remove the backup node so that it is no longer used by other backup tasks. In other words, when the target backup node completes any backup task, other backup nodes except the target backup node of any backup task are used as new multiple backup nodes, that is, each backup node can be excluded after completing a backup task. This ensures the load balancing of each backup node, avoids node overload or resource waste, and improves the reliability and stability of the entire backup system. e. Update the list of processed backup tasks. f. Update the status of the remaining backup nodes, recalculate the efficiency parameters of each backup node for the next backup task, and then return to step a until all constraints are met or all backup tasks are processed. g. Output the final selected backup node list and processing results, including the total deduplication savings and the total mobility cost.

[0131] In another embodiment, the backup center server determines the target backup node for executing the backup task from multiple backup nodes in sequence according to the similarity and the performance information of the backup nodes, including: obtaining the storage cost of the backup node and the efficiency cost of the backup node; the storage cost is determined according to the deduplication saving amount of the backup node, and the deduplication saving amount includes the amount of data that can be saved by the backup node after deleting duplicate data in the data to be backed up and the cached data; the efficiency cost is determined according to the network performance and backup efficiency of the backup node; the transmission cost of the backup node is determined according to the similarity and the data movement cost of the backup node; the efficiency parameters of the multiple backup nodes are determined according to the storage cost, the efficiency cost and the transmission cost; for any backup task in the new multiple backup tasks, according to the efficiency parameters of the multiple backup nodes, the target backup node for executing any backup task is determined in sequence from the multiple backup nodes; when the target backup node completes any backup task, the other backup tasks except any backup task are used as new multiple backup tasks, and the other backup nodes except the target backup node of any backup task are used as new multiple backup nodes, and the step of determining the efficiency parameters of the multiple backup nodes according to the similarity between any backup task and the multiple backup nodes and the performance information of the backup nodes for any backup task in the new multiple backup tasks is returned.

[0132] The technical solution of this embodiment reasonably selects backup nodes through data similarity analysis and performance information, reduces the amount of redundant data transmitted in the network, thereby reducing the global redundant data degree, reducing network bandwidth occupancy, and improving the efficiency of data backup and data recovery.

[0133] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0134] Based on the same inventive concept, the embodiment of the present application also provides a device for determining backup nodes in a global deduplication storage scenario for implementing the method for determining backup nodes in the global deduplication storage scenario involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the device for determining backup nodes in one or more global deduplication storage scenarios provided below can be referred to the limitations of the method for determining backup nodes in a global deduplication storage scenario above, and will not be repeated here.

[0135] In an exemplary embodiment, as shown in FIG7 , a device for determining a backup node in a global deduplication storage scenario is provided, including:

[0136] The task module 710 is used to determine a plurality of backup tasks; the backup tasks include data to be backed up; each of the backup tasks corresponds to a fingerprint feature vector determined according to the data to be backed up.

[0137] The merging module 720 is used to merge the to-be-backed-up data of the backup tasks that meet the merging condition among the backup tasks according to the feature distances between the fingerprint feature vectors corresponding to the backup tasks, so as to generate multiple new backup tasks.

[0138] Determination module 730 is used to determine the similarity between the new multiple backup tasks and the multiple backup nodes based on the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes; the backup nodes are used to store the data to be backed up and delete duplicate data in the data to be backed up and the cached data.

[0139] The backup module 740 is used to determine, in sequence from the multiple backup nodes, a target backup node for executing the backup task according to the similarity and the performance information of the backup node.

[0140] In one embodiment, the merging module 720 is specifically used to determine the two backup tasks with the smallest characteristic distance based on the characteristic distance between the fingerprint feature vectors corresponding to each of the backup tasks; if the total amount of data to be backed up included in the two backup tasks after merging is less than the preset total amount of data, the two backup tasks are merged to obtain new multiple backup tasks, and the step of determining the two backup tasks with the smallest characteristic distance based on the characteristic distance between the fingerprint feature vectors corresponding to each of the backup tasks is returned until the number of tasks in the new multiple backup tasks is less than the preset number of categories.

[0141] In one of the embodiments, the preset total amount of data is determined based on the total amount of data of the multiple backup tasks and the preset number of categories; the preset number of categories is determined based on the performance information and the number of nodes of the multiple backup nodes.

[0142] In one embodiment, the backup module 740 is used to determine, for any backup task among the new multiple backup tasks, a target backup node for executing the any backup task from the multiple backup nodes according to the similarity between the any backup task and the multiple backup nodes and the performance information of the backup nodes;

[0143] In the case that the target backup node completes executing any one of the backup tasks, the backup tasks other than the any one of the backup tasks are taken as the new multiple backup tasks, and the backup nodes other than the target backup node of the any one of the backup tasks are taken as the new multiple backup nodes, and the step of returning to any one of the new multiple backup tasks, and determining the efficiency parameters of the multiple backup nodes according to the similarity between the any one of the backup tasks and the multiple backup nodes and the performance information of the backup nodes.

[0144] In one embodiment, the backup module 740 is used to obtain the storage cost of the backup node and the efficiency cost of the backup node; the storage cost is determined based on the deduplication savings of the backup node, and the deduplication savings include the amount of data that can be saved by the backup node after deleting duplicate data in the data to be backed up and the cached data; the efficiency cost is determined based on the network performance and backup efficiency of the backup node; the transmission cost of the backup node is determined based on the similarity and the data movement cost of the backup node; the efficiency parameters of the multiple backup nodes are determined based on the storage cost, the efficiency cost and the transmission cost; and the target backup nodes for executing the backup task are determined in turn from the multiple backup nodes based on the efficiency parameters of the multiple backup nodes.

[0145] In one embodiment, the task module 710 is used to determine multiple data blocks to be backed up from the data to be backed up in the backup task; determine multiple hash values ​​corresponding to any of the data blocks to be backed up; use the minimum hash value among the multiple hash values ​​as the characteristic value of any of the data blocks to be backed up; and construct a fingerprint feature vector corresponding to the backup task based on the characteristic values ​​of each of the data blocks to be backed up.

[0146] In one embodiment, the determination module 730 is used to determine the similarity between the new multiple backup tasks and the multiple backup nodes based on the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes, and the method also includes: determining the cached data based on the hot data in the backup node; the hot data includes data in the backup node with an access frequency higher than a frequency threshold; determining multiple node data blocks from the cached data; and determining the fingerprint feature vector of the cached data based on the fingerprints of the multiple node data blocks.

[0147] Each module in the device for determining the backup node in the above-mentioned global deduplication storage scenario can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0148] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store backup data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external backup center server via a network connection. When the computer program is executed by the processor, a method for determining a backup node in a global deduplication storage scenario is implemented.

[0149] Those skilled in the art will understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0152] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0154] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0155] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for determining a backup node in a global deduplication storage scenario, characterized in that: The method comprises: Determine a plurality of backup tasks; the backup tasks include data to be backed up; each of the backup tasks corresponds to a fingerprint feature vector determined according to the data to be backed up; According to the characteristic distances between the fingerprint characteristic vectors corresponding to the backup tasks, the to-be-backed-up data of the backup tasks that meet the merging conditions in the backup tasks are merged to generate multiple new backup tasks; Determining similarities between the new multiple backup tasks and the multiple backup nodes according to the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes; According to the similarity and the performance information of the backup node, a target backup node for executing the backup task is determined in sequence from the multiple backup nodes; the target backup node is used to store the data to be backed up of the backup task and delete duplicate data in the data to be backed up and the cached data.

2. The method according to claim 1, characterized in that The step of merging the to-be-backed-up data of the backup tasks that meet the merging condition in the backup tasks according to the feature distances between the fingerprint feature vectors corresponding to the backup tasks to generate multiple new backup tasks includes: Determine two backup tasks with the smallest characteristic distances according to characteristic distances between fingerprint characteristic vectors corresponding to the backup tasks; If the total amount of data to be backed up after merging the two backup tasks is less than the preset total amount of data, the two backup tasks are merged to obtain new multiple backup tasks, and the step of determining the two backup tasks with the smallest feature distance based on the feature distance between the fingerprint feature vectors corresponding to each of the backup tasks is returned until the number of tasks in the new multiple backup tasks is less than the preset number of categories.

3. The method according to claim 2, characterized in that The preset total amount of data is determined according to the total amount of data of the multiple backup tasks and the preset number of categories; the preset number of categories is determined according to the performance information and the number of nodes of the multiple backup nodes.

4. The method according to claim 1, characterized in that The step of sequentially determining a target backup node for executing the backup task from the plurality of backup nodes according to the similarity and the performance information of the backup nodes includes: For any backup task among the new multiple backup tasks, according to the similarity between the any backup task and the multiple backup nodes and the performance information of the backup nodes, determine a target backup node for executing the any backup task from the multiple backup nodes; In the case that the target backup node completes executing any one of the backup tasks, the backup tasks other than the any one of the backup tasks are taken as the new multiple backup tasks, and the backup nodes other than the target backup node of the any one of the backup tasks are taken as the new multiple backup nodes, and the step of returning to any one of the new multiple backup tasks, and determining the efficiency parameters of the multiple backup nodes according to the similarity between the any one of the backup tasks and the multiple backup nodes and the performance information of the backup nodes.

5. The method according to claim 1, characterized in that The step of sequentially determining a target backup node for executing the backup task from the plurality of backup nodes according to the similarity and the performance information of the backup nodes includes: Obtaining the storage cost of the backup node and the efficiency cost of the backup node; the storage cost is determined according to the deduplication saving amount of the backup node, the deduplication saving amount includes the amount of data that can be saved by the backup node after deleting duplicate data in the to-be-backed-up data and the cached data; the efficiency cost is determined according to the network performance and backup efficiency of the backup node; Determining a transmission cost of the backup node according to the similarity and the data movement cost of the backup node; Determining efficiency parameters of the plurality of backup nodes according to the storage cost, the efficiency cost and the transmission cost; According to the efficiency parameters of the multiple backup nodes, a target backup node for executing the backup task is determined in sequence from the multiple backup nodes.

6. The method according to claim 1, characterized in that Before merging the to-be-backed-up data of the backup tasks that meet the merging condition among the backup tasks according to the feature distances between the fingerprint feature vectors corresponding to the backup tasks, the method further includes: Determining a plurality of data blocks to be backed up from the data to be backed up in the backup task; Determine multiple hash values ​​corresponding to any of the data blocks to be backed up; Using the minimum hash value among the multiple hash values ​​as the characteristic value of any of the data blocks to be backed up; According to the characteristic values ​​of each of the data blocks to be backed up, a fingerprint characteristic vector corresponding to the backup task is constructed.

7. The method according to claim 1, characterized in that Before determining the similarity between the new multiple backup tasks and the multiple backup nodes according to the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes, the method further includes: Determine the cache data according to the hot data in the backup node; the hot data includes data in the backup node whose access frequency is higher than a frequency threshold; Determining a plurality of node data blocks from the cache data; A fingerprint feature vector of the cache data is determined according to the fingerprints of the multiple node data blocks.

8. A device for determining a backup node in a global deduplication storage scenario, characterized in that: The device comprises: A task module, used to determine a plurality of backup tasks; the backup tasks include data to be backed up; each of the backup tasks corresponds to a fingerprint feature vector determined according to the data to be backed up; A merging module, configured to merge the to-be-backed-up data of the backup tasks that meet the merging conditions among the backup tasks according to the feature distances between the fingerprint feature vectors corresponding to the backup tasks, and generate multiple new backup tasks; A determination module, used to determine the similarity between the new multiple backup tasks and the multiple backup nodes according to the fingerprint feature vectors corresponding to the new multiple backup tasks and the fingerprint feature vectors of the cached data in the multiple backup nodes; the backup nodes are used to store the data to be backed up and delete duplicate data in the data to be backed up and the cached data; The backup module is used to determine the target backup node for executing the backup task from the multiple backup nodes in sequence according to the similarity and the performance information of the backup node.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images