Data processing method, device, server and storage medium

Abstract

The present application discloses a data processing method, device, server and storage medium, the data processing method comprising: determining a first parameter of each data block in at least one data block; assigning a first index table to each data block in the at least one data block according to the corresponding first parameter and according to the first number of at least two first index tables created in advance; according to the first fingerprint value of each data block in the at least one data block and the corresponding assigned first index table, if the first fingerprint value of the data block does not exist in the corresponding assigned first index table, storing the data block in the first server. The scheme provided in the embodiment of the present application realizes data block fingerprint value processing through parallel indexing, evenly assigns data blocks to index tables according to a specified algorithm, makes the process of assigning data blocks to index tables random, and thus speeds up the data deduplication rate.

Description

Technical Field

[0001] The present application relates to the field of computer storage technology, and in particular to a data processing method, device, server and storage medium. Background Art

[0002] With the development of informatization, the amount of data is growing rapidly. How to effectively protect and store data is particularly important. In order to ensure data security or archive and protect infrequently used data, it is often necessary to combine backup software for data protection. The amount of data growth is getting bigger and bigger, and there are more and more duplicate data. Whether it is backing up or archiving data, the excessive amount of data that needs to be transmitted will cause excessive pressure on network bandwidth and long backup time, and also cause a huge waste of storage space. Therefore, many backup software provide data deduplication functions to reduce the amount of data transmission or storage. However, deduplication involves data segmentation, fingerprint calculation, fingerprint duplication detection and other processes. The execution rate of the whole process is slow, which seriously affects the backup rate of backup software. Generally speaking, data deduplication can be divided into file deduplication, fixed-length block deduplication, and variable-length block deduplication. In terms of execution rate, fixed-length block deduplication is lower than file deduplication, but higher than variable-length block deduplication. The deduplication rate that reflects the proportion of duplicate data is exactly the opposite of the execution rate. How to improve the execution rate of deduplication without affecting the deduplication rate of data is a major problem.

[0003] In the related technology, parallel processing can solve the I / O access bottleneck of the traditional single-thread disk index table to a certain extent. In the process of parallel deduplication, the global fingerprint index table can be accessed and modified by multiple threads to ensure uniqueness, and a lock synchronization mechanism can be used. However, the global data block index table under parallel deduplication consumes a lot of CPU resources and I / O resources to ensure consistency, affecting the normal access of other services, and the data deduplication rate is not high. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a data processing method, device, server and storage medium to at least solve the problem in the related technology that parallel deduplication affects the normal access of other services and the data deduplication rate is not high, reduce resource consumption in the parallel deduplication process, and improve the data deduplication rate.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data processing method, the method comprising:

[0007] Determine a first parameter of each data block in at least one data block; the first parameter is obtained by calculating the sum of ASCII values ​​of at least two characters in a first fingerprint value of the data block; the first fingerprint value is obtained by a first hash algorithm;

[0008] According to the corresponding first parameter and according to the first number of at least two pre-created first index tables, a first index table is allocated to each data block in the at least one data block; the first index table is used to record the first index and the corresponding first fingerprint value corresponding to the data block stored in the first server;

[0009] According to the first fingerprint value of each data block in the at least one data block and the corresponding allocated first index table, if the first fingerprint value of the data block does not exist in the corresponding allocated first index table, the data block is stored in the first server.

[0010] Wherein, in the above scheme, the method further comprises:

[0011] In the case where the first fingerprint value of the data block already exists in the corresponding allocated first index table, judging whether the corresponding data block is stored in the first server based on the Bloom filter, and obtaining a corresponding judgment result;

[0012] When the corresponding judgment result indicates that the corresponding data block is not backed up in the first server, the data block is stored in the first server.

[0013] In the above scheme, the method further comprises:

[0014] If the corresponding judgment result cannot indicate that the corresponding data block is not backed up in the first server, according to the second fingerprint value and the second index table of the corresponding data block, if the second fingerprint value of the data block does not exist in the second index table, the data block is stored in the first server; wherein,

[0015] The second fingerprint value is obtained by a second hash algorithm; the second index table is used to record the second index and the corresponding second fingerprint value corresponding to the data block stored in the first server.

[0016] In the above solution, the first hash algorithm represents a weak hash algorithm; the second hash algorithm represents a strong hash algorithm.

[0017] In the above scheme, the method further comprises:

[0018] If the second fingerprint value of the data block already exists in the second index table, the data block is not stored in the first server.

[0019] In the above solution, when determining the first parameter of each data block in the at least one data block, the method includes:

[0020] The first parameter of the data block is obtained by calculating the sum of the ASCII values ​​of the first character and the last character of the first fingerprint value of the data block.

[0021] In the above scheme, the method further comprises:

[0022] Backing up the data blocks stored in the first server.

[0023] In the above solution, before determining the first parameter of each data block in the at least one data block, the method further includes:

[0024] Deduplication processing is performed on the at least one data block according to the data stored in the first server.

[0025] In the above solution, before determining the first parameter of each data block in the at least one data block, the method further includes:

[0026] assigning a thread to each data block in the at least one data block;

[0027] When the second number of allocated threads is greater than the first number, a synchronization lock is allocated to the at least two first index tables.

[0028] The present application also provides a data processing device, including:

[0029] A first determining unit, configured to determine a first parameter of each data block in at least one data block; the first parameter is obtained by calculating the sum of ASCII values ​​of at least two characters in a first fingerprint value of the data block; the first fingerprint value is obtained by a first hash algorithm;

[0030] A first allocation unit is configured to allocate a first index table to each data block in the at least one data block according to a corresponding first parameter and a first number of at least two pre-created first index tables; the first index table is configured to record a first index and a first fingerprint value corresponding to the data block stored in the first server;

[0031] The first storage unit is used to store the data block in the first server according to the first fingerprint value of each data block in the at least one data block and the corresponding first index table if the first fingerprint value of the data block does not exist in the corresponding first index table.

[0032] The embodiment of the present application further provides a server, comprising: a processor and a memory for storing a computer program that can be run on the processor,

[0033] Wherein, the processor is used to execute the steps of the above-mentioned data processing method when running the computer program.

[0034] An embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0035] In an embodiment of the present application, a first fingerprint value is obtained by a first hash algorithm, the sum of the ASCII values ​​of at least two characters in the first fingerprint value of a data block is calculated, and a first parameter of each data block in at least one data block is obtained. A first index table is allocated to each data block according to the corresponding first parameter and the first number of at least two first index tables created in advance. The first index table is used to record the first index and the corresponding first fingerprint value corresponding to the data block stored in the first server. According to the first fingerprint value of each data block and the corresponding allocated first index table, if the first fingerprint value of the data block does not exist in the corresponding allocated first index table, the data block is stored in the first server. In this way, data block fingerprint value processing is realized through parallel indexing, and data blocks are evenly allocated to index tables according to a specified algorithm, so that the process of allocating index tables to data blocks is randomized, thereby achieving the effect of accelerating the rate of data deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0037] Figure 2 A flowchart of a data processing method provided for an application embodiment of the present application;

[0038] Figure 3 A flowchart of another data processing method provided for an application embodiment of the present application;

[0039] Figure 4 A flowchart of another data processing method provided for an application embodiment of the present application;

[0040] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the related art, the global data block index table under parallel deduplication consumes a large amount of CPU resources and I / O resources to ensure consistency, affecting the normal access of other services, and the data deduplication rate is not high.

[0043] Based on this, in various embodiments of the present application, a first fingerprint value is obtained by a first hash algorithm, the sum of the ASCII values ​​of at least two characters in the first fingerprint value of the data block is calculated to obtain a first parameter of each data block in at least one data block, and a first index table is assigned to each data block according to the corresponding first parameter and the first number of at least two pre-created first index tables. The first index table is used to record the first index and the corresponding first fingerprint value corresponding to the data block stored in the first server. According to the first fingerprint value of each data block and the corresponding assigned first index table, if the first fingerprint value of the data block does not exist in the corresponding assigned first index table, the data block is stored in the first server.

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] Figure 1 The following is a schematic diagram of the implementation flow of the data processing method provided in the embodiment of the present application. Figure 1 The data processing method shown includes:

[0046] Step 101: Determine a first parameter of each data block in at least one data block.

[0047] The first parameter is obtained by calculating the sum of ASCII values ​​of at least two characters in a first fingerprint value of the data block; and the first fingerprint value is obtained by a first hash algorithm.

[0048] The file to be backed up is divided into blocks to obtain at least one data block, a first fingerprint value of each data block in the at least one data block is calculated using a first hash algorithm, characters of at least two set positions in the first fingerprint value calculated by the first hash algorithm are extracted, and the sum of the ASCII values ​​of the extracted characters is calculated, thereby determining a first parameter of each data block. Here, the first hash algorithm can be a strong hash algorithm or a weak hash algorithm; the characters include numbers or letters; in actual applications, the set positions can be the first and last positions, the first two positions, or the fourth to sixth positions, and the set positions for character extraction are set as needed.

[0049] In one embodiment, when determining the first parameter of each data block in at least one data block, the method includes:

[0050] The first parameter of the data block is obtained by calculating the sum of the ASCII values ​​of the first character and the last character of the first fingerprint value of the data block.

[0051] Here, a first fingerprint value of each data block in at least one data block is calculated using a first hash algorithm, the first and last characters of the first fingerprint value are extracted, the sum of the ASCII values ​​of the extracted characters is calculated, and the first parameter of each data block is determined. Using parallel indexing can reduce the consumption of machine resources for backup.

[0052] In one embodiment, before determining the first parameter of each data block in the at least one data block, the method further includes:

[0053] Deduplication processing is performed on the at least one data block according to the data stored in the first server.

[0054] Before determining the first parameter of each data block in the at least one data block, the at least one data block is deduplicated at the source end according to the data stored in the first server, where the first server stores the backed-up data. Because the data that has been backed up in the first server no longer needs to be backed up, by deduplicating the data at the source end, the amount of data to be deduplicated and backed up can be reduced, thereby improving the deduplication execution rate and the backup rate.

[0055] In one embodiment, before determining the first parameter of each data block in the at least one data block, the method further includes:

[0056] assigning a thread to each data block in the at least one data block;

[0057] When the second number of allocated threads is greater than the first number, a synchronization lock is allocated to the at least two first index tables.

[0058] Before determining the first parameter of each data block in at least one data block, a thread is assigned to each data block. When the second number of assigned threads is greater than the first number of the first index table, more than one thread may share a first index table at the same time, resulting in a write synchronization problem. By assigning a synchronization lock to each first index table, uniqueness can be ensured under multi-threaded access and modification.

[0059] Step 102: Allocate a first index table to each data block in the at least one data block according to the corresponding first parameter and according to the first number of at least two pre-created first index tables.

[0060] If two data blocks are identical, the first fingerprint values ​​obtained by the first hash algorithm are identical, and thus the first parameters calculated by extracting the set bit characters are also identical. In this way, when each data block is subsequently assigned a first index table, identical data blocks will be assigned to the same first index table, enabling all data blocks to be evenly assigned the first index table. Here, the first index table is used to record the first index and the corresponding first fingerprint value corresponding to the data block stored in the first server.

[0061] According to a first parameter λ of each data block in at least one data block and a first number n of at least two pre-created first index tables, a first index table is assigned to each data block by taking a remainder ξ=λmodn of the first parameter with the number of first index tables, wherein the first index corresponding to the data block stored in the first server and the corresponding first fingerprint value are recorded in the first index table.

[0062] For example, five first index tables are created in advance, and the first parameters of the six data blocks are 11, 15, 6, 38, 2, and 29 respectively, and the remainders are 1, 0, 1, 3, 2, and 4 respectively, and the second, first, second, fourth, third, and fifth first index tables are allocated accordingly. Since the results of calculating the ASCII values ​​of characters with the same set bits for the same data block are the same, when allocating the first index table, the idea of ​​distributed node balancing is applied to the parallel processing module of deduplication. The same data blocks will be divided into the same first index table, and different data blocks can be evenly allocated to different first index tables according to the specified algorithm, so that parallel deduplication of data blocks can be fully realized, and each sub-index table does not interfere with each other, which can achieve a certain load balance and speed up the data deduplication rate.

[0063] Step 103: based on the first fingerprint value of each data block in the at least one data block and the corresponding first index table, if the first fingerprint value of the data block does not exist in the corresponding first index table, store the data block in the first server.

[0064] The data block is stored in the first server only when the first fingerprint value of the data block does not exist in the data block corresponding to the first index table. That is, the data block stored in the first server is a data block without duplicate data blocks, and the first index table records the first index and the corresponding first fingerprint value corresponding to the data block assigned to the first index table. When the first fingerprint value of the data block does not exist in the corresponding first index table, it means that the data block is different from the data block assigned to the first index table, and the data block is stored in the first server for subsequent backup.

[0065] In one embodiment, the method further comprises:

[0066] In the case where the first fingerprint value of the data block already exists in the corresponding allocated first index table, judging whether the corresponding data block is stored in the first server based on the Bloom filter, and obtaining a corresponding judgment result;

[0067] When the corresponding judgment result indicates that the corresponding data block is not backed up in the first server, the data block is stored in the first server.

[0068] Bloom filter is an algorithm based on hash table. When the first fingerprint value of a data block already exists in the corresponding first index table, it is screened by Bloom filter to determine whether the data block is a non-repeating data block. It can quickly determine most of the elements that do not exist in the set and obtain the corresponding judgment result. If the judgment result is that the data block is non-repeating, the data block is stored in the first server. In practical applications, at least one third hash algorithm is used to map the bit data group. When it is found that all the mapping bits of the data block are not all 1, the data block can be judged as a non-repeating data block and saved. Here, the third hash algorithm can be a weak hash algorithm or a strong hash algorithm. In this way, using Bloom filter, it is possible to quickly determine that the data block does not exist in the fingerprint index table, thereby speeding up the retrieval rate.

[0069] Considering that data deduplication involves data security, by identifying the fingerprint of the data block, if the fingerprints of two data blocks are the same, the data block is considered to be duplicated and not retained. When restoring the data, the first fingerprint value of each data block is found according to the metadata table of the file, and then the data block is restored according to the data block fingerprint. If two different data blocks collide, that is, the data blocks collide, and are considered to be the same data block, data loss and rewriting will occur, resulting in distortion of data information. In one embodiment, the method further includes:

[0070] If the corresponding judgment result cannot indicate that the corresponding data block is not backed up in the first server, according to the second fingerprint value of the corresponding data block and the second index table, if the second fingerprint value of the data block does not exist in the second index table, storing the data block in the first server;

[0071] The second fingerprint value is obtained by a second hash algorithm; the second index table is used to record the second index and the corresponding second fingerprint value corresponding to the data block stored in the first server.

[0072] When the Bloom filter cannot give a conclusion that the data block is not repeated, a second hash algorithm is used to calculate the second fingerprint value of each data block in at least one data block, and at least two characters of the set position in the second fingerprint value calculated by the second hash algorithm are extracted, and the sum of the ASCII values ​​of the extracted characters is calculated, thereby determining the second parameter of each data block. Here, the second hash algorithm can be a strong hash algorithm or a weak hash algorithm; the characters include numbers or letters; in actual applications, the set position can be the first and last positions, the first two positions, or the fourth to sixth positions, and the set position of the character extraction is set as needed.

[0073] In this embodiment, if two data blocks are identical, the second fingerprint values ​​obtained by the second hash algorithm are identical, and thus the second parameters for extracting the set bit character calculation are also identical. In this way, when a second index table is subsequently assigned to each data block, the same data block will be assigned to the same second index table, so that all data blocks can be evenly assigned to the second index table. The second index table records the second index and the corresponding second fingerprint value corresponding to the data block to which the second index table has been assigned. When the second fingerprint value of a data block does not exist in the second index table assigned according to the specified algorithm, it means that the data block is different from the data block to which the second index table has been assigned, and the data block is stored in the first server for subsequent backup. The combined use of multiple Hash algorithm combinations can better protect data consistency, reduce the probability of accidental data deletion due to hash conflicts, and improve the reliability and security of data deduplication.

[0074] In one embodiment, the method further comprises:

[0075] If the second fingerprint value of the data block already exists in the second index table, the data block is not stored in the first server.

[0076] When the Bloom filter cannot give a conclusion that the data block is not repeated, a second hash algorithm is used to calculate the second fingerprint value of each data block in at least one data block. The second parameter of each data block can be determined by extracting at least two set characters in the second fingerprint value calculated by the second hash algorithm and calculating the sum of the ASCII values ​​of the extracted characters. When allocating the second index table, the same data block will be assigned to the same second index table, which can achieve balanced distribution of the second index table for all data blocks. The second index table records the second index and the corresponding second fingerprint value of the deduplicated data block. When the second fingerprint value of the data block exists in the second index table allocated according to the specified algorithm, it means that the data block is the same as the data block that has been deduplicated, and the data block is abandoned. Only the index and metadata are recorded. By combining the use of hash algorithms, the effect of improving the data deduplication rate can be achieved.

[0077] Considering that data deduplication involves data security, mainly including data block collision, it is through the identification of data block fingerprints. If the fingerprints of two data blocks are the same, the data block is considered to be duplicated and will not be retained. When restoring the data, the corresponding fingerprint of each data block is found according to the metadata table of the file, and then the data block is restored according to the data block fingerprint. If two different data blocks collide and are considered to be the same data block, data loss and rewriting will occur, resulting in data information distortion.

[0078] In one embodiment, the first hash algorithm represents a weak hash algorithm; and the second hash algorithm represents a strong hash algorithm.

[0079] Here, when allocating the first index table, a weak hash algorithm is used to quickly and evenly allocate different data blocks to different deduplication channels, and then a strong hash algorithm is used to double-judge whether the data block is a duplicate data block. At the same time, by combining strong and weak hashes, the probability of accidental data deletion due to hash conflicts is reduced, thereby improving the reliability and security of data deduplication.

[0080] In one embodiment, the method further comprises:

[0081] Backing up the data blocks stored in the first server.

[0082] Here, the data blocks stored in the first server are backed up. The data can be backed up to other storage disks of the first server or to the storage disk of the second server, which plays a role of backup after data processing and can also be used for source-side deduplication when data processing is performed next time.

[0083] The present application will be further described in detail below in conjunction with application examples.

[0084] Reference Figure 2 Application examples shown:

[0085] Step 1: The user sets the backup strategy on the front-end page of the backup software management platform to trigger the backup command.

[0086] Step 2: The backup software management platform sends a backup instruction to make the backup module function effective.

[0087] Step 3: Send backup / reverse proxy instructions to the media management server according to user settings, and send deduplication instructions to the deduplication server at the same time.

[0088] Step 4: The media management server receives the backup / reverse proxy instruction and allocates a storage volume of a corresponding size according to the allocation parameters or default parameters in the instruction.

[0089] Step 5: After receiving the deduplication instruction, the deduplication server clones the deduplication pool snapshot.

[0090] Step 6: The media management server maps the storage volume created in step 4 to the client, completing the reverse proxy of the storage volume of the media management server, so that the client can access the proxied storage volume.

[0091] Step 7: The deduplication server maps the deduplication pool snapshot cloned in step 5 to the client, completing the reverse proxy of the deduplication pool, so that the client can access the proxied deduplication pool.

[0092] Step 8: The client receives the backup instruction sent by the backup software management platform, formats the storage volume of the media management server reverse proxy, and mounts the storage volume so that the client can read and write to the storage volume; and executes a direct mount command on the mapped deduplication pool so that the client has the ability to read and write to the deduplication pool snapshot.

[0093] Step 9: While the client is formatting the storage volume of the reverse proxy of the media management server, the client uses the deduplication pool from the proxy to perform source-side deduplication on the backup data source.

[0094] Step 10: The client implements the parallel index design module according to a certain number of set threads. In the parallel deduplication channel, the data blocks are evenly distributed to the respective threads and the corresponding sub-indexes for deduplication.

[0095] Step 11: Back up the deduplicated data and save it to the storage volume. Unmount the deduplication pool from the reverse proxy and cancel the mapping.

[0096] Step 12: After the backup is complete, unmount and unmap the reverse proxy storage volume.

[0097] In step 10 of the above application embodiment, the client implements the parallel index design module according to the set number of threads, such as Figure 3 As shown, it includes the following steps:

[0098] Step 10A-1: Divide the backup file into blocks of indefinite length and enable multi-threading, that is, the following steps are processed in parallel;

[0099] Step 10A-2: Calculate the fingerprint value str of the divided data block using a weak hash algorithm;

[0100] Step 10A-3: extract the first digit or letter α and the last digit or letter β of the fingerprint value str calculated by the weak hash algorithm; if the two data blocks are the same, the calculated λ is also the same;

[0101] Step 10A-4: Take the ASCII values ​​of α and β, and add them to obtain λ=ascii(α)+ascii(β);

[0102] Step 10A-5: Assuming that the number of threads designed is n, and the number of parallel index tables is also n, mapping is performed in n parallel index tables through ξ=λmodn to achieve the same data block being assigned to the same sub-index table. The randomness of ξ can achieve balanced distribution of sub-indexes for all data blocks.

[0103] In step 10 of the above application embodiment, the client implements the parallel index design module according to the set number of threads, and evenly distributes the data blocks to the respective threads and the corresponding sub-indexes for deduplication in the parallel deduplication channel, such as Figure 4 As shown, the specific steps are as follows:

[0104] Step 10B-1: Allocate sub-index tables: According to the parallel index design module process of the above steps 10A-1 to 10A-5, allocate sub-index tables for the data blocks under each thread. According to the analysis of the parallel index design module, the same data block will be allocated the same sub-index table, and different data blocks can be evenly allocated different sub-index tables. Among them, if the data block fingerprints under m threads (m≤n, n is the total number of threads) are hashed to the same sub-index table, then these m threads share one sub-index table, and there will be a write synchronization problem. At this time, a synchronization lock needs to be allocated to each individual sub-index table; if the data block fingerprints under m threads are hashed to m different sub-index tables, then the m sub-index tables are read and written independently of each other, and no write synchronization lock is required between different sub-indexes;

[0105] Step 10B-2: Preliminary duplication judgment of weak hash fingerprint: Perform preliminary duplication judgment based on the weak hash fingerprint used in the sub-index table allocated in step 10B-1. If it is determined that the fingerprint does not exist, the data block is considered to be non-duplicate and needs to be saved, and the index and metadata are recorded. Then, the data backup function is executed, and the backup software completes the subsequent backup operation. If it is determined that the fingerprint exists, it is impossible to determine whether the data block is duplicated and it is necessary to proceed to step 10B-3.

[0106] Step 10B-3: The Bloom filter determines whether the data is non-duplicate data: 3 different weak hash algorithms are used to map the bit data group: When it is found that all the mapping bits of the data block are not all 1, according to the characteristics of the Bloom filter, it can be determined that the element does not exist, and the data block can be determined to be a non-duplicate data block. The data block needs to be saved, and the index and metadata are recorded, and the data backup function is performed, and the data deduplication phase ends; when it is found that all the mapping bits of the data block are 1, it is impossible to determine whether the data block is duplicate data, and step 10B-4 needs to be executed.

[0107] Step 10B-4: Strong hash double judgment: When the data cannot be judged after filtering by the Bloom filter in step 10B-3, a double judgment will be performed in this step: first, the fingerprint of the data block is calculated using the strong hash algorithm, and the sub-index table is retrieved. If the calculated fingerprint can be retrieved in the sub-index table, the data block is considered to be duplicated, so the data block is abandoned and only the index and metadata are recorded; if it is determined that the data block is not duplicated, the data block is saved, and the index and metadata are recorded at the same time, entering the data backup stage.

[0108] The application embodiment proposes a data processing method based on parallel indexing, which enables the client program to access the storage volume by reverse proxying the storage volume to the client; parallel indexing is implemented according to a certain number of threads set, and the file data to be backed up by the client is divided into different data blocks, and a parallel sub-index table is allocated, so that the data blocks can be evenly distributed to the respective threads and the corresponding sub-indexes for deduplication; the deduplicated data is further backed up to the storage volume from the reverse proxy, and after all data is backed up, the storage volume is unloaded and unmapped. The method mainly speeds up the data block retrieval speed through parallel indexing, reduces the amount of backup data through data deduplication, and reduces the bandwidth pressure and storage pressure in the entire backup process.

[0109] Combination Figure 2 The structural diagram of the corresponding data processing device includes the following modules:

[0110] 1. Backup software management platform: The business processes corresponding to the software functions are all connected in series through the front-end of the backup software management platform. Normal scheduling configuration, time window, job monitoring, data management and other functions are all completed by the front-end configuration management of the backup software management platform.

[0111] 2. The backup module receives the backup instructions sent by the backup software management platform, sends the backup / reverse proxy instructions to the media management server, and sends the deduplication instructions to the deduplication server.

[0112] 3. Media management server: Unify the management of storage media with different speeds, such as random access memory (RAM), non-volatile random access memory (NV-RAM), NVMe-SSD, PCIe-SSD, SATA / SAS-SSD, SATA / SAS-HDD, etc., and automatically complete the tiering and hot migration operations of data between media with different speeds.

[0113] 4. The deduplication server clones the deduplication pool snapshot. The client can use the deduplication pool sent by the proxy to perform source-side deduplication on the backup data source, and determine the data blocks that need to be deduplicated based on the snapshot that has been backed up in the deduplication pool. Here, the deduplication server and the media management server can be the same server;

[0114] 5. Client: Deployed in the business system host to be backed up, communicates with the backup software management platform and media management server to complete data backup and recovery functions.

[0115] 6. Parallel index design module: This module processes the data block fingerprint value, assigns different data blocks to different sub-index tables according to the specified algorithm, adds the ASCII values ​​of the first and last characters of the data block fingerprint, and modulo the number of threads to allocate the same data block to the same sub-index table (the fingerprints of the same data block are the same, so the ASCII values ​​of the first and last characters are the same, and the values ​​modulo the specified number of threads after addition are also the same). In the same sub-index table, except for the small probability of duplicate judgment errors caused by hash conflicts, it can efficiently determine whether the data block is duplicate data. The function of the sub-index table is equivalent to that of the global index table. In parallel threads, each thread has only one sub-index table and the data block assigned to the thread. The reading and writing of each sub-index table are independent of each other. At the same time, the value modulo the addition of the ASCII value has a large randomness, so different sub-index tables can be evenly assigned to data blocks, increasing the feasibility and efficiency of parallel index tables.

[0116] 7. Single-channel deduplication module: It adopts parallel channel deduplication, which is composed of n (n threads) single-channel deduplication modules. The main function of each single-channel deduplication module is to use the sub-index table to speed up the deduplication fingerprint retrieval time for the data block under the single thread, determine whether the data block is a duplicate data block according to the specified deduplication algorithm, and finally output the non-duplicate data block.

[0117] 8. Data backup: It occurs in the media management server. The deduplicated data is backed up and backed up to the storage volume. The deduplication pool from the reverse proxy is unmounted and unmapped. After the backup is completed, the storage volume of the reverse proxy is unmounted and unmapped.

[0118] To implement the method of the embodiment of the present application, the embodiment of the present application also provides a data processing device, such as Figure 5 As shown, the device comprises:

[0119] A first determining unit 501 is used to determine a first parameter of each data block in at least one data block; the first parameter is obtained by calculating the sum of ASCII values ​​of at least two characters in a first fingerprint value of the data block; the first fingerprint value is obtained by a first hash algorithm;

[0120] A first allocation unit 502 is configured to allocate a first index table to each data block in the at least one data block according to the corresponding first parameter and the first number of at least two pre-created first index tables; the first index table is used to record the first index and the corresponding first fingerprint value corresponding to the data block stored in the first server;

[0121] The first storage unit 503 is used to store the data block in the first server according to the first fingerprint value of each data block in the at least one data block and the corresponding first index table if the first fingerprint value of the data block does not exist in the corresponding first index table.

[0122] In one embodiment, the device further comprises:

[0123] The first judgment unit is used to judge whether the corresponding data block is stored in the first server based on the Bloom filter when the first fingerprint value of the data block already exists in the corresponding allocated first index table, so as to obtain a corresponding judgment result.

[0124] The second storage unit is configured to store the data block in the first server when the corresponding judgment result indicates that the corresponding data block is not backed up in the first server.

[0125] In one embodiment, the apparatus further comprises:

[0126] The third storage unit is used to store the data block in the first server according to the second fingerprint value and the second index table of the corresponding data block when the corresponding judgment result cannot represent that the corresponding data block is not backed up in the first server and when the second fingerprint value of the data block does not exist in the second index table.

[0127] In one embodiment, the first hash algorithm represents a weak hash algorithm; and the second hash algorithm represents a strong hash algorithm.

[0128] In one embodiment, the apparatus further comprises:

[0129] The abandoning unit is used to abandon storing the data block in the first server when the second fingerprint value of the data block already exists in the second index table.

[0130] In one embodiment, the first determining unit 501 is used to:

[0131] The first parameter of the data block is obtained by calculating the sum of the ASCII values ​​of the first character and the last character of the first fingerprint value of the data block.

[0132] In one embodiment, the apparatus further comprises:

[0133] The backup unit is used to back up the data blocks stored in the first server.

[0134] In one embodiment, the apparatus further comprises:

[0135] The deduplication unit is used to perform deduplication processing on the at least one data block according to the data stored in the first server.

[0136] In one embodiment, the apparatus further comprises:

[0137] The second allocation unit is configured to allocate a thread to each data block in the at least one data block.

[0138] The third allocation unit is configured to allocate synchronization locks to the at least two first index tables when a second number of allocated threads is greater than the first number.

[0139] In actual application, the first determination unit 501, the first allocation unit 502, the first judgment unit, the abandonment unit, the second allocation unit, and the third allocation unit can be implemented by a processor based on a data processing device, and the first storage unit 503, the second storage unit, the third storage unit, the backup unit, and the deduplication unit can be implemented by a processor based on a data processing device combined with a communication interface.

[0140] It should be noted that: when the data processing device provided in the above embodiment performs data processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0141] Based on the hardware implementation of the above program modules, and in order to implement the data processing method of the embodiment of the present application, the embodiment of the present application also provides a server, such as Figure 6 As shown, the server 600 includes:

[0142] Communication interface 610, capable of exchanging information with other devices such as network devices;

[0143] The processor 620 is connected to the communication interface 610 to implement information exchange with other devices and is used to execute the method provided by one or more technical solutions when running a computer program. The computer program is stored in the memory 630.

[0144] Specifically, the processor 620 is configured to determine a first parameter of each data block in at least one data block;

[0145] The first parameter is obtained by calculating the sum of ASCII values ​​of at least two characters in a first fingerprint value of the data block; and the first fingerprint value is obtained by a first hash algorithm.

[0146] The processor 620 is further configured to allocate a first index table to each data block in the at least one data block according to the corresponding first parameter and according to a first number of at least two pre-created first index tables;

[0147] The first index table is used to record the first index and the corresponding first fingerprint value corresponding to the data block stored in the first server.

[0148] The communication interface 610 is used to determine, when the processor 620 determines based on the first fingerprint value of each data block in the at least one data block and the corresponding first index table, that the data block is stored in the first server if the first fingerprint value of the data block does not exist in the corresponding first index table;

[0149] In one embodiment, the processor 620 is further used to determine whether the corresponding data block is stored in the first server based on the Bloom filter to obtain a corresponding judgment result when the first fingerprint value of the data block already exists in the corresponding allocated first index table.

[0150] The communication interface 610 is further configured to store the data block in the first server when the processor 620 determines, based on the corresponding judgment result representation, that the corresponding data block is not backed up in the first server.

[0151] In one embodiment, the communication interface 610 is also used to store the data block in the first server when the processor 620 determines, based on the corresponding judgment result, that it cannot represent that the corresponding data block is not backed up in the first server, and when the processor 620 determines, based on the second fingerprint value and second index table of the corresponding data block, that the second fingerprint value of the data block does not exist in the second index table.

[0152] In one embodiment, the first hash algorithm represents a weak hash algorithm; and the second hash algorithm represents a strong hash algorithm.

[0153] In one embodiment, the processor 620 is further configured to abandon storing the data block in the first server when it is determined that the second fingerprint value of the data block already exists in the second index table.

[0154] In one embodiment, the processor 620 is further configured to obtain a first parameter of the data block by calculating a sum of ASCII values ​​of a first character and a last character of a first fingerprint value of the data block.

[0155] In one embodiment, the processor 620 is further configured to back up the data blocks stored in the first server.

[0156] In one embodiment, the processor 620 is further configured to perform deduplication processing on the at least one data block according to data stored in the first server.

[0157] In one embodiment, the processor 620 is further configured to allocate a thread to each data block in the at least one data block.

[0158] The processor 620 is further configured to allocate synchronization locks to the at least two first index tables when the second number of allocated threads is greater than the first number.

[0159] Of course, in actual application, the various components in the server 600 are coupled together through the bus system 640. It can be understood that the bus system 640 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 640 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 640 .

[0160] The memory 630 in the embodiment of the present application is used to store various types of data to support the operation of the server 600. Examples of such data include: any computer program used to operate on the server 600.

[0161] It can be understood that the memory 630 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a RAM, which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory 630 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.

[0162] The method disclosed in the above embodiment of the present application can be applied to the processor 620, or implemented by the processor 620. The processor 620 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 620 or the instruction in the form of software. The above processor 620 can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 620 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 630, and the processor 620 reads the program in the memory 630 and completes the steps of the above method in combination with its hardware.

[0163] Optionally, when the processor 620 executes the program, it implements the corresponding processes implemented by the server in each method of the embodiments of the present application, which will not be repeated here for the sake of brevity.

[0164] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 630 storing a computer program, and the computer program can be executed by a processor 620 of an electronic device to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0165] In the several embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0166] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0167] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0168] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.

[0169] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0170] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict. Unless otherwise specified and limited, the term "connection" should be understood in a broad sense, for example, it can be an electrical connection, or it can be the internal connection of two components, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0171] In addition, in the examples of the present application, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects distinguished by "first\second\third" can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0172] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0173] The various specific technical features in the various embodiments described in the specific implementation methods can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this application will not be described separately.

Claims

1. A data processing method, characterized in that: The method comprises: Determine a first parameter of each data block in at least one data block; the first parameter is obtained by calculating the sum of ASCII values ​​of at least two characters in a first fingerprint value of the data block; the first fingerprint value is obtained by a first hash algorithm; According to the corresponding first parameter and the first number of at least two pre-created first index tables, a first index table is allocated to each data block in the at least one data block; wherein, according to the first parameter of each data block in the at least one data block and the first number of at least two pre-created first index tables, a first index table is allocated to each data block by taking the remainder of the first parameter by the number of first index tables; the first index table is used to record the first index and the corresponding first fingerprint value corresponding to the data block stored in the first server; According to the first fingerprint value of each data block in the at least one data block and the corresponding allocated first index table, if the first fingerprint value of the data block does not exist in the corresponding allocated first index table, the data block is stored in the first server.

2. The data processing method according to claim 1, characterized in that: The method further comprises: In the case where the first fingerprint value of the data block already exists in the corresponding allocated first index table, judging whether the corresponding data block is stored in the first server based on the Bloom filter, and obtaining a corresponding judgment result; When the corresponding judgment result indicates that the corresponding data block is not backed up in the first server, the data block is stored in the first server.

3. The data processing method according to claim 2, characterized in that: The method further comprises: If the corresponding judgment result cannot indicate that the corresponding data block is not backed up in the first server, according to the second fingerprint value and the second index table of the corresponding data block, if the second fingerprint value of the data block does not exist in the second index table, the data block is stored in the first server; wherein, The second fingerprint value is obtained by a second hash algorithm; the second index table is used to record the second index and the corresponding second fingerprint value corresponding to the data block stored in the first server.

4. The data processing method according to claim 3, characterized in that: The first hash algorithm represents a weak hash algorithm; the second hash algorithm represents a strong hash algorithm.

5. The data processing method according to claim 3, characterized in that: The method further comprises: If the second fingerprint value of the data block already exists in the second index table, the data block is not stored in the first server.

6. The data processing method according to claim 1, characterized in that: When determining the first parameter of each data block in at least one data block, the method comprises: The first parameter of the data block is obtained by calculating the sum of the ASCII values ​​of the first character and the last character of the first fingerprint value of the data block.

7. The data processing method according to claim 1, characterized in that: The method further comprises: Backing up the data blocks stored in the first server.

8. The data processing method according to claim 1, characterized in that: Before determining the first parameter of each data block in the at least one data block, the method further includes: Deduplication processing is performed on the at least one data block according to the data stored in the first server.

9. The data processing method according to claim 1, characterized in that: Before determining the first parameter of each data block in the at least one data block, the method further includes: assigning a thread to each data block in the at least one data block; When the second number of allocated threads is greater than the first number, a synchronization lock is allocated to the at least two first index tables.

10. A data processing device, characterized in that: include: A first determining unit, configured to determine a first parameter of each data block in at least one data block; The first parameter is obtained by calculating the sum of ASCII values ​​of at least two characters in a first fingerprint value of the data block; the first fingerprint value is obtained by a first hash algorithm; A first allocation unit is used to allocate a first index table to each data block in the at least one data block according to the corresponding first parameter and the first number of at least two first index tables created in advance; wherein, according to the first parameter of each data block in the at least one data block and the first number of at least two first index tables created in advance, a first index table is allocated to each data block by taking the remainder of the first parameter by the number of first index tables; the first index table is used to record the first index and the corresponding first fingerprint value corresponding to the data block stored in the first server; The first storage unit is used to store the data block in the first server according to the first fingerprint value of each data block in the at least one data block and the corresponding first index table if the first fingerprint value of the data block does not exist in the corresponding first index table.

11. A server, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the data processing method described in any one of claims 1 to 9.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 9 are implemented.

Images