A data management method based on data migration and related equipment

Abstract

Embodiments of the present application provide a data management method based on data migration and related equipment, which is used to arrange data blocks with relevance to improve migration efficiency as much as possible. The method of the embodiments of the present application comprises: obtaining a first to-be-managed file and a second to-be-managed file; wherein the first to-be-managed file comprises first to-be-classified data blocks corresponding to different categories, and the second to-be-managed file comprises second to-be-classified data blocks corresponding to different categories; determining a first label classification result of the first to-be-classified data blocks in the first to-be-managed file; determining a second label classification result of the second to-be-classified data blocks in the second to-be-managed file based on the first label classification result; and clustering and arranging the first to-be-classified data blocks and the second to-be-classified data blocks according to the second label classification result, to determine target data blocks with relevance, so as to migrate and store the target data blocks.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data processing, and in particular to a data management method based on data migration and related equipment. BACKGROUND

[0002] Data fragmentation has always been a prominent problem in deduplication data storage. Data fragmentation causes different file sharing repeated data blocks, resulting in the mismatch between the logical data block order of the file and the physical data block order on the storage device, which seriously affects the performance of data recovery.

[0003] There are mainly two ways to handle the data fragmentation problem in the deduplication system at present: one is data block rewriting, giving up the elimination of part of the repeated data blocks to maintain the physical locality of these data blocks, for example, a kind of operation Capping limiting or limiting a certain value or value range. When Capping writes a new file, the use frequency of each data block is recorded, and the data block with low use frequency is given up and re-deleted; the other is a data block organization method based on the life cycle, which takes the access situation (i.e. the life cycle) of each version of the data block in the backup system as its characteristic, and classifies and manages the data block according to the characteristic, such as the MFdedup system of storage layout and evolution algorithm, which can record the continuous life cycle of each data block and classify and manage the data block according to the continuous life cycle of the data block. SUMMARY

[0004] Embodiments of the present application provide a data management method based on data migration and related equipment for arranging data blocks with relevance to improve migration efficiency as much as possible.

[0005] The first aspect of the embodiments of the present application provides a data management method based on data migration, comprising:

[0006] Obtaining a first to-be-managed file and a second to-be-managed file; wherein the first to-be-managed file includes first to-be-classified data blocks corresponding to different categories, and the second to-be-managed file includes second to-be-classified data blocks corresponding to different categories;

[0007] Determine the first label classification result of the first to-be-classified data block in the first to-be-managed file; based on the first label classification result, determine the second label classification result of the second to-be-classified data block in the second to-be-managed file;

[0008] According to the second label classification result, the first to-be-classified data block and the second to-be-classified data block are clustered and arranged, and the target data block with relevance is determined, so as to migrate and store the target data block.

[0009] Optionally, the determining the first label classification result of the first to-be-classified data block in the first to-be-managed file comprises:

[0010] setting a classification root node;

[0011] setting a first classification sub-node according to the classification root node and the first to-be-managed file; wherein the first classification sub-node comprises a first referenced sub-node and a first unreferenced sub-node;

[0012] determining a first label reference result of the first to-be-classified data block in the first to-be-managed file, and determining a first classification result corresponding to the first referenced sub-node according to the first label reference result;

[0013] determining the first label classification result according to the first classification result.

[0014] Optionally, the determining the second label classification result of the second to-be-classified data block in the second to-be-managed file based on the first label classification result comprises:

[0015] setting a second classification sub-node under the first referenced sub-node and the first unreferenced sub-node according to the first classification sub-node and the second to-be-managed file; wherein the second classification sub-node comprises a second referenced sub-node and a second unreferenced sub-node;

[0016] determining a second label reference result of the first to-be-classified data block and the second to-be-classified data block in the second to-be-managed file, and determining a second classification result corresponding to the second referenced sub-node and the second unreferenced sub-node respectively according to the second label reference result;

[0017] determining the second label classification result according to the second classification result.

[0018] Optionally, the method further comprises:

[0019] obtaining a third to-be-managed file; wherein the third to-be-managed file comprises third to-be-classified data blocks corresponding to different categories;

[0020] judging whether the number of data blocks corresponding to the second label classification result in the third to-be-classified data blocks exceeds a classification node threshold value;

[0021] when the number of data blocks exceeds the classification node threshold value, maintaining the second referenced sub-node or the second unreferenced sub-node corresponding to the second classification sub-node, and determining a third classification sub-result;

[0022] Or, when the number of data blocks does not exceed the classification node threshold, a third classification sub-node is set under the second reference sub-node and the second unreferenced sub-node according to the second classification sub-node and the third to-be-managed file; wherein the third classification sub-node includes a third reference sub-node and a third unreferenced sub-node; a third label reference result of the first to-be-classified data block, the second to-be-classified data block and the third to-be-classified data block in the third to-be-managed file is determined, and a fourth classification sub-result corresponding to the third reference sub-node and the third unreferenced sub-node is determined according to the third label reference result;

[0023] The third label classification result is determined according to the third classification sub-result and the fourth classification sub-result.

[0024] Optionally, the clustering and arrangement of the first to-be-classified data block and the second to-be-classified data block according to the second label classification result to determine the target data block that exists in association includes:

[0025] According to the second label classification result, the data block attributes of all to-be-classified data blocks in the first to-be-managed file and the second to-be-managed file are marked; wherein the data block attributes include valid attributes and invalid attributes, the invalid attribute data block is used to describe a data block that is not marked by the second label classification result, and the valid attribute is used to describe a data block that is marked by the second label classification result;

[0026] The to-be-classified data block in the invalid attribute is deleted, and the to-be-classified data block in the valid attribute is determined as the target data block;

[0027] According to the second label classification result, the target data block is attributed to different storage spaces.

[0028] Optionally, the first to-be-managed file is obtained, including:

[0029] The first file data of the first to-be-managed file is obtained, and the first data feature of the first file data is determined;

[0030] The first to-be-managed file is segmented according to the first data feature to obtain the first to-be-classified data block, and the first fingerprint index information of all first to-be-classified data blocks is determined;

[0031] The clustering and arrangement of the first to-be-classified data block and the second to-be-classified data block according to the second label classification result includes:

[0032] When the second fingerprint index information corresponding to the second to-be-classified data block exists, the first fingerprint index information that exists in association with the second fingerprint index information is determined.

[0033] stopping writing the to-be-classified data block corresponding to the first fingerprint index information in the second to-be-classified data block, and determining the remaining to-be-classified data block as the target data block.

[0034] Optionally, the clustering and arranging of the first to-be-classified data block and the second to-be-classified data block according to the second label classification result comprises:

[0035] when there is second fingerprint index information corresponding to the second to-be-classified data block, determining third fingerprint index information same as the first fingerprint index information, wherein the third fingerprint index information is used to indicate the to-be-classified data block in the first to-be-classified data block having the same fingerprint index information as the second fingerprint index information;

[0036] calculating fourth label classification results of the to-be-classified data block corresponding to the second to-be-classified data block, the first to-be-classified data block and the third fingerprint index information, respectively;

[0037] setting the fourth label classification result as the second label classification result, and performing the step of clustering and arranging the first to-be-classified data block and the second to-be-classified data block.

[0038] The second aspect of the embodiment of the present application provides a data management system based on data migration, comprising:

[0039] an acquisition unit configured to acquire a first to-be-managed file and a second to-be-managed file; wherein the first to-be-managed file comprises first to-be-classified data blocks corresponding to different categories, and the second to-be-managed file comprises second to-be-classified data blocks corresponding to different categories;

[0040] a determination unit configured to determine a first label classification result of the first to-be-classified data block in the first to-be-managed file, and determine a second label classification result of the second to-be-classified data block in the second to-be-managed file based on the first label classification result;

[0041] an arrangement unit configured to cluster and arrange the first to-be-classified data block and the second to-be-classified data block according to the second label classification result, and determine a target data block having an association, so as to migrate and store the target data block.

[0042] The second aspect of the embodiment of the present application provides a data management method based on data migration for executing the first aspect.

[0043] The third aspect of the embodiment of the present application provides a data management device based on data migration, comprising:

[0044] a central processing unit, a memory, an input and output interface, a wired or wireless network interface, and a power supply;

[0045] The memory is a volatile memory or a persistent memory.

[0046] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the data management method based on data migration of the first aspect.

[0047] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium includes instructions, when the instructions run on the computer, make the computer execute the data management method based on data migration of the first aspect.

[0048] The fifth aspect of the embodiment of the present application provides a computer program product, the computer program product includes instructions, when the instructions run on the computer, make the computer execute the data management method based on data migration of the first aspect.

[0049] From the above technical solutions, the embodiment of the present application has the following advantages: by the data management method based on data migration disclosed in the embodiment of the present application, the first to be managed file and the second to be managed file are obtained first; wherein the first to be managed file includes the first to be classified data block corresponding to different categories, and the second to be managed file includes the second to be classified data block corresponding to different categories; then the first label classification result of the first to be classified data block in the first to be managed file is determined; based on the first label classification result, the second label classification result of the second to be classified data block in the second to be managed file is determined; finally, the first to be classified data block and the second to be classified data block are clustered and arranged according to the second label classification result, and the target data block existing the association is determined, so as to migrate and store the target data block. Therefore, the overhead of data block arrangement and migration can be effectively reduced. At the same time, since the association with the running environment is not great in the process of file management, it has universality in different scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0051] Figure 1 The flowchart of the data management method based on data migration disclosed in the embodiment of the present application;

[0052] Figure 2Another flowchart of a data management method based on data migration disclosed by an embodiment of the present application;

[0053] Figure 3 Another flowchart of a data management method based on data migration disclosed by an embodiment of the present application;

[0054] Figure 4 A demonstration diagram of a clustering process disclosed by an embodiment of the present application;

[0055] Figure 5 A demonstration diagram of a data block clustering method disclosed by an embodiment of the present application;

[0056] Figure 6 A comparison diagram between an existing garbage collection and the garbage collection disclosed by an embodiment of the present application;

[0057] Figure 7 A flowchart of a data management system based on data migration disclosed by an embodiment of the present application;

[0058] Figure 8 A structure diagram of a data management system based on data migration disclosed by an embodiment of the present application;

[0059] Figure 9 A structure diagram of a data management device based on data migration disclosed by an embodiment of the present application. DETAILED DESCRIPTION

[0060] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, if any, are used for distinguishing between similar objects talking about the embodiments and not necessarily for describing a specific sequential or chronological order. It is to be understood that the data so described can be interchanged, under appropriate circumstances, without departing from the scope of the embodiments described herein. Moreover, the terms "comprise", "comprising", "include", "including", and the like, are typically used herein to indicate the presence of stated features, integers, steps, or components, but not to the exclusion of others. It is to be understood that the data so described can be interchanged, under appropriate circumstances, without departing from the scope of the embodiments described herein.

[0061] It should be noted that the description herein involving "first", "second", etc. is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the technical features indicated, or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope required by the present application.

[0062] Data fragmentation has been a prominent problem in deduplicated data storage. Data fragmentation causes different file shares to duplicate data blocks, resulting in the mismatch between the logical data block order of files and the physical data block order on storage devices, which seriously affects the performance of data recovery. There are mainly two ways to handle the data fragmentation problem in deduplication systems at present: one is data block rewriting, which gives up eliminating some duplicate data blocks to maintain the physical locality of these data blocks, for example, Capping records the usage frequency of each data block when writing a new file, and gives up deduplication for data blocks with low usage frequency; the other is a data block organization method based on life cycle, which takes the access situation of each version of data blocks in the backup system (i.e. life cycle) as its characteristic, and classifies and manages data blocks according to the characteristic, such as MFDedup records the continuous life cycle of each data block, and classifies and manages data blocks according to the continuous life cycle of data blocks.

[0063] The existing solutions have many problems. The data block rewriting method has limited improvement on recovery speed, and often at the cost of significant deduplication efficiency reduction. For example, Capping can alleviate the fragmentation problem to a certain extent and improve the recovery speed, but it significantly reduces the deduplication rate, up to one third. The data block organization method based on life cycle is limited to backup scenarios, and only has obvious effect on deduplication fragmentation in backup systems, and is not applicable to other scenarios (such as primary storage scenarios). In addition, the data block organization of MFDedup requires a special migration link, which increases additional overhead.

[0064] Therefore, the present application designs a data management method based on data migration for data deduplication, which provides a new perspective for the management of deduplicated data blocks. It uses data block clustering based on approximate accurate attribution, which can aggregate data blocks with strong logical correlation without relying on specific scenarios. At the same time, in the garbage collection process, the existing data blocks can be sorted and migrated according to the previous clustering results to generate a new data block layout, without the need for additional migration process, thereby reducing the overhead.

[0065] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0066] To conveniently solve the technical problems described in the above and describe the data management method based on data migration described in the present application, please refer to Figure 1 , Figure 1 A flowchart of a data management method based on data migration disclosed in the embodiments of the present application is shown. The method comprises steps 101-103.

[0067] 101, obtaining a first to-be-managed file and a second to-be-managed file.

[0068] To realize the management of file data, the to-be-managed files need to be obtained, i.e. the first to-be-managed file and the second to-be-managed file. It is not difficult to understand that the first to-be-managed file and the second to-be-managed file described in the embodiments refer to the file 1 or the file 2 obtained in the system. At this time, the file 1 and the file 2 do not have an obvious acquisition order, and it is not meant that the file 1 and the file 2 are the same or different. It should also be understood that the first to-be-managed file comprises the first to-be-classified data block corresponding to different categories, and the second to-be-managed file comprises the second to-be-classified data block corresponding to different categories.

[0069] In one specific embodiment, in the process of system running, the file can be written in real time, so that the system can obtain the corresponding to-be-managed file. Further, the system can also traverse the storage space of the system or other data interfaces, so as to obtain the corresponding file data and obtain the to-be-managed file. The way of obtaining the to-be-managed file is not limited here, and will not be described in detail hereinafter.

[0070] 102, determining a first label classification result of the first to-be-classified data block in the first to-be-managed file; and determining a second label classification result of the second to-be-classified data block in the second to-be-managed file based on the first label classification result.

[0071] Thus, based on step 101, the first label classification result of the first to-be-classified database in the first to-be-managed file can be determined. Then, based on the first label classification result, the second label classification result of the second to-be-classified database in the second to-be-managed file is determined. It is not difficult to understand that in the embodiment, a database clustering method of approximate accurate attribution is mainly used. It should be noted that attribution refers to the case that a data block is referenced by each file in the data storage. The data blocks with similar attribution are similar to the files referencing them, and thus are logically related. Therefore, classifying and organizing the data blocks based on the attribution can enable each class of data blocks to have good logical correlation, thereby alleviating the fragmentation problem caused by deduplication. At the most fundamental level, the classification is based on the logical correlation of the data blocks. That is, the purpose of classification is to want the logical correlation of the data blocks between classes to be strong, and the data blocks are accessed at the same time with a high probability during reading and writing. However, the correlation needs to be quantified. The attribution, that is, the case that the data block is applied by each file as mentioned above, is a kind of measurement method. Specifically, all the files in the deduplication system can be numbered from 1, and the attribution of the data block is expressed as a binary digital string, referred to as a reference vector. The nth bit of the reference vector indicates whether the data block is referenced by the file n, 1 for referenced and 0 for not referenced. It is easy to conclude that the number of types of attribution is of the order of magnitude of the number of system files. If the classification is accurately performed according to the attribution, the number of classifications of the system is extremely large, which is not feasible in practice.

[0072] Thus, the approximate accurate attribution proposed in the application refers to that the data blocks in a category contain different but as similar as possible attributions, so as to ensure the logical correlation of the data blocks in the same category as much as possible in the case of category reduction. In order to achieve this goal, the intergenerational relationship and genetic characteristics in the evolution of attribution are needed to be used to remove the same data blocks (deduplication) and then dynamically classify the data blocks. It should be pointed out in advance that the intergenerational relationship of attribution refers to that after the system writes a new file, the attribution of the data blocks in the system will change and a new generation will be generated. When the system writes a new file, the reference vector of each data block needs to be updated, thereby generating a new generation of reference vector, that is, a new generation of attribution. Therefore, the attribution has intergenerational relationship. It is not difficult to understand that the intergenerational relationship can be simply understood as that the attribution of the existing data blocks needs to be updated every time a new file is written, so that every time a new file is written, it is a change time. The genetic characteristics of attribution refer to that most of the bits in the reference vector are inherited by the new generation of attribution when generating the new generation of attribution. When the system writes a new file each time, a bit needs to be added to the reference vector of each data block, and the latest bit is determined to be 0 or 1 according to whether it is referenced by the new file. In addition, the original part of the reference vector is retained. That is, most of the bits of the reference vector are retained by the new generation of reference vector, which is equivalent to being "inherited" by the new generation. Therefore, the attribution has genetic characteristics. Obviously, if all categories are continuously subdivided every time a file is written, the number of categories will reach an exponential level, and the data block clustering method based on approximate accurate attribution needs to be used to effectively limit the number of categories while ensuring the logical correlation of the data blocks in the same category as much as possible. For the convenience of understanding and description, the attribution and the intergenerational relationship and genetic characteristics of the attribution will not be described in detail hereinafter.

[0073] In one specific embodiment, when the first to-be-managed file is written, all the data blocks in the first to-be-classified data block are referenced by the first to-be-managed file, so the corresponding first label classification result can be understood as all references. As described above, the related reference vector is 1. That is, the first to-be-classified data block belongs to node "1", forming the first generation of classification. Then, the second to-be-managed file is written, and since there may be the same data blocks as the first to-be-classified data block in the second to-be-classified data block, and there may be data blocks different from the first to-be-classified data block, and the number of data blocks contained in the unique category in the system is greater than 2, the corresponding node needs to be branched. The repeated data blocks and the non-repeated data blocks need to be counted and calculated respectively. Thus, the second label classification result can be generated based on the first label classification result.

[0074] 103、According to the second label classification result, the first to-be-classified data block and the second to-be-classified data block are clustered and arranged, the target data blocks with correlation are determined, and the target data blocks are migrated and stored.

[0075] Thus, based on step 102, the reference vector of all data blocks can be determined after obtaining the second label classification result, and thus the first to-be-classified data block and the second to-be-classified data block can be clustered and arranged, and the target data block with the association is determined, so as to migrate and store the target data block.

[0076] It should be noted that after the data blocks in the system are classified, the arranged and migrated data blocks in the system need to be classified according to the classification. Specifically, the application utilizes the opportunity of migration in the scanning process of garbage collection, arranges and migrates the data blocks according to the clustering result based on the approximate accurate attribution, and eliminates the additional migration overhead. In one specific embodiment, after the label classification result of all data blocks is determined, the data structure of the file to be managed at this time is updated to form the corresponding clustering result. Thus, by traversing all data blocks through the scanning process, the effective data blocks are migrated and the invalid data blocks are discarded, and thus the target database that needs to be migrated and stored can be obtained.

[0077] Further, the effective data blocks can also be aggregated and migrated according to the clustering result, so as to ensure that the data blocks with the same or similar attribution are aggregated as much as possible. Thus, the migration and storage of the data blocks are finally completed.

[0078] According to the data management method based on data migration disclosed in the embodiment, the first to-be-managed file and the second to-be-managed file are first obtained; the first to-be-managed file includes first to-be-classified data blocks corresponding to different categories, and the second to-be-managed file includes second to-be-classified data blocks corresponding to different categories; then the first label classification result of the first to-be-classified data block in the first to-be-managed file is determined; based on the first label classification result, the second label classification result of the second to-be-classified data block in the second to-be-managed file is determined; finally, the first to-be-classified data block and the second to-be-classified data block are clustered and arranged according to the second label classification result, the target data block with the association is determined, and the target data block is migrated and stored. Thus, the overhead of data block arrangement and migration can be effectively reduced. At the same time, since the association with the running environment is not great in the process of file management, the method has universality in different scenarios.

[0079] For the convenience of detailed description of the data management method based on data migration described above Figure 1 , please refer to Figure 2 , Figure 2 The flowchart of another data management method based on data migration disclosed in the embodiment of the application is shown. It includes steps 201-210.

[0080] 201、obtain first file data of the first to-be-managed file, determine a first data feature of the first file data, and segment the first to-be-managed file according to the first data feature to obtain first to-be-classified data blocks, and determine first fingerprint index information of all the first to-be-classified data blocks.

[0081] The step 201 in this embodiment is similar to the step 101 in the foregoing Figure 1 embodiment, and details are not described herein. It should be noted that, in this embodiment, the manner of obtaining the first to-be-managed file is similar to the manner of obtaining other to-be-managed files, which will not be described herein. Specifically, first file data of the first to-be-managed file is obtained, a first data feature of the first file data is determined, and the first to-be-managed file is segmented according to the first data feature to obtain first to-be-classified data blocks, and finally first fingerprint index information of all the first to-be-classified data blocks is determined.

[0082] In one specific embodiment, after obtaining the first to-be-managed file, the first file data is obtained through analysis of the to-be-managed file, and a data feature of the file data is determined. Thus, the to-be-managed file is segmented based on the data feature to obtain corresponding data blocks, and fingerprint index information of each data block is calculated.

[0083] Specifically, the chunking process can use a content-defined chunking (CDC) algorithm. The input to-be-managed file is divided into data blocks of different lengths, i.e., the to-be-classified data blocks described in the foregoing, and then the fingerprint of each data block is calculated. It should be noted that the CDC algorithm is an algorithm for dividing data into blocks of different sizes. This algorithm is usually used when data is transmitted or stored to improve efficiency and reduce resource consumption. The basic idea of the content-defined chunking algorithm is to divide data into blocks of appropriate sizes according to the characteristics and needs of the data. In this way, data can be better managed during transmission or storage, and transmission speed and storage efficiency can be improved.

[0084] Further, for the second to-be-managed file or the third to-be-managed file or other to-be-managed files, the specific segmentation manner is similar to that of the step 201, which will not be described herein.

[0085] 202、set a classification root node, and set a first classification sub-node according to the classification root node and the first to-be-managed file.

[0086] As shown in the step 201, after obtaining the first to-be-managed file and the second to-be-managed file, the to-be-classified data blocks can be classified by using the data block clustering method based on approximate accurate attribution. Further, for convenience of explanation, please refer to Figure 4 ,Figure 4 A demonstration diagram of a clustering process disclosed in an embodiment of the present application. Specifically, the process of each classification can be organized in the form of a binary tree, and the left and right child nodes of each node are the classification results of the original data blocks after writing a new file, and the bitmap indicated by each node (except the root node) is the common prefix of the reference vector of all leaf nodes in the tree with the node as the root node. Here, the branch threshold is set to 2, that is, each classified data block is greater than 2 to continue branching. It is not difficult to understand that the branch threshold described in the embodiment can also be set to the size of all data blocks of the classification node. The first leaf node is defined according to the reference vector, which is always empty, but the node needs to branch each time it is written to the file, because the new non-repeated data block introduced in the new file needs to be written to the right child node of the node. For example, the first file to be managed (file 1) includes data blocks 1-6, the second file to be managed (file 2) includes data blocks 7, 3, 8, 1, 2 and 4, and the third file to be managed (file 3) includes data blocks 9, 10, 3, 8, 7 and 1. Therefore, it can be simply understood that the reference vector corresponding to the data block at this time is as shown on the right Figure 4 The right side is not described here.

[0087] Specifically, a classification root node can be set, and a first classification child node can be set according to the classification root node and the first file to be managed. It is not difficult to understand that the first classification child node includes a first reference sub-node and a first unreferenced sub-node.

[0088] In one specific embodiment, for the convenience of understanding and description, please refer to Figure 5 , Figure 5 A demonstration diagram of a data block clustering method disclosed in an embodiment of the present application. From Figure 5 It can be seen that when file 1 is written, the root node is set. Then, after setting the point according to file 1 is introduced, the corresponding first classification child node can be generated, that is, the first generation. At this time, there is a node "1", which is the first reference sub-node; node "0", which is the first unreferenced sub-node. Among them, the reference sub-node is that the corresponding data block to be classified is referenced in the file to be managed, and the unreferenced sub-node is that the corresponding data block to be classified is not referenced in the file to be managed. For the convenience of understanding and description, this will not be described in detail.

[0089] 203, determine the first tag reference result of the first data block to be classified in the first file to be managed, and determine the first classification result corresponding to the first reference sub-node according to the first tag reference result, to determine the first tag classification result according to the first classification result.

[0090] Thus, based on the classification of the first data block to be classified in step 202, the first label reference result of the first data block to be classified in the first file to be managed is determined, and the first classification result corresponding to the first reference sub-node is determined according to the first label reference result, so that the first label classification result is determined according to the first classification result.

[0091] In one specific embodiment, since the first data block to be classified can directly constitute the first file to be managed, it can be simply understood that all the data blocks are referenced by the file 1. Thus, referring to the above table, it can be seen that all the data blocks are referenced by the file 1, so that the reference vector is 1, and all the data blocks belong to the node “1”, forming the first generation classification. The first generation classification result is: {block 1, block 2, block 3, block 4, block 5, block 6}. Figure 5 When the file 1 is written, all the data blocks are referenced by the file 1, so that the reference vector is 1, and all the data blocks belong to the node “1”, forming the first generation classification. The first generation classification result is: {block 1, block 2, block 3, block 4, block 5, block 6}.

[0092] 204、According to the first classification sub-node and the second file to be managed, the second classification sub-node is set under the first reference sub-node and the first non-reference sub-node respectively, the second label reference result of the first data block to be classified and the second data block to be classified in the second file to be managed is determined, and the second classification result corresponding to the second reference sub-node and the second non-reference sub-node respectively is determined according to the second label reference result.

[0093] Based on step 203, after the first classification sub-node and the second file to be managed are determined, the second classification sub-node can be set under the first reference sub-node and the first non-reference sub-node respectively, so that the second label reference result of the first data block to be classified and the second data block to be classified in the second file to be managed is determined, and the second classification result corresponding to the second reference sub-node and the second non-reference sub-node respectively is determined according to the second label reference result. It is not difficult to understand that the second classification sub-node includes the second reference sub-node and the second non-reference sub-node.

[0094] In one specific embodiment, when the file 2 is written, the corresponding second reference sub-node and the second non-reference sub-node need to be set under the first reference sub-node or the first non-reference sub-node respectively. That is Figure 5 The “00”, “01”, “10” and “11” of the nodes are shown. Then, the second data block to be classified and the first data block to be classified can be placed under the corresponding second reference sub-node and the second non-reference sub-node, so that the corresponding second label reference result is determined, and the second classification result is finally determined. Further, referring to the corresponding file and its constituent blocks in Figure 4 Thus, the system contains only the classification containing data blocks, the number of which is greater than 2, so that the corresponding node needs to branch. Blocks 1-4 are referenced by the file 2, so they are placed in the right sub-node 11 of the node 1, and blocks 5 and 6 are placed in the left sub-node 10. In addition, the non-repeated data blocks 7 and 8 of the file 2 need to be added to the tree.

[0095] 205. determining a second label classification result according to the second classification result.

[0096] Thus, based on the classification of step 204, the second label classification result can be determined according to the second classification result.

[0097] In one specific embodiment, the second label classification result is determined according to the second classification result by Figure 5 As shown in FIG. 2B, the second classification result is that node "01" corresponds to {block 7, block 8}, node "10" corresponds to {block 5, block 6}, and node "11" corresponds to {block 1, block 2, block 3, block 4}.

[0098] Further, when there are other files to be managed, steps 301-305 in FIG. 3 can be referred to, which will not be described here in detail. Figure 3 It should be noted that at this time, the first file to be managed can also be understood as the file to be managed obtained at the beginning, and the second file to be managed can be understood as the file to be managed obtained subsequently, which will not be described here one by one.

[0099] Further, different migration modes can be implemented based on the second label classification result. Specifically, steps 206-207 can be executed, or step 208 can be executed, or steps 209-210 can be executed.

[0100] 206. marking data block attributes of all data blocks to be classified in the first file to be managed and the second file to be managed according to the second label classification result.

[0101] Step 206 in this embodiment can also be understood as marking data block attributes of all data blocks to be classified in the first file to be managed and the second file to be managed according to the second label classification result by using the CDC algorithm. It is not difficult to understand that the data block attributes include valid attributes and invalid attributes, and the invalid attribute data blocks are used to describe data blocks that are not marked by the second label classification result, and the valid attributes are used to describe data blocks that are marked by the second label classification result.

[0102]

[0103] ​In one specific embodiment, the characteristics and requirements of all data blocks to be classified can be identified. Specifically, there are many ways to implement the variable-length chunking algorithm, and one common implementation is as follows: 1. Determine the size of the chunk: according to the characteristics and requirements of the data, determine the size of each chunk. The size of the data, the transmission speed, the storage capacity, etc. can be adjusted. 2. Split the data: split the data according to the determined chunk size. The data can be split into multiple blocks using loops or recursion. 3. Handle the last block: if the size of the data cannot be divided by the chunk size, the last block may be smaller than the other blocks. Special processing can be performed as needed, such as adding padding data or marking the block as the last block. 4. Transmit or store data: transmit or store the segmented data blocks. Network transmission protocols or file systems can be used for operation. 5. Receive or read data: when receiving or reading data, process according to the chunk information. The data can be reorganized or processed according to the size and order of the blocks. For the sake of understanding and description, this will not be described in detail.

[0104] Further, by determining the data block attributes of the data blocks to be classified, it can be determined whether the corresponding data blocks to be classified are invalid attributes or valid attributes.

[0105] 207, delete the data blocks to be classified in the invalid attribute, and determine the data blocks to be classified in the valid attribute as target data blocks, and according to the second label classification result, the target data blocks are attributed to different storage spaces.

[0106] Based on step 206, the data blocks to be classified in the invalid attribute can be deleted, and the data blocks to be classified in the valid attribute are determined as target data blocks, so that the target data blocks are attributed to different storage spaces according to the second label classification result.

[0107] In one specific embodiment, the garbage collection process is generally divided into marking and scanning processes: the marking process marks invalid data blocks and determines the containers that need to be recycled, and the scanning process migrates the data blocks in the recycled containers to new containers according to the results of the marking process. Specifically, refer to Figure 6 , Figure 6 A comparison diagram of the existing garbage collection disclosed in the embodiments of the present application and the garbage collection of the present application. The garbage collection process of the general system only identifies valid data blocks and puts them in order into new containers, but ignores the logical correlation between the data blocks. The method proposed in the present application can arrange the data blocks according to the clustering results before migration, and then migrate according to the order of the arranged data blocks. It can be seen that this migration method realizes the functions of invalid block removal and valid data block arrangement simultaneously using the garbage collection process. It is not difficult to understand that for Figure 6In the case of garbage collection, the left side shows the migration using the existing method, and the data blocks can be organized in the new container in the original arrangement order. The present application can organize the data blocks in the new container according to the data block attribution, and specifically, the data blocks can be classified by the second label classification result.

[0108] Further, if there is a third label classification result, the attribution or classification can be performed according to the third label classification result. That is, the attribution or classification is performed according to the latest label classification result.

[0109] 208、When the second fingerprint index information corresponding to the second data block to be classified exists, the first fingerprint index information associated with the second fingerprint index information is determined to stop writing the data block to be classified corresponding to the first fingerprint index information in the second data block to be classified, and the remaining data block to be classified is determined as the target data block.

[0110] Corresponding to step 205, when the second fingerprint index information corresponding to the second data block to be classified exists, the first fingerprint index information associated with the second fingerprint index information can be determined to stop writing the data block to be classified corresponding to the first fingerprint index information in the second data block to be classified, and the remaining data block to be classified is determined as the target data block.

[0111] In one specific embodiment, when the second fingerprint index information corresponding to the second data block to be classified is also calculated in the deduplication process, the first fingerprint index information associated with the second fingerprint index information can be determined to stop writing the data block to be classified corresponding to the first fingerprint index information, and the remaining data block to be classified is determined as the target data block.

[0112] In one specific embodiment, the deduplication process can identify and delete duplicate data blocks according to the information in the fingerprint index. Further, temporary attribution information needs to be recorded in the process to form a new clustering result later. It is not difficult to understand that the recorded is the reference vector information of the data block after writing to the new file, which is used for clustering later. It should be noted that the deduplication system uses a cryptographic hash function (such as SHA-1 secure hash algorithm) to calculate the fingerprint of the data block, which can be used as a unique identifier of the data block. For the convenience of understanding and description, this will not be described in detail hereinafter.

[0113] For example, file 2 has the same data block 1, data block 2, data block 3 and data block 4 as file 1. Therefore, when writing the data blocks of the file to the database, data block 1, data block 2, data block 3 and data block 4 in file 2 can not be written. Further, data block 1, data block 2, data block 3 and data block 4 in file 1 can also not be written. This is not limited here.

[0114] Further, after obtaining the target data block, a similar process as step 207 can be performed, i.e. attributing the target data block to different storage spaces.

[0115] It should be further noted that the second fingerprint index information in the embodiment can be replaced by the third fingerprint index information, and the specific implementation manner is similar to that of the second fingerprint index information, and details are not described herein.

[0116] 209、When the second fingerprint index information corresponding to the second to-be-classified data block exists, third fingerprint index information identical to the first fingerprint index information is determined.

[0117] Corresponding to step 205, when the second fingerprint index information corresponding to the second to-be-classified data block exists, third fingerprint index information identical to the first fingerprint index information in the second fingerprint index information can be determined. It can be understood that the third fingerprint index information is used to indicate the to-be-classified data block in the first to-be-classified data block that has the same fingerprint index information as the second fingerprint index information.

[0118] In one specific embodiment, steps 209-210 can be understood as similar to step 208, and specifically, by identifying the same part in the first fingerprint index information and the second fingerprint index information, the same to-be-classified data block can be identified and determined.

[0119] Further, when the fourth fingerprint index information corresponding to the third to-be-classified data block exists, the same part in any two of the first fingerprint index information, the second fingerprint index information, or the fourth fingerprint index information can be found, and the corresponding to-be-classified data block can be identified.

[0120] 210、Respectively calculate fourth label classification results of the second to-be-classified data block, the first to-be-classified data block, and the to-be-classified data block corresponding to the third fingerprint index information.

[0121] Thus, the fourth label classification results of the second to-be-classified data block, the first to-be-classified data block, and the to-be-classified data block corresponding to the third fingerprint index information can be respectively calculated.

[0122] In one specific embodiment, the label classification result of the second to-be-classified data block, the label classification result of the first to-be-classified data block, and the label classification result of the to-be-classified data block corresponding to the third fingerprint index information can be respectively calculated, and all the label classification results can be integrated to generate the fourth label classification result.

[0123] Further, the data block remaining after the deduplication can be stored in the system, and a file spectrum can be generated according to the order of the data blocks composed of the input file.

[0124] Further, the fourth label classification result can be set as the second label classification result, and step 205 is performed, which is not limited here.

[0125] In another possible implementation, the method provided by the present application can not only cluster and migrate the data blocks after deduplication, but also abandon the deduplication of some data blocks during the deduplication process by using the data rewriting method. Further, the data blocks with the same fingerprints after data rewriting can be regarded as different data blocks, and the ownership information of each data block is calculated separately, so that the classification and arrangement results can be achieved.

[0126] It should be noted that, please refer to Figure 7 , Figure 7 The present application discloses a flowchart of a data management system based on data migration. As can be seen from Figure 7 and in combination with the above Figure 2 described, first, when a file is written, the system needs to go through the processes of blocking, deduplication and storage. Specifically, the blocking process uses the CDC algorithm to cut the input file into data blocks of different lengths and calculate the fingerprints of the data blocks. The deduplication process identifies and deletes duplicate data blocks according to the information in the fingerprint index. The storage process stores the data blocks remaining after deduplication into the system and generates a file spectrum according to the order in which the input file is composed of data blocks.

[0127] Then, when certain conditions are exceeded, the garbage collection process is triggered. Specifically,

[0128] First, the data block clustering based on accurate ownership needs to be performed. The temporary ownership information recorded before is used to update the data structure related to clustering, and a new clustering result is formed. The marking process checks whether each data block in the system is valid. The scanning process traverses all the data blocks, migrates the valid data blocks and discards the invalid data blocks. The method provided by the present application needs to perform the data block arrangement and migration fused with the garbage collection process, and the valid data blocks are aggregated and migrated according to the clustering result, so that the data blocks with the same or similar ownership can be aggregated together as much as possible.

[0129] It should be noted that the general deduplication method does not need to record the ownership information, does not perform classification, and therefore does not need to adjust the migration order according to the classification during the garbage collection process.

[0130] By the data management method based on data migration disclosed in the embodiment, the logically related data blocks can be aggregated together, and the classification number can be effectively reduced. Meanwhile, the data blocks are sorted and migrated according to the clustering result based on approximate accurate attribution by using the migration opportunity in the scanning process of garbage collection, the additional migration overhead is eliminated, and the invalid block cleaning and the sorting of the effective data blocks are simultaneously realized by using the garbage collection process.

[0131] Further, when there is a third to-be-managed file in addition to the first to-be-managed file and the second to-be-managed file, the third to-be-managed file can be referred to Figure 3 , Figure 3 The flowchart of another data management method based on data migration disclosed in the embodiment is shown in FIG. 3. The method comprises steps 301-305. It should be noted that the execution order of steps 301-305 is not obvious, and the specific execution order is the logical order. Details will be described later. Figure 2

[0132] 301. Obtain a third to-be-managed file, and determine whether the number of data blocks corresponding to the second label classification result in the third to-be-classified data block exceeds the classification node threshold. When the number of data blocks exceeds the classification node threshold, perform step 302; when the number of data blocks does not exceed the classification node threshold, perform steps 303-304.

[0133] As described in the foregoing embodiment, when there is a third to-be-managed file in addition to the first to-be-managed file and the second to-be-managed file, it is necessary to determine whether the number of data blocks corresponding to the second label classification result in the third to-be-classified data block exceeds the classification node threshold. Therefore, when the number of data blocks exceeds the classification node threshold, step 302 is performed; when the number of data blocks does not exceed the classification node threshold, steps 303-304 are performed. Figure 2 In one specific embodiment, the classification node threshold is set to 2, that is, the data blocks of each classification are greater than 2, and then the branching is continued. Further, the third to-be-managed file comprises third to-be-classified data blocks corresponding to different categories.

[0134] For the convenience of understanding and description, the third to-be-managed file can be referred to

[0135] . The third to-be-managed file comprises data blocks 9, 10, 3, 8, 7, and 1. Figure 4 It is not difficult to understand that step 301 in the embodiment can be executed synchronously with step 201, or can be executed after step 201, or can be executed before step 201, and the specific execution time is not limited here.

[0136]

[0137] ​​302、when the data block quantity exceeds the classification node threshold value, maintaining the second reference sub-node or the second unreferenced sub-node corresponding to the second classification sub-node, and determining a third classification sub-result.

[0138] Based on step 301, when the data block quantity exceeds the classification node threshold value, the second reference sub-node or the second unreferenced sub-node corresponding to the second classification sub-node can be maintained, and a third classification sub-result is determined.

[0139] In one specific embodiment, the data block quantities of the classifications {block 7, block 8} and {block 5, block 6} do not exceed the branch threshold value 2, so the corresponding nodes do not branch. Therefore, the corresponding second reference sub-node or second unreferenced sub-node can be directly referenced, thereby determining the corresponding third classification sub-result.

[0140] 303、when the data block quantity does not exceed the classification node threshold value, setting a third classification sub-node under the second reference sub-node and the second unreferenced sub-node according to the second classification sub-node and the third to-be-managed file.

[0141] Based on step 301, when the data block quantity does not exceed the classification node threshold value, a third classification sub-node can be set under the second reference sub-node and the second unreferenced sub-node according to the second classification sub-node and the third to-be-managed file. It is not difficult to understand that the third classification sub-node includes a third reference sub-node and a third unreferenced sub-node.

[0142] In one specific embodiment, the data block quantity of the classification {block 1, block 2, block 3, block 4} exceeds the branch threshold value, so the corresponding node needs to be branched. Therefore, it can be seen from Figure 3 that the corresponding third generation nodes are “000”, “001”, “110”, and “111”.

[0143] 304、determining a third tag reference result of the first to-be-classified data block, the second to-be-classified data block, and the third to-be-classified data block in the third to-be-managed file, and determining a fourth classification sub-result corresponding to the third reference sub-node and the third unreferenced sub-node according to the third tag reference result.

[0144] Therefore, based on step 303, a third tag reference result of the first to-be-classified data block, the second to-be-classified data block, and the third to-be-classified data block in the third to-be-managed file can be determined, and a fourth classification sub-result corresponding to the third reference sub-node and the third unreferenced sub-node according to the third tag reference result is determined.

[0145] In one specific embodiment, block 1 and block 3 are referenced by file 3, and thus are placed in the right child node 111 of node 11, and block 2 and block 4 are placed in the left child node 110. In addition, non-repeated data blocks 9 and 10 of file 3 need to be added into the tree. Thus, the fourth classification sub-result can be identified.

[0146] 305. Determine the third label classification result according to the third classification sub-result and the fourth classification sub-result.

[0147] Thus, the third label classification result can be determined based on the third classification sub-result and the fourth classification sub-result of steps 302 and 304.

[0148] In one specific embodiment, the third generation classification result is: {block 9, block 10}, {block 7, block 8}, {block 5, block 6}, {block 2, block 4}, {block 1, block 3}.

[0149] It is understood that steps 302-305 can be executed synchronously with step 205, or after step 205, or before step 205, and the specific embodiments are not limited here. Correspondingly, when the third label classification result is obtained, the third label classification result can replace the second label classification result, and step 205 can be executed.

[0150] By the data management method based on data migration disclosed in the embodiment, the branch threshold is set to prevent the generation of too small classification to avoid the explosive growth of the number of classifications, and the logical correlation performance of the data blocks in the classification is guaranteed as much as possible.

[0151] It should be understood that, although each step in the flowchart involved in each embodiment described above is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0152] Please refer to Figure 8 , Figure 8 The structure of the data management system based on data migration disclosed in the embodiments of the present application is shown in the schematic diagram.

[0153] The acquisition unit 801 is configured to acquire a first to-be-managed file and a second to-be-managed file; wherein the first to-be-managed file comprises first to-be-classified data blocks corresponding to different categories, and the second to-be-managed file comprises second to-be-classified data blocks corresponding to different categories;

[0154] The determination unit 802 is configured to determine a first label classification result of the first to-be-classified data block in the first to-be-managed file; and determine a second label classification result of the second to-be-classified data block in the second to-be-managed file based on the first label classification result.

[0155] The arrangement unit 803 is configured to cluster and arrange the first to-be-classified data block and the second to-be-classified data block according to the second label classification result, determine target data blocks that exist in association, and migrate and store the target data blocks.

[0156] Exemplarily, the system further comprises a setting unit 804.

[0157] The setting unit 804 is configured to set a classification root node.

[0158] The setting unit 804 is further configured to set a first classification sub-node according to the classification root node and the first to-be-managed file; wherein the first classification sub-node comprises a first reference sub-node and a first unreferenced sub-node.

[0159] The determination unit 802 is specifically configured to determine a first label reference result of the first to-be-classified data block in the first to-be-managed file, and determine a first classification result corresponding to the first reference sub-node according to the first label reference result.

[0160] The determination unit 802 is further configured to determine the first label classification result according to the first classification result.

[0161] Exemplarily, the system comprises:

[0162] The setting unit 804 is specifically configured to set a second classification sub-node under the first reference sub-node and the first unreferenced sub-node according to the first classification sub-node and the second to-be-managed file; wherein the second classification sub-node comprises a second reference sub-node and a second unreferenced sub-node.

[0163] The determination unit 802 is specifically configured to determine a second label reference result of the first to-be-classified data block and the second to-be-classified data block in the second to-be-managed file, and determine a second classification result corresponding to the second reference sub-node and the second unreferenced sub-node respectively according to the second label reference result.

[0164] The determination unit 802 is further configured to determine the second label classification result according to the second classification result.

[0165] Exemplarily, the system further comprises a judgment unit 805.

[0166] The acquisition unit 801 is specifically configured to acquire a third to-be-managed file; wherein the third to-be-managed file comprises third to-be-classified data blocks corresponding to different categories;

[0167] The judgment unit 805 is configured to judge whether the number of data blocks corresponding to the second label classification result in the third to-be-classified data blocks exceeds a classification node threshold value;

[0168] The determination unit 802 is further configured to maintain the second reference sub-node or the second unreferenced sub-node corresponding to the second classification sub-node when the number of data blocks exceeds the classification node threshold value, and determine a third classification sub-result;

[0169] Or, the setting unit 804 is further configured to set, when the number of data blocks does not exceed the classification node threshold value, a third classification sub-node under the second reference sub-node and the second unreferenced sub-node according to the second classification sub-node and the third to-be-managed file; wherein the third classification sub-node comprises a third reference sub-node and a third unreferenced sub-node; determine a third label reference result of the first to-be-classified data block, the second to-be-classified data block and the third to-be-classified data block in the third to-be-managed file, and determine a fourth classification sub-result corresponding to the third reference sub-node and the third unreferenced sub-node according to the third label reference result;

[0170] The determination unit 802 is further configured to determine a third label classification result according to the third classification sub-result and the fourth classification sub-result.

[0171] Exemplarily, the system further comprises a deletion unit 806 and an execution unit 807;

[0172] The determination unit 802 is specifically configured to mark data block attributes of all to-be-classified data blocks in the first to-be-managed file and the second to-be-managed file according to the second label classification result; wherein the data block attributes comprise valid attributes and invalid attributes, and the invalid attribute data block is used to describe data blocks not marked by the second label classification result, and the valid attribute is used to describe data blocks marked by the second label classification result;

[0173] The deletion unit 806 is configured to delete the to-be-classified data block in the invalid attribute, and determine the to-be-classified data block in the valid attribute as a target data block;

[0174] The execution unit 807 is configured to attribute the target data block to different storage spaces according to the second label classification result.

[0175] Exemplarily, the system comprises:

[0176] The acquisition unit 801 is specifically configured to acquire first file data of the first to-be-managed file, and determine a first data feature of the first file data;

[0177] The acquisition unit 801 is further configured to segment the first to-be-managed file according to the first data feature to obtain a first to-be-classified data block, and determine first fingerprint index information of all the first to-be-classified data blocks.

[0178] The determination unit 802 is specifically configured to determine first fingerprint index information associated with the second fingerprint index information when the second fingerprint index information corresponding to the second to-be-classified data block exists.

[0179] The deletion unit 806 is specifically configured to stop writing the to-be-classified data block corresponding to the first fingerprint index information in the second to-be-classified data block, and determine the remaining to-be-classified data block as a target data block.

[0180] Exemplarily, the system further includes a calculation unit 808.

[0181] The determination unit 802 is specifically configured to determine third fingerprint index information that is the same as the first fingerprint index information when the second fingerprint index information corresponding to the second to-be-classified data block exists, wherein the third fingerprint index information is used to indicate the to-be-classified data block in the first to-be-classified data block that has the same fingerprint index information as the second fingerprint index information.

[0182] The calculation unit 808 is configured to calculate a fourth label classification result of the to-be-classified data block corresponding to the second to-be-classified data block, the first to-be-classified data block and the third fingerprint index information, respectively.

[0183] The execution unit 807 is specifically configured to set the fourth label classification result as the second label classification result, and perform the step of clustering and arranging the first to-be-classified data block and the second to-be-classified data block.

[0184] Please refer to Figure 9 The embodiment of the application discloses a structure schematic diagram of a data management device based on data migration, which includes:

[0185] The central processor 901, the memory 905, the input and output interface 904, the wired or wireless network interface 903 and the power supply 902;

[0186] The memory 905 is a transitory storage memory or a persistent storage memory;

[0187] The central processor 901 is configured to communicate with the memory 905, and execute the instruction operation in the memory 905 to perform the foregoing Figures 1 to 3 The data management method based on data migration in any of the embodiments.

[0188] The chip system includes at least one processor and a communication interface. The communication interface and the at least one processor are interconnected through a line. The at least one processor is configured to run computer programs or instructions to execute the above-mentioned Figures 1 to 3 The data management method based on data migration in any of the embodiments.

[0189] The computer readable storage medium includes instructions. When the instructions are run on a computer, the computer is caused to execute the above-mentioned Figures 1 to 3 The data management method based on data migration in any of the embodiments.

[0190] The computer program product includes instructions. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned Figures 1 to 3 The data management method based on data migration in any of the embodiments.

[0191] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0192] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, another division mode can be used. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0193] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0194] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0195] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A data management method based on data migration, characterized by, The method comprises: obtaining a first to-be-managed file and a second to-be-managed file; wherein the first to-be-managed file comprises a plurality of first to-be-classified data blocks corresponding to different categories, and the second to-be-managed file comprises a plurality of second to-be-classified data blocks corresponding to different categories; determining a first reference vector of the first to-be-classified data blocks in the first to-be-managed file, and generating a first label classification result according to the first reference vector; based on the first label classification result, determining a second reference vector of the second to-be-classified data blocks in the second to-be-managed file, and generating a second label classification result according to the second reference vector; clustering and arranging the first to-be-classified data blocks and the second to-be-classified data blocks according to the second label classification result, determining target data blocks that exist in association, and migrating and storing the target data blocks.

2. The data management method based on data migration according to claim 1, characterized in that, The determination of the first reference vector of the first to-be-classified data blocks in the first to-be-managed file, and the generation of the first label classification result according to the first reference vector, comprise: setting a classification root node; setting a first classification sub-node according to the classification root node and the first to-be-managed file; wherein the first classification sub-node comprises a first reference sub-node and a first unreferenced sub-node; determining the first reference vector of the first to-be-classified data blocks in the first to-be-managed file, and determining a first classification result corresponding to the first reference sub-node according to the first reference vector; determining the first label classification result according to the first classification result.

3. The data management method based on data migration according to claim 2, characterized in that, The determination of the second reference vector of the second to-be-classified data blocks in the second to-be-managed file based on the first label classification result, and the generation of the second label classification result according to the second reference vector, comprise: setting a second classification sub-node under the first reference sub-node and the first unreferenced sub-node according to the first classification sub-node and the second to-be-managed file; wherein the second classification sub-node comprises a second reference sub-node and a second unreferenced sub-node; determining a second reference vector of the first to-be-classified data blocks and the second to-be-classified data blocks in the second to-be-managed file, and determining a second classification result corresponding to the second reference sub-node and the second unreferenced sub-node respectively according to the second reference vector; determining the second label classification result according to the second classification result.

4. The data management method based on data migration according to claim 3, characterized in that, The method further comprises: obtaining a third to-be-managed file; wherein the third to-be-managed file comprises a plurality of third to-be-classified data blocks corresponding to different categories; judging whether the number of data blocks corresponding to the second label classification result in the third to-be-classified data blocks exceeds a classification node threshold value; when the number of data blocks exceeds the classification node threshold value, maintaining the second reference sub-node or the second unreferenced sub-node corresponding to the second classification sub-node, and determining a third classification sub-result; Or, when the number of data blocks does not exceed the classification node threshold, a third classification subnode is set under the second reference subnode and the second unreferenced subnode according to the second classification subnode and the third to-be-managed file; wherein the third classification subnode comprises a third reference subnode and a third unreferenced subnode; A third reference vector of the first to-be-classified data block, the second to-be-classified data block and the third to-be-classified data block in the third to-be-managed file is determined, and a fourth classification subresult corresponding to the third reference subnode and the third unreferenced subnode is determined according to the third reference vector; A third label classification result is determined according to the third classification subresult and the fourth classification subresult.

5. The data management method based on data migration according to claim 1, characterized in that, The clustering and arrangement of the first to-be-classified data block and the second to-be-classified data block according to the second label classification result comprises: According to the second label classification result, the data block attributes of all to-be-classified data blocks in the first to-be-managed file and the second to-be-managed file are marked; wherein the data block attributes comprise valid attributes and invalid attributes, the invalid attribute data block is used to describe a data block not marked by the second label classification result, and the valid attribute is used to describe a data block marked by the second label classification result; The to-be-classified data block in the invalid attribute is deleted, and the to-be-classified data block in the valid attribute is determined as the target data block; According to the second label classification result, the target data block is attributed to different storage spaces.

6. The data management method based on data migration according to claim 1, characterized in that, The first to-be-managed file is obtained, comprising: The first file data of the first to-be-managed file is obtained, and the first data feature of the first file data is determined; According to the first data feature, the first to-be-managed file is segmented to obtain the first to-be-classified data block, and the first fingerprint index information of all first to-be-classified data blocks is determined; The clustering and arrangement of the first to-be-classified data block and the second to-be-classified data block according to the second label classification result comprises: When there is second fingerprint index information corresponding to the second to-be-classified data block, first fingerprint index information associated with the second fingerprint index information is determined; Writing of the to-be-classified data block corresponding to the first fingerprint index information in the second to-be-classified data block is stopped, and the remaining to-be-classified data block is determined as the target data block.

7. The data management method based on data migration according to claim 6, characterized in that, The clustering and arrangement of the first to-be-classified data block and the second to-be-classified data block according to the second label classification result comprises: When there is second fingerprint index information corresponding to the second to-be-classified data block, third fingerprint index information identical to the second fingerprint index information is determined; wherein the third fingerprint index information is used to indicate the to-be-classified data block in the first to-be-classified data block having the same fingerprint index information as the second fingerprint index information; Fourth label classification results of the second to-be-classified data block, the first to-be-classified data block and the to-be-classified data block corresponding to the third fingerprint index information are calculated respectively; The fourth label classification result is set as the second label classification result, and the step of clustering and arranging the first to-be-classified data block and the second to-be-classified data block is performed.

8. A data management system based on data migration, characterized by, The system comprises: An acquisition unit is configured to acquire a first to-be-managed file and a second to-be-managed file. The first to-be-managed file comprises a plurality of first to-be-classified data blocks corresponding to different categories, and the second to-be-managed file comprises a plurality of second to-be-classified data blocks corresponding to different categories. A determination unit is configured to determine a first reference vector of the first to-be-classified data block in the first to-be-managed file, and generate a first label classification result according to the first reference vector. Based on the first label classification result, a second reference vector of the second to-be-classified data block in the second to-be-managed file is determined, and a second label classification result is generated according to the second reference vector. An arrangement unit is configured to cluster and arrange the first to-be-classified data block and the second to-be-classified data block according to the second label classification result, determine a target data block with an association, and migrate and store the target data block.

9. A data management apparatus based on data migration, characterized by, The device comprises: A central processing unit, a memory, an input and output interface, a wired or wireless network interface, and a power supply; The memory is a transitory storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the data management method based on data migration according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises instructions which, when executed on a computer, cause the computer to perform the data management method based on data migration according to any one of claims 1 to 7.

Images