Cross-port data migration management system and method

Abstract

The invention discloses a cross-port data migration management system and method, and relates to the technical field of data migration. According to the method, the representative fingerprints BL are selected after the similar data are clustered, the representative fingerprints BL are fingerprints b with the minimum fingerprint value, and the representative fingerprints BL form the representative fingerprint set, so that the fingerprint comparison range is expanded, the probability of finding repeated fingerprints is improved, similarity calculation does not need to be carried out on each piece of data once, and the accuracy of fingerprint comparison is improved. The cluster similar to the to-be-input data in mouth can be found in the buffer database for de-duplication, so that the cluster most similar to the to-be-processed data can be efficiently positioned in each de-duplication operation; after the buffered data fingerprints are clustered, the fingerprints are classified in a layered manner through the classification labels, so that the fingerprint comparison amount in the similarity calculation process is reduced, the hit rate of the fingerprints accessing data in the future is increased, the query efficiency of fingerprint indexes in a buffer database is ensured while the maximum utilization of the memory space is ensured, and the user experience is improved. And the performance of deleting the repeated data is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a cross-port data migration management system and method. Background Art

[0002] Cross-port data migration is achieved by solving system compatibility issues during cross-network transmission. Transmission software is set up in both the source port and the target port, and standardized data formats and protocols are used to ensure that the cross-port data migration management system correctly parses and processes data.

[0003] In the prior art, frequent data backup in the source database within the source port often results in the generation of a large amount of redundant data. The existing cross-port data migration management system will delete duplicate data in the source database before the source data is output, which will increase the preprocessing time required for the early stage of data migration. Duplicate data is generally identified with the help of data fingerprints, but duplicate data is not identified in multiple storage levels, and it is impossible to ensure that only one copy of the same data is retained in different storage layers, resulting in low efficiency in identifying duplicate data. Summary of the invention

[0004] The purpose of the present invention is to provide a cross-port data migration management system and method to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a cross-port data migration management method, the method comprising:

[0006] Step S1, synchronizing the source data of the source port database to the buffer database in real time;

[0007] Step S2: Synchronize the updated data of the source port database to the buffer database in real time through the source port synchronization tool, and remove duplicate data in the data processing module during the synchronization process;

[0008] The step S2 of removing duplicate data during the synchronization process includes:

[0009] Step S21, the data block clustering unit divides the data in the input source database into blocks to obtain data blocks a, and merges them to obtain a data block set a+. In the data block set a+, every n data blocks a are aggregated to obtain a super block A+, and merged to obtain a super block set A++;

[0010] Step S22: The fingerprint block clustering unit calculates the fingerprint b of each data block a and adds the fingerprint b to the fingerprint sequence B. The smallest fingerprint b in each super block A+ minAs the representative fingerprint BL of the super block A+, all representative fingerprints BL are recorded in the super block set A++; when the number of all fingerprints b in the super block set A++ reaches the set value M, the similarity S of all representative fingerprints BL in the super block set A++ is calculated, and the similarity S is compared with the threshold. If the similarity S ≥ the threshold, step S23 is performed; if the similarity S < the threshold, step S24 is performed;

[0011] Step S23: determine that the cluster is similar to the data to be input, select the fingerprint sequence B of the most similar super block set A++ to form a cluster, and use the representative fingerprint BL of the selected super block set A++ as the cluster center. Extract the fingerprint b in the super block set A++ into the data comparison unit and perform fingerprint comparison; if the fingerprint b in the comparison unit is the same as the fingerprint b in the fingerprint sequence B, delete the same fingerprint b in the fingerprint sequence B; if they are different, replace the different fingerprint b min Store in fingerprint sequence B, fingerprint sequence B corresponds to the fingerprint index; repeat step S22 until all super blocks A+ are processed; compare whether the size of all fingerprint sequences B exceeds the maximum cache; if exceeded, proceed to step S25; if not exceeded, end.

[0012] The deduplication of data during synchronization in step S2 further includes:

[0013] Step S24: determine that the cluster is not similar to the data to be input, merge the fingerprints b in all current super block sets A++ into a new cluster and cluster center, update the fingerprint sequence B, store all data and fingerprints b in the super block set A++; compare whether the size of all fingerprint sequences B exceeds the maximum cache; if so, proceed to step S15; if not, end.

[0014] The deduplication of data during synchronization in step S2 further includes:

[0015] Step S25, the cache optimization module optimizes the selected cluster centers to be optimized according to the characteristics of the cluster centers, retains the cluster center with the highest data access rate for the next wave of writing, and thereby adjusts the number of fingerprint indexes in the data comparison unit.

[0016] The step S25 comprises:

[0017] Step S251: The future data access unit classifies each cluster according to the characteristics of the data set and adds classification labels to construct a data set with labels, and normalizes all the characteristic values ​​in the labeled data set to obtain the final data set E. After evaluating the accuracy of the classification, the data set e with the highest classification accuracy in the data set E is recorded as the optimal data set;

[0018] Step S252, repeatedly inputting the optimal data set into step S151 for classification, executing the cycle, and iteratively updating the optimal data set;

[0019] Step S153: When the loop is completed, the final future data access model will be output.

[0020] The calculation formula of the similarity S is:

[0021] When there is no fingerprint sequence B1 to be recorded in the super block set A++:

[0022] Where i represents the i-th fingerprint sequence B, j represents the j-th fingerprint sequence B, and the fingerprint sequence B is obtained. i and fingerprint sequence B j The similarity S;

[0023] When the fingerprint sequence B1 is to be recorded in the super block set A++:

[0024] Among them, G is a cluster, k represents a cluster G k , G h Represents cluster G k The cluster center of B represents the fingerprint sequence, which is the fingerprint sequence to be recorded at this time, and B k Indicates the representative fingerprint sequence of the fingerprint to be recorded, B j Represents the j fingerprints in the fingerprint sequence to be recorded.

[0025] Step S3: In the buffer database, the data processing module performs data migration processing on the buffer data, wherein the data migration processing includes transaction splitting, data format conversion and data synchronization;

[0026] Step S4: Finally, the buffered data is synchronized to the target port database in real time through the target port synchronization tool.

[0027] The characteristics of the cluster center include the access frequency of the cluster, the periodic access frequency, the cluster creation time and the latest access time; the classification label includes the cluster category, the cluster occupied storage, the cluster center fingerprint standard deviation and whether to delete.

[0028] A cross-port data migration management system, the system comprising:

[0029] The connected source port database, buffer database and destination port database;

[0030] The data migration module is used to synchronize the source data of the source port database to the buffer database in real time; synchronize the updated data of the source port database to the buffer database in real time through the source port synchronization tool, and the data processing module removes duplicate data during the synchronization process; in the buffer database, the data processing module performs data migration processing on the buffer data, and the data migration processing includes transaction splitting, data format conversion and data synchronization; and synchronize the buffer data to the target port database in real time through the target port synchronization tool.

[0031] The data processing module comprises:

[0032] The data block clustering unit is used to divide the data in the input source database into blocks to obtain data blocks a, merge them to obtain a data block set a+, and in the data block set a+, every n data blocks a are aggregated to obtain a super block A+, and merged to obtain a super block set A++;

[0033] The fingerprint block clustering unit is used to calculate the fingerprint b of each data block a, calculate the similarity S of all representative fingerprints BL in the super block set A++, compare the similarity S with the threshold, if the similarity S≥threshold, proceed to step S23; if the similarity S<threshold, proceed to step S24;

[0034] The data comparison unit is used to compare the fingerprint b in the super block set A++; if the fingerprint b in the comparison unit is the same as the fingerprint b in the fingerprint sequence B, the same fingerprint b in the fingerprint sequence B is deleted; if they are different, the different fingerprint b is replaced. min Stored in fingerprint sequence B, fingerprint sequence B corresponds to the fingerprint index.

[0035] The data processing module also includes:

[0036] The future data access unit is used to classify each cluster according to the characteristics of the data set and add classification labels, build a labeled data set, and normalize all the characteristic values ​​in the labeled data set to obtain the final data set E. After evaluating the accuracy of the classification, the data group e with the highest classification accuracy in the data set E is recorded as the optimal data set.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention selects a representative fingerprint BL after clustering similar data. The representative fingerprint BL is the fingerprint b with the smallest fingerprint value. Several representative fingerprints BL constitute a representative fingerprint collection. The number of representative fingerprints BL is much smaller than the number of fingerprints in the fingerprint sequence B, which expands the scope of fingerprint comparison and improves the probability of finding repeated fingerprints. Then, representative data is selected. It is not necessary to perform a similarity calculation on each data. Clusters similar to the data to be entered in the buffer database can be found for deduplication. The larger the similarity value, the higher the similarity, so that each deduplication operation can efficiently locate the cluster most similar to the data to be processed.

[0039] 2. After the buffered data fingerprints are clustered, the present invention classifies the fingerprints in layers by clustering category, clustering storage occupation, clustering center fingerprint standard deviation and classification label of whether to delete, thereby reducing the amount of fingerprint comparison in the similarity calculation process, improving the hit rate of fingerprints for future access to data, ensuring the query efficiency of fingerprint indexes in the buffer database while ensuring maximum utilization of memory space, timely eliminating clustering centers that exceed the capacity of the cache database, and ensuring the performance of deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of a cross-port data migration management method of the present invention;

[0041] Figure 2 The present invention is a schematic diagram of the structure of a cross-port data migration management system. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example: Figure 1 As shown, the present invention provides a cross-port data migration management method, the method comprising:

[0044] Step S1, synchronizing the source data of the source port database to the buffer database in real time;

[0045] Step S2: Synchronize the updated data of the source port database to the buffer database in real time through the source port synchronization tool, and remove duplicate data in the data processing module during the synchronization process;

[0046] Step S3: In the buffer database, the data processing module performs data migration processing on the buffer data, wherein the data migration processing includes transaction splitting, data format conversion and data synchronization;

[0047] Step S4: Finally, the buffered data is synchronized to the target port database in real time through the target port synchronization tool.

[0048] The step S2 of removing duplicate data during the synchronization process includes:

[0049] Step S21, the data block clustering unit divides the data in the input source database into blocks to obtain data blocks a, and merges them to obtain a data block set a+. In the data block set a+, every n data blocks a are aggregated to obtain a super block A+, and merged to obtain a super block set A++;

[0050] Step S22: The fingerprint block clustering unit calculates the fingerprint b of each data block a and adds the fingerprint b to the fingerprint sequence B. The smallest fingerprint b in each super block A+ min As the representative fingerprint BL of the super block A+, when the minimum fingerprints in the two fingerprint sequences B are equal, the two fingerprint sequences B have a considerable number of repeated fingerprints, and the number of representative fingerprints BL is much smaller than the number of fingerprints in the fingerprint sequence B, which expands the scope of fingerprint comparison and improves the probability of finding repeated fingerprints. Then, representative data is selected and all representative fingerprints BL are recorded in the super block set A++; when the number of all fingerprints b in the super block set A++ reaches the set value M, the similarity S of all representative fingerprints BL in the super block set A++ is calculated, and the similarity S is compared with the threshold. If the similarity S ≥ the threshold, step S23 is performed; if the similarity S < the threshold, step S24 is performed;

[0051] Step S23: determine that the cluster is similar to the data to be input, select the fingerprint sequence B of the most similar super block set A++ to form a cluster, and use the representative fingerprint BL of the selected super block set A++ as the cluster center. Extract the fingerprint b in the super block set A++ into the data comparison unit and perform fingerprint comparison; if the fingerprint b in the comparison unit is the same as the fingerprint b in the fingerprint sequence B, delete the same fingerprint b in the fingerprint sequence B; if they are different, replace the different fingerprint b min Store in fingerprint sequence B, fingerprint sequence B corresponds to the fingerprint index; repeat step S22 until all super blocks A+ are processed; compare whether the size of all fingerprint sequences B exceeds the maximum cache; if exceeded, proceed to step S25; if not exceeded, end. It is not necessary to perform a similarity calculation for each data, and the clusters in the buffer database that are similar to the data to be entered can be found for deduplication. The larger the similarity value, the higher the similarity, so that each deduplication operation can efficiently locate the cluster that is most similar to the data to be processed.

[0052] The deduplication of data during synchronization in step S2 further includes:

[0053] Step S24: determine that the cluster is not similar to the data to be input, merge the fingerprints b in all current super block sets A++ into a new cluster and cluster center, update the fingerprint sequence B, store all data and fingerprints b in the super block set A++; compare whether the size of all fingerprint sequences B exceeds the maximum cache; if so, proceed to step S15; if not, end.

[0054] The deduplication of data during synchronization in step S2 further includes:

[0055] Step S25, the cache optimization module optimizes the selected cluster centers to be optimized according to the characteristics of the cluster centers, retains the cluster center with the highest data access rate for the next wave of writing, and thereby adjusts the number of fingerprint indexes in the data comparison unit.

[0056] The step S25 comprises:

[0057] Step S251, the future data access unit classifies each cluster according to the characteristics of the data set and adds classification labels, constructs a labeled data set, and normalizes all the characteristic values ​​in the labeled data set to obtain the final data set E. After evaluating the accuracy of the classification, the data group e with the highest classification accuracy in the data set E is recorded as the optimal data set; by hierarchical classification of fingerprints b, the amount of fingerprint comparison in the similarity calculation process is reduced, thereby improving the hit rate of fingerprints for future access to data.

[0058] Step S252, repeatedly inputting the optimal data set into step S151 for classification, executing the cycle, and iteratively updating the optimal data set;

[0059] Step S153: After the cycle is completed, the final future data access model is outputted. While ensuring the maximum utilization of memory space, the query efficiency of fingerprint index in the buffer database is ensured, and the clustering centers that exceed the capacity of the cache database are eliminated in time to ensure the performance of deduplication.

[0060] The calculation formula of the similarity S is:

[0061] When there is no fingerprint sequence B1 to be recorded in the super block set A++:

[0062] Where i represents the i-th fingerprint sequence B, j represents the j-th fingerprint sequence B, and the fingerprint sequence B is obtained. i and fingerprint sequence B j The similarity S;

[0063] When the fingerprint sequence B1 is to be recorded in the super block set A++:

[0064] Among them, G is a cluster, k represents a cluster G k , G h Represents cluster G k The cluster center of B represents the fingerprint sequence, which is the fingerprint sequence to be recorded at this time, and B k Indicates the representative fingerprint sequence of the fingerprint to be recorded, B j Represents the j fingerprints in the fingerprint sequence to be recorded.

[0065] The characteristics of the cluster center include the access frequency of the cluster, the periodic access frequency, the cluster creation time and the latest access time; the classification label includes the cluster category, the cluster occupied storage, the cluster center fingerprint standard deviation and whether to delete.

[0066] Figure 2 , a cross-port data migration management system, the system comprising:

[0067] The connected source port database, buffer database and destination port database;

[0068] The data migration module is used to synchronize the source data of the source port database to the buffer database in real time; synchronize the updated data of the source port database to the buffer database in real time through the source port synchronization tool, and the data processing module removes duplicate data during the synchronization process; in the buffer database, the data processing module performs data migration processing on the buffer data, and the data migration processing includes transaction splitting, data format conversion and data synchronization; and synchronize the buffer data to the target port database in real time through the target port synchronization tool.

[0069] The data processing module comprises:

[0070] The data block clustering unit is used to divide the data in the input source database into blocks to obtain data blocks a, merge them to obtain a data block set a+, and in the data block set a+, every n data blocks a are aggregated to obtain a super block A+, and merged to obtain a super block set A++;

[0071] The fingerprint block clustering unit is used to calculate the fingerprint b of each data block a, calculate the similarity S of all representative fingerprints BL in the super block set A++, compare the similarity S with the threshold, if the similarity S≥threshold, proceed to step S23; if the similarity S<threshold, proceed to step S24;

[0072] The data comparison unit is used to compare the fingerprint b in the super block set A++; if the fingerprint b in the comparison unit is the same as the fingerprint b in the fingerprint sequence B, the same fingerprint b in the fingerprint sequence B is deleted; if they are different, the different fingerprint b is replaced. min Stored in fingerprint sequence B, fingerprint sequence B corresponds to the fingerprint index.

[0073] The data processing module also includes:

[0074] The future data access unit is used to classify each cluster according to the characteristics of the data set and add classification labels, build a labeled data set, and normalize all the characteristic values ​​in the labeled data set to obtain the final data set E. After evaluating the accuracy of the classification, the data group e with the highest classification accuracy in the data set E is recorded as the optimal data set.

[0075] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A cross-port data migration management method, characterized in that: The method includes: Step S1, synchronizing the source data of the source port database to the buffer database in real time; Step S2: Synchronize the updated data of the source port database to the buffer database in real time through the source port synchronization tool, and remove duplicate data in the data processing module during the synchronization process; Step S3: In the buffer database, the data processing module performs data migration processing on the buffer data, wherein the data migration processing includes transaction splitting, data format conversion and data synchronization; Step S4: Finally, the buffered data is synchronized to the target port database in real time through the target port synchronization tool.

2. A cross-port data migration management method according to claim 1, characterized in that: The step S2 of removing duplicate data during the synchronization process includes: Step S21, the data block clustering unit divides the data in the input source database into blocks to obtain data blocks a, and merges them to obtain a data block set a+. In the data block set a+, every n data blocks a are aggregated to obtain a super block A+, and merged to obtain a super block set A++; Step S22: The fingerprint block clustering unit calculates the fingerprint b of each data block a and adds the fingerprint b to the fingerprint sequence B. The smallest fingerprint b in each super block A+ min As the representative fingerprint BL of the super block A+, all representative fingerprints BL are recorded in the super block set A++; when the number of all fingerprints b in the super block set A++ reaches the set value M, the similarity S of all representative fingerprints BL in the super block set A++ is calculated, and the similarity S is compared with the threshold value. If the similarity S ≥ the threshold value, step S23 is performed; if the similarity S < the threshold value, step S24 is performed; Step S23: determine that the cluster is similar to the data to be input, select the fingerprint sequence B of the most similar super block set A++ to form a cluster, and use the representative fingerprint BL of the selected super block set A++ as the cluster center. Extract the fingerprint b in the super block set A++ into the data comparison unit and perform fingerprint comparison; if the fingerprint b in the comparison unit is the same as the fingerprint b in the fingerprint sequence B, delete the same fingerprint b in the fingerprint sequence B; if they are different, replace the different fingerprint b min Store in fingerprint sequence B, fingerprint sequence B corresponds to the fingerprint index; repeat step S22 until all super blocks A+ are processed; compare whether the size of all fingerprint sequences B exceeds the maximum cache; if exceeded, proceed to step S25; if not exceeded, end.

3. A cross-port data migration management method according to claim 2, characterized in that: The deduplication of data during synchronization in step S2 further includes: Step S24: determine that the cluster is not similar to the data to be input, merge the fingerprints b in all current super block sets A++ into a new cluster and cluster center, update the fingerprint sequence B, store all data and fingerprints b in the super block set A++; compare whether the size of all fingerprint sequences B exceeds the maximum cache; if so, proceed to step S15; if not, end.

4. A cross-port data migration management method according to claim 3, characterized in that: The deduplication of data during synchronization in step S2 further includes: Step S25, the cache optimization module optimizes the selected cluster centers to be optimized according to the characteristics of the cluster centers, retains the cluster center with the highest data access rate for the next wave of writing, and thereby adjusts the number of fingerprint indexes in the data comparison unit.

5. The cross-port data migration management method according to claim 1, characterized in that: The step S25 comprises: Step S251: The future data access unit classifies each cluster according to the characteristics of the data set and adds classification labels to construct a data set with labels, and normalizes all the characteristic values ​​in the labeled data set to obtain the final data set E. After evaluating the accuracy of the classification, the data set e with the highest classification accuracy in the data set E is recorded as the optimal data set; Step S252, repeatedly inputting the optimal data set into step S151 for classification, executing the cycle, and iteratively updating the optimal data set; Step S153: When the loop is completed, the final future data access model will be output.

6. A cross-port data migration management method according to claim 2, characterized in that: The calculation formula of the similarity S is: When there is no fingerprint sequence B1 to be recorded in the super block set A++: Where i represents the i-th fingerprint sequence B, j represents the j-th fingerprint sequence B, and the fingerprint sequence B is obtained. i and fingerprint sequence B j The similarity S; When the fingerprint sequence B1 is to be recorded in the super block set A++: Among them, G is a cluster, k represents a cluster G k , G h Represents cluster G k The cluster center of B represents the fingerprint sequence, which is the fingerprint sequence to be recorded at this time, and B k Indicates the representative fingerprint sequence of the fingerprint to be recorded, B j Represents the j fingerprints in the fingerprint sequence to be recorded.

7. A cross-port data migration management method according to claim 1, characterized in that: The characteristics of the cluster center include the access frequency of the cluster, the periodic access frequency, the cluster creation time and the latest access time; the classification label includes the cluster category, the cluster occupied storage, the cluster center fingerprint standard deviation and whether to delete.

8. A cross-port data migration management system, applied to a cross-port data migration management method according to claims 1-7, characterized in that: The system includes: The connected source port database, buffer database and destination port database; The data migration module is used to synchronize the source data of the source port database to the buffer database in real time; synchronize the updated data of the source port database to the buffer database in real time through the source port synchronization tool, and the data processing module removes duplicate data during the synchronization process; in the buffer database, the data processing module performs data migration processing on the buffer data, and the data migration processing includes transaction splitting, data format conversion and data synchronization; and synchronize the buffer data to the target port database in real time through the target port synchronization tool.

9. The cross-port data migration management system according to claim 8, characterized in that: The data processing module comprises: The data block clustering unit is used to divide the data in the input source database into blocks to obtain data blocks a, merge them to obtain a data block set a+, and in the data block set a+, every n data blocks a are aggregated to obtain a super block A+, and merged to obtain a super block set A++; The fingerprint block clustering unit is used to calculate the fingerprint b of each data block a, calculate the similarity S of all representative fingerprints BL in the super block set A++, compare the similarity S with the threshold, if the similarity S≥threshold, proceed to step S23; if the similarity S<threshold, proceed to step S24; The data comparison unit is used to compare the fingerprint b in the super block set A++; if the fingerprint b in the comparison unit is the same as the fingerprint b in the fingerprint sequence B, the same fingerprint b in the fingerprint sequence B is deleted; if they are different, the different fingerprint b is replaced. min Stored in fingerprint sequence B, fingerprint sequence B corresponds to the fingerprint index.

10. The cross-port data migration management system according to claim 9, characterized in that: The data processing module also includes: The future data access unit is used to classify each cluster according to the characteristics of the data set and add classification labels, build a labeled data set, and normalize all the characteristic values ​​in the labeled data set to obtain the final data set E. After evaluating the accuracy of the classification, the data group e with the highest classification accuracy in the data set E is recorded as the optimal data set.

Images