Data migration method and system, computer equipment and storage medium

Abstract

According to the data migration method and system, the computer equipment and the storage medium, in the data migration process, data missing detection is carried out through comparison of data migration transmission time, whether a migrated data center has complete data or not can be verified, and therefore an enterprise can conveniently carry out data migration backup, and the data migration efficiency is improved. A more reliable basis is provided for subsequent data analysis and mining; through data error detection, the accuracy of the data in the migrated data center can be ensured, and an enterprise can be assisted to make a proper business decision based on the accurate data; through data repeated detection, data redundancy can be reduced, data storage is optimized, and storage cost is reduced. Besides, in the data migration process, for the problems of network transmission delay, bandwidth bottleneck and the like, the link bandwidth between the data centers can be improved by optimizing the bandwidth configuration of the data centers or upgrading network equipment, so that the bandwidth bottleneck is broken through, and the transmission efficiency in the data migration process is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data migration method and system, a computer device and a storage medium. Background Art

[0002] With the development of information technology, global data is also growing rapidly. In the era of big data, the single-machine approach cannot meet the growing application needs. In order to cope with the exponential growth of data scale and meet the needs of massive data analysis, most companies will centrally place computing equipment, server equipment, network equipment, storage equipment, etc. to form a set of infrastructure for data processing, storage, exchange and management, namely data centers.

[0003] However, as the scale of data gradually increases, more and more distributed applications are migrated to data centers. Since data centers are distributed in different locations, the same migration task may need to transmit data between different data centers. Network transmission delays and bandwidth bottlenecks will affect the scheduling efficiency and completion time of the task, resulting in data missing, data errors, data duplication and other problems. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a data migration method and system, a computer device and a storage medium, which are used to solve the problems of data missing, data errors and data duplication in the data migration process in the prior art.

[0005] To achieve the above purpose and other related purposes, the present application provides a data migration method, comprising the following steps:

[0006] The data centers distributed in multiple geographical locations are regarded as a data center set D, D = {D1, D2, ..., Dn}, where D1 to Dn represent data centers in different geographical locations respectively; wherein the data is encapsulated into data files and stored in the data center;

[0007] The same or different data processing capabilities are configured for each data center, which are respectively denoted as PD={PD1, PD2, ..., PDn}, where PD1 represents the data processing capability of data center D1, PD2 represents the data processing capability of data center D2, and PDn represents the data processing capability of data center Dn;

[0008] Analyze the current data migration task, and select at least one data center from the data center set as the task data center according to the data processing capacity of each data center and the link bandwidth between the data centers, and record the remaining data centers in the data center set as other data centers;

[0009] Migrate the data files stored in other data centers to the task data center, and calculate the transmission time for other data centers to complete the data file migration based on the link bandwidth between the task data center and other data centers, the data files stored in other data centers, and the data processing capacity of other data centers, and record it as the reference transmission time;

[0010] Recording the actual transmission time of other data centers when the data file migration is completed, and comparing the actual transmission time with the reference transmission time;

[0011] When the actual transmission time is greater than the reference transmission time, performing data duplication detection on the data files in the task data center;

[0012] When the actual transmission time is less than the reference transmission time, performing data missing detection on the data files in the task data center;

[0013] When the actual transmission time is equal to the reference transmission time, data duplication detection, data missing detection and data error detection are performed on the data files in the task data center.

[0014] Optionally, the process of performing data missing detection on the data files in the task data center includes:

[0015] Obtain all data types of all data files stored in other data centers, recorded as reference data types; and obtain all data types of all data files in the task data center, recorded as actual data types;

[0016] Compare the reference data type with the actual data type to determine whether the actual data type is the same as the reference data type;

[0017] When the actual data type is different from the reference data type, marking that data files in the task data center are missing;

[0018] When the actual data type is the same as the reference data type, all data formats of all data files stored in other data centers are obtained, recorded as reference data formats; and all data formats of all data files in the task data center are obtained, recorded as actual data formats;

[0019] Comparing the reference data format with the actual data format to determine whether the actual data format is the same as the reference data format;

[0020] When the actual data format is different from the reference data format, marking that data files in the task data center are missing;

[0021] When the actual data format is the same as the reference data format, the total data volume of all data files stored in other data centers is obtained, which is recorded as the total reference data volume; and the total data volume of all data files in the task data center is obtained, which is recorded as the total actual data volume;

[0022] When the total amount of actual data is less than the total amount of reference data, marking that data files in the task data center have data missing;

[0023] When the total amount of actual data is greater than or equal to the total amount of reference data, data duplication detection is performed on the data files in the task data center.

[0024] Optionally, the process of performing data duplication detection on the data files in the task data center includes:

[0025] According to the geographical location of the data center, multiple data migration file sets are created in the task data center, and each data migration file set only stores data files of a single data center;

[0026] The data files stored in the data migration file set are taken as target data files, and the data files migrated to the data center of the data migration file set are recorded as data files to be migrated;

[0027] Splitting the target data file and the to-be-migrated data file according to single data to obtain multiple data segments;

[0028] According to the data splitting and sorting result, the target data file is combined with the corresponding data segments in the data file to be migrated, and the similarity between the two data segments in each combination is calculated;

[0029] When there are redundant data segments that cannot be combined in the target data file, and when there are data segment combinations whose similarity is less than a preset similarity, marking that there are redundant data in the target data file; and,

[0030] Recording a data segment combination whose similarity is less than a preset similarity as an abnormal data segment combination, selecting the next-order data segments from the target data file in turn to replace the data segments in the abnormal data segment combination, and recombining them with the data segments corresponding to the data file to be migrated, and calculating the similarity of the two data segments in the new combination;

[0031] If the similarities of all new combinations are greater than or equal to the preset similarity, the data corresponding to the replaced data segments in all abnormal data segment combinations are regarded as duplicate data;

[0032] If there are new combinations whose similarities are all less than the preset similarity, the next-order data segments are selected from the target data file to replace the data segments in the abnormal data segment combination, and are recombined with the data segments corresponding to the data file to be migrated, until the similarities of all new combinations are greater than or equal to the preset similarity, the data corresponding to the replaced data segments in all abnormal data segment combinations are treated as duplicate data.

[0033] Optionally, the process of performing data error detection on the data files in the task data center includes:

[0034] According to the geographical location of the data center, multiple data migration file sets are created in the task data center, and each data migration file set only stores data files of a single data center;

[0035] The data files stored in the data migration file set are taken as target data files, and the data files migrated to the data center of the data migration file set are recorded as data files to be migrated;

[0036] The target data file and the to-be-migrated data file are split according to the same data splitting standard to obtain a plurality of data segments; wherein the data splitting standard includes time, length, and symbol mark;

[0037] Based on the data splitting and sorting results, respectively calculating the similarities of all data segments in the target data file and the data file to be migrated;

[0038] The similarity of each data segment is compared with the preset similarity respectively. If the similarity of one or more data segments is less than the preset similarity, it is marked that the data corresponding to the one or more data segments have migration errors.

[0039] Optionally, when migrating data files stored in other data centers to the task data center, the method further includes:

[0040] Sorting each of the other data centers according to their geographical locations relative to the task data center, and setting a geographical location score for each of the other data centers based on the geographical location sorting result; and,

[0041] sorting each of the other data centers according to the link bandwidth with the task data center, and setting a link bandwidth score for each of the other data centers based on the link bandwidth sorting result; and,

[0042] sorting each of the other data centers according to data processing capability, and setting a data processing capability score for each of the other data centers based on the data processing capability sorting result; and,

[0043] sorting the data files of each data center among the other data centers according to the data file capacity, and setting a data file capacity score for each data center among the other data centers based on the sorting result of the data file capacity;

[0044] A geographical location score, a link bandwidth score, a data processing capability score, and a data file capacity score are set for each of the other data centers for weighted calculation to obtain a comprehensive score for each of the other data centers;

[0045] Based on the comprehensive score of each data center in other data centers from high to low, the stored data files are migrated to the task data center in sequence.

[0046] Optionally, the data type of the data file includes at least one of an integer type, a floating point type, a string type, a list type, a dictionary type, a Boolean type, a tuple type, and a set type.

[0047] Optionally, the data format of the data file includes at least one of a text format, a binary format, a database format, a markup language format and a serialization format.

[0048] The present application also provides a data migration system, the system comprising:

[0049] A data center module is used to treat data centers distributed in multiple geographical locations as a data center set D, where D = {D1, D2, ..., Dn}, where D1 to Dn represent data centers in different geographical locations respectively; wherein data is encapsulated into data files and stored in the data center;

[0050] The data processing capability module is used to configure the same or different data processing capabilities for each data center, which are respectively denoted as PD={PD1, PD2, ..., PDn}, where PD1 represents the data processing capability of the data center D1, PD2 represents the data processing capability of the data center D2, and PDn represents the data processing capability of the data center Dn;

[0051] A task parsing module, used to parse the current data migration task, and select at least one data center from the data center set as the task data center according to the data processing capacity of each data center and the link bandwidth between data centers, and record the remaining data centers in the data center set as other data centers;

[0052] A data migration module is used to migrate data files stored in other data centers to the task data center, and calculate the transmission time for other data centers to complete the data file migration based on the link bandwidth between the task data center and other data centers, the data files stored in other data centers, and the data processing capacity of other data centers, and record it as a reference transmission time; and record the actual transmission time of other data centers when the data file migration is completed, and compare the actual transmission time with the reference transmission time;

[0053] A data migration detection module is used to perform data duplication detection on the data files in the task data center when the actual transmission time is greater than the reference transmission time; or to perform data missing detection on the data files in the task data center when the actual transmission time is less than the reference transmission time; or to perform data duplication detection, data missing detection and data error detection on the data files in the task data center when the actual transmission time is equal to the reference transmission time.

[0054] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the data migration methods described above.

[0055] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the above-mentioned data migration methods are implemented.

[0056] As described above, the present application provides a data migration method and system, a computer device and a storage medium, which have the following beneficial effects: first, data centers distributed in multiple geographical locations are regarded as a data center set, and then the same or different data processing capabilities are configured for each data center; then the current data migration task is parsed, and according to the data processing capability of each data center and the link bandwidth between the data centers, at least one data center is selected from the data center set as the task data center, and the remaining data centers in the data center set are recorded as other data centers; then the data files stored in the other data centers are migrated to the task data center, and according to the link bandwidth between the task data center and the other data centers, the data files stored in the other data centers are According to the file and the data processing capacity of other data centers, calculate the transmission time for other data centers to complete the data file migration, and record it as the reference transmission time; at the same time, record the actual transmission time of other data centers when the data file migration is completed, and compare the actual transmission time with the reference transmission time; when the actual transmission time is greater than the reference transmission time, perform data duplication detection on the data files in the task data center; or, when the actual transmission time is less than the reference transmission time, perform data missing detection on the data files in the task data center; or, when the actual transmission time is equal to the reference transmission time, perform data duplication detection, data missing detection and data error detection on the data files in the task data center. It can be seen that in the process of data migration, the present application performs data missing detection by comparing the data migration transmission time, which can verify whether the migrated data center has complete data, thereby facilitating the enterprise to perform data migration backup and provide a more reliable basis for subsequent data analysis and mining; at the same time, by performing data error detection, it can not only ensure the accuracy of the data in the migrated data center, but also assist the enterprise to make appropriate business decisions based on accurate data; and, by performing data duplication detection, it can reduce data redundancy, optimize data storage, and reduce storage costs. In addition, during the data migration process of the present application, problems such as network transmission delay and bandwidth bottleneck can be solved by optimizing the bandwidth configuration of the data center or upgrading the network equipment to increase the link bandwidth between data centers, thereby breaking through the bandwidth bottleneck and improving the transmission efficiency of the data migration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of a data migration method provided in an embodiment of the present application;

[0058] Figure 2 A schematic diagram of calculating a reference transmission time provided by an embodiment of the present application;

[0059] Figure 3A schematic diagram of the data migration principle provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of the hardware structure of a data migration system provided in one embodiment of the present application.

[0061] Figure 5 A schematic diagram of the hardware structure of a terminal device provided by an embodiment;

[0062] Figure 6 A schematic diagram of the hardware structure of a terminal device provided in another embodiment. DETAILED DESCRIPTION

[0063] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0064] It should be noted that the illustrations provided in this embodiment only illustrate the basic concept of the present application in a schematic manner, and therefore the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0065] See also Figure 1 As shown, this embodiment provides a data migration method, comprising the following steps:

[0066] S110, the data centers distributed in multiple geographical locations are taken as a data center set D, D = {D1, D2, ..., Dn}, where D1 ~ Dn represent data centers in different geographical locations respectively, and the data is encapsulated into data files and stored in the data center. In this embodiment or other embodiments, the data encapsulated into data files includes but is not limited to log data, code data, application data, software data, etc. In this embodiment or other embodiments, the geographical location can be divided according to the actual longitude and latitude, or according to the administrative region. In this embodiment or other embodiments, the data centers in the data center set can be managed by a central controller and connected through heterogeneous network links, and each link has a certain link bandwidth. As an example, for example, the geographical location in this embodiment includes but is not limited to WZ District, C City, YB District, C City, KZ County, G City, GA District, G City, YC County, etc., where the data center in WZ District, C City can be called D1, the data center in YB District, C City can be called D2, the data center in KZ County, C City can be called D3, the data center in GA District, G City can be called D4, and the data center in YC County, G City can be called D5.

[0067] S120, configure the same or different data processing capabilities for each data center, respectively recorded as PD = {PD1, PD2, ..., PDn}, where PD1 represents the data processing capability of data center D1, PD2 represents the data processing capability of data center D2, and PDn represents the data processing capability of data center Dn. In this embodiment or other embodiments, data processing capability may refer to the ability of a data center to reasonably collect, organize, and analyze data, extract valuable information from the acquired data, make judgments, and make reasonable decisions.

[0068] S130, parsing the current data migration task, and selecting at least one data center from the data center set as the task data center according to the data processing capacity of each data center and the link bandwidth between the data centers, and recording the remaining data centers in the data center set as other data centers. As an example, in this embodiment, after the staff initiates a data migration job request, the central processor responds to the data migration job request, and then extracts the corresponding data migration task according to the data migration job request response result, obtains the data migration task at the current moment, and records it as the current data migration task. At the same time, the central processor parses the current data migration task, and then selects at least one data center from the data center set as the task data center to execute the current data migration task according to the data processing capacity of each data center and the link bandwidth between the data centers, and records the remaining data centers in the data center set as other data centers. Among them, when selecting the task data center, the central processor can select the most suitable data center according to the processing capacity and link bandwidth of each data center, as well as the resource requirements and execution time of each data migration task, so as to ensure that the data migration task is correctly allocated and executed. In this embodiment or other embodiments, how the central processor specifically selects the task data center can be determined according to the actual application scenario, and will not be repeated here. As an example, in this embodiment, data center D1 in WZ district of city C can be used as the task data center, and data center D2 in YB district of city C, data center D3 in KZ county of city C, data center D4 in GA district of city G, data center D5 in YC county of city G and other data centers can be used as other data centers.

[0069] S140, migrate the data files stored in other data centers to the task data center, and calculate the transmission time for other data centers to complete the data file migration based on the link bandwidth between the task data center and other data centers, the data files stored in other data centers, and the data processing capacity of other data centers, and record it as the reference transmission time. Figure 2As shown, there are two data centers, denoted as data center 1 and data center 2. The data processing capacity of data center 1 is 50MB / s, and the data processing capacity of data center 2 is 40MB / s. The two data centers are linked by a link bandwidth of 50MB / s, that is, data center 1 and data center 2 are linked by a link bandwidth of 50MB / s. Data file f1 is stored in data center 1, and data file f2 is stored in data center 2. The file size of data file f1 is 40MB, and the file size of data file f2 is 50MB. At a certain time, when the data migration task starts to request data files, data migration task t1 needs to migrate data file f1, and data migration task t2 needs to migrate data file f2. At this time, if the CPU assigns data migration task t1 to data center 2 and data migration task t2 to data center 1, that is, when executing data migration task t1, data center 2 is the task data center, and data center 1 is the other data center at this time, the reference transmission time corresponding to data migration task t1 is: (file size of data file f1) / (data processing capability of data center 1)+(file size of data file f1) / (link bandwidth between data center 1 and data center 2)=40 / 50+40 / 50=1.6 seconds. When executing data migration task t2, data center 2 is the other data center, and data center 1 is the task data center at this time, the reference transmission time corresponding to data migration task t2 is: (file size of data file f2) / (data processing capability of data center 2)+(file size of data file f2) / (link bandwidth between data center 1 and data center 2)=50 / 50+50 / 50=2 seconds.

[0070] S150, recording the actual transmission time of other data centers when the data file migration is completed, and comparing the actual transmission time with the reference transmission time;

[0071] S160-1, when the actual transmission time is greater than the reference transmission time, performing data duplication detection on the data files in the task data center;

[0072] S160-2, when the actual transmission time is less than the reference transmission time, performing data missing detection on the data files in the task data center;

[0073] S160-3, when the actual transmission time is equal to the reference transmission time, data duplication detection, data missing detection and data error detection are performed on the data files in the task data center.

[0074] According to the above records, in an exemplary embodiment of the present application, the process of performing data loss detection on the data files in the task data center includes:

[0075] All data types of all data files stored in other data centers are obtained and recorded as reference data types; and all data types of all data files in the task data center are obtained and recorded as actual data types. In this embodiment or other embodiments, the data types of data files include but are not limited to integer type, floating point type, string type, list type, dictionary type, Boolean type, tuple type and set type.

[0076] Compare the reference data type with the actual data type to determine whether the actual data type is the same as the reference data type;

[0077] When the actual data type is different from the reference data type, the data files in the task data center are marked as missing data. If the actual data type is different from the reference data type, it means that the data types corresponding to all data files in the task data center do not include all data types corresponding to all data files in other data centers, which can indicate that data files in the task data center are missing data types. As an example, for example, the reference data type includes integer type, floating point type, string type, list type, dictionary type, Boolean type, tuple type and set type, and the actual data type includes integer type, floating point type, list type, dictionary type, Boolean type, tuple type and set type. Since the actual data type lacks a string type compared to the reference data type, it means that data files in the task data center are missing data types.

[0078] When the actual data type is the same as the reference data type, all data formats of all data files stored in other data centers are obtained and recorded as reference data formats; and all data formats of all data files in the task data center are obtained and recorded as actual data formats. In this embodiment or other embodiments, the data format of the data file includes but is not limited to text format, binary format, database format, markup language format and serialization format. Among them, the text format includes but is not limited to CSV (Comma-Separated Values, CSV for short) format, TSV (Tab-Separated Values, TSV for short) format, JSON (JavaScript Object Notation, JSON for short) format, XML (eXtensible MarkupLanguage, XML for short) format, YAML (YAML Ain't Markup Language, YAML for short) format, etc. Binary formats include but are not limited to binary formats suitable for Excel, AVI, MP4, MKV, PDF, PEG, PNG, GIF and other files. Database formats include but are not limited to SQL, MySQL and other data formats, etc. Markup language formats include but are not limited to HTML / CSS / JavaScript formats, LaTeX formats, Markdown formats, etc. Serialization formats include but are not limited to Protobuf (Protocol Buffers) format, Avro format, Thrift format, etc.

[0079] Compare the reference data format with the actual data format to determine whether the actual data format is the same as the reference data format;

[0080] When the actual data format is different from the reference data format, the data files in the task data center are marked as missing data. If the actual data format is different from the reference data format, it means that the data formats corresponding to all data files in the task data center do not include all data formats corresponding to all data files in other data centers, which can indicate that data files in the task data center are missing data. As an example, for example, the reference data format includes text format, binary format, database format, markup language format and serialization format, and the actual data format includes text format, binary format, database format and serialization format. Since the actual data format lacks a markup language format compared to the reference data format, it means that data files in the task data center are missing data formats.

[0081] When the actual data format is the same as the reference data format, the total data volume of all data files stored in other data centers is obtained, which is recorded as the total reference data volume; and the total data volume of all data files in the task data center is obtained, which is recorded as the total actual data volume;

[0082] When the total amount of actual data is less than the total amount of reference data, the data files in the task data center are marked as missing data. If the total amount of actual data is less than the total amount of reference data, it means that the total amount of data corresponding to all data files in the task data center does not include the total amount of data corresponding to all data files in other data centers, which can indicate that there are missing data in the data files in the task data center.

[0083] When the total amount of actual data is greater than or equal to the total amount of reference data, data duplication detection is performed on the data files in the mission data center. If the total amount of actual data is greater than the total amount of reference data, it means that the total amount of data corresponding to all data files in the mission data center exceeds the total amount of data corresponding to all data files in other data centers, which indicates that data files in the mission data center may have data duplication.

[0084] It can be seen that during the data migration process, data missing detection can be performed by comparing the data migration transmission time, which can verify whether the data center after migration has complete data, thereby facilitating the enterprise to perform data migration backup and provide a more reliable foundation for subsequent data analysis and mining.

[0085] According to the above records, in an exemplary embodiment of the present application, the process of performing data duplication detection on the data files in the task data center includes:

[0086] According to the geographical location of the data center, multiple data migration file sets are created in the task data center, and each data migration file set only stores data files of a single data center;

[0087] The data files stored in the data migration file set are taken as target data files, and the data files migrated to the data center of the data migration file set are recorded as data files to be migrated;

[0088] The target data file and the data file to be migrated are split according to single data to obtain multiple data segments. In this embodiment or other embodiments, when the target data file and the data file to be migrated are split according to single data, one data segment obtained can be represented as one data.

[0089] According to the data splitting and sorting results, the target data file is combined with the corresponding data segments in the data file to be migrated, and the similarity of the two data segments in each combination is calculated. In this embodiment or other embodiments, the Euclidean distance or cosine distance of the two data segments can be calculated as the similarity of the two data segments.

[0090] When there are redundant data segments that cannot be combined in the target data file, and there are data segment combinations whose similarity is less than a preset similarity, marking that there are redundant data in the target data file; and,

[0091] Recording a data segment combination whose similarity is less than a preset similarity as an abnormal data segment combination, selecting the next-order data segments from the target data file in turn to replace the data segments in the abnormal data segment combination, and recombining them with the data segments corresponding to the data file to be migrated, and calculating the similarity of the two data segments in the new combination;

[0092] If the similarities of all new combinations are greater than or equal to the preset similarity, the data corresponding to the replaced data segments in all abnormal data segment combinations are regarded as duplicate data;

[0093] If there are new combinations whose similarities are all less than the preset similarity, the next-order data segments are selected from the target data file to replace the data segments in the abnormal data segment combination, and they are recombined with the data segments corresponding to the data file to be migrated, until the similarities of all new combinations are greater than or equal to the preset similarity, the data corresponding to the replaced data segments in all abnormal data segment combinations are treated as duplicate data.

[0094] It can be seen that during the data migration process, by performing data duplication detection, data redundancy can be reduced, data storage can be optimized, and storage costs can be reduced.

[0095] According to the above records, in an exemplary embodiment of the present application, the process of performing data error detection on the data files in the task data center includes:

[0096] According to the geographical location of the data center, multiple data migration file sets are created in the task data center, and each data migration file set only stores data files of a single data center;

[0097] The data files stored in the data migration file set are taken as target data files, and the data files migrated to the data center of the data migration file set are recorded as data files to be migrated;

[0098] The target data file and the data file to be migrated are split according to the same data splitting standard to obtain multiple data segments; wherein the data splitting standard includes time, length, and symbol mark. In this embodiment or other embodiments, when the target data file and the data file to be migrated are split according to time, length, and symbol mark, a data segment obtained can be represented as a segment of one or more data with a certain length.

[0099] Based on the data splitting and sorting results, the similarities of all data segments in the target data file and the data file to be migrated are calculated respectively;

[0100] The similarity of each data segment is compared with the preset similarity respectively. If the similarity of one or more data segments is less than the preset similarity, it is marked that the data corresponding to the one or more data segments have migration errors.

[0101] From this, we can see that during the data migration process, by performing data error detection, we can not only ensure the accuracy of the data in the migrated data center, but also assist enterprises in making appropriate business decisions based on accurate data.

[0102] According to the above records, in an exemplary embodiment of the present application, when migrating data files stored in other data centers to the task data center, it can include: sorting each data center in the other data centers according to the geographical location with the task data center, and setting a geographical location score for each data center in the other data centers based on the geographical location sorting result; and, sorting each data center in the other data centers according to the link bandwidth with the task data center, and setting a link bandwidth score for each data center in the other data centers based on the link bandwidth sorting result; and, sorting each data center in the other data centers according to the data processing capacity, and setting a data processing capacity score for each data center in the other data centers based on the data processing capacity sorting result; and, sorting each data center in the other data centers according to the data file capacity, and setting a data file capacity score for each data center in the other data centers based on the data file capacity sorting result; setting a geographical location score, a link bandwidth score, a data processing capacity score and a data file capacity score for each data center in the other data centers, and performing weighted calculation to obtain a comprehensive score for each data center in the other data centers; and migrating the stored data files to the task data center in order from high to low based on the comprehensive score of each data center in the other data centers.

[0103] According to the above records, in an exemplary embodiment of the present application, if the actual transmission time when other data centers migrate stored data files to task data is greater than the reference transmission time, this situation may be caused by problems such as network transmission delay and bandwidth bottleneck. At this time, the bandwidth configuration of the data center can be optimized or the network equipment can be upgraded to increase the link bandwidth between the data centers, thereby breaking through the bandwidth bottleneck and improving the transmission efficiency of the data migration process. As an example, for example, the bandwidth resources are divided into normal bandwidth and idle bandwidth, and the normal bandwidth is used to be normally allocated on the links between data centers. When the actual transmission time corresponding to a certain link or a plurality of links is greater than the reference transmission time, the central processor can schedule the idle bandwidth allocation on these links whose actual transmission time is greater than the reference transmission time, so that the link bandwidth of these links can appear in a certain period of time. Normal configuration bandwidth + idle scheduling bandwidth, thereby improving the link bandwidth between data centers, breaking through the bandwidth bottleneck, and improving the transmission efficiency of the data migration process.

[0104] According to the above records, in a specific example, the example provides a data migration principle process, such as Figure 3 As shown, the central controller mainly pre-processes the migration task job requests, and extracts and parses the data migration tasks based on these job requests. At the same time, the central controller can consider the data processing capacity and link bandwidth of each data center, as well as the resource requirements and execution time of each data migration task, and select at least one data center from the data center set as the task data center, and place the data migration task in the task data center for execution to ensure that the task is correctly allocated and executed.

[0105] The data migration unit is mainly responsible for migrating data files from other data centers to the mission data center. At the same time, based on the link bandwidth between the mission data center and other data centers, the data files stored in other data centers, and the data processing capabilities of other data centers, the data migration unit calculates the transmission time for other data centers to complete the data file migration, which is recorded as the reference transmission time; and records the actual transmission time of other data centers when the data file migration is completed, and compares the actual transmission time with the reference transmission time. At the same time, the data migration unit feeds back the actual transmission time and the reference transmission time to the central controller, which performs data duplication detection, data missing detection, and data error detection on the data files in the mission data center.

[0106] Data centers mainly store data files and provide data processing capabilities to perform data migration tasks. Each data center has the same or different data processing capabilities. Figure 3 As shown, data center D1 serves as the task data center, and then data center D2, data center D3, data center D4, data center D5 and other data centers serve as other data centers.

[0107] It can be seen that during the entire data migration process, data migration tasks and data files can be properly scheduled and allocated so that the data migration tasks can be executed in the appropriate data center. At the same time, issues such as data file transmission and storage also need to be considered to ensure the performance and efficiency of the entire data migration process.

[0108] In summary, the present application provides a data migration method, which first regards data centers distributed in multiple geographical locations as a data center set, and then configures the same or different data processing capabilities for each data center; then analyzes the current data migration task, and selects at least one data center from the data center set as the task data center according to the data processing capability of each data center and the link bandwidth between the data centers, and records the remaining data centers in the data center set as other data centers; then migrates the data files stored in other data centers to the task data center, and calculates the transmission time for other data centers to complete the data file migration based on the link bandwidth between the task data center and other data centers, the data files stored in other data centers and the data processing capabilities of other data centers, and records it as a reference transmission time; at the same time, records the actual transmission time of other data centers when the data file migration is completed, and compares the actual transmission time with the reference transmission time; when the actual transmission time is greater than the reference transmission time, performs data duplication detection on the data files in the task data center; or, when the actual transmission time is less than the reference transmission time, performs data missing detection on the data files in the task data center; or, when the actual transmission time is equal to the reference transmission time, performs data duplication detection, data missing detection and data error detection on the data files in the task data center. It can be seen that during the data migration process, this method can verify whether the migrated data center has complete data by comparing the data migration transmission time to detect data missing, so as to facilitate the enterprise to perform data migration backup and provide a more reliable basis for subsequent data analysis and mining; at the same time, by performing data error detection, it can not only ensure the accuracy of the data in the migrated data center, but also assist the enterprise to make appropriate business decisions based on accurate data; and, by performing data duplication detection, it can reduce data redundancy, optimize data storage, and reduce storage costs. In addition, during the data migration process, this method can also optimize the bandwidth configuration of the data center or upgrade the network equipment to improve the link bandwidth between data centers to break through the bandwidth bottleneck and improve the transmission efficiency of the data migration process.

[0109] In another exemplary embodiment of the present application, Figure 4As shown, a data migration system is provided, comprising:

[0110] The data center module 410 is used to treat data centers distributed in multiple geographical locations as a data center set D, D = {D1, D2, ..., Dn}, where D1 ~ Dn represent data centers in different geographical locations respectively; wherein the data is encapsulated into data files and stored in the data center. In this embodiment or other embodiments, the data encapsulated into data files includes but is not limited to log data, code data, application data, software data, etc. In this embodiment or other embodiments, the geographical location can be divided according to the actual longitude and latitude, or according to the administrative region. In this embodiment or other embodiments, the data centers in the data center set can be managed by a central controller and connected through heterogeneous network links, and each link has a certain link bandwidth. As an example, for example, the geographical location in this embodiment includes but is not limited to WZ District, C City, YB District, C City, KZ County, G City, GA District, G City, YC County, etc., where the data center in WZ District, C City can be called D1, the data center in YB District, C City can be called D2, the data center in KZ County, C City can be called D3, the data center in GA District, G City can be called D4, and the data center in YC County, G City can be called D5.

[0111] The data processing capability module 420 is used to configure the same or different data processing capabilities for each data center, which are respectively recorded as PD={PD1, PD2, ..., PDn}, where PD1 represents the data processing capability of the data center D1, PD2 represents the data processing capability of the data center D2, and PDn represents the data processing capability of the data center Dn. In this embodiment or other embodiments, the data processing capability may refer to the ability of the data center to reasonably collect, organize, and analyze data, extract valuable information from the obtained data, make judgments, and make reasonable decisions.

[0112] The task parsing module 430 is used to parse the current data migration task, and select at least one data center from the data center set as the task data center according to the data processing capacity of each data center and the link bandwidth between the data centers, and record the remaining data centers in the data center set as other data centers. As an example, in this embodiment, after the staff initiates a data migration job request, the central processor responds to the data migration job request, and then extracts the corresponding data migration task according to the data migration job request response result, obtains the data migration task at the current moment, and records it as the current data migration task. At the same time, the central processor parses the current data migration task, and then selects at least one data center from the data center set as the task data center to execute the current data migration task according to the data processing capacity of each data center and the link bandwidth between the data centers, and records the remaining data centers in the data center set as other data centers. Among them, when selecting the task data center, the central processor can select the most suitable data center according to the processing capacity and link bandwidth of each data center, as well as the resource requirements and execution time of each data migration task, so as to ensure that the data migration task is correctly allocated and executed. In this embodiment or other embodiments, how the central processor specifically selects the task data center can be determined according to the actual application scenario, which will not be repeated here. As an example, in this embodiment, data center D1 in WZ district of city C can be used as the task data center, and data center D2 in YB district of city C, data center D3 in KZ county of city C, data center D4 in GA district of city G, data center D5 in YC county of city G and other data centers can be used as other data centers.

[0113] The data migration module 440 is used to migrate the data files stored in other data centers to the task data center, and calculate the transmission time for other data centers to complete the data file migration based on the link bandwidth between the task data center and other data centers, the data files stored in other data centers, and the data processing capabilities of other data centers, and record it as the reference transmission time; and record the actual transmission time of other data centers when the data file migration is completed, and compare the actual transmission time with the reference transmission time. As an example, if Figure 2As shown, there are two data centers, denoted as data center 1 and data center 2. The data processing capacity of data center 1 is 50MB / s, and the data processing capacity of data center 2 is 40MB / s. The two data centers are linked by a link bandwidth of 50MB / s, that is, data center 1 and data center 2 are linked by a link bandwidth of 50MB / s. Data file f1 is stored in data center 1, and data file f2 is stored in data center 2. The file size of data file f1 is 40MB, and the file size of data file f2 is 50MB. At a certain time, when the data migration task starts to request data files, data migration task t1 needs to migrate data file f1, and data migration task t2 needs to migrate data file f2. At this time, if the CPU assigns data migration task t1 to data center 2 and data migration task t2 to data center 1, that is, when executing data migration task t1, data center 2 is the task data center, and data center 1 is the other data center at this time, the reference transmission time corresponding to data migration task t1 is: (file size of data file f1) / (data processing capability of data center 1)+(file size of data file f1) / (link bandwidth between data center 1 and data center 2)=40 / 50+40 / 50=1.6 seconds. When executing data migration task t2, data center 2 is the other data center, and data center 1 is the task data center at this time, the reference transmission time corresponding to data migration task t2 is: (file size of data file f2) / (data processing capability of data center 2)+(file size of data file f2) / (link bandwidth between data center 1 and data center 2)=50 / 50+50 / 50=2 seconds.

[0114] The data migration detection module 450 is used to perform data duplication detection on the data files in the task data center when the actual transmission time is greater than the reference transmission time; or, when the actual transmission time is less than the reference transmission time, perform data missing detection on the data files in the task data center; or, when the actual transmission time is equal to the reference transmission time, perform data duplication detection, data missing detection and data error detection on the data files in the task data center.

[0115] According to the above records, in an exemplary embodiment of the present application, the process of performing data loss detection on the data files in the task data center includes:

[0116] All data types of all data files stored in other data centers are obtained and recorded as reference data types; and all data types of all data files in the task data center are obtained and recorded as actual data types. In this embodiment or other embodiments, the data types of data files include but are not limited to integer type, floating point type, string type, list type, dictionary type, Boolean type, tuple type and set type.

[0117] Compare the reference data type with the actual data type to determine whether the actual data type is the same as the reference data type;

[0118] When the actual data type is different from the reference data type, the data files in the task data center are marked as missing data. If the actual data type is different from the reference data type, it means that the data types corresponding to all data files in the task data center do not include all data types corresponding to all data files in other data centers, which can indicate that data files in the task data center are missing data types. As an example, for example, the reference data type includes integer type, floating point type, string type, list type, dictionary type, Boolean type, tuple type and set type, and the actual data type includes integer type, floating point type, list type, dictionary type, Boolean type, tuple type and set type. Since the actual data type lacks a string type compared to the reference data type, it means that data files in the task data center are missing data types.

[0119] When the actual data type is the same as the reference data type, all data formats of all data files stored in other data centers are obtained and recorded as reference data formats; and all data formats of all data files in the task data center are obtained and recorded as actual data formats. In this embodiment or other embodiments, the data format of the data file includes but is not limited to text format, binary format, database format, markup language format and serialization format. Among them, the text format includes but is not limited to CSV (Comma-Separated Values, CSV for short) format, TSV (Tab-Separated Values, TSV for short) format, JSON (JavaScript Object Notation, JSON for short) format, XML (eXtensible MarkupLanguage, XML for short) format, YAML (YAML Ain't Markup Language, YAML for short) format, etc. Binary formats include but are not limited to binary formats suitable for Excel, AVI, MP4, MKV, PDF, PEG, PNG, GIF and other files. Database formats include but are not limited to SQL, MySQL and other data formats, etc. Markup language formats include but are not limited to HTML / CSS / JavaScript formats, LaTeX formats, Markdown formats, etc. Serialization formats include but are not limited to Protobuf (Protocol Buffers) format, Avro format, Thrift format, etc.

[0120] Compare the reference data format with the actual data format to determine whether the actual data format is the same as the reference data format;

[0121] When the actual data format is different from the reference data format, the data files in the task data center are marked as missing data. If the actual data format is different from the reference data format, it means that the data formats corresponding to all data files in the task data center do not include all data formats corresponding to all data files in other data centers, which can indicate that data files in the task data center are missing data. As an example, for example, the reference data format includes text format, binary format, database format, markup language format and serialization format, and the actual data format includes text format, binary format, database format and serialization format. Since the actual data format lacks a markup language format compared to the reference data format, it means that data files in the task data center are missing data formats.

[0122] When the actual data format is the same as the reference data format, the total data volume of all data files stored in other data centers is obtained, which is recorded as the total reference data volume; and the total data volume of all data files in the task data center is obtained, which is recorded as the total actual data volume;

[0123] When the total amount of actual data is less than the total amount of reference data, the data files in the task data center are marked as missing data. If the total amount of actual data is less than the total amount of reference data, it means that the total amount of data corresponding to all data files in the task data center does not include the total amount of data corresponding to all data files in other data centers, which can indicate that there are missing data in the data files in the task data center.

[0124] When the total amount of actual data is greater than or equal to the total amount of reference data, data duplication detection is performed on the data files in the mission data center. If the total amount of actual data is greater than the total amount of reference data, it means that the total amount of data corresponding to all data files in the mission data center exceeds the total amount of data corresponding to all data files in other data centers, which indicates that data files in the mission data center may have data duplication.

[0125] It can be seen that during the data migration process, data missing detection can be performed by comparing the data migration transmission time, which can verify whether the data center after migration has complete data, thereby facilitating the enterprise to perform data migration backup and provide a more reliable foundation for subsequent data analysis and mining.

[0126] According to the above records, in an exemplary embodiment of the present application, the process of performing data duplication detection on the data files in the task data center includes:

[0127] According to the geographical location of the data center, multiple data migration file sets are created in the task data center, and each data migration file set only stores data files of a single data center;

[0128] The data files stored in the data migration file set are taken as target data files, and the data files migrated to the data center of the data migration file set are recorded as data files to be migrated;

[0129] The target data file and the data file to be migrated are split according to single data to obtain multiple data segments. In this embodiment or other embodiments, when the target data file and the data file to be migrated are split according to single data, one data segment obtained can be represented as one data.

[0130] According to the data splitting and sorting results, the target data file is combined with the corresponding data segments in the data file to be migrated, and the similarity of the two data segments in each combination is calculated. In this embodiment or other embodiments, the Euclidean distance or cosine distance of the two data segments can be calculated as the similarity of the two data segments.

[0131] When there are redundant data segments that cannot be combined in the target data file, and there are data segment combinations whose similarity is less than a preset similarity, marking that there are redundant data in the target data file; and,

[0132] Recording a data segment combination whose similarity is less than a preset similarity as an abnormal data segment combination, selecting the next-order data segments from the target data file in turn to replace the data segments in the abnormal data segment combination, and recombining them with the data segments corresponding to the data file to be migrated, and calculating the similarity of the two data segments in the new combination;

[0133] If the similarities of all new combinations are greater than or equal to the preset similarity, the data corresponding to the replaced data segments in all abnormal data segment combinations are regarded as duplicate data;

[0134] If there are new combinations whose similarities are all less than the preset similarity, the next-order data segments are selected from the target data file to replace the data segments in the abnormal data segment combination, and they are recombined with the data segments corresponding to the data file to be migrated, until the similarities of all new combinations are greater than or equal to the preset similarity, the data corresponding to the replaced data segments in all abnormal data segment combinations are treated as duplicate data.

[0135] It can be seen that during the data migration process, by performing data duplication detection, data redundancy can be reduced, data storage can be optimized, and storage costs can be reduced.

[0136] According to the above records, in an exemplary embodiment of the present application, the process of performing data error detection on the data files in the task data center includes:

[0137] According to the geographical location of the data center, multiple data migration file sets are created in the task data center, and each data migration file set only stores data files of a single data center;

[0138] The data files stored in the data migration file set are taken as target data files, and the data files migrated to the data center of the data migration file set are recorded as data files to be migrated;

[0139] The target data file and the data file to be migrated are split according to the same data splitting standard to obtain multiple data segments; wherein the data splitting standard includes time, length, and symbol mark. In this embodiment or other embodiments, when the target data file and the data file to be migrated are split according to time, length, and symbol mark, a data segment obtained can be represented as a segment of one or more data with a certain length.

[0140] Based on the data splitting and sorting results, the similarities of all data segments in the target data file and the data file to be migrated are calculated respectively;

[0141] The similarity of each data segment is compared with the preset similarity respectively. If the similarity of one or more data segments is less than the preset similarity, it is marked that the data corresponding to the one or more data segments have migration errors.

[0142] From this, we can see that during the data migration process, by performing data error detection, we can not only ensure the accuracy of the data in the migrated data center, but also assist enterprises in making appropriate business decisions based on accurate data.

[0143] According to the above records, in an exemplary embodiment of the present application, when migrating data files stored in other data centers to the task data center, it can include: sorting each data center in the other data centers according to the geographical location with the task data center, and setting a geographical location score for each data center in the other data centers based on the geographical location sorting result; and, sorting each data center in the other data centers according to the link bandwidth with the task data center, and setting a link bandwidth score for each data center in the other data centers based on the link bandwidth sorting result; and, sorting each data center in the other data centers according to the data processing capacity, and setting a data processing capacity score for each data center in the other data centers based on the data processing capacity sorting result; and, sorting each data center in the other data centers according to the data file capacity, and setting a data file capacity score for each data center in the other data centers based on the data file capacity sorting result; setting a geographical location score, a link bandwidth score, a data processing capacity score and a data file capacity score for each data center in the other data centers, and performing weighted calculation to obtain a comprehensive score for each data center in the other data centers; and migrating the stored data files to the task data center in order from high to low based on the comprehensive score of each data center in the other data centers.

[0144] According to the above records, in an exemplary embodiment of the present application, if the actual transmission time when other data centers migrate stored data files to task data is greater than the reference transmission time, this situation may be caused by problems such as network transmission delay and bandwidth bottleneck. At this time, the bandwidth configuration of the data center can be optimized or the network equipment can be upgraded to increase the link bandwidth between the data centers, thereby breaking through the bandwidth bottleneck and improving the transmission efficiency of the data migration process. As an example, for example, the bandwidth resources are divided into normal bandwidth and idle bandwidth, and the normal bandwidth is used to be normally allocated on the links between data centers. When the actual transmission time corresponding to a certain link or a plurality of links is greater than the reference transmission time, the central processor can schedule the idle bandwidth allocation on these links whose actual transmission time is greater than the reference transmission time, so that the link bandwidth of these links can appear in a certain period of time. Normal configuration bandwidth + idle scheduling bandwidth, thereby improving the link bandwidth between data centers, breaking through the bandwidth bottleneck, and improving the transmission efficiency of the data migration process.

[0145] According to the above records, in a specific example, the example provides a data migration principle process, such as Figure 3As shown, the central controller mainly pre-processes the migration task job requests, and extracts and parses the data migration tasks based on these job requests. At the same time, the central controller can consider the data processing capacity and link bandwidth of each data center, as well as the resource requirements and execution time of each data migration task, and select at least one data center from the data center set as the task data center, and place the data migration task in the task data center for execution to ensure that the task is correctly allocated and executed.

[0146] The data migration unit is mainly responsible for migrating data files from other data centers to the mission data center. At the same time, based on the link bandwidth between the mission data center and other data centers, the data files stored in other data centers, and the data processing capabilities of other data centers, the data migration unit calculates the transmission time for other data centers to complete the data file migration, which is recorded as the reference transmission time; and records the actual transmission time of other data centers when the data file migration is completed, and compares the actual transmission time with the reference transmission time. At the same time, the data migration unit feeds back the actual transmission time and the reference transmission time to the central controller, which performs data duplication detection, data missing detection, and data error detection on the data files in the mission data center.

[0147] Data centers mainly store data files and provide data processing capabilities to perform data migration tasks. Each data center has the same or different data processing capabilities. Figure 3 As shown, data center D1 serves as the task data center, and then data center D2, data center D3, data center D4, data center D5 and other data centers serve as other data centers.

[0148] It can be seen that during the entire data migration process, data migration tasks and data files can be properly scheduled and allocated so that the data migration tasks can be executed in the appropriate data center. At the same time, issues such as data file transmission and storage also need to be considered to ensure the performance and efficiency of the entire data migration process.

[0149] In summary, the present application provides a data migration system, which first treats data centers distributed in multiple geographical locations as a data center set, and then configures the same or different data processing capabilities for each data center; then analyzes the current data migration task, and selects at least one data center from the data center set as the task data center according to the data processing capability of each data center and the link bandwidth between the data centers, and records the remaining data centers in the data center set as other data centers; then migrates the data files stored in other data centers to the task data center, and calculates the transmission time for other data centers to complete the data file migration based on the link bandwidth between the task data center and other data centers, the data files stored in other data centers, and the data processing capabilities of other data centers, and records it as a reference transmission time; at the same time, records the actual transmission time of other data centers when the data file migration is completed, and compares the actual transmission time with the reference transmission time; when the actual transmission time is greater than the reference transmission time, performs data duplication detection on the data files in the task data center; or, when the actual transmission time is less than the reference transmission time, performs data missing detection on the data files in the task data center; or, when the actual transmission time is equal to the reference transmission time, performs data duplication detection, data missing detection and data error detection on the data files in the task data center. It can be seen that during the data migration process, this system can detect missing data by comparing the data migration transmission time, and verify whether the migrated data center has complete data, so as to facilitate the enterprise to perform data migration backup and provide a more reliable basis for subsequent data analysis and mining; at the same time, by performing data error detection, it can not only ensure the accuracy of the data in the migrated data center, but also assist the enterprise to make appropriate business decisions based on accurate data; and, by performing data duplication detection, it can reduce data redundancy, optimize data storage, and reduce storage costs. In addition, during the data migration process, this system can also optimize the bandwidth configuration of the data center or upgrade the network equipment to improve the link bandwidth between data centers to break through the bandwidth bottleneck and improve the transmission efficiency of the data migration process.

[0150] It can be explained that the data migration system provided in the above embodiment and the data migration method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the data migration system provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the system into different functional modules to complete all or part of the functions described above, which is not limited here. Therefore, the present application effectively overcomes various shortcomings in the prior art and has a high industrial utilization value.

[0151] In an exemplary embodiment of the present application, a computer device is further provided. The computer device may include a memory, a processor, and a computer program stored in the memory. The processor executes the computer program so that the computer device performs the following steps: Figure 1 The data migration method. In practical applications, the computer device can be used as a terminal device or as a server. Examples of terminal devices may include: smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc. The embodiments of the present application do not limit specific computer devices.

[0152] In an exemplary embodiment of the present application, a computer-readable storage medium is further provided, wherein a computer program / instruction is stored in the storage medium, and when the computer program / instruction is executed by a processor, the above-mentioned device can execute the above-mentioned Figure 1 The data migration method described.

[0153] Figure 5 This is a schematic diagram of the hardware structure of a terminal device provided in one embodiment of the present application. Figure 5 As shown, the terminal device may include: an input device 1100, a first processor 1101, an output device 1102, a first memory 1103 and at least one communication bus 1104. The communication bus 1104 is used to realize the communication connection between the components. The first memory 1103 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage. Various programs may be stored in the first memory 1103 to complete various processing functions and implement the method steps of this embodiment.

[0154] As an example, the first processor 1101 may be implemented as a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components. The first processor 1101 is coupled to the input device 1100 and the output device 1102 via a wired or wireless connection.

[0155] As an example, the input device 1100 may include multiple input devices, such as a user interface for users, a device interface for devices, a software programmable interface, a camera, and a sensor. Optionally, the device interface for devices may be a wired interface for data transmission between devices, or a hardware insertion interface for data transmission between devices (such as a USB interface, a serial port, etc.); Optionally, the user interface for users may be, for example, a control button for users, a voice input device for receiving voice input, and a touch sensing device for users to receive user touch input (such as a touch screen with a touch sensing function, a touchpad, etc.); Optionally, the software programmable interface may be, for example, an entry for users to edit or modify programs, such as an input pin interface or an input interface of a chip; the output device 1102 may include output devices such as a display and a speaker.

[0156] In this embodiment, the processor of the terminal device includes functions for executing each module of the voice recognition device in each device. The specific functions and technical effects can be referred to the above embodiments and will not be repeated here.

[0157] Figure 6 A schematic diagram of the hardware structure of a terminal device provided for one embodiment of the present application. Figure 6 Yes Figure 5 A specific embodiment in the implementation process. As shown in the figure, the terminal device of this embodiment may include a second processor 1201 and a second memory 1202.

[0158] The second processor 1201 executes the computer program code stored in the second memory 1202 to implement the above embodiment. Figure 1 The method.

[0159] The second memory 1202 is configured to store various types of data to support operations on the terminal device. Examples of such data include instructions for any application or method used to operate on the terminal device, such as messages, pictures, videos, etc. The second memory 1202 may include a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage.

[0160] As an example, the second processor 1201 is set in the processing component 1200. The terminal device may also include: a communication component 1203, a power component 1204, a multimedia component 1205, a voice component 1206, an input / output interface 1207 and / or a sensor component 1208. The specific components included in the terminal device are set according to actual needs, and this embodiment does not limit this.

[0161] The processing component 1200 generally controls the overall operation of the terminal device. The processing component 1200 may include one or more second processors 1201 to execute instructions to complete all or part of the steps in the above-mentioned data processing method. In addition, the processing component 1200 may include one or more modules to facilitate the interaction between the processing component 1200 and other components. For example, the processing component 1200 may include a multimedia module to facilitate the interaction between the multimedia component 1205 and the processing component 1200.

[0162] The power supply component 1204 provides power to various components of the terminal device. The power supply component 1204 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0163] The multimedia component 1205 includes a display screen that provides an output interface between the terminal device and the user. In some embodiments, the display screen may include a liquid crystal display (LCD) and a touch panel (TP). If the display screen includes a touch panel, the display screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0164] The voice component 1206 is configured to output and / or input voice signals. For example, the voice component 1206 includes a microphone (MIC), and when the terminal device is in an operating mode, such as a voice recognition mode, the microphone is configured to receive an external voice signal. The received voice signal can be further stored in the second memory 1202 or sent via the communication component 1203. In some embodiments, the voice component 1206 also includes a speaker for outputting voice signals.

[0165] The input / output interface 1207 provides an interface between the processing component 1200 and the peripheral interface modules, which may be click wheels, buttons, etc. These buttons may include but are not limited to: volume buttons, start buttons, and lock buttons.

[0166] The sensor component 1208 includes one or more sensors for providing various aspects of status assessment for the terminal device. For example, the sensor component 1208 can detect the open / closed state of the terminal device, the relative positioning of the components, and the presence or absence of contact between the user and the terminal device. The sensor component 1208 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, the sensor component 1208 may also include a camera, etc.

[0167] The communication component 1203 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In one embodiment, the terminal device may include a SIM card slot for inserting a SIM card, so that the terminal device can log in to a GPRS network and establish communication with a server via the Internet.

[0168] From the above, we can see that Figure 6 The communication component 1203, the voice component 1206, the input / output interface 1207, and the sensor component 1208 involved in the embodiment can all be used as Figure 5 Implementation method of the input device in the embodiment.

[0169] It is understandable that the above embodiments collect, store, use, process, transmit, provide, disclose, delete, etc. related data (such as log data, code data, application data, software data, etc.) with or with the consent of the user. For example, log data, code data, application data, software data, etc. are authorized with the knowledge and consent of the user; or are actively provided by the user after reading the relevant instructions, or are actively authorized / provided / uploaded by the user when using some or all of the functions described in the above embodiments, or are obtained in other ways / channels with or with the consent of the user.

[0170] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A data migration method, characterized in that: The method comprises the following steps: The data centers distributed in multiple geographical locations are regarded as a data center set D, D = {D1, D2, ..., Dn}, where D1 to Dn represent data centers in different geographical locations respectively; wherein the data is encapsulated into data files and stored in the data center; The same or different data processing capabilities are configured for each data center, which are respectively denoted as PD={PD1, PD2, ..., PDn}, where PD1 represents the data processing capability of data center D1, PD2 represents the data processing capability of data center D2, and PDn represents the data processing capability of data center Dn; Analyze the current data migration task, and select at least one data center from the data center set as the task data center according to the data processing capacity of each data center and the link bandwidth between the data centers, and record the remaining data centers in the data center set as other data centers; Migrate the data files stored in other data centers to the task data center, and calculate the transmission time for other data centers to complete the data file migration based on the link bandwidth between the task data center and other data centers, the data files stored in other data centers, and the data processing capacity of other data centers, and record it as the reference transmission time; Recording the actual transmission time of other data centers when the data file migration is completed, and comparing the actual transmission time with the reference transmission time; When the actual transmission time is greater than the reference transmission time, performing data duplication detection on the data files in the task data center; When the actual transmission time is less than the reference transmission time, performing data missing detection on the data files in the task data center; When the actual transmission time is equal to the reference transmission time, data duplication detection, data missing detection and data error detection are performed on the data files in the task data center.

2. The data migration method according to claim 1, characterized in that: The process of performing data missing detection on the data files in the task data center includes: Obtain all data types of all data files stored in other data centers, recorded as reference data types; and obtain all data types of all data files in the task data center, recorded as actual data types; Compare the reference data type with the actual data type to determine whether the actual data type is the same as the reference data type; When the actual data type is different from the reference data type, marking that data files in the task data center are missing; When the actual data type is the same as the reference data type, all data formats of all data files stored in other data centers are obtained, recorded as reference data formats; and all data formats of all data files in the task data center are obtained, recorded as actual data formats; Comparing the reference data format with the actual data format to determine whether the actual data format is the same as the reference data format; When the actual data format is different from the reference data format, marking that data files in the task data center are missing; When the actual data format is the same as the reference data format, the total data volume of all data files stored in other data centers is obtained, which is recorded as the total reference data volume; and the total data volume of all data files in the task data center is obtained, which is recorded as the total actual data volume; When the total amount of actual data is less than the total amount of reference data, marking that data files in the task data center have data missing; When the total amount of actual data is greater than or equal to the total amount of reference data, data duplication detection is performed on the data files in the task data center.

3. The data migration method according to claim 1 or 2, characterized in that: The process of performing data duplication detection on the data files in the task data center includes: According to the geographical location of the data center, multiple data migration file sets are created in the task data center, and each data migration file set only stores data files of a single data center; The data files stored in the data migration file set are taken as target data files, and the data files migrated to the data center of the data migration file set are recorded as data files to be migrated; Splitting the target data file and the to-be-migrated data file according to single data to obtain multiple data segments; According to the data splitting and sorting result, the target data file is combined with the corresponding data segments in the data file to be migrated, and the similarity between the two data segments in each combination is calculated; When there are redundant data segments that cannot be combined in the target data file, and when there are data segment combinations whose similarity is less than a preset similarity, marking that there are redundant data in the target data file; and, Recording a data segment combination whose similarity is less than a preset similarity as an abnormal data segment combination, selecting the next-order data segments from the target data file in turn to replace the data segments in the abnormal data segment combination, and recombining them with the data segments corresponding to the data file to be migrated, and calculating the similarity of the two data segments in the new combination; If the similarities of all new combinations are greater than or equal to the preset similarity, the data corresponding to the replaced data segments in all abnormal data segment combinations are regarded as duplicate data; If there are new combinations whose similarities are all less than the preset similarity, the next-order data segments are selected from the target data file to replace the data segments in the abnormal data segment combination, and are recombined with the data segments corresponding to the data file to be migrated, until the similarities of all new combinations are greater than or equal to the preset similarity, the data corresponding to the replaced data segments in all abnormal data segment combinations are treated as duplicate data.

4. The data migration method according to claim 1, characterized in that: The process of performing data error detection on the data files in the task data center includes: According to the geographical location of the data center, multiple data migration file sets are created in the task data center, and each data migration file set only stores data files of a single data center; The data files stored in the data migration file set are taken as target data files, and the data files migrated to the data center of the data migration file set are recorded as data files to be migrated; The target data file and the to-be-migrated data file are split according to the same data splitting standard to obtain a plurality of data segments; wherein the data splitting standard includes time, length, and symbol mark; Based on the data splitting and sorting results, respectively calculating the similarities of all data segments in the target data file and the data file to be migrated; The similarity of each data segment is compared with the preset similarity respectively. If the similarity of one or more data segments is less than the preset similarity, it is marked that the data corresponding to the one or more data segments have migration errors.

5. The data migration method according to claim 1, characterized in that: When migrating data files stored in other data centers to the task data center, the method further includes: Sorting each of the other data centers according to their geographical locations relative to the task data center, and setting a geographical location score for each of the other data centers based on the geographical location sorting result; and, sorting each of the other data centers according to the link bandwidth with the task data center, and setting a link bandwidth score for each of the other data centers based on the link bandwidth sorting result; and, sorting each of the other data centers according to data processing capability, and setting a data processing capability score for each of the other data centers based on the data processing capability sorting result; and, sorting the data files of each data center among the other data centers according to the data file capacity, and setting a data file capacity score for each data center among the other data centers based on the sorting result of the data file capacity; A geographical location score, a link bandwidth score, a data processing capability score, and a data file capacity score are set for each of the other data centers for weighted calculation to obtain a comprehensive score for each of the other data centers; Based on the comprehensive score of each data center in other data centers from high to low, the stored data files are migrated to the task data center in sequence.

6. The data migration method according to claim 2, characterized in that: The data type of the data file includes at least one of an integer type, a floating point type, a string type, a list type, a dictionary type, a Boolean type, a tuple type, and a set type.

7. The data migration method according to claim 2, characterized in that: The data format of the data file includes at least one of a text format, a binary format, a database format, a markup language format, and a serialization format.

8. A data migration system, characterized in that: The system comprises: A data center module is used to treat data centers distributed in multiple geographical locations as a data center set D, where D = {D1, D2, ..., Dn}, where D1 to Dn represent data centers in different geographical locations respectively; wherein data is encapsulated into data files and stored in the data center; The data processing capability module is used to configure the same or different data processing capabilities for each data center, which are respectively denoted as PD={PD1, PD2, ..., PDn}, where PD1 represents the data processing capability of the data center D1, PD2 represents the data processing capability of the data center D2, and PDn represents the data processing capability of the data center Dn; A task parsing module, used to parse the current data migration task, and select at least one data center from the data center set as the task data center according to the data processing capacity of each data center and the link bandwidth between data centers, and record the remaining data centers in the data center set as other data centers; A data migration module is used to migrate data files stored in other data centers to the task data center, and calculate the transmission time for other data centers to complete the data file migration based on the link bandwidth between the task data center and other data centers, the data files stored in other data centers, and the data processing capacity of other data centers, and record it as a reference transmission time; and record the actual transmission time of other data centers when the data file migration is completed, and compare the actual transmission time with the reference transmission time; A data migration detection module is used to perform data duplication detection on the data files in the task data center when the actual transmission time is greater than the reference transmission time; or to perform data missing detection on the data files in the task data center when the actual transmission time is less than the reference transmission time; or to perform data duplication detection, data missing detection and data error detection on the data files in the task data center when the actual transmission time is equal to the reference transmission time.

9. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data migration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the data migration method described in any one of claims 1 to 7 are implemented.

Images