Data migration method and system, medium and electronic equipment

Abstract

The invention belongs to the technical field of data migration, and particularly relates to a data migration method and system, a medium and electronic equipment, and the method comprises the steps: obtaining a source medium data volume and a target medium data volume; transferring source medium data into a target medium, performing data cleaning, and executing data migration; comparing the data volume of the source medium with the data volume of the target medium after data migration, and performing preliminary data integrity verification of data migration; after the preliminary data integrity verification is passed, the cleaned data is divided into duplicate value deletion and abnormal value deletion, data integrity verification of data migration is carried out, and data migration is completed; when the cleaned data is duplicated value deletion, re-comparing whether the same duplicated data exists in the target medium after data migration; and when the cleaned data is abnormal value deletion, analyzing the metadata of the deleted abnormal value data, finding out the timestamp of the abnormal value, and confirming the authenticity of the abnormal value.

Description

Technical Field

[0001] The present invention belongs to the technical field of data migration, and in particular relates to a data migration method, system, medium and electronic equipment. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of information technology, databases, as core components of information systems, play a vital role in data storage, management and analysis. Data migration refers to the process of transferring data stored in one storage system or database to another storage system or database. With the development of business and the updating of technology, data migration has become an indispensable part of enterprise IT systems. Data migration usually involves the transmission, conversion and verification of large amounts of data. It is crucial to ensure the integrity, accuracy and consistency of data during the migration process. The purpose of data migration may be to upgrade the database system, integrate multiple data sources, optimize the data storage structure or meet regulatory compliance requirements.

[0004] Although data migration technology is constantly developing, existing data migration methods still have some significant shortcomings; among them, the problem of data cleaning is particularly prominent. In the process of data migration, in order to ensure the quality and consistency of data, it is usually necessary to clean the source data. However, existing data cleaning methods are often too simple and crude, and it is easy to misjudge real data as abnormal data and delete it, resulting in missing and incomplete data. This situation of mistakenly deleting real data is not uncommon in data migration, especially when dealing with large-scale data sets.

[0005] Existing data migration methods also have shortcomings in data integrity verification. After the migration is completed, it is often only possible to simply compare the total amount of source data and target data, but it is impossible to effectively verify the accuracy and consistency of the data. This extensive verification method is difficult to find subtle differences and errors hidden in the data, which brings potential risks to subsequent data analysis and business decisions. Therefore, the existing data migration methods need to be improved in terms of data cleaning and integrity verification. Summary of the invention

[0006] To solve the above problems, the present invention proposes a data migration method, system, medium and electronic device, which solves the problem that data cleaning in existing data migration easily leads to deletion of real data by introducing an abnormal data processing mechanism, so as to ensure the effective identification of data during the data migration process and improve the accuracy and reliability of data migration.

[0007] According to some embodiments, a first solution of the present invention provides a data migration method, which adopts the following technical solution:

[0008] A data migration method, comprising:

[0009] Obtaining the source medium data volume and the target medium data volume;

[0010] Transfer source media data to target media for data cleansing and data migration;

[0011] Compare the data volume of the source media with the data volume of the target media after data migration to conduct preliminary data integrity verification of data migration;

[0012] After the preliminary data integrity verification is passed, the cleaned data is divided into duplicate value deletion and outlier deletion, and the data integrity verification of data migration is performed to complete the data migration;

[0013] When the cleaned data is duplicate value deletion, re-compare whether there is the same duplicate data in the target medium after data migration; when the cleaned data is outlier deletion, analyze the metadata of the deleted outlier data, find out the timestamp of the outlier, and confirm the authenticity of the outlier.

[0014] As a further technical limitation, the process of finding the timestamp of the outlier value to confirm the authenticity of the outlier value is: if there is an abnormal event that can explain the outlier value at the timestamp where the outlier value occurs, the outlier value is marked as being caused by a real event and the outlier value is re-imported into the target medium; otherwise, the outlier value is deleted from the temporary storage location.

[0015] Furthermore, the process of confirming the authenticity of the abnormal value is as follows: if there is an abnormal event that can explain the abnormal value under the timestamp of the abnormal value, it means that the abnormal data is caused by a real event, and the authenticity of the abnormal data is reconfirmed through the metadata management system; the confirmed real abnormal value is re-imported into the target medium using the data migration tool; for abnormal values ​​that cannot be reasonably explained or are not related to external factors, they are permanently deleted from the temporary storage location using the data management tool.

[0016] As a further technical limitation, the process of transferring source medium data to target medium for data cleaning is: using a data cleaning tool to clean the data to be migrated in the source medium so that the data format in the source medium is the same as the data format in the target medium; using a temporary database file system to store the cleaned data during the data migration process, and using a mark field to record the deleted data information during the data cleaning process.

[0017] As a further technical limitation, the process of preliminary data integrity verification of the data migration is: using a data verification tool to compare the total data volume of the source medium and the total data volume of the target medium and the deleted data, wherein the data verification tool includes but is not limited to Apache Spark and Pandas; analyzing whether there is any data loss or corruption during the data migration process, and completing the preliminary data integrity verification of the data migration.

[0018] As a further technical limitation, the process of data integrity verification of the data migration is: using a database query tool to re-compare whether there is duplicate data identical to the deleted duplicate values ​​in the target medium, wherein the database query tool includes but is not limited to SQL JOIN and Python's pandas.merge; for outliers, using a metadata management system to find the timestamp when the outlier value appears, and verify it in combination with an external data source, wherein the metadata management system includes HiveMetastore and Spark SQL's Catalog.

[0019] According to some embodiments, a second solution of the present invention provides a data migration system, which adopts the following technical solution:

[0020] A data migration system, comprising:

[0021] A data volume acquisition module, configured to acquire source medium data volume and target medium data volume;

[0022] A data migration module, which is configured to transfer source medium data to target medium for data cleaning and perform data migration;

[0023] The data verification module is configured to compare the data volume of the source medium with the data volume of the target medium after the data migration, and perform preliminary data integrity verification of the data migration; after the preliminary data integrity verification is passed, the cleaned data is divided into duplicate value deletion and outlier deletion, and the data integrity verification of the data migration is performed to complete the data migration; when the cleaned data is duplicate value deletion, re-comparison is made as to whether there is the same duplicate data in the target medium after the data migration; when the cleaned data is outlier deletion, the metadata of the deleted outlier data is analyzed, the timestamp of the outlier is found, and the authenticity of the outlier is confirmed.

[0024] As a further technical limitation, in the data verification module, the process of finding the timestamp of the outlier value to confirm the authenticity of the outlier value is: if there is an abnormal event that can explain the outlier value at the timestamp where the outlier value occurs, the outlier value is marked as being caused by a real event, and the outlier value is re-imported into the target medium; otherwise, the outlier value is deleted from the temporary storage location.

[0025] Furthermore, the process of confirming the authenticity of the abnormal value is as follows: if there is an abnormal event that can explain the abnormal value under the timestamp of the abnormal value, it means that the abnormal data is caused by a real event, and the authenticity of the abnormal data is reconfirmed through the metadata management system; the confirmed real abnormal value is re-imported into the target medium using the data migration tool; for abnormal values ​​that cannot be reasonably explained or are not related to external factors, they are permanently deleted from the temporary storage location using the data management tool.

[0026] As a further technical limitation, in the data migration module, the process of transferring the source medium data to the target medium for data cleaning is: using a data cleaning tool to clean the data to be migrated in the source medium so that the data format in the source medium is the same as the data format in the target medium; using a temporary database file system to store the cleaned data during the data migration process, and using a mark field to record the deleted data information during the data cleaning process.

[0027] As a further technical limitation, in the data verification module, the process of preliminary data integrity verification of the data migration is: using a data verification tool to compare the total data volume of the source medium and the total data volume of the target medium and the deleted data, wherein the data verification tool includes but is not limited to Apache Spark and Pandas; analyzing whether there is any data loss or damage during the data migration process, and completing the preliminary data integrity verification of the data migration.

[0028] As a further technical limitation, in the data verification module, the process of data integrity verification of the data migration is: using a database query tool to re-compare whether there is duplicate data identical to the deleted duplicate values ​​in the target medium, wherein the database query tool includes but is not limited to SQL JOIN and Python's pandas.merge; for outliers, using a metadata management system to find the timestamp when the outlier value occurs, and verifying it in combination with an external data source, wherein the metadata management system includes Hive Metastore and Spark SQL's Catalog.

[0029] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution:

[0030] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps of a data migration method as described in the first solution of the present invention.

[0031] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution:

[0032] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in a data migration method as described in the first solution of the present invention are implemented.

[0033] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution:

[0034] A computer program product includes software codes, wherein the program in the software codes executes the steps in a data migration method as described in the first solution of the present invention.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention introduces an abnormal data handling mechanism to effectively identify and retain real data during the data migration process, while removing useless abnormal data, thereby improving the accuracy and reliability of data migration while ensuring the continuity of enterprise business and the availability of data. In addition, the present invention is also flexible and scalable, and can adapt to changes in different scenarios and requirements, providing a more comprehensive and reliable solution for enterprise data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0038] Figure 1 This is a flow chart of a data migration method in Embodiment 1 of the present invention;

[0039] Figure 2 This is a structural block diagram of a data migration system in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0042] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0043] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0044] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant scientific research or technical personnel in this field, the specific meanings of the above terms in the present invention can be determined according to specific circumstances, and they cannot be understood as limitations on the present invention.

[0045] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0046] Embodiment 1

[0047] Embodiment 1 of the present invention introduces a data migration method.

[0048] like Figure 1 A data migration method is shown, comprising:

[0049] Step S01: Before data migration, the initial data volume of the source medium and the target medium is recorded;

[0050] Step S02: Execute data migration, transfer the data of the source medium to the target medium, and perform a data cleaning step during the data migration. At the same time, record the data cleaned (i.e., deleted) during the migration process, including duplicate values ​​and abnormal values, and store them in a temporary storage location. After the data migration is completed, record the amount of data on the target medium after the migration and the amount of cleaned data.

[0051] Step S03: Compare the total data volume of the source medium and the target medium, taking into account the amount of deleted data, and perform preliminary data integrity verification:

[0052] Step S04: After the preliminary data integrity verification is passed, the deleted data is divided into two categories: duplicate value deletion and outlier value deletion. For duplicate value deletion, re-check whether there is the same duplicate data in the target medium; for outlier value deletion, further analyze external factors or system logs to confirm the authenticity of the outlier value;

[0053] Step S05: for deleting duplicate values, re-check whether there is the same duplicate data in the target medium; for deleting outliers, further analyze the metadata of the data, find the timestamp of the outlier, and confirm the authenticity of the outlier;

[0054] Step S06: If an abnormal event that can explain the abnormal value occurs at the timestamp of the abnormal value, the abnormal value is marked as being caused by a real event, and the abnormal value is re-imported into the target medium. If there is no abnormal value that cannot be reasonably explained or is not related to external factors, the abnormal value is deleted from the temporary storage location.

[0055] As one or more implementation modes, in step S01, ensure that the physical storage devices (such as hard disks, servers) of the source medium and the target medium are operating normally and have sufficient storage space, install and configure data migration tools (such as ETL tools, database migration software), ensure that they are compatible with the databases of the source medium and the target medium, use database management tools or scripts to record the initial data volume of the source medium and the target medium, including but not limited to the number of tables, the number of records, the data size, etc., and configure a log system to record key events and error information during the data migration process.

[0056] As one or more implementation modes, in step S02, the specific steps of cleaning data during data migration are:

[0057] Step S201: clean the data to be migrated in the source medium by using a data cleaning tool, and set cleaning rules, including removing null values, correcting erroneous data, and standardizing the format, so that the data format in the source medium is the same as the data format in the target medium, wherein the data cleaning tool includes but is not limited to OpenRefine and Trifacta;

[0058] Step S202: using a temporary database file system (MySQL temporary table or Redis cache) to store the cleaned data during the migration process, including duplicate values ​​and abnormal values;

[0059] Step S203: During the cleaning process, the tag field is used to record detailed information of the deleted data, including data ID, deletion reason (duplicate value or abnormal value), and deletion time.

[0060] It should be noted that during the data cleaning process, the tag field is used to record the detailed information of the deleted data, including data ID, deletion reason (duplicate value or abnormal value), deletion time, etc. At the same time, a data management table or log file can be established to store this information for subsequent analysis and processing. In addition, the security and integrity of these records should be ensured, and encryption, access control and other measures should be used to prevent data from being illegally tampered with or deleted;

[0061] If regular routine data migration is required, you can automate the data migration and cleaning process by writing scripts or using existing ETL tools (such as Apache Nifi, Talend, etc.). The scripts or ETL tools can be configured to run regularly to automatically extract data from the source medium, clean the data, and then migrate the cleaned data to the target medium.

[0062] As one or more implementation modes, in step S03, the specific steps of preliminarily verifying the data integrity are:

[0063] Step S301: using a data verification tool to compare the total data volume of the source medium and the total data volume of the target medium and the deleted data, wherein the data verification tool includes but is not limited to Apache Spark and Pandas;

[0064] Step S302: Analyze the migration log to check whether there is any data loss or damage.

[0065] It should be noted that in step S301, in addition to mentioning the use of data verification tools such as Apache Spark and Pandas, you can also use Apache Spark's DataFrameAPI to perform data reading, filtering, aggregation and other operations to calculate the total data volume of the source medium and the target medium, and consider the amount of deleted data. In addition to directly comparing the data size, you can also calculate the hash values ​​of the data items in the source medium and the target medium, and compare whether these hash values ​​are consistent, to determine whether the data has been tampered with or lost during the migration process.

[0066] As one or more implementation modes, a database query tool is used to re-compare whether there is duplicate data identical to the deleted duplicate value in the target medium, wherein the database query tool includes but is not limited to SQL JOIN and Python's pandas.merge; for outliers, a metadata management system is used to find the timestamp of the outlier and verify it in combination with an external data source (such as a meteorological system, business log), wherein the metadata management system includes Hive Metastore and Spark SQL's Catalog.

[0067] As one or more implementation modes, in step S06, the specific steps of the specific method for handling abnormal data are:

[0068] Step S601: If there is an abnormal event (extreme weather, promotional activity) that can explain the abnormal value under the timestamp where the abnormal value appears, it means that the abnormal data is caused by a real event, and the authenticity of the abnormal data is reconfirmed through the metadata management system;

[0069] Step S602: Use a data migration tool to re-import the confirmed true abnormal values ​​into the target medium to ensure the integrity and accuracy of the data;

[0070] Step S603: For abnormal values ​​that cannot be reasonably explained or are not related to external factors, use a data management tool to permanently delete them from a temporary storage location to avoid interference with subsequent analysis, where the data management tool includes but is not limited to a data cleaning script.

[0071] In this embodiment, if the abnormal value deleted during data migration is meteorological data, then firstly, the timestamp of the deleted meteorological data is extracted from the data migration log or the abnormal data record, and the API or query interface provided by the meteorological system is used to query the corresponding meteorological data according to the extracted timestamp. The meteorological data may include temperature, humidity, etc. The queried meteorological data is compared with the adjacent timestamps (i.e., the data before and after the timestamp) to find out whether various meteorological indicators have changed significantly or fluctuated abnormally at the timestamp;

[0072] The specific method of judging the abnormality of meteorological data is to mark the temperature data in the environmental factor data of the day before the abnormal timestamp as , and mark the temperature data of the day after as , and calculate the temperature difference value by the following formula;

[0073] The threshold of the temperature difference value is set to 10. When the temperature difference value is greater than 10, the temperature data recorded by the environmental factor collection unit of the day is marked as abnormal.

[0074] The air humidity of the previous day is marked as , and the air humidity of the next day is marked as , and then the difference in air humidity percentage is calculated by the following formula:

[0075] Mark the temperature data in the environmental factor data of the previous day as X , the temperature data for the next day is marked as Y , the temperature difference is calculated by the following formula:

[0076] Temperature difference = | YX | ;

[0077] The threshold of the temperature difference value is set to 10. When the temperature difference value is greater than 10, the temperature data recorded by the environmental factor collection unit of the day is marked as abnormal.

[0078] The air humidity of the previous day is marked as A, and the air humidity of the next day is marked as B , and then calculate the percentage difference in air humidity using the following formula:

[0079] Humidity Difference Percentage = | BA / A×100%;

[0080] The threshold of the air humidity difference percentage is set to 20%. When the temperature difference percentage is greater than 20%, the air humidity data recorded by the environmental factor collection unit of the day is marked as abnormal.

[0081] The average wind speed of the previous day is marked as α, and the average wind speed of the next day is marked as β. The difference in wind speed is calculated using the following formula:

[0082] Wind speed difference = α-β;

[0083] The threshold of the light intensity difference is set to 5, and when the light intensity difference is greater than 5, it is marked as abnormal light intensity in that day;

[0084] In the embodiment, if the data with an abnormality is system migration data, operation logs are collected, and then the following method is used to determine whether the abnormality is caused by a real event: using the API or query tool of the log management system to extract the operation log from the system; the operation log usually contains information such as timestamp, user ID, operation type, and operation object;

[0085] First, check whether the timestamp in the operation log is within a reasonable time range; check whether the operation type complies with the system design and standard operating specifications; check whether the operation object is within the legal range, such as file path, database table name, etc.; based on the comparison results, mark the operation log that does not comply with the normal operating specifications as an abnormal operation log; abnormal operation logs may contain illegal operations, unauthorized access, abnormally frequent operations, etc., or the operation log may contain system maintenance that is no longer planned, etc., which are all abnormal operation logs. If the timestamps of these logs are the same as the timestamps of the abnormal system data, it means that the abnormal data is caused by real events and cannot be easily cleaned.

[0086] This embodiment introduces an abnormal data handling mechanism to effectively identify and retain real data during the data migration process, while removing useless abnormal data, thereby improving the accuracy and reliability of data migration while ensuring the continuity of enterprise business and the availability of data. In addition, the present invention is also flexible and scalable, and can adapt to changes in different scenarios and requirements, providing a more comprehensive and reliable solution for enterprise data migration.

[0087] Embodiment 2

[0088] Embodiment 2 of the present invention introduces a data migration system.

[0089] like Figure 2 A data migration system shown includes:

[0090] A data volume acquisition module, configured to acquire source medium data volume and target medium data volume;

[0091] A data migration module, which is configured to transfer source medium data to target medium for data cleaning and perform data migration;

[0092] The data verification module is configured to compare the data volume of the source medium with the data volume of the target medium after the data migration, and perform preliminary data integrity verification of the data migration; after the preliminary data integrity verification is passed, the cleaned data is divided into duplicate value deletion and outlier deletion, and the data integrity verification of the data migration is performed to complete the data migration; when the cleaned data is duplicate value deletion, re-comparison is made as to whether there is the same duplicate data in the target medium after the data migration; when the cleaned data is outlier deletion, the metadata of the deleted outlier data is analyzed, the timestamp of the outlier is found, and the authenticity of the outlier is confirmed.

[0093] As one or more implementation modes, in the data verification module, the process of finding the timestamp of the outlier value to confirm the authenticity of the outlier value is: if there is an abnormal event that can explain the outlier value at the timestamp where the outlier value occurs, the outlier value is marked as caused by a real event, and the outlier value is re-imported into the target medium; otherwise, the outlier value is deleted from the temporary storage location.

[0094] As one or more implementation modes, the process of confirming the authenticity of the abnormal value is as follows: if there is an abnormal event that can explain the abnormal value under the timestamp of the abnormal value, it means that the abnormal data is caused by a real event, and the authenticity of the abnormal data is reconfirmed through the metadata management system; the confirmed real abnormal value is re-imported into the target medium using a data migration tool; for abnormal values ​​that cannot be reasonably explained or are not related to external factors, they are permanently deleted from the temporary storage location using a data management tool.

[0095] As one or more implementation modes, in the data migration module, the process of transferring the source medium data to the target medium for data cleaning is: using a data cleaning tool to clean the data to be migrated in the source medium so that the data format in the source medium is the same as the data format in the target medium; using a temporary database file system to store the cleaned data during the data migration process, and using a mark field to record the deleted data information during the data cleaning process.

[0096] As one or more implementation modes, in the data verification module, the process of preliminary data integrity verification of the data migration is: using a data verification tool to compare the total data volume of the source medium and the total data volume of the target medium and the deleted data, wherein the data verification tool includes but is not limited to Apache Spark and Pandas; analyzing whether there is any data loss or damage during the data migration process, and completing the preliminary data integrity verification of the data migration.

[0097] As one or more implementation modes, in the data verification module, the process of data integrity verification of the data migration is: using a database query tool to re-compare whether there is duplicate data identical to the deleted duplicate values ​​in the target medium, wherein the database query tool includes but is not limited to SQL JOIN and Python's pandas.merge; for outliers, using a metadata management system to find the timestamp when the outlier value appears, and verifying it in combination with an external data source, wherein the metadata management system includes the Hive Metastore and the Spark SQL Catalog.

[0098] The detailed steps are the same as those of the data migration method provided in Example 1 and will not be repeated here.

[0099] Embodiment 3

[0100] Embodiment 3 of the present invention provides a computer-readable storage medium.

[0101] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in a data migration method as described in Embodiment 1 of the present invention.

[0102] The detailed steps are the same as those of the data migration method provided in Example 1 and will not be repeated here.

[0103] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0108] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0109] Embodiment 4

[0110] A fourth embodiment of the present invention provides an electronic device.

[0111] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in a data migration method as described in Embodiment 1 of the present invention are implemented.

[0112] The detailed steps are the same as those of the data migration method provided in Example 1 and will not be repeated here.

[0113] Embodiment 5

[0114] Embodiment 5 of the present invention provides a computer program product.

[0115] A computer program product includes software codes, wherein the program in the software codes executes the steps in a data migration method as described in Embodiment 1 of the present invention.

[0116] The detailed steps are the same as those of the data migration method provided in Example 1 and will not be repeated here.

[0117] It can be understood that the computer-readable storage device here can include both the built-in storage medium in the computer device and the extended storage medium supported by the computer device. The computer-readable storage device provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage device here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage device to implement the steps of a distribution network extreme scenario identification method in the above embodiment.

[0118] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A data migration method, characterized in that: include: Obtaining source medium data volume and target medium data volume; Transfer source media data to target media for data cleansing and data migration; Compare the data volume of the source media with the data volume of the target media after data migration to conduct preliminary data integrity verification of data migration; After the preliminary data integrity verification is passed, the cleaned data is divided into duplicate value deletion and outlier deletion, and the data integrity verification of data migration is performed to complete the data migration; When the cleaned data is duplicate value deletion, re-compare whether there is the same duplicate data in the target medium after data migration; when the cleaned data is outlier deletion, analyze the metadata of the deleted outlier data, find out the timestamp of the outlier, and confirm the authenticity of the outlier.

2. A data migration method as claimed in claim 1, characterized in that: The process of finding the timestamp of the abnormal value to confirm the authenticity of the abnormal value is as follows: if there is an abnormal event that can explain the abnormal value at the timestamp of the abnormal value, then the abnormal value is marked as caused by a real event, and the abnormal value is re-imported into the target medium; Otherwise, the abnormal value is deleted from the temporary storage location.

3. A data migration method as claimed in claim 2, characterized in that: The process of confirming the authenticity of the abnormal value is as follows: if there is an abnormal event that can explain the abnormal value under the timestamp of the abnormal value, it means that the abnormal data is caused by a real event, and the authenticity of the abnormal data is reconfirmed through the metadata management system; the confirmed real abnormal value is re-imported into the target medium using a data migration tool; For outliers that cannot be reasonably explained or are not related to external factors, permanently delete them from the temporary storage location using data management tools.

4. A data migration method as claimed in claim 1, characterized in that: The process of transferring the source medium data to the target medium for data cleaning is: using a data cleaning tool to clean the data to be migrated in the source medium so that the data format in the source medium is the same as the data format in the target medium; Use a temporary database file system to store the cleaned data during the data migration process, and use a marker field to record deleted data information during the data cleansing process.

5. A data migration method as claimed in claim 1, characterized in that: The process of preliminary data integrity verification of the data migration is: using a data verification tool to compare the total data volume of the source medium and the total data volume of the target medium and the deleted data, wherein the data verification tool includes but is not limited to Apache Spark and Pandas; analyzing whether there is any data loss or damage during the data migration process, and completing the preliminary data integrity verification of the data migration.

6. A data migration method as claimed in claim 1, characterized in that: The data integrity verification process of the data migration is as follows: using a database query tool to re-compare whether there are duplicate data identical to the deleted duplicate values ​​in the target medium, wherein the database query tool includes but is not limited to SQL JOIN and Python's pandas.merge; for abnormal values, using a metadata management system to find the timestamp of the abnormal value, and verifying it in combination with an external data source, wherein the metadata management system includes Hive Metastore and Spark SQL's Catalog.

7. A data migration system, characterized in that: include: A data volume acquisition module, configured to acquire source medium data volume and target medium data volume; A data migration module, which is configured to transfer source medium data to target medium for data cleaning and perform data migration; The data verification module is configured to compare the data volume of the source medium with the data volume of the target medium after the data migration, and perform preliminary data integrity verification of the data migration; after the preliminary data integrity verification is passed, the cleaned data is divided into duplicate value deletion and outlier deletion, and the data integrity verification of the data migration is performed to complete the data migration; when the cleaned data is duplicate value deletion, re-comparison is made as to whether there is the same duplicate data in the target medium after the data migration; when the cleaned data is outlier deletion, the metadata of the deleted outlier data is analyzed, the timestamp of the outlier is found, and the authenticity of the outlier is confirmed.

8. A data migration system as claimed in claim 7, characterized in that: In the data verification module, the process of finding the timestamp of the abnormal value to confirm the authenticity of the abnormal value is: if there is an abnormal event that can explain the abnormal value under the timestamp of the abnormal value, then mark the abnormal value as caused by a real event, and re-import the abnormal value into the target medium; Otherwise, the abnormal value is deleted from the temporary storage location.

9. A data migration system as claimed in claim 8, characterized in that: The process of confirming the authenticity of the abnormal value is as follows: if there is an abnormal event that can explain the abnormal value under the timestamp of the abnormal value, it means that the abnormal data is caused by a real event, and the authenticity of the abnormal data is reconfirmed through the metadata management system; the confirmed real abnormal value is re-imported into the target medium using a data migration tool; For outliers that cannot be reasonably explained or are not related to external factors, permanently delete them from the temporary storage location using data management tools.

10. A data migration system as claimed in claim 7, characterized in that: In the data migration module, the process of transferring the source medium data to the target medium for data cleaning is: using a data cleaning tool to clean the data to be migrated in the source medium so that the data format in the source medium is the same as the data format in the target medium; Use a temporary database file system to store the cleaned data during the data migration process, and use a marker field to record deleted data information during the data cleansing process.

11. A data migration system as claimed in claim 7, characterized in that: In the data verification module, the process of preliminary data integrity verification of the data migration is: using a data verification tool to compare the total data volume of the source medium and the total data volume of the target medium and the deleted data, wherein the data verification tool includes but is not limited to Apache Spark and Pandas; analyzing whether there is any data loss or damage during the data migration process, and completing the preliminary data integrity verification of the data migration.

12. A data migration system as claimed in claim 7, characterized in that: In the data verification module, the process of data integrity verification of the data migration is: using a database query tool to re-compare whether there is duplicate data identical to the deleted duplicate values ​​in the target medium, wherein the database query tool includes but is not limited to SQL JOIN and Python's pandas.merge; for outliers, using a metadata management system to find the timestamp when the outlier appears, and verifying it in combination with an external data source, wherein the metadata management system includes Hive Metastore and Spark SQL's Catalog.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a data migration method as described in any one of claims 1 to 6 are implemented.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of a data migration method as described in any one of claims 1-6 are implemented.

15. A computer program product comprising software code, characterized in that The program in the software code executes the steps of a data migration method as described in any one of claims 1-6.

Images