Data migration method and device, equipment and storage medium

Abstract

The invention provides a data migration method and device, equipment and a storage medium, and belongs to the technical field of databases. The method comprises the following steps: extracting to-be-migrated data through a data source plug-in; performing data cleaning and data conversion on the to-be-migrated data by using components in the data migration framework, wherein the data cleaning comprises at least one of complete duplicated data cleaning and similar duplicated data cleaning; and loading the to-be-migrated data subjected to data cleaning and data conversion to a target data warehouse. A data cleaning component is introduced into a data migration framework to clean completely duplicated data and similar duplicated data, so that diversified data cleaning requirements are met. High expandability is achieved through basic function framing and special service function modularization, and the requirements for integration and data cleaning of multi-source heterogeneous data are met. And real-time data cleaning and transmission efficiency optimization are realized under the conditions of complex construction site conditions and non-uniform data standards of various regions.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data migration method, apparatus, device, and storage medium. Background Technology

[0002] Traditional data processing, storage methods, and data analysis solutions can no longer meet the ever-increasing business demands. As a result, data warehouse solutions have rapidly gained dominance in various fields. Various data warehouse technologies (Extract Transform Load, ETL) provide data warehouses with data migration support capabilities, and their main data processing steps are data extraction, data transformation, and data loading.

[0003] Currently, the mainstream ETL middleware in the industry includes Datapipline, Kettle, DataX, Sqoop, Oracle Goldengate, Datastage, and Flume. These seven ETL middleware programs are open-source and support custom development.

[0004] However, the aforementioned middleware programs each have their own focus on different use cases. For example, Datapipline, Oracle Goldengate, and Datastage do not support data migration between different databases; Sqoop is mainly used for data transfer between Hadoop and traditional databases; Kettle has poor scalability and cannot meet the development needs of the infrastructure layer; and Flume is complex to deploy and cannot be plugged in. Therefore, none of these middleware programs can meet the diverse cleaning needs of wellbore data during the migration process from remote construction sites to data centers. Summary of the Invention

[0005] This application provides a data migration method, apparatus, device, and storage medium, capable of meeting diverse data cleaning needs. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a data migration method, the method comprising:

[0007] Extract the data to be migrated using the data source plugin;

[0008] The data to be migrated is cleaned and transformed using components in the data migration framework. The data cleaning includes at least one of completely duplicate data cleaning and similar duplicate data cleaning. The similar duplicate data cleaning refers to cleaning data that is similar in appearance and has the same semantics.

[0009] The data to be migrated, after data cleaning and transformation, is loaded into the target data warehouse.

[0010] On the other hand, embodiments of this application provide a data migration apparatus, the apparatus comprising:

[0011] The extraction module is used to extract data to be migrated through the data source plugin.

[0012] The cleaning module is used to perform data cleaning and data transformation on the data to be migrated using components in the data migration framework. The data cleaning includes at least one of completely duplicate data cleaning and similar duplicate data cleaning. The similar duplicate data cleaning refers to cleaning data that is similar in appearance and has the same semantics.

[0013] The loading module is used to load the data to be migrated, after data cleaning and data transformation, into the target data warehouse.

[0014] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement a data migration method as described above.

[0015] On the other hand, embodiments of this application provide a computer program product that runs on a processor of a computer device, causing the computer device to perform a data migration method as described above.

[0016] The technical solution provided in this application includes at least the following beneficial effects:

[0017] This application provides a data migration method, apparatus, device, and storage medium. It introduces a data cleaning component into the data migration framework to clean completely duplicated data and similarly duplicated data, meeting diverse data cleaning needs. Through a framework of basic functions and componentization of special business functions, it achieves high scalability, meeting the integration and data cleaning needs of multi-source heterogeneous data. Even in complex construction site conditions and with inconsistent data standards across regions, it enables real-time data cleaning and optimizes transmission efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0019] Figure 1 This is a flowchart of a data migration method provided in an exemplary embodiment of this application;

[0020] Figure 2 This is a flowchart of a data migration method provided by another exemplary embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the structure of DataX provided in an exemplary embodiment of this application;

[0022] Figure 4 This is a schematic diagram of a DataX data cleaning and consumption model provided in an exemplary embodiment of this application;

[0023] Figure 5 This is a schematic diagram of a data migration framework provided in an exemplary embodiment of this application;

[0024] Figure 6 This is a structural block diagram of a data migration apparatus provided in an exemplary embodiment of this application;

[0025] Figure 7 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] If the application documents contain similar descriptions such as "first, second, third", the following explanation shall be added: In the following description, the terms "first, second, third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0030] Please refer to Figure 1 The diagram illustrates a flowchart of a control method for an air conditioning unit provided in an exemplary embodiment of this application. The method includes the following steps:

[0031] Step 101: Extract the data to be migrated through the data source plugin.

[0032] The data to be migrated is distributed and heterogeneous. In the scenario of migrating real-time wellbore data, the data to be migrated includes data manually entered by technicians at the construction site and data automatically collected by well site instruments and equipment, such as logging data and well logging data. Due to the complexity of the construction site conditions, the data collection methods and recording standards differ between different regions, teams, and even different wellbores, and there is a large amount of duplicate data. Therefore, it is necessary to clean and transform the data to be migrated.

[0033] In one possible implementation, this application uses a data migration framework to implement business functions such as data extraction, cleaning, transformation, and loading. Various business functions and data sources are connected to the data migration framework in the form of plug-ins. The data source plug-in encapsulates the data source connection method and some special operations, and the computer device extracts the data to be migrated from the data source through the data source plug-in.

[0034] As an illustration, data sources include relational databases like MySQL, distributed databases like DB2, and data management systems like Open Data Processing Service (ODPS).

[0035] Step 102: Use the components in the data migration framework to perform data cleaning and data transformation on the data to be migrated. Data cleaning includes completely duplicate data cleaning and similar duplicate data cleaning. Similar duplicate data cleaning refers to cleaning data that is similar in appearance and has the same semantics.

[0036] In one possible implementation, the data migration framework is integrated with a data cleaning component to perform data cleaning. This data cleaning includes at least one of completely duplicate data cleaning and similarly duplicate data cleaning. Completely duplicate data cleaning refers to cleaning data that is identical in both appearance and semantics. Similarly duplicate data cleaning refers to cleaning data that is similar in appearance and semantics, i.e., data that is identical but differs in appearance, such as the full name and abbreviation of the same thing (e.g., "Computer Science" vs. "Software Engineering and Computer Science"), or typos from manual input (e.g., "date" vs. "date"). Normalizing similarly duplicate data can further reduce the amount of data.

[0037] Optionally, the computer equipment may perform only full duplicate data cleaning, only similar duplicate data cleaning, or both on the data to be migrated. This application does not limit the order of full duplicate data cleaning and similar duplicate data cleaning. To reduce data processing volume, the computer equipment may perform full duplicate data cleaning first, followed by similar duplicate data cleaning.

[0038] Data transformation includes changing the way data is stored. For example, it can transform the granularity of data, such as converting "city" or "district" to the corresponding "province". It can also perform calculations based on preset rules, such as converting "oil production" to "cost" based on a preset cost calculation method. Furthermore, it can perform operations such as row-to-column and column-to-row transformations on data tables.

[0039] Step 103: Load the data to be migrated, after data cleaning and data transformation, into the target data warehouse.

[0040] After the computer equipment performs data cleaning and transformation on the data to be migrated, it uses the loading component in the data migration framework to load the data into the corresponding target data warehouse.

[0041] As an illustration, in the scenario of wellbore data migration, the above framework can be used to migrate distributed heterogeneous data from multiple construction sites to the headquarters data center after cleaning and transformation.

[0042] In summary, the data migration method, apparatus, device, and storage medium provided in this application introduce a data cleaning component into the data migration framework to clean completely duplicated data and similarly duplicated data, meeting diverse data cleaning needs. High scalability is achieved through a framework of basic functions and componentization of special business functions, satisfying the integration and data cleaning requirements of multi-source heterogeneous data. Real-time data cleaning and optimized transmission efficiency are achieved even in complex construction site conditions and where data standards vary across regions.

[0043] Please refer to Figure 2 The diagram illustrates a flowchart of a control method for an air conditioning unit provided in another exemplary embodiment of this application. The method includes the following steps:

[0044] Step 201: Write the data to be migrated into the blocking queue through the data source plugin.

[0045] In one possible implementation, the data migration framework in this application embodiment is built using the middleware DataX. For example... Figure 3As shown in the figure, the structural design of DataX simplifies the complex network synchronization link into a star data link, which serves as a transmission carrier during the entire data migration. Therefore, when adding a new service that requires introducing a new data source, only need to encapsulate the data source link method and some special operations into a data source plugin and dock it on the DataX framework. By using the Framework + Plugin architecture adopted by DataX in the design architecture, the data transmission carrier and data heterogeneous processing and other logics are encapsulated into the DataX framework. Such a design is conducive to the functional precipitation of the common part being encapsulated into the framework layer. For the differential processing logics of different data source ends or destination ends, the business code is encapsulated in the form of a plugin, which is conducive to handling various data source links and realizing the isolation of business layer codes such as data extraction, data cleaning, and data insertion, thereby achieving the high scalability of the DataX plugin.

[0046] Among them, the processing flow of data cleaning and migration can be regarded as a production and consumption model after simplification, as Figure 4 shown. The data source end plugin is the producer of data. The data channel uses a blocking queue. The data source plugin writes the data to be migrated into the blocking queue and waits for processing.

[0047] Step 202: Extract the data to be migrated from the blocking queue in the order of data writing.

[0048] The computer device extracts the data to be migrated from the blocking queue in sequence according to the data writing order for data cleaning and conversion. Using the blocking queue ensures that it is executed according to the writing order of the data to be migrated, and with the help of the plugin-based design, the decoupling of the data acquisition and insertion ends is achieved, which is conducive to solving the problem of inconsistent IO speeds between the upstream and downstream of data migration, effectively controlling the data migration speed and protecting the data source end and the destination end.

[0049] In a possible implementation manner, before data cleaning, the computer device first performs preprocessing before data cleaning on the data to be migrated by using the preprocessing plugin in the framework based on the preprocessing rules in the rule library, and performs character replacement on the data to improve the integrity of the data. For example, uniformly processing "River 1 Well" into "River 1" to avoid different representations with the same semantics.

[0050] Step 203: Perform data cleaning on the data to be migrated through the data cleaning component in the first sub-framework.

[0051] In a possible implementation manner, the data migration framework of the embodiment of the present application includes two parts, as Figure 5As shown in the diagram. The first sub-framework is the ECL framework, which includes data extraction, data cleaning, and data loading components, loading the cleaned data to be migrated into the intermediate database. The second sub-framework is the TL component, which includes data transformation and data loading components, transforming the data in the intermediate database and then loading it into the target data warehouse.

[0052] The ECL-TL data migration architecture provided in this application splits the data migration process into two stages, which can improve the stability of the data migration system. If the ECL stage succeeds but the TL stage fails, the entire data migration task will not fail, and the computer device can reproduce the data from the intermediate library. This also indirectly improves the system's operating efficiency, as it is not necessary to go through the entire process from beginning to end again when errors occur in subsequent stages.

[0053] Specifically, the data cleaning component includes a fully repeatable cleaning component, and step 203 includes the following steps:

[0054] Step 203a: Construct a hash table using the fully repeatable cleaning component in the first sub-frame. The hash table stores preset key fields and data identifiers of the data to be migrated. Data identifiers of data with the same preset key fields are stored in the same linear table.

[0055] Complete duplicate record cleaning components can utilize hash tables for data storage, offering extremely high efficiency. Users first select one or more key fields; if two records have the same key field, it indicates that one of them is a duplicate record.

[0056] This application uses the concept of hashing to store the values ​​of key fields and their corresponding IDs in a hash table, and retrieves completely duplicate data based on the hash table.

[0057] Step 203b: Traverse the hash table and obtain the length of each linear list.

[0058] Step 203c: In response to the existence of a linear list to be cleaned, based on the cleaning rules in the rule base, perform complete duplicate data cleaning on the data to be migrated corresponding to the linear list to be cleaned. The linear list to be cleaned is a linear list with a length greater than 1.

[0059] Records with the same key field are stored in the same linear table. Then, the hash table is traversed. If the length of the linear table is greater than 1, it means that there are duplicate records. Finally, the records are cleaned according to the set cleaning rules.

[0060] In another possible implementation, since the hash table is stored in the device's memory, the memory usage of this method increases with the amount of data. Therefore, hash table-based cleaning is suitable for handling small amounts of data to be cleaned. If the amount of data is large, other data storage methods can be used, such as storing large amounts of data on a disk.

[0061] Specifically, the data cleaning component also includes a similar repetitive cleaning component, and step 203 further includes the following steps:

[0062] Step 203d: Based on data attributes, cluster the data to be migrated using the similarity and repetition cleaning component in the second sub-framework to obtain the dataset.

[0063] Step 203e: Compare the similarity of the data to be cleaned in the dataset, and clean the similar duplicate data of the data to be cleaned according to the cleaning rules in the rule base.

[0064] Similar and duplicate record cleaning components can perform data fusion through clustering. By clustering data records according to the importance of their attributes, the amount of data is greatly reduced. Then, the similarity of records in the clustered small dataset is compared to detect similar and duplicate records.

[0065] When clustering based on attributes, different processing methods are selected to improve clustering performance depending on the type of attribute in the record. For example, for attributes that are mainly Chinese, have a small number of characters, and are prone to input errors due to homophones, the computer device converts them into phonetic codes for clustering, such as "hash" or "process type"; for attributes that contain multiple words, have a large number of characters, and are prone to abbreviations, the computer device converts them into spatial vectors for clustering, such as "unit name", "working condition", or "address"; for attributes that are not mainly Chinese, they are directly clustered, such as various codes.

[0066] Specifically, the data cleaning component can also include user-provided custom components, meaning users can write custom functions based on business needs. Step 203 also includes the following steps:

[0067] Step 203f: In response to the existence of a custom component in the first subframe, custom data cleaning is performed on the data to be cleaned using the custom component. The custom component is written in a preset scripting language and imported into the first subframe.

[0068] By introducing a pre-defined scripting language (such as JavaScript) into the ECL-TL framework, users can easily perform tasks such as data cleaning and calculation. For example, in the process of processing non-standard hash numbers, hash data in the form of "XX area YY hash" needs to be customized using JavaScript to convert it into "YY" hash numbers.

[0069] Step 204: Load the cleaned data to be migrated into the intermediate library through the loading component in the first sub-frame.

[0070] In one possible implementation, before data loading, the computer device also uses a quality assessment plugin in the framework to perform a quality assessment on the cleaned data to be migrated, and to check the compliance of the original data with existing data standards.

[0071] The computer equipment temporarily loads the cleaned data to be migrated into an intermediate database through the loading component in the first sub-frame, which can improve the stability of the data migration system. If the ECL phase succeeds but the TL phase fails, the entire data migration task will not fail, and the computer equipment can reproduce the data from the intermediate database. This also indirectly improves the system's operating efficiency, as it is not necessary to go through the entire process from beginning to end again if errors occur in subsequent stages.

[0072] Step 205: Extract the data to be migrated from the intermediate library and perform data transformation through the data transformation component in the second sub-framework.

[0073] Step 206: Load the transformed data to be migrated into the target data warehouse through the loading component in the second sub-frame.

[0074] In one possible implementation, the computer device uses the TL framework to transform the data temporarily stored in the intermediate library and finally load it into the target data warehouse. The specific transformation and loading process can be found in step 103 above, and will not be repeated here in this embodiment.

[0075] In one possible implementation, as business expands, users can add new data sources to the framework. The method also includes the following steps:

[0076] Step 1: In response to the data source addition command, obtain the data source plugin, which encapsulates the data source connection method and operation commands.

[0077] Step 2: Connect the data source plugin to the DataX middleware.

[0078] Based on DataX's architecture, new data sources can be added. When adding new business logic requires introducing a new data source, simply encapsulate the data source connection method and some special operations into a data source plugin and integrate it with the DataX framework.

[0079] Figure 6 This is a structural block diagram of a data migration apparatus provided in an exemplary embodiment of this application. The apparatus includes the following structure:

[0080] Extraction module 601 is used to extract data to be migrated through the data source plugin;

[0081] The cleaning module 602 is used to perform data cleaning and data transformation on the data to be migrated using components in the data migration framework. The data cleaning includes at least one of completely duplicate data cleaning and similar duplicate data cleaning. The similar duplicate data cleaning refers to cleaning data that are similar in appearance and have the same semantics.

[0082] Loading module 603 is used to load the data to be migrated, after data cleaning and data transformation, into the target data warehouse.

[0083] Optionally, the cleaning module 602 is further configured to:

[0084] The data to be migrated is cleaned using the data cleaning component in the first sub-frame.

[0085] The data to be migrated, after data cleaning, is loaded into the intermediate library through the loading component in the first sub-frame.

[0086] The data to be migrated is extracted from the intermediate library and transformed using the data transformation component in the second sub-framework;

[0087] The loading module 603 is also used for:

[0088] The transformed data to be migrated is loaded into the target data warehouse using the loading component in the second sub-framework.

[0089] Optionally, the loading module 603 is further configured to:

[0090] A hash table is constructed using the fully repeatable cleaning component in the first sub-framework. The hash table stores preset key fields and data identifiers of the data to be migrated. The data identifiers of data with the same preset key fields are stored in the same linear table.

[0091] Traverse the hash table to obtain the length of each linear list;

[0092] In response to the existence of a linear list to be cleaned, based on the cleaning rules in the rule base, the data to be migrated corresponding to the linear list to be cleaned is subjected to complete duplicate data cleaning, wherein the linear list to be cleaned is a linear list with a length greater than 1.

[0093] Optionally, the loading module 603 is further configured to:

[0094] Based on data attributes, the data to be migrated is clustered using the similarity and duplication cleaning component in the second sub-frame to obtain a dataset;

[0095] The similarity of the data to be cleaned in the dataset is compared, and the data with similarity higher than the threshold is cleaned according to the cleaning rules in the rule base.

[0096] Optionally, the loading module 603 is further configured to:

[0097] In response to the existence of a custom component in the first sub-framework, the custom component is used to perform custom data cleaning on the data to be cleaned. The custom component is written in a preset scripting language and imported into the first sub-framework.

[0098] Optionally, the extraction module 601 is further configured to:

[0099] The data to be migrated is written into a blocking queue via the data source plugin;

[0100] The data to be migrated is extracted from the blocking queue according to the data writing order.

[0101] Optionally, the data migration framework is built using the DataX middleware;

[0102] The device further includes:

[0103] The acquisition module is used to acquire the data source plugin in response to the data source addition command. The data source plugin encapsulates the data source connection method and operation instructions.

[0104] Add a module to interface the data source plugin into the DataX middleware.

[0105] In summary, the data migration device provided in this application introduces a data cleaning component into the data migration framework to clean completely duplicated data and similarly duplicated data, meeting diverse data cleaning needs. High scalability is achieved through a framework of basic functions and componentization of special business functions, satisfying the integration and data cleaning requirements of multi-source heterogeneous data. Real-time data cleaning and transmission efficiency optimization are achieved even in complex construction site conditions and where data standards vary across regions.

[0106] Please refer to Figure 7 The diagram illustrates the structure of a computer device provided in one embodiment of this application.

[0107] The computer device 700 in this application may include one or more of the following components: processor 710 and memory 720.

[0108] Processor 710 may include one or more processing cores. Processor 710 connects to various parts within the computer device 700 using various interfaces and lines, and performs various functions and processes data of the computer device 700 by running or executing instructions, programs, code sets, or instruction sets stored in memory 720, and by calling data stored in memory 720. Optionally, processor 710 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 710 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem.

[0109] The memory 720 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 720 may include a non-transitory computer-readable storage medium. The memory 720 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created by the computer device 700 during use.

[0110] In addition, those skilled in the art will understand that the structure of the computer device 700 shown in the above figures does not constitute a limitation on the computer device 700. The computer device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the computer device 700 also includes radio frequency circuits, audio circuits, Wi-Fi components, power supplies, Bluetooth components, etc., which will not be described in detail here.

[0111] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement a data migration method as described in the above embodiments.

[0112] This application also provides a computer program product that runs on the processor of a computer device, causing the computer device to perform a data migration method as described in the above embodiments.

[0113] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0114] The scope of protection of this disclosure is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its scope and spirit. If such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A data migration method, characterized in that, The method includes: Extract the data to be migrated using the data source plugin; The data to be migrated is cleaned and transformed using components in the data migration framework. The data cleaning includes at least one of completely duplicate data cleaning and similar duplicate data cleaning. The similar duplicate data cleaning refers to cleaning data that is similar in appearance and has the same semantics. The data to be migrated, after data cleaning and transformation, is loaded into the target data warehouse.

2. The method according to claim 1, characterized in that, The process of using components within the data migration framework to perform data cleaning and transformation on the data to be migrated includes: The data to be migrated is cleaned using the data cleaning component in the first sub-frame. The data to be migrated, after data cleaning, is loaded into the intermediate library through the loading component in the first sub-frame. The data to be migrated is extracted from the intermediate library and transformed using the data transformation component in the second sub-framework; The step of loading the data to be migrated, after data cleaning and transformation, into the target data warehouse includes: The transformed data to be migrated is loaded into the target data warehouse using the loading component in the second sub-framework.

3. The method according to claim 2, characterized in that, The data cleaning process, performed on the data to be migrated using the data cleaning component in the first sub-frame, includes: A hash table is constructed using the fully repeatable cleaning component in the first sub-framework. The hash table stores preset key fields and data identifiers of the data to be migrated. The data identifiers of data with the same preset key fields are stored in the same linear table. Traverse the hash table to obtain the length of each linear list; In response to the existence of a linear list to be cleaned, based on the cleaning rules in the rule base, the data to be migrated corresponding to the linear list to be cleaned is subjected to complete duplicate data cleaning, wherein the linear list to be cleaned is a linear list with a length greater than 1.

4. The method according to claim 2, characterized in that, The data cleaning process, performed on the data to be migrated using the data cleaning component in the first sub-frame, includes: Based on data attributes, the data to be migrated is clustered using the similarity and duplication cleaning component in the second sub-frame to obtain a dataset; The similarity of the data to be cleaned in the dataset is compared, and the data with similarity higher than the threshold is cleaned according to the cleaning rules in the rule base.

5. The method according to claim 2, characterized in that, The data cleaning process, performed on the data to be migrated using the data cleaning component in the first sub-frame, includes: In response to the presence of a custom component in the first sub-framework, custom data cleaning is performed on the data to be migrated using the custom component. The custom component is written in a preset scripting language and imported into the first sub-framework.

6. The method according to any one of claims 1 to 5, characterized in that, The process of extracting the data to be migrated through the data source plugin includes: The data to be migrated is written into a blocking queue via the data source plugin; The data to be migrated is extracted from the blocking queue according to the data writing order.

7. The method according to any one of claims 1 to 5, characterized in that, The data migration framework is built using the DataX middleware; The method further includes: In response to a data source addition command, a data source plugin is obtained, which encapsulates the data source connection method and operation commands. Connect the data source plugin to the DataX middleware.

8. A data migration device, characterized in that, The device includes: The extraction module is used to extract data to be migrated through the data source plugin; The cleaning module is used to perform data cleaning and data transformation on the data to be migrated using components in the data migration framework. The data cleaning includes at least one of completely duplicate data cleaning and similar duplicate data cleaning. The similar duplicate data cleaning refers to cleaning data that is similar in appearance and has the same semantics. The loading module is used to load the data to be migrated, after data cleaning and data transformation, into the target data warehouse.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, performs a data migration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement a data migration method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product runs on the processor of a computer device, causing the computer device to perform a data migration method as described in any one of claims 1 to 7.

Images