Data migration method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of data migration, can be applied to service system platforms of medical health, financial science and technology and the like, and discloses a data migration method and device, computer equipment and a storage medium. Constructing a data migration strategy of each source data source and the target data source according to the connection information; constructing a joint query statement, and reading to-be-migrated data from each source data source according to the joint query statement; acquiring configuration information of the target data source, and converting the to-be-migrated data into intermediate data adaptive to the target data source according to the configuration information; writing the intermediate data into the target data source according to the data migration strategy; therefore, the efficiency and the accuracy of data migration in a data migration scene of multiple data sources can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a data migration method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] With the rapid development of information technology, data plays a vital role in corporate decision-making, business process optimization, and innovative applications. Efficient data migration and synchronization are key to achieving data integration, system upgrades, cross-platform operations, and multi-system collaboration.

[0003] However, existing data migration and synchronization tools still have many shortcomings when it comes to migrating data from multiple data sources, particularly in terms of efficiency and accuracy, making it difficult to meet the growing demands of enterprises. In healthcare, data accuracy and integrity are crucial. Vast amounts of patient data are stored in systems such as healthcare information systems (HIS), electronic medical records (EMRs), and photographic imaging systems (PACS). This data needs to be migrated and synchronized between these systems to support applications such as telemedicine, medical big data analytics, and clinical decision support. In fintech, efficient data migration and synchronization are key to achieving financial innovation and risk management. Financial institutions need to migrate data from traditional core systems to new big data platforms, cloud computing environments, or blockchain systems to support applications such as risk assessment, transaction monitoring, and robo-advisory. However, both healthcare and fintech face challenges in data migration efficiency and accuracy in multi-data source scenarios.

[0004] Based on this, how to provide a data migration method, apparatus, computer device and computer-readable storage medium that can effectively improve the efficiency and accuracy of data migration in data migration scenarios with multiple data sources is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a data migration method, apparatus, computer device and computer-readable storage medium, aiming to solve the problem of how to effectively improve the efficiency and accuracy of data migration in data migration scenarios with multiple data sources.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a data migration method, comprising:

[0008] Acquire connection information between a target data source and multiple source data sources, and construct a data migration strategy between each of the source data sources and the target data source based on the connection information;

[0009] Constructing a joint query statement, and reading the data to be migrated from each of the source data sources according to the joint query statement;

[0010] Acquiring configuration information of the target data source, and converting the data to be migrated into intermediate data adapted to the target data source according to the configuration information;

[0011] According to the data migration strategy, the intermediate data is written into the target data source.

[0012] In a second aspect, the present invention provides a data migration device, comprising:

[0013] a strategy building module, configured to obtain connection information between a target data source and a plurality of source data sources, and to build a data migration strategy between each of the source data sources and the target data source according to the connection information;

[0014] A data reading module, configured to construct a joint query statement and read the data to be migrated from each of the source data sources according to the joint query statement;

[0015] a data conversion module, configured to obtain configuration information of the target data source and convert the data to be migrated into intermediate data adapted to the target data source according to the configuration information;

[0016] A data writing module is used to write the intermediate data into the target data source according to the data migration strategy.

[0017] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data migration method as described above when executing the computer program.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program implements the data migration method as described above when executed by a processor.

[0019] Compared with the prior art, the present invention provides a data migration method, apparatus, computer device and computer-readable storage medium, wherein the method obtains connection information between a target data source and multiple source data sources, and constructs a data migration strategy for each of the source data sources and the target data source based on the connection information; constructs a joint query statement, and reads the data to be migrated from each of the source data sources according to the joint query statement; obtains configuration information of the target data source, and converts the data to be migrated into intermediate data adapted to the target data source according to the configuration information; and writes the intermediate data into the target data source according to the data migration strategy; thereby, the present invention can effectively improve the efficiency and accuracy of data migration in a data migration scenario with multiple data sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of an application environment of a data migration method provided by an embodiment of the present invention.

[0022] Figure 2 A flowchart of a data migration method provided by one embodiment of the present invention.

[0023] Figure 3 A schematic diagram of program modules of a data migration device provided by one embodiment of the present invention.

[0024] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present invention.

[0025] Figure 5 Another structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0028] It will also be understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0030] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" in the present specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] It should be understood that the order of execution of the steps in the following embodiments does not necessarily mean the order in which they are executed. The order in which each process is executed should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0033] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0034] An embodiment of the present invention provides a data migration method that can be applied to Figure 1In the application environment shown, the client and server communicate via a network. The client includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other computer devices. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0035] See also Figure 2 An embodiment of the present invention provides a data migration method, wherein the method includes the following steps:

[0036] S100: Acquire connection information between a target data source and multiple source data sources, and construct a data migration strategy between each of the source data sources and the target data source based on the connection information;

[0037] S200: Construct a joint query statement, and read the data to be migrated from each of the source data sources according to the joint query statement;

[0038] S300: Acquire configuration information of the target data source, and convert the data to be migrated into intermediate data adapted to the target data source according to the configuration information;

[0039] S400: Writing the intermediate data into the target data source according to the data migration strategy.

[0040] In specific implementation, the data migration method of this embodiment effectively improves the efficiency and accuracy of data migration for data migration scenarios with multiple data sources through a series of systematic steps. The specific analysis is as follows:

[0041] 1. Joint processing of multiple data sources

[0042] In step S100, by obtaining connection information between the target data source and multiple source data sources and constructing a data migration policy, this method can simultaneously handle data migration requirements from multiple data sources. Compared to traditional single-source, single-target migration tools, this significantly reduces configuration complexity and development workload in multi-data source scenarios. Through unified policy management, data migration tasks across multiple data sources can be efficiently coordinated, avoiding the tedious process of individual configuration and management.

[0043] 2. Joint query optimization

[0044] In step S200, by constructing a joint query statement and reading the data to be migrated from multiple source data sources, this method can directly aggregate and integrate data at the data source level. This joint query approach avoids the inefficient operations of multiple individual queries and data splicing at the application layer, greatly improving data reading efficiency. Furthermore, the joint query statement can be optimized based on the characteristics of the data source, further improving performance.

[0045] 3. Data conversion and adaptation

[0046] In step S300, the method ensures data consistency and accuracy during the migration process by obtaining the target data source's configuration information and converting the data to be migrated into intermediate data compatible with the target data source. By performing data conversion (e.g., format conversion) before migration, write failures or data errors caused by issues like format mismatches can be effectively avoided, thereby improving the success rate and quality of data migration.

[0047] 4. Policy-driven efficient writing

[0048] In step S400, the intermediate data is written to the target data source according to the data migration policy. This method allows for flexible adjustment of write operations based on pre-set policies. Furthermore, policy-driven write operations can be optimized based on the performance characteristics of the target data source, such as by adopting batch writes or batch processing, to further improve write efficiency.

[0049] By combining the above steps, this data migration method effectively addresses the complexity of data migration in scenarios with multiple data sources. It not only improves data migration efficiency and reduces data reading and writing time, but also enhances data accuracy through data conversion. This systematic migration approach is particularly suitable for complex business scenarios such as healthcare and fintech, significantly reducing development and maintenance costs while improving the reliability and stability of data migration.

[0050] It is understandable that the data migration method provided in the embodiment of the present invention can be applied to data migration scenarios related to the medical and health field. The following is a specific example:

[0051] Scenario Description: In the healthcare sector, hospital information systems (HIS), electronic medical records (EMR), and picture acquisition systems (PACS) are typically stored in separate databases and may utilize different database management systems. To enable telemedicine and medical big data analysis, this dispersed data needs to be integrated into a unified big data platform.

[0052] Specific examples:

[0053] 1. Data source:

[0054] Source Data source 1: HIS system, which stores the patient's basic information (such as name, age, gender, etc.).

[0055] Source data source 2: EMR system, which stores patients' medical records (such as diagnosis results, treatment plans, etc.).

[0056] Source Data Source 3: PACS system, which stores patients' medical imaging data (such as X-ray, CT, etc.).

[0057] 2. Target data source: Big data platform, used to store integrated data and support telemedicine and data analysis.

[0058] 3. Application of the method of the present invention:

[0059] S100: Obtain the connection information between the big data platform and HIS, EMR, and PACS systems, and build a data migration strategy.

[0060] S200: Construct a joint query statement to read the data to be migrated from the HIS, EMR, and PACS systems. For example, a joint query statement can link a patient's name, medical history information, and imaging data to form a complete data set.

[0061] S300: Obtain the configuration information of the big data platform and convert the read data into intermediate data that is compatible with the big data platform. For example, the storage format of the image data is converted from the proprietary format of the PACS system to a universal format supported by the big data platform.

[0062] S400: Write the intermediate data to the big data platform according to the data migration strategy. Ensure the integrity and consistency of the data, for example, maintain the relationship between patient information and imaging data.

[0063] Through the data migration method of the present invention, medical data scattered in different systems can be efficiently integrated into a unified big data platform, providing a solid data foundation for telemedicine and medical big data analysis.

[0064] It is understandable that the data migration method provided in the embodiment of the present invention can also be applied to data migration scenarios related to the financial technology field. The following is a specific example:

[0065] Scenario Description: In the field of FinTech, financial institutions need to migrate data from the core banking system (CBS), risk management platform (RMP), and transaction monitoring system (TMS) to a new big data analytics platform to support applications such as risk assessment, transaction monitoring, and smart investment advisory.

[0066] Specific examples:

[0067] 1. Data source:

[0068] Source Data Source 1: Core Banking System (CBS), which stores customer account information, transaction records, etc.

[0069] Source Data Source 2: Risk Management Platform (RMP), which stores risk assessment models and risk indicators.

[0070] Source Data Source 3: Transaction Monitoring System (TMS), which stores real-time transaction data and abnormal transaction alerts.

[0071] 2. Target data source: Big data analysis platform, used to store integrated data and support risk assessment and transaction monitoring.

[0072] 3. Application of the method of the present invention:

[0073] S100: Obtain the connection information between the big data analysis platform and CBS, RMP, and TMS, and build a data migration strategy.

[0074] S200: Construct a joint query statement to read the data to be migrated from CBS, RMP, and TMS. For example, the joint query statement can link customer account information, risk assessment results, and transaction data to form a complete data set.

[0075] S300: Obtain the configuration information of the big data analysis platform and convert the read data into intermediate data that is compatible with the big data analysis platform. For example, the timestamp format of the transaction data may be converted from the proprietary format of the TMS system to a universal format supported by the big data analysis platform.

[0076] S400: Write the intermediate data to the big data analysis platform according to the data migration strategy. Ensure the integrity and consistency of the data, for example, maintain the relationship between customer account information and transaction data.

[0077] Through the data migration method of the present invention, financial data scattered in different systems can be efficiently integrated into a unified big data analysis platform, providing a solid data foundation for applications such as risk assessment, transaction monitoring and smart investment consulting.

[0078] These two examples respectively demonstrate the specific applications of the present invention in the fields of healthcare and financial technology, and illustrate how to solve data migration problems in multi-data source scenarios through a systematic data migration method, thereby improving the efficiency and accuracy of data migration.

[0079] Furthermore, in one embodiment, the data migration method, wherein obtaining connection information between a target data source and multiple source data sources, and constructing a data migration strategy between each of the source data sources and the target data source based on the connection information, specifically includes the following steps:

[0080] Get the configuration files of the target data source and multiple source data sources;

[0081] Extracting connection parameters of the target data source and each of the source data sources according to the configuration file;

[0082] Determining connection information between each of the source data sources and the target data source according to the connection parameters;

[0083] The connection information is analyzed, and a data migration strategy of a mapping relationship between each of the source data sources and the target data source and a transmission path is constructed according to the analysis result.

[0084] During specific implementation, the specific implementation process of the steps in this embodiment is roughly as follows:

[0085] 1. Get the configuration file:

[0086] Obtain the target data source and multiple source data source configuration files from a preset storage location (such as a local file system, cloud storage, or configuration management tool). These configuration files contain information such as the data source type, address, port, username, password, table structure, etc., providing the basis for subsequent connection parameter extraction.

[0087] 2. Extract connection parameters:

[0088] Parse the obtained configuration file and extract the connection parameters for the target data source and each source data source. These parameters include the data source type (such as MySQL, Oracle, PostgreSQL), address (IP address or domain name), port, username and password, table structure, and other information. Organize these parameters into a ready-to-use format in preparation for establishing the data connection.

[0089] 3. Confirm the connection information:

[0090] Based on the extracted connection parameters, verify the validity of the connection parameters (such as whether the user name and password are correct), establish connections between the target data source and each source data source, and determine the connection information between each source data source and the target data source to provide support for subsequent data migration strategy construction.

[0091] 4. Build a data migration strategy:

[0092] Analyze the acquired connection information and, based on the analysis results, build a data migration strategy for the mapping relationship between each source data source and the target data source and the transmission path.

[0093] Through the above process, this embodiment can systematically obtain the connection information between the target data source and multiple source data sources, and build an efficient and accurate data migration strategy to ensure the smooth progress of the data migration process.

[0094] Furthermore, in one embodiment, the data migration method, wherein the constructing of a joint query statement and reading the data to be migrated from each of the source data sources according to the joint query statement, specifically comprises the steps of:

[0095] Analyze the table structure and field information of each source data source, and determine the tables and fields in each source data source that participate in the joint query based on the analysis results;

[0096] Based on the tables and fields, a joint query statement is generated using preset SQL rules;

[0097] The joint query statement is executed to read the data to be migrated from each of the source data sources.

[0098] During specific implementation, the specific implementation process of the steps in this embodiment is roughly as follows:

[0099] 1. Analyze the table structure and field information of the source data source:

[0100] Traverse each source data source, connect to the corresponding database and obtain its table structure and field information. This includes metadata such as table name, field name, field type, primary key, foreign key, etc.

[0101] Compare the target data source's table structure and field requirements to determine which tables and fields in the source data source need to participate in the federated query. For example, if the target data source requires a patient's name and diagnosis, and this information is stored in a HIS system and an EMR system, respectively, you need to extract the corresponding tables and fields from both systems.

[0102] 2. Determine the tables and fields involved in the union query:

[0103] Based on the analysis results, identify the tables and fields in each source data source that need to participate in the federated query.

[0104] 3. Generate a joint query statement:

[0105] Generate a join query statement based on the specified tables and fields using pre-set SQL rules. For example, use the SQL JOIN statement to connect multiple tables and select the required fields.

[0106] Ensure that the union query statement can correctly handle data associations between different source data sources and complies with SQL syntax specifications.

[0107] 5. Execute the union query statement and read the data:

[0108] Execute the prepared joint query statement to read the data to be migrated from each source data source. Ensure the completeness and accuracy of the query results.

[0109] Through the above process, this embodiment can systematically construct a joint query statement and efficiently and accurately read the data to be migrated from multiple source data sources, providing a solid foundation for subsequent data conversion and writing.

[0110] Furthermore, in one embodiment, the data migration method, wherein the step of obtaining configuration information of the target data source and converting the data to be migrated into intermediate data adapted to the target data source according to the configuration information, specifically comprises the following steps:

[0111] Obtaining configuration information of the target data source, and determining data format requirements of the target data source according to the configuration information;

[0112] According to the data format requirement, the data to be migrated is converted into intermediate data that conforms to the data format of the target data source.

[0113] During specific implementation, the specific implementation process of the steps in this embodiment is roughly as follows:

[0114] 1. Get the configuration information of the target data source:

[0115] Read the target data source's configuration information from a pre-set storage location (such as a configuration file, database management system, or cloud service). This configuration information typically includes the target data source's data source type, table structure, field type, field length, and data constraints (such as primary keys, foreign keys, and unique constraints).

[0116] 2. Determine the data format requirements of the target data source:

[0117] Parse the configuration information to extract the target data source's field mapping rules, data type requirements, field length restrictions, etc. For example, the target data source may require certain fields to be of specific data types (such as integers, strings, dates, etc.), or have field length restrictions (such as a maximum length of 255 characters for a string field).

[0118] Based on the table structure of the target data source, determine the fields and conversion rules that need to be converted for the data to be migrated. For example, if the target data source's date field format is YYYY-MM-DD, while the source data source's date format is DD / MM / YYYY, you need to define the corresponding conversion rules.

[0119] 3. Analyze the structure and content of the data to be migrated:

[0120] Analyze the data to be migrated and extract its field structure and data types. This can be done by reading the table structure of the source data source or directly analyzing data samples.

[0121] Compare the field structure of the data to be migrated with the field structure of the target data source to determine the required field mapping and data type conversion. For example, if the fields in the source data source are named first_name and last_name, and the field in the target data source is named full_name, you need to merge these two fields into one field.

[0122] 4. Convert the data to be migrated into intermediate data:

[0123] Convert the data to be migrated based on the specified field mapping and data type conversion rules. For example, you can convert the date format from DD / MM / YYYY to YYYY-MM-DD, merge two fields into one, or truncate string data to meet the field length restrictions of the target data source.

[0124] Store the converted data as intermediate data. The format of the intermediate data should meet the requirements of the target data source. Intermediate data can be stored in memory, temporary tables, or files to facilitate subsequent data write operations.

[0125] Through the above process, this embodiment can systematically obtain the configuration information of the target data source and convert the data to be migrated into intermediate data that conforms to the data format of the target data source, thereby providing a solid foundation for subsequent data writing.

[0126] Furthermore, in one embodiment, the data migration method, wherein the step of writing the intermediate data to the target data source according to the data migration strategy, specifically comprises the steps of:

[0127] Determining a key for data migration based on data security requirements of the data to be migrated, and distributing the key to the target data source;

[0128] Encrypting the intermediate data using the key and a corresponding encryption algorithm to generate encrypted data;

[0129] According to the data migration strategy, the encrypted data is written into the target data source.

[0130] Furthermore, the data migration method, wherein the step of writing the encrypted data to the target data source according to the data migration strategy, specifically comprises the steps of:

[0131] Writing the encrypted data into the target data source using the data migration strategy, and monitoring the network bandwidth of the server where the target data source is deployed and the load of the server;

[0132] When at least one of the network bandwidth and the load is abnormal, adjusting the data migration strategy;

[0133] According to the adjusted data migration strategy, the encrypted data is rewritten into the target data source.

[0134] Furthermore, the data migration method, wherein, after writing the encrypted data to the target data source according to the data migration strategy, further specifically comprises the steps of:

[0135] Decrypting the encrypted data using the key and a corresponding decryption algorithm to generate decrypted data;

[0136] Performing a consistency check on the decrypted data and the data to be migrated to obtain a check result;

[0137] A data migration report of the data to be migrated is generated according to the verification result, and the data migration report is formatted and then output or displayed.

[0138] During specific implementation, the specific implementation process of the steps in this embodiment is roughly as follows:

[0139] 1. Determine the key and distribute it:

[0140] Based on the data security requirements of the data to be migrated, select an appropriate encryption algorithm and generate a key for data migration.

[0141] Securely distribute the generated key to the target data source. The distribution process can use secure communication protocols (such as HTTPS, SSH) or use a key management system (KMS) to ensure the secure transmission and storage of the key.

[0142] 2. Encrypt intermediate data:

[0143] The generated key and the corresponding encryption algorithm are used to encrypt the intermediate data to generate the encrypted data. The encryption process can use a symmetric encryption algorithm (such as AES) or an asymmetric encryption algorithm (such as RSA), the specific choice depends on the data security requirements and performance considerations.

[0144] Ensure that encrypted data remains secure during transmission and storage to prevent data leakage or tampering.

[0145] 3. Write encrypted data and monitor performance:

[0146] According to the data migration strategy, the encrypted data is written to the target data source. The writing process can adopt batch writing or batch processing to improve writing efficiency.

[0147] During the write process, monitor the network bandwidth and server load of the server where the target data source is deployed. You can use system monitoring tools (such as Prometheus and Grafana) or monitoring services provided by cloud platforms to obtain these indicators in real time.

[0148] 4. Adjust data migration strategy:

[0149] When abnormalities in network bandwidth and / or server load are detected (such as network congestion or server overload), the data migration strategy is adjusted according to pre-set rules. The adjustment strategy may include:

[0150] Reduce the amount of data written each time;

[0151] Increase the write interval time;

[0152] Pause the write operation and wait until the network or server status returns to normal before continuing.

[0153] According to the adjusted data migration strategy, the encrypted data is rewritten into the target data source to ensure smooth data migration.

[0154] 5. Decryption and consistency check:

[0155] The encrypted data is decrypted using the key and the corresponding decryption algorithm to generate decrypted data.

[0156] Perform a consistency check between the decrypted data and the original data to be migrated to ensure that no data loss or errors occurred during the migration process. Verification methods can include data hash value comparison, field value comparison, etc.

[0157] Based on the verification results, a data migration report is generated to record detailed information about the data migration, including migration time, data volume, verification results, etc.

[0158] The data migration report is formatted and then output or displayed. It can be displayed in text format, HTML format or through visualization tools to facilitate user review and auditing.

[0159] Through the above process, this embodiment can ensure the security, integrity and accuracy of data during the migration process, and at the same time optimize the performance and reliability of the migration process by dynamically adjusting the data migration strategy.

[0160] As can be seen from the above method embodiments, the data migration method provided by the present invention includes: obtaining connection information between a target data source and multiple source data sources, and constructing a data migration strategy for each of the source data sources and the target data source based on the connection information; constructing a joint query statement, and reading the data to be migrated from each of the source data sources based on the joint query statement; obtaining configuration information of the target data source, and converting the data to be migrated into intermediate data that is adapted to the target data source based on the configuration information; and writing the intermediate data into the target data source based on the data migration strategy. In this way, the method of the present invention can effectively improve the efficiency and accuracy of data migration in a data migration scenario with multiple data sources.

[0161] It should be understood that although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative work, and these operation steps are not necessarily performed in the order of the embodiments or flowcharts. The order of steps listed in the embodiments or flowcharts is only one way of executing the steps among many steps and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. Those of ordinary skill in the art can understand from the description of the embodiments of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in an interchangeable manner, etc. Moreover, at least a portion of the steps in the embodiments or flowcharts may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a portion of the sub-steps or stages of other steps.

[0162] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a data migration device, wherein the device includes:

[0163] A strategy building module 11 is configured to obtain connection information between a target data source and multiple source data sources, and to build a data migration strategy between each of the source data sources and the target data source based on the connection information;

[0164] The data reading module 12 is used to construct a joint query statement and read the data to be migrated from each source data source according to the joint query statement;

[0165] The data conversion module 13 is configured to obtain configuration information of the target data source and convert the data to be migrated into intermediate data adapted to the target data source according to the configuration information;

[0166] The data writing module 14 is configured to write the intermediate data into the target data source according to the data migration strategy.

[0167] Furthermore, in one embodiment, in the data migration device, the policy building module 11 is specifically configured to:

[0168] Get the configuration files of the target data source and multiple source data sources;

[0169] Extracting connection parameters of the target data source and each of the source data sources according to the configuration file;

[0170] Determining connection information between each of the source data sources and the target data source according to the connection parameters;

[0171] The connection information is analyzed, and a data migration strategy of a mapping relationship between each of the source data sources and the target data source and a transmission path is constructed according to the analysis result.

[0172] Furthermore, in one embodiment, in the data migration device, the data reading module 12 is specifically configured to:

[0173] Analyze the table structure and field information of each source data source, and determine the tables and fields in each source data source that participate in the joint query based on the analysis results;

[0174] Based on the tables and fields, a joint query statement is generated using preset SQL rules;

[0175] The joint query statement is executed to read the data to be migrated from each of the source data sources.

[0176] Furthermore, in one embodiment, in the data migration device, the data conversion module 13 is specifically configured to:

[0177] Obtaining configuration information of the target data source, and determining data format requirements of the target data source according to the configuration information;

[0178] According to the data format requirement, the data to be migrated is converted into intermediate data that conforms to the data format of the target data source.

[0179] Furthermore, in one embodiment, in the data migration device, the data writing module 14 is specifically configured to:

[0180] Determining a key for data migration based on data security requirements of the data to be migrated, and distributing the key to the target data source;

[0181] Encrypting the intermediate data using the key and a corresponding encryption algorithm to generate encrypted data;

[0182] According to the data migration strategy, the encrypted data is written into the target data source.

[0183] Furthermore, the data migration device, wherein writing the encrypted data to the target data source according to the data migration strategy, specifically includes:

[0184] Writing the encrypted data into the target data source using the data migration strategy, and monitoring the network bandwidth of the server where the target data source is deployed and the load of the server;

[0185] When at least one of the network bandwidth and the load is abnormal, adjusting the data migration strategy;

[0186] According to the adjusted data migration strategy, the encrypted data is rewritten into the target data source.

[0187] Furthermore, the data migration device, after writing the encrypted data into the target data source according to the data migration strategy, further specifically includes:

[0188] Decrypting the encrypted data using the key and a corresponding decryption algorithm to generate decrypted data;

[0189] Performing a consistency check on the decrypted data and the data to be migrated to obtain a check result;

[0190] A data migration report of the data to be migrated is generated according to the verification result, and the data migration report is formatted and then output or displayed.

[0191] It should be noted that, in the embodiment of the device of the present invention, the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the aforementioned method embodiment part and will not be repeated here.

[0192] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which can be a server, and its internal structure diagram can be as follows: Figure 4As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of the data migration method in any of the above method embodiments are implemented.

[0193] Based on the above method embodiment, another embodiment of the present invention further provides a computer device, which can be a client, and its internal structure diagram can be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions or steps of the client side of the data migration method in any of the above method embodiments are implemented.

[0194] Those skilled in the art will understand that Figure 4 and Figure 5 The structural diagram shown in the figure is only a schematic diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than shown in the figure, or combine certain components, or have a different component arrangement.

[0195] The processor referred to herein may be a CPU, other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or any conventional processor, etc.

[0196] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of the computer device, and in other embodiments, it can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory can also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program. The memory can also be used to temporarily store data that has been output or is about to be output.

[0197] Based on the above method embodiments, another embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the data migration method described in any of the above method embodiments. The computer-readable storage medium may be non-volatile or volatile.

[0198] It should be noted that the above-mentioned functions or steps that can be implemented by computer-readable storage media or computer devices, and the technical effects brought about by the functions / steps, can be found in the relevant descriptions in the aforementioned method embodiments. To avoid repetition, they will not be described one by one here.

[0199] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM). The disclosed memory components or memories of the operating environments described herein are intended to comprise one or more of these and / or any other suitable types of memory.

[0200] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, in the embodiment of the device of the present invention, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual application, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0201] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0202] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0204] It should be noted that if software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A data migration method, characterized in that: include: Acquire connection information between a target data source and multiple source data sources, and construct a data migration strategy between each of the source data sources and the target data source based on the connection information; Constructing a joint query statement, and reading the data to be migrated from each of the source data sources according to the joint query statement; Acquiring configuration information of the target data source, and converting the data to be migrated into intermediate data adapted to the target data source according to the configuration information; According to the data migration strategy, the intermediate data is written into the target data source.

2. The data migration method according to claim 1, characterized in that: The obtaining of connection information between the target data source and the plurality of source data sources, and constructing a data migration strategy between each of the source data sources and the target data source according to the connection information, includes: Get the configuration files of the target data source and multiple source data sources; Extracting connection parameters of the target data source and each of the source data sources according to the configuration file; Determining connection information between each of the source data sources and the target data source according to the connection parameters; The connection information is analyzed, and a data migration strategy of a mapping relationship between each of the source data sources and the target data source and a transmission path is constructed according to the analysis result.

3. The data migration method according to claim 1, wherein: The constructing of a joint query statement and reading the data to be migrated from each of the source data sources according to the joint query statement includes: Analyze the table structure and field information of each source data source, and determine the tables and fields in each source data source that participate in the joint query based on the analysis results; Based on the tables and fields, a joint query statement is generated using preset SQL rules; The joint query statement is executed to read the data to be migrated from each of the source data sources.

4. The data migration method according to claim 1, wherein: The acquiring of the configuration information of the target data source and converting the to-be-migrated data into intermediate data adapted to the target data source according to the configuration information includes: Obtaining configuration information of the target data source, and determining data format requirements of the target data source according to the configuration information; According to the data format requirement, the data to be migrated is converted into intermediate data that conforms to the data format of the target data source.

5. The data migration method according to claim 1, wherein: Writing the intermediate data into the target data source according to the data migration strategy includes: Determining a key for data migration based on data security requirements of the data to be migrated, and distributing the key to the target data source; Encrypting the intermediate data using the key and a corresponding encryption algorithm to generate encrypted data; According to the data migration strategy, the encrypted data is written into the target data source.

6. The data migration method according to claim 5, characterized in that: Writing the encrypted data into the target data source according to the data migration strategy includes: Writing the encrypted data into the target data source using the data migration strategy, and monitoring the network bandwidth of the server where the target data source is deployed and the load of the server; When at least one of the network bandwidth and the load is abnormal, adjusting the data migration strategy; According to the adjusted data migration strategy, the encrypted data is rewritten into the target data source.

7. The data migration method according to claim 5 or 6, characterized in that: After writing the encrypted data into the target data source according to the data migration strategy, the method further includes: Decrypting the encrypted data using the key and a corresponding decryption algorithm to generate decrypted data; Performing a consistency check on the decrypted data and the data to be migrated to obtain a check result; A data migration report of the data to be migrated is generated according to the verification result, and the data migration report is formatted and then output or displayed.

8. A data migration device, characterized in that: include: a strategy building module, configured to obtain connection information between a target data source and a plurality of source data sources, and to build a data migration strategy between each of the source data sources and the target data source according to the connection information; A data reading module, configured to construct a joint query statement and read the data to be migrated from each of the source data sources according to the joint query statement; a data conversion module, configured to obtain configuration information of the target data source and convert the data to be migrated into intermediate data adapted to the target data source according to the configuration information; A data writing module is used to write the intermediate data into the target data source according to the data migration strategy.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data migration method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data migration method according to any one of claims 1 to 7 is implemented.

Images