Data migration method and device, electronic equipment and medium

Abstract

The invention provides a data migration method and device, electronic equipment and a medium, and is applied to a data migration system, and the data migration system comprises a GaussDB database and a scheduling platform; the method comprises the steps that a database structure and an SSIS operation process of data assets stored in an SQL Server database by a target enterprise are obtained, the database structure is converted into a first target SQL script, and the SSIS operation process is converted into a second target SQL script; the scheduling platform determines a task scheduling strategy and a resource configuration strategy for each data asset, executes a first target SQL script and a second target SQL script, migrates a database structure and an SSIS operation process of the data assets to a GaussDB database, migrates data of the data assets to the database structure of the GaussDB database, and transmits the data of the data assets to the GaussDB database; and safe and efficient data migration is realized.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data migration method, device, electronic device and medium. Background Art

[0002] With the rapid development of information technology, data has become the core asset of enterprises. However, traditional data processing processes often rely on foreign technology stacks, such as SQL Server databases and SSIS (SQL Server Integration Services) data integration services, which not only poses risks in data security and privacy protection, but also limits the further improvement of data processing efficiency and flexibility. At the same time, facing the processing needs of massive data, traditional methods are difficult to quickly respond to market changes and business needs. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a data migration method, device, electronic device and medium to safely and efficiently migrate the data assets of an enterprise to another database to ensure data security.

[0004] A data migration method provided in an embodiment of the present application is applied to a data migration system, wherein the data migration system includes a GaussDB database and a scheduling platform; the method includes:

[0005] Obtain the database structure and SSIS operation flow of the data assets of the target enterprise stored in the SQL Server database, convert the database structure into a first target SQL script, and convert the SSIS operation flow into a second target SQL script; the first target SQL script and the second target SQL script are compatible with the GaussDB database;

[0006] The scheduling platform determines the task scheduling strategy and resource allocation strategy for each data asset based on the pre-configured scheduling rules;

[0007] Execute the first target SQL script and the second target SQL script based on the task scheduling strategy and the resource allocation strategy to migrate the database structure and SSIS job flow of the data assets in the SQL Server database to the GaussDB database;

[0008] Based on the task scheduling strategy and resource allocation strategy, the data of the data assets is migrated to the database structure of the GaussDB database.

[0009] In some embodiments, in the data migration method, converting the database structure into a first target SQL script includes:

[0010] Export the database structure information of data assets in the SQL Server database;

[0011] Determine whether the data attributes and / or business requirements of the data assets in the SQL Server database and the GaussDB database match;

[0012] If there is no match, determine the target conversion logic between the mismatched data attributes and / or business requirements;

[0013] Determine whether there is any missing index for the data asset in the SQL Server database;

[0014] If present, determine the missing index;

[0015] Based on the database structure information of the data asset, the target conversion logic between the mismatched data attributes and the missing indexes, the database structure is converted into a first target SQL script.

[0016] In some embodiments, in the data migration method, converting the SSIS job flow into a second target SQL script includes:

[0017] Determine new operational processes based on new business requirements for data assets;

[0018] Based on the SSIS job flow and the newly added job flow, the SSIS job flow is converted into a second target SQL script.

[0019] In some embodiments, in the data migration method, converting the database structure into a first target SQL script and converting the SSIS job flow into a second target SQL script include:

[0020] Based on the pre-configured script conversion tool, the database structure is converted into the first target SQL script, and the SSIS job flow is converted into the second target SQL script;

[0021] Execute the first target SQL script and the second target SQL script in the test environment to obtain test results of the first target SQL script and the second target SQL script;

[0022] The first target SQL script and the second target SQL script are adjusted based on the test result to obtain adjusted first target SQL script and second target SQL script.

[0023] In some embodiments, in the data migration method, the scheduling platform determines the task scheduling strategy and resource allocation strategy for each data asset based on the pre-configured scheduling rules; including:

[0024] Obtain the task priority of each data asset, and obtain the real-time system load and remaining resource information of the scheduling platform, SQL Server database, and GaussDB database;

[0025] Based on the task priority of each data asset, and obtaining the real-time system load and remaining resource information of the scheduling platform, SQL Server database and GaussDB database, determine the task scheduling strategy and resource allocation strategy for each data asset.

[0026] In some embodiments, in the data migration method, after determining the task scheduling strategy and resource allocation strategy for each data asset, the method includes:

[0027] Sending the task scheduling strategy and resource allocation strategy of each data asset to the target terminal device, and displaying the task scheduling strategy and resource allocation strategy of each data asset on the graphical user interface of the target terminal device;

[0028] Receive adjustment instructions for task scheduling policies and / or resource allocation policies for at least one type of data asset sent by a target terminal device, and update the task scheduling policies and / or resource allocation policies for the corresponding type of data assets; the adjustment instructions are triggered based on adjustment controls of a graphical user interface of the target terminal device.

[0029] In some embodiments, in the data migration method, migrating the data of the data asset to the database structure of the GaussDB database includes:

[0030] Obtaining data to be encrypted from the data of the data asset;

[0031] Encrypting the data to be encrypted of the data asset based on a pre-configured encryption algorithm to obtain encrypted data;

[0032] The encrypted data is migrated to the database structure of the GaussDB database.

[0033] In some embodiments, a data migration device is also provided, which is applied to a data migration system, wherein the data migration system includes a GaussDB database and a scheduling platform; the device includes:

[0034] An acquisition module is used to acquire the database structure and SSIS operation flow of the data assets of the target enterprise stored in the SQL Server database, convert the database structure into a first target SQL script, and convert the SSIS operation flow into a second target SQL script; the first target SQL script and the second target SQL script are compatible with the GaussDB database;

[0035] A determination module, configured to enable the scheduling platform to determine a task scheduling strategy and a resource allocation strategy for each data asset based on a pre-configured scheduling rule;

[0036] A first migration module is used to execute the first target SQL script and the second target SQL script based on the task scheduling strategy and the resource allocation strategy, and migrate the database structure and SSIS operation process of the data assets in the SQL Server database to the GaussDB database;

[0037] The second migration module migrates the data of the data assets to the database structure of the GaussDB database based on the task scheduling strategy and resource allocation strategy.

[0038] In some embodiments, an electronic device is also provided, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the data migration method are performed.

[0039] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data migration method are executed.

[0040] The present application provides a data migration method, device, electronic device and medium. The data migration method is applied to a data migration system, wherein the data migration system includes a GaussDB database and a scheduling platform. The method obtains the database structure and SSIS operation flow of the data assets stored in the SQL Server database of the target enterprise, converts the database structure into a first target SQL script, and converts the SSIS operation flow into a second target SQL script. The first target SQL script and the second target SQL script are compatible with the GaussDB database. The scheduling platform determines the task scheduling strategy and resource allocation strategy for each data asset based on the pre-configured scheduling rules. The first target SQL script and the second target SQL script are executed based on the task scheduling strategy and the resource allocation strategy to migrate the database structure and the SSIS operation flow of the data assets in the SQL Server database to the GaussDB database. Based on the task scheduling strategy and the resource allocation strategy, the data of the data assets is migrated to the database structure of the GaussDB database. The data migration method described in the present application automatically analyzes SQL Server databases. The Server database structure and SSIS job flow are converted into SQL scripts and DolphinScheduler job configurations that are compatible with the GaussDB database. This has high processing efficiency and introduces auxiliary configuration functions into the scheduling platform to intelligently recommend the best task scheduling strategies and resource allocation solutions, reduce manual intervention, and improve configuration efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flow chart of the data migration method according to an embodiment of the present application is shown;

[0043] Figure 2 A flow chart of a method for converting a database structure into a first target SQL script according to an embodiment of the present application is shown;

[0044] Figure 3 A flowchart of a method for converting an SSIS job flow into a second target SQL script according to an embodiment of the present application is shown;

[0045] Figure 4 A flow chart of a method for determining a task scheduling strategy and a resource allocation strategy for each data asset according to an embodiment of the present application is shown;

[0046] Figure 5 A schematic diagram of the structure of the data migration device according to an embodiment of the present application is shown;

[0047] Figure 6 A schematic structural diagram of an electronic device described in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0049] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0050] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0051] With the rapid development of information technology, data has become the core asset of enterprises. However, traditional data processing processes often rely on foreign technology stacks, such as SQL Server databases and SSIS (SQL Server Integration Services) data integration services, which not only poses risks in data security and privacy protection, but also limits the further improvement of data processing efficiency and flexibility. At the same time, facing the processing needs of massive data, traditional methods are difficult to quickly respond to market changes and business needs.

[0052] Based on this, the embodiments of the present application provide a data migration method, device, electronic device and medium. The data migration method is applied to a data migration system, and the data migration system includes a GaussDB database and a scheduling platform; the method obtains the database structure and SSIS operation flow of the data assets stored in the SQL Server database of the target enterprise, converts the database structure into a first target SQL script, and converts the SSIS operation flow into a second target SQL script; the first target SQL script and the second target SQL script are compatible with the GaussDB database; the scheduling platform determines the task scheduling strategy and resource allocation strategy for each data asset based on the pre-configured scheduling rules; executes the first target SQL script and the second target SQL script based on the task scheduling strategy and the resource allocation strategy, and migrates the database structure and the SSIS operation flow of the data assets in the SQL Server database to the GaussDB database; based on the task scheduling strategy and the resource allocation strategy, migrates the data of the data assets to the database structure of the GaussDB database; the data migration method described in the embodiments of the present application automatically analyzes SQL The Server database structure and SSIS job flow are converted into SQL scripts and DolphinScheduler job configurations that are compatible with the GaussDB database. This has high processing efficiency and introduces auxiliary configuration functions into the scheduling platform to intelligently recommend the best task scheduling strategies and resource allocation solutions, reduce manual intervention, and improve configuration efficiency and accuracy.

[0053] Please refer to Figure 1 , Figure 1 A flow chart of a data migration method according to an embodiment of the present application is shown; the data migration method is applied to a data migration system, the data migration system includes a GaussDB database and a scheduling platform; the method includes the following steps S101-S104:

[0054] S101, obtaining the database structure and SSIS operation flow of the data assets of the target enterprise stored in the SQL Server database, converting the database structure into a first target SQL script, and converting the SSIS operation flow into a second target SQL script; the first target SQL script and the second target SQL script are compatible with the GaussDB database;

[0055] S102, the scheduling platform determines a task scheduling strategy and a resource allocation strategy for each data asset based on pre-configured scheduling rules;

[0056] S103, executing the first target SQL script and the second target SQL script based on the task scheduling strategy and the resource allocation strategy, and migrating the database structure and SSIS operation process of the data assets in the SQL Server database to the GaussDB database;

[0057] S104. Based on the task scheduling strategy and resource allocation strategy, migrate the data of the data assets to the database structure of the GaussDB database.

[0058] In step S101, the database structure and SSIS job flow of the target enterprise's data assets stored in the SQL Server database are obtained, the database structure is converted into a first target SQL script, and the SSIS job flow is converted into a second target SQL script; the first target SQL script and the second target SQL script are compatible with the GaussDB database.

[0059] Data assets refer to data resources that are owned or controlled by individuals or enterprises and recorded in physical or electronic form. Data assets have the characteristics of data ownership (exploration rights, use rights, ownership), value, measurability, and readability. In modern enterprises, data assets are of great significance to the decision-making, business operations, and market competition of enterprises.

[0060] In the embodiment of the present application, the data assets specifically refer to data assets in a SQL Server database.

[0061] The database structure refers to the structure of an associated and structured data set reasonably stored on a computer's storage device. The database structure includes a database, a data table, a field, an index, and the like.

[0062] The database is a collection of related data. A database consists of one or a group of data tables and is used to store and manage data.

[0063] The data table is the basic unit for storing data in a database, and is composed of a group of data records. Each table contains the same type of information. A table is actually a two-dimensional table. For example, the test scores of all students in a class can be stored in a table. Each row in the table corresponds to a student, and this row includes the student's student number, name, and grades in each course.

[0064] Each column in a data table is called a field. Each field has corresponding descriptive information, such as data attributes, data width, etc. Fields are the basic components of a data table and are used to store specific data values.

[0065] In order to improve the efficiency of accessing the database, you can use indexes on the database. An index is actually a special type of table that contains the values ​​of key fields (defined by the user) and pointers to the actual record locations. These values ​​and pointers are stored in a specific order (also defined by the user), so that the required data records can be found at a faster speed.

[0066] The SSIS (SQL Server Integration Services) operation process mainly involves the data extraction, transformation and loading (ETL) process.

[0067] Obtain the database structure and SSIS job flow of the target enterprise's data assets stored in the SQL Server database. Specifically, you can use tools such as SQL Server Management Studio (SSMS) to view the database structure, analyze SSIS packages, and understand the job flow.

[0068] In step S101, the SQL Server database structure and SSIS job flow are automatically analyzed and converted into SQL scripts and DolphinScheduler job configurations that are compatible with the GaussDB database; further, potential problems in the data migration process are identified and optimized during the conversion process, such as data attribute mismatch, missing indexes, etc., to ensure the smoothness of the migration process and data integrity.

[0069] For details, please refer to Figure 2 , the converting of the database structure into the first target SQL script comprises the following steps S201-S206:

[0070] S201, exporting database structure information of data assets in the SQL Server database;

[0071] S202, determining whether the data attributes and / or business requirements of the data assets in the SQL Server database and the GaussDB database match;

[0072] S203, if there is no match, determining the target conversion logic between the unmatched data attributes and / or business requirements;

[0073] S204, determining whether there is any missing index for the data asset in the SQL Server database;

[0074] S205. If it exists, determine the missing index;

[0075] S206: Based on the database structure information of the data asset, the target conversion logic between the mismatched data attributes, and the missing indexes, convert the database structure into a first target SQL script.

[0076] Here, the data attributes include data type, data format, etc.

[0077] The business needs, including subsequent data analysis, etc., may discover new business needs or data needs that are different from the original ones based on the properties of the GaussDB database during the data migration process. For example, the requirements for connecting with external systems are different; new business needs such as machine learning and modeling.

[0078] Determine whether the data attributes and / or business requirements of the data assets in the SQL Server database and the GaussDB database match, that is, determine whether the data attributes of the data assets in the SQL Server database and the GaussDB database match, and / or determine whether the business requirements of the data assets in the SQL Server database and the GaussDB database match.

[0079] When performing data migration, if there are differences between the SQL Server database and the GaussDB database in terms of data type, data format, etc., data conversion and cleaning are required; based on this, the target conversion logic between the mismatched data attributes is configured. The target conversion logic includes data type conversion, data format adjustment, null value processing, as well as complex tasks such as denoising and feature extraction.

[0080] When the data attributes in the SQL Server database and the GaussDB database do not match, the configured target conversion logic is used to convert the attributes of the data in the SQL Server database into the data attributes required in the GaussDB database, for example, converting data in format A into data in format B.

[0081] When the business requirements in the SQL Server database and the GaussDB database do not match, the configured target conversion logic is used to process the data in the SQL Server database based on the business requirements in the GaussDB database, such as adding a new table to store the extracted new features.

[0082] In the database environment, index missing may occur due to human operational errors, file corruption, deletion and reconstruction of index objects, and incompatibility between SQLServer database and GaussDB database. Based on this, reorganizing the original indexes during migration can not only ensure the smoothness of the migration process and data integrity, but also sort out and solve the problems of the original indexes.

[0083] Please refer to Figure 3 The step of converting the SSIS job flow into a second target SQL script includes the following steps S301-S302:

[0084] S301. Determine new operation processes based on new business requirements for data assets;

[0085] S302: Based on the SSIS job flow and the newly added job flow, convert the SSIS job flow into a second target SQL script.

[0086] While migrating data, we fully sorted out business needs and solved the business needs that were not realized by the original database by adding new operating processes and solving the original business that could not be realized.

[0087] In any optional embodiment, converting the database structure into a first target SQL script and converting the SSIS job flow into a second target SQL script includes:

[0088] Based on the pre-configured script conversion tool, the database structure is converted into the first target SQL script, and the SSIS job flow is converted into the second target SQL script;

[0089] Execute the first target SQL script and the second target SQL script in the test environment to obtain test results of the first target SQL script and the second target SQL script;

[0090] The first target SQL script and the second target SQL script are adjusted based on the test result to obtain adjusted first target SQL script and second target SQL script.

[0091] The script conversion tool, such as AI large model, etc.

[0092] In other words, script conversion tools such as AI models can be used to automatically convert SQL Server database structures and SSIS job processes into SQL scripts compatible with the GaussDB database, reducing manual operations and improving work efficiency.

[0093] Executing the converted SQL script in a test environment can detect possible errors, performance issues, or compatibility issues in advance, helping to resolve these issues before deployment in the actual production environment, reducing risks and costs; adjusting the SQL script based on the test results can ensure the correctness and efficiency of the script in the target environment.

[0094] In some embodiments, the first target SQL script and the second target SQL script are adjusted by, specifically, customizing and optimizing the scripts according to actual needs.

[0095] In the step S102, the scheduling platform determines a task scheduling strategy and a resource allocation strategy for each data asset based on pre-configured scheduling rules.

[0096] The task scheduling strategy includes the execution order, execution time and execution frequency of the migration tasks.

[0097] Exemplarily, the task scheduling strategies include: one-time scheduling: applicable to data migration tasks that need to be executed immediately and only once; interval scheduling: periodically executing data migration tasks at set time intervals, such as once every hour or every day; fixed time point scheduling: executing data migration tasks at specific time points, such as 2 a.m. every day; Cron expression scheduling: defining complex scheduling rules through Cron expressions, such as "execute at 9 a.m. and 5 p.m. every Monday to Friday."

[0098] The resource allocation strategy includes the computing resources, storage resources, and network resources required during the migration process.

[0099] Exemplarily, specific resource allocation strategies include: computing resource configuration: reasonably configure CPU, memory and other computing resources according to the complexity of the data migration task and the amount of data. For large-scale data migration tasks for enterprise data assets in the embodiments of the present application, in some optional embodiments, high-performance computing nodes or clusters are configured to accelerate the migration process; storage resource configuration: ensure that there is sufficient storage space to store temporary data and target data during the migration process; network resource configuration: optimize network bandwidth and latency to ensure the efficiency and stability of data migration. During the migration process, it is necessary to monitor the usage of network resources and make adjustments and optimizations as needed.

[0100] Please refer to Figure 4 The scheduling platform determines the task scheduling strategy and resource allocation strategy for each data asset based on the pre-configured scheduling rules; including the following steps S401-S402:

[0101] S401, obtaining the task priority of each data asset, and obtaining the real-time system load and remaining resource information of the scheduling platform, SQL Server database and GaussDB database;

[0102] S402. Based on the task priority of each data asset, and obtaining the real-time system load and remaining resource information of the scheduling platform, SQL Server database and GaussDB database, determine the task scheduling strategy and resource allocation strategy for each data asset.

[0103] The migration of each data asset is regarded as a migration task. Different data assets have different task priorities, so that resources are allocated to data assets with higher priorities and the migration order of data assets is determined.

[0104] Generally speaking, the higher the priority of a data asset, the higher its order during migration.

[0105] At the same time, in the embodiments of the present application, when determining the task scheduling strategy and resource allocation strategy in real time, by understanding the system load and remaining resources in real time, it is possible to ensure that resources are reasonably allocated to avoid resource idleness or overuse. For example, during high-load periods, resources can be allocated to key tasks first, while during low-load periods, some non-critical but resource-intensive tasks can be executed.

[0106] Real-time resource information helps system administrators or automated tools adjust resource configuration according to current load conditions, thereby optimizing system performance. For example, in SQL Server or GaussDB, memory allocation, CPU usage, etc. can be dynamically adjusted according to real-time load to improve query speed and processing power.

[0107] In the embodiment of the present application, the scheduling platform is DolphinScheduler.

[0108] The AI-assisted configuration function is introduced in DolphinScheduler. It intelligently recommends the best task scheduling strategy and resource allocation strategy based on historical job data, resource usage, etc., reduces manual intervention, and improves configuration efficiency and accuracy.

[0109] Specifically, by combining AI algorithms such as reinforcement learning and genetic algorithms, the task execution order of migration tasks and the resource allocation of each migration task are dynamically adjusted according to factors such as real-time system load, task priority, and resource availability, so as to achieve efficient use of resources and rapid completion of tasks.

[0110] In some embodiments, in the data migration method, after determining the task scheduling strategy and resource allocation strategy for each data asset, the method includes:

[0111] Sending the task scheduling strategy and resource allocation strategy of each data asset to the target terminal device, and displaying the task scheduling strategy and resource allocation strategy of each data asset on the graphical user interface of the target terminal device;

[0112] Receive adjustment instructions for task scheduling policies and / or resource allocation policies for at least one type of data asset sent by a target terminal device, and update the task scheduling policies and / or resource allocation policies for the corresponding type of data assets; the adjustment instructions are triggered based on adjustment controls of a graphical user interface of the target terminal device.

[0113] That is to say, in the embodiment of the present application, the task scheduling strategy and / or resource allocation strategy of the data asset is adjusted through the visual interface, for example, the order of the migration tasks is adjusted, the resource allocation of the migration tasks is adjusted, and the like.

[0114] In some optional embodiments, a visual management platform integrating AI and deep learning technologies is developed to provide users with an intuitive operating interface and intelligent analysis tools. The platform supports real-time monitoring of data processing flows, historical data analysis, performance tuning suggestions and other functions, helping users to quickly locate problems, optimize processes, and realize intelligent management of data processing flows.

[0115] In the step S103, the first target SQL script and the second target SQL script are executed based on the task scheduling strategy and the resource allocation strategy to migrate the database structure and SSIS job flow of the data assets in the SQL Server database to the GaussDB database.

[0116] Through precise task scheduling and resource allocation, resource contention and task conflicts during the migration process can be avoided, ensuring the smooth progress of the migration; reasonable resource allocation strategies can maximize the use of system resources, improve migration efficiency, and reduce migration costs; ensure business continuity: through migration verification and adjustment, it can be ensured that the database structure and operation process after migration can run correctly, ensuring business continuity.

[0117] It should be noted that when executing the first target SQL script and the second target SQL script based on the task scheduling strategy and resource allocation strategy, and migrating the database structure and SSIS job process of the data assets in the SQL Server database to the GaussDB database, it is still necessary to monitor the real-time system load and remaining resource information of the scheduling platform, SQL Server database and GaussDB database in real time, so as to adjust the task scheduling strategy and resource allocation strategy in real time. Through systematic migration steps and monitoring mechanisms, problems arising during the migration process can be discovered and handled in a timely manner, thereby reducing migration risks.

[0118] Migrate the database structure of the data assets in the SQL Server database to the GaussDB database, that is, build the database, table, index, etc. corresponding to the data assets in the GaussDB database.

[0119] In the step S104, based on the task scheduling strategy and resource allocation strategy, the data of the data assets is migrated to the database structure of the GaussDB database.

[0120] Here, the specific data of the data assets is migrated to the constructed database. The migration process also follows the task scheduling strategy and resource allocation strategy.

[0121] The step of migrating the data of the data asset to the database structure of the GaussDB database includes:

[0122] Obtaining data to be encrypted from the data of the data asset;

[0123] Encrypting the data to be encrypted of the data asset based on a pre-configured encryption algorithm to obtain encrypted data;

[0124] The encrypted data is migrated to the database structure of the GaussDB database.

[0125] Here, the pre-configured encryption algorithm is a domestic encryption algorithm.

[0126] Encrypt data during the migration process to ensure the security of data assets during the migration process.

[0127] When data is migrated to the GaussDB database, data transmission, storage, and processing must all be encrypted and protected using domestic encryption algorithms to ensure data security in the trusted computing environment.

[0128] By migrating data assets to the GaussDB database and encrypting them using domestic encryption algorithms, it is easier to integrate domestic CPU, operating system, database, middleware and other software and hardware resources, realize domestic substitution of the entire data processing process, reduce dependence on external technology, and reduce the risk of data leakage; at the same time, the application of data encryption and privacy protection technology ensures the security of data during transmission, storage and processing, and enhances the company's data protection capabilities; it also combines AI technology to continuously optimize and adapt the performance of domestic software and hardware to ensure that performance is not degraded or even improved during the replacement process.

[0129] Based on the same inventive concept, a data migration device corresponding to the data migration method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned data migration method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0130] Please refer to Figure 5 , Figure 5The structural diagram of the data migration device described in the embodiment of the present application is shown; the data migration device is applied to a data migration system, and the data migration system includes a GaussDB database and a scheduling platform; the device includes:

[0131] The acquisition module 501 is used to acquire the database structure and SSIS operation flow of the data assets of the target enterprise stored in the SQL Server database, convert the database structure into a first target SQL script, and convert the SSIS operation flow into a second target SQL script; the first target SQL script and the second target SQL script are compatible with the GaussDB database;

[0132] A determination module 502 is used to enable the scheduling platform to determine a task scheduling strategy and a resource allocation strategy for each data asset based on a pre-configured scheduling rule;

[0133] The first migration module 503 is used to execute the first target SQL script and the second target SQL script based on the task scheduling strategy and the resource allocation strategy to migrate the database structure and SSIS operation process of the data assets in the SQL Server database to the GaussDB database;

[0134] The second migration module 504 migrates the data of the data assets to the database structure of the GaussDB database based on the task scheduling strategy and resource allocation strategy.

[0135] In some embodiments, in the data migration device, the acquisition module, when converting the database structure into the first target SQL script, is specifically used to:

[0136] Export the database structure information of data assets in the SQL Server database;

[0137] Determine whether the data attributes and / or business requirements of the data assets in the SQL Server database and the GaussDB database match;

[0138] If there is no match, determine the target conversion logic between the mismatched data attributes and / or business requirements;

[0139] Determine whether there is any missing index for the data asset in the SQL Server database;

[0140] If it exists, determine the missing index;

[0141] Based on the database structure information of the data asset, the target conversion logic between the mismatched data attributes and the missing indexes, the database structure is converted into a first target SQL script.

[0142] In some embodiments, in the data migration device, the acquisition module, when converting the SSIS job flow into the second target SQL script, is specifically used to:

[0143] Determine new operational processes based on new business requirements for data assets;

[0144] Based on the SSIS job flow and the newly added job flow, the SSIS job flow is converted into a second target SQL script.

[0145] In some embodiments, in the data migration device, the acquisition module, when converting the database structure into the first target SQL script and converting the SSIS job flow into the second target SQL script, is specifically used to:

[0146] Based on the pre-configured script conversion tool, the database structure is converted into the first target SQL script, and the SSIS job flow is converted into the second target SQL script;

[0147] Execute the first target SQL script and the second target SQL script in the test environment to obtain test results of the first target SQL script and the second target SQL script;

[0148] The first target SQL script and the second target SQL script are adjusted based on the test result to obtain adjusted first target SQL script and second target SQL script.

[0149] In some embodiments, in the data migration device, the determination module is specifically used to:

[0150] Obtain the task priority of each data asset, and obtain the real-time system load and remaining resource information of the scheduling platform, SQL Server database, and GaussDB database;

[0151] Based on the task priority of each data asset, and obtaining the real-time system load and remaining resource information of the scheduling platform, SQL Server database and GaussDB database, determine the task scheduling strategy and resource allocation strategy for each data asset.

[0152] In some embodiments, in the data migration device, the determination module is further used to, after determining the task scheduling strategy and resource allocation strategy for each data asset, send the task scheduling strategy and resource allocation strategy for each data asset to the target terminal device, and display the task scheduling strategy and resource allocation strategy for each data asset on the graphical user interface of the target terminal device;

[0153] Receive adjustment instructions for task scheduling policies and / or resource allocation policies for at least one type of data asset sent by a target terminal device, and update the task scheduling policies and / or resource allocation policies for the corresponding type of data assets; the adjustment instructions are triggered based on adjustment controls of a graphical user interface of the target terminal device.

[0154] In some embodiments, in the data migration device, the second migration module, when migrating the data of the data asset to the database structure of the GaussDB database, is specifically used to:

[0155] Obtaining data to be encrypted from the data of the data asset;

[0156] Encrypting the data to be encrypted of the data asset based on a pre-configured encryption algorithm to obtain encrypted data;

[0157] The encrypted data is migrated to the database structure of the GaussDB database.

[0158] Based on the same inventive concept, an electronic device corresponding to the data migration method is also provided in the embodiment of the present application. Since the principle of solving the problem by the electronic device in the embodiment of the present application is similar to the above-mentioned data migration method in the embodiment of the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0159] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of the electronic device described in an embodiment of the present application is shown; the electronic device 600 includes: a processor 601, a memory 602 and a bus, the memory 602 stores machine-readable instructions executable by the processor 601, and when the electronic device 600 is running, the processor 601 communicates with the memory 602 through the bus, and when the machine-readable instructions are executed by the processor 601, the steps of the data migration method are executed.

[0160] Based on the same inventive concept, a computer-readable storage medium corresponding to the data migration method is also provided in the embodiment of the present application. Since the principle of solving the problem by the computer-readable storage medium in the embodiment of the present application is similar to the above-mentioned data migration method in the embodiment of the present application, the implementation of the computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0161] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data migration method are executed.

[0162] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0163] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0165] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0166] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data migration method, characterized in that: Applied to a data migration system, the data migration system includes a GaussDB database and a scheduling platform; the method includes: Obtain the database structure and SSIS operation flow of the data assets of the target enterprise stored in the SQL Server database, convert the database structure into a first target SQL script, and convert the SSIS operation flow into a second target SQL script; the first target SQL script and the second target SQL script are compatible with the GaussDB database; The scheduling platform determines the task scheduling strategy and resource allocation strategy for each data asset based on the pre-configured scheduling rules; Execute the first target SQL script and the second target SQL script based on the task scheduling strategy and the resource allocation strategy to migrate the database structure and SSIS job flow of the data assets in the SQL Server database to the GaussDB database; Based on the task scheduling strategy and resource allocation strategy, the data of the data assets is migrated to the database structure of the GaussDB database.

2. The data migration method according to claim 1, characterized in that: The converting of the database structure into a first target SQL script comprises: Export the database structure information of data assets in the SQL Server database; Determine whether the data attributes and / or business requirements of the data assets in the SQL Server database and the GaussDB database match; If there is no match, determine the target conversion logic between the mismatched data attributes and / or business requirements; Determine whether there is any missing index for the data asset in the SQL Server database; If present, determine the missing index; Based on the database structure information of the data asset, the target conversion logic between the mismatched data attributes and the missing indexes, the database structure is converted into a first target SQL script.

3. The data migration method according to claim 1, characterized in that: The converting of the SSIS job flow into a second target SQL script comprises: Determine new operational processes based on new business requirements for data assets; Based on the SSIS job flow and the newly added job flow, the SSIS job flow is converted into a second target SQL script.

4. The data migration method according to claim 1, characterized in that: Convert the database structure into the first target SQL script and convert the SSIS job flow into the second target SQL script, including: Based on the pre-configured script conversion tool, the database structure is converted into the first target SQL script, and the SSIS job flow is converted into the second target SQL script; Execute the first target SQL script and the second target SQL script in the test environment to obtain test results of the first target SQL script and the second target SQL script; The first target SQL script and the second target SQL script are adjusted based on the test result to obtain adjusted first target SQL script and second target SQL script.

5. The data migration method according to claim 1, characterized in that: The scheduling platform determines the task scheduling strategy and resource allocation strategy for each data asset based on the pre-configured scheduling rules; including: Obtain the task priority of each data asset, and obtain the real-time system load and remaining resource information of the scheduling platform, SQL Server database, and GaussDB database; Based on the task priority of each data asset, and obtaining the real-time system load and remaining resource information of the scheduling platform, SQL Server database and GaussDB database, determine the task scheduling strategy and resource allocation strategy for each data asset.

6. The data migration method according to claim 5, characterized in that: After determining the task scheduling strategy and resource allocation strategy for each data asset, the method includes: Sending the task scheduling strategy and resource allocation strategy of each data asset to the target terminal device, and displaying the task scheduling strategy and resource allocation strategy of each data asset on the graphical user interface of the target terminal device; Receive adjustment instructions for task scheduling policies and / or resource allocation policies for at least one type of data asset sent by a target terminal device, and update the task scheduling policies and / or resource allocation policies for the corresponding type of data assets; the adjustment instructions are triggered based on adjustment controls of a graphical user interface of the target terminal device.

7. The data migration method according to claim 1, characterized in that: The step of migrating the data of the data asset to the database structure of the GaussDB database includes: Obtaining data to be encrypted from the data of the data asset; Encrypting the data to be encrypted of the data asset based on a pre-configured encryption algorithm to obtain encrypted data; The encrypted data is migrated to the database structure of the GaussDB database.

8. A data migration device, characterized in that: Applied to a data migration system, the data migration system includes a GaussDB database and a scheduling platform; the device includes: An acquisition module is used to acquire the database structure and SSIS operation flow of the data assets of the target enterprise stored in the SQL Server database, convert the database structure into a first target SQL script, and convert the SSIS operation flow into a second target SQL script; the first target SQL script and the second target SQL script are compatible with the GaussDB database; A determination module, configured to enable the scheduling platform to determine a task scheduling strategy and a resource allocation strategy for each data asset based on a pre-configured scheduling rule; A first migration module is used to execute the first target SQL script and the second target SQL script based on the task scheduling strategy and the resource allocation strategy, and migrate the database structure and SSIS operation process of the data assets in the SQL Server database to the GaussDB database; The second migration module migrates the data of the data assets to the database structure of the GaussDB database based on the task scheduling strategy and resource allocation strategy.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the data migration method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data migration method as described in any one of claims 1 to 7 are executed.

Images