Data migration method and device, computer device and storage medium

Abstract

The application relates to a cloud technology-based data migration method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining migration information configured for original data migration, and splitting the migration information to generate a plurality of processing tasks that allow concurrent execution; obtaining the original data from an original database based on the migration information; performing sharding processing on the original data to obtain data shards corresponding to each processing task; calling a plurality of cluster applications corresponding to a target cluster in parallel, and concurrently executing each processing task through the plurality of cluster applications to convert the corresponding data shards into storage data suitable for the target cluster; and migrating each storage data to an online service of the target cluster through the called plurality of cluster applications. The method can improve the efficiency of data automatic migration by deploying a plurality of cluster applications.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data migration method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, computer devices are processing increasingly more data. While a single computer can generally meet the needs of computing small amounts of data, large-scale data processing requires specialized big data services. For example, data generated by the distributed computing framework Hadoop can be imported into the columnar database management system ClickHouse online service for data storage, querying, and other processing.

[0003] Currently, the industry standard for importing data generated by distributed computing frameworks into columnar database management systems typically involves using the ClickHouse-local tool to generate storage data for the corresponding ClickHouse tables from specified files and then migrating it to the online tables. However, the traditional ClickHouse-local tool is limited by single-machine capabilities, resulting in a limited data processing capacity per second and thus low data migration efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a data migration method, apparatus, computer equipment, storage medium, and computer program product that can improve migration efficiency in response to the above-mentioned technical problems.

[0005] A data migration method, the method comprising:

[0006] Obtain the migration information configured for the original data migration, and split the migration information to generate multiple processing tasks that can be executed concurrently;

[0007] The original data is obtained from the original database based on the migration information;

[0008] The original data is split into data fragments to obtain data fragments corresponding to each processing task.

[0009] Multiple cluster applications corresponding to the target cluster are invoked in parallel, and the processing tasks are executed concurrently by the multiple cluster applications to convert the corresponding data shards into storage data suitable for the target cluster.

[0010] By invoking the multiple cluster applications, the stored data is migrated to the online service of the target cluster.

[0011] A data migration apparatus, the apparatus comprising:

[0012] The splitting module is used to obtain migration information configured for the original data migration and split the migration information to generate multiple processing tasks that can be executed concurrently.

[0013] The acquisition module is used to acquire the original data from the original database based on the migration information;

[0014] The processing module is used to perform data sharding on the original data to obtain data shards corresponding to each processing task.

[0015] The concurrency module is used to call multiple cluster applications corresponding to the target cluster in parallel, and execute each processing task concurrently through the multiple cluster applications to convert the corresponding data shards into storage data suitable for the target cluster.

[0016] The migration module is used to migrate the stored data to the online service of the target cluster by calling the multiple cluster applications.

[0017] In one embodiment, the splitting module is further configured to display a migration configuration interface in response to a migration operation triggered on the original data; and to obtain migration information generated by the editing operation in response to an editing operation on the migration configuration interface.

[0018] In one embodiment, the splitting module is further configured to obtain migration information configured for the original data migration, obtain concurrency information and the number of target nodes in the target cluster from the migration information, and split the migration information according to the concurrency information and the number of target nodes to generate multiple processing tasks that can be executed concurrently.

[0019] In one embodiment, the acquisition module is further configured to generate a configuration file based on the migration information, and acquire the original data from the original database based on the configuration file; the configuration file includes conversion rules and partitioning rules;

[0020] The processing module is further configured to perform data transformation on the original data according to the transformation rules, and divide the data obtained from the data transformation according to the partitioning rules to obtain multiple data fragments; and associate each data fragment with each processing task, wherein different processing tasks are associated with different data fragments.

[0021] In one embodiment, the acquisition module is further configured to generate a configuration file based on the migration information via a scheduler, and transmit the configuration file to the computing engine; and obtain the original data from the original database via the computing engine based on the configuration file.

[0022] The processing module is further configured to perform data transformation on the original data according to the transformation rules through the computing engine, and to divide the data obtained from the data transformation according to the partitioning rules to obtain multiple data fragments.

[0023] In one embodiment, the splitting module is further configured to generate task information corresponding to each processing task, the task information including data sharding information and corresponding target node information, the target node being a node in the target cluster;

[0024] The concurrency module is also used to call multiple cluster applications corresponding to the target cluster in parallel, and execute each processing task concurrently by the multiple cluster applications based on the shard information, so as to convert the corresponding data shards into storage data suitable for the target cluster.

[0025] The migration module is further configured to migrate the stored data to the online service of the target node corresponding to the corresponding node information by invoking the multiple cluster applications based on the node information.

[0026] In one embodiment, the splitting module is further configured to generate task information corresponding to each of the processing tasks through a scheduler;

[0027] The concurrency module is also used to call multiple application containers in parallel through the scheduler and send the task information corresponding to each processing task to each application container; and to call their respective cluster applications through each application container to concurrently execute each processing task based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster and store them in the storage space of the corresponding application container.

[0028] The migration module is further configured to call its respective cluster application through each application container to determine the target node corresponding to each data shard from the node information received by each of them, and to copy the stored data in its respective storage space to the online service of the target node corresponding to the data shard.

[0029] In one embodiment, the apparatus further includes a recycling module, which is used to recycle each application container when the scheduler detects that all processing tasks have been completed.

[0030] In one embodiment, the splitting module is further configured to generate task information corresponding to each of the processing tasks through a scheduler;

[0031] The concurrency module is further configured to send task information corresponding to each processing task to the computing engine through the scheduler; and to call multiple cluster applications corresponding to the target cluster in parallel through the computing engine, and to execute each processing task concurrently through the multiple cluster applications based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster.

[0032] The migration module is further configured to determine the target node corresponding to each data shard from the node information of each task information by calling the multiple cluster applications, and to copy the stored data to the online service of the corresponding target node.

[0033] In one embodiment, the splitting module is further configured to generate task information corresponding to each of the processing tasks through a scheduler; the task information includes data sharding information and node information of the corresponding target node, wherein the target node is a node in the target cluster; and the scheduler sends the task information corresponding to each of the processing tasks to the computing engine.

[0034] The processing module is further configured to perform sharding processing on the original data through a computing engine to obtain data shards corresponding to each processing task.

[0035] The concurrency module is further configured to concurrently call multiple cluster applications corresponding to the target cluster through the computing engine, and send each data shard to the input pipe of the corresponding cluster application through the output pipe of the computing engine; and concurrently execute each processing task through the multiple cluster applications based on the shard information, so as to convert the corresponding data shards received from the input pipe into storage data suitable for the target cluster.

[0036] The migration module is further configured to determine the target node corresponding to each data shard from the node information by calling the multiple cluster applications, and to copy the stored data to the online service of the corresponding target node.

[0037] In one embodiment, the original database is a database corresponding to a distributed computing framework, and the original data is data corresponding to the distributed computing framework; the target cluster is a columnar storage database cluster, and the stored data is data that meets the storage format of the columnar storage database cluster.

[0038] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0039] Obtain the migration information configured for the original data migration, and split the migration information to generate multiple processing tasks that can be executed concurrently;

[0040] The original data is obtained from the original database based on the migration information;

[0041] The original data is split into data fragments to obtain data fragments corresponding to each processing task.

[0042] Multiple cluster applications corresponding to the target cluster are invoked in parallel, and the processing tasks are executed concurrently by the multiple cluster applications to convert the corresponding data shards into storage data suitable for the target cluster.

[0043] By invoking the multiple cluster applications, the stored data is migrated to the online service of the target cluster.

[0044] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0045] Obtain the migration information configured for the original data migration, and split the migration information to generate multiple processing tasks that can be executed concurrently;

[0046] The original data is obtained from the original database based on the migration information;

[0047] The original data is split into data fragments to obtain data fragments corresponding to each processing task.

[0048] Multiple cluster applications corresponding to the target cluster are invoked in parallel, and the processing tasks are executed concurrently by the multiple cluster applications to convert the corresponding data shards into storage data suitable for the target cluster.

[0049] By invoking the multiple cluster applications, the stored data is migrated to the online service of the target cluster.

[0050] A computer program product, comprising a computer program, wherein the computer program, when executed by a processor, performs the following steps:

[0051] Obtain the migration information configured for the original data migration, and split the migration information to generate multiple processing tasks that can be executed concurrently;

[0052] The original data is obtained from the original database based on the migration information;

[0053] The original data is split into data fragments to obtain data fragments corresponding to each processing task.

[0054] Multiple cluster applications corresponding to the target cluster are invoked in parallel, and the processing tasks are executed concurrently by the multiple cluster applications to convert the corresponding data shards into storage data suitable for the target cluster.

[0055] By invoking the multiple cluster applications, the stored data is migrated to the online service of the target cluster.

[0056] The aforementioned data migration method, apparatus, computer equipment, and storage medium acquire migration information configured for the original data migration and split this information to generate multiple processing tasks that can be executed concurrently, allowing each processing task to handle one data shard. Based on the migration information, original data is retrieved from the original database, and the original data is sharded to obtain data shards corresponding to each processing task. Multiple cluster applications corresponding to the target cluster are invoked in parallel. These cluster applications concurrently execute each processing task to convert the corresponding data shards into storage data suitable for the target cluster. This allows for the simultaneous and rapid conversion of multiple data shards into their respective corresponding storage data, effectively improving the efficiency of data shard conversion. Through the invoked cluster applications, each storage data shard is migrated to the online service of the target cluster, thereby enabling the rapid migration of original data from the original database to the online service of the target cluster, improving the efficiency of data migration. Attached Figure Description

[0057] Figure 1 This is a diagram illustrating the application environment of a data migration method in one embodiment.

[0058] Figure 2 This is a flowchart illustrating a data migration method in one embodiment;

[0059] Figure 3 This is a schematic diagram of the migration configuration interface in one embodiment;

[0060] Figure 4 This is a schematic diagram of a configuration file in one embodiment;

[0061] Figure 5 This is a schematic diagram of migration information and task information corresponding to multiple processing tasks generated by splitting in one embodiment;

[0062] Figure 6 This is a schematic diagram of a process in one embodiment where multiple cluster applications concurrently execute processing tasks based on each shard information to convert the corresponding data shards into storage data suitable for the target cluster.

[0063] Figure 7 This is a schematic diagram illustrating the process of a scheduler scheduling application containers to concurrently execute processing tasks in one embodiment.

[0064] Figure 8 This is a schematic diagram illustrating the process of an application container calling a cluster application to execute a processing task in one embodiment.

[0065] Figure 9 This is a schematic diagram illustrating the process of concurrently executing various processing tasks by multiple cluster applications to convert corresponding data shards into storage data suitable for the target cluster, as described in another embodiment.

[0066] Figure 10 This is a framework diagram of a data migration method in one embodiment;

[0067] Figure 11 This is a structural block diagram of a data migration device in one embodiment;

[0068] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] The data migration method provided in this application can be applied to, for example... Figure 1 In the application environment shown, the data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on the cloud or other network servers. Both terminal 102 and server 104 can independently execute the data migration method provided in this embodiment. Terminal 102 and server 104 can also work together to execute the data migration method provided in this embodiment. When terminal 102 and server 104 work together to execute the data migration method provided in this embodiment, terminal 102 obtains migration information configured for the original data migration and splits the migration information to generate multiple processing tasks that can be executed concurrently. Terminal 102 sends the migration information and each processing task to server 104, and server 104 obtains the original data from the original database based on the migration information. Server 104 performs sharding processing on the original data to obtain data shards corresponding to each processing task. Server 104 calls multiple cluster applications corresponding to the target cluster in parallel, and executes each processing task concurrently through multiple cluster applications to convert the corresponding data shards into storage data suitable for the target cluster. Server 104 migrates stored data to the online service of the target cluster by calling multiple cluster applications.

[0071] Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 104 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0072] In one embodiment, multiple servers can form a blockchain, with each server acting as a node on the blockchain. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0073] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operations management. The user management module is responsible for managing the identity information of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between user real identities and blockchain addresses (access management). Under authorization, it also manages and audits transactions of certain real identities and provides risk control rule configuration (risk control audit). The basic services module is deployed on all blockchain node devices to verify the validity of business requests. After consensus is reached on valid requests, they are recorded in storage. For a new business request, the basic services first perform interface adaptation parsing and authentication (interface adaptation), and then encrypt the business information using a consensus algorithm (consensus management). The encryption process ensures that the encrypted data is transmitted completely and consistently to the shared ledger (network communication) and recorded and stored. The smart contract module is responsible for contract registration, issuance, triggering, and execution. Developers can define contract logic using a programming language and publish it to the blockchain (contract registration). Based on the contract terms, the module calls keys or other events to trigger execution and complete the contract logic. It also provides functions for contract upgrades and cancellations. The operations management module is mainly responsible for deployment, configuration modification, contract settings, cloud adaptation, and real-time status visualization during product launch, such as alarms, network status management, and node device health status management.

[0074] In one embodiment, data related to the data migration method can be stored on a blockchain, such as raw data, migration information, processing tasks, data shards, and storage data.

[0075] It should be noted that the quantities of "multiple" mentioned in the embodiments of this application all refer to the quantity of "at least two".

[0076] In one embodiment, such as Figure 2 As shown, a data migration method is provided, which is applied to a computer device (the computer device may be...). Figure 1 Taking the terminal or server shown as an example, the following steps are included:

[0077] Step S202: Obtain the migration information configured for the original data migration, and split the migration information to generate multiple processing tasks that can be executed concurrently.

[0078] The original data refers to the data that needs to be migrated from the original database to the online service of the target cluster. The original data can be the data corresponding to the distributed computing framework, and the original database is the database used by the distributed computing framework to store the data. The original data is stored in the original database. In short, a database can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, which can be shared by multiple users, has as little redundancy as possible, and is independent of the application.

[0079] Distributed computing frameworks, such as Hadoop, are open-source frameworks written in Java that store massive amounts of data and run distributed analytics applications on distributed server clusters. Their core components are HDFS and MapReduce. HDFS stands for Distributed File System, designed to run on commodity hardware. It consists of a management server (Namenode) and worker servers (Datanodes). The Namenode is a central server responsible for managing the file system's namespace and client access to files. A Datanode is typically a single node responsible for managing the data stored on its node. MapReduce is a computing framework whose core idea is to distribute computational tasks across servers within the cluster. By splitting these tasks, a JobTracker performs distributed computation. It's important to understand that the original data can also be from any other type of storage framework, such as distributed storage, direct-attached storage, centralized storage, or symmetric storage, but is not limited to these.

[0080] Migration information refers to the migration-related information configured by a user object in the migration configuration interface using an object identifier. For example, migration information might be the job information entered by the user object. A user object refers to the user of the migration configuration interface, which is a user interface provided to the user object to configure information related to data migration. An object identifier is a unique identifier for the user object, including but not limited to user identifiers and device identifiers. Specifically, a user identifier can be a username or user account. A user account refers to an account registered in the application. A device identifier can be a unique identifier for the electronic device used by the user object. Migration information may include attribute information and the corresponding text content for each attribute. Attribute information refers to information representing attributes, specifically task type, target cluster, database name, table name, partition, data repartitioning rules, target cluster field names, etc., but not limited to these. For example, the text content corresponding to the task type might be "Data Export" or "Data Import," indicating a task to export data from or import data into the database. The text content corresponding to the database name might be "AAA," indicating that the database name is "AAA."

[0081] A processing task refers to the task of transforming the data corresponding to data shards, that is, converting data from one database into data suitable for another database. In this embodiment, a processing task may refer to dividing the original data in the original database into data shards and converting the data corresponding to the data shards into data suitable for the target cluster.

[0082] Specifically, users can configure relevant information, i.e., migration information, for the original data to be migrated through object identifiers. The computer device can obtain this migration information, determine the number of shards needed to divide the original data, and then determine the number of processing tasks to be executed concurrently based on the number of shards. According to the determined number of processing tasks, the migration information is split to generate a number of processing tasks that can be executed concurrently. Each of the concurrently executed processing tasks is independent of the others in its processing.

[0083] In one embodiment, the number of shards is used as the number of processing tasks, with each data shard of the original data corresponding to one processing task.

[0084] Step S204: Obtain the original data from the original database based on the migration information.

[0085] Specifically, the computer device can determine the location information corresponding to the original data from the migration information. This location information may include the address of the database where the original data is located, i.e., the address of the original database, and may also include at least one of the following: the node address, partition address, and table name of the original data within the original database. Based on the location information of the original data, the computer device searches for the storage location corresponding to that location information in the original database and retrieves the data stored at that storage location. The data stored at that storage location is the original data.

[0086] Step S206: The original data is split into data fragments to obtain data fragments corresponding to each processing task.

[0087] Specifically, after acquiring the raw data, the computer equipment can segment the raw data to obtain data segments equal to the number of processing tasks. Furthermore, each processing task corresponds to one data segment.

[0088] In one embodiment, a computer device can obtain the partitioning rules for data sharding from migration information. These rules can be random or hash-based. The original data can be sharded according to the processing logic of these rules to obtain multiple data shards. For example, if the partitioning rule is random, the data can be sharded according to its size, resulting in shards of equal size. Furthermore, the number of data shards needs to be the same as the number of tasks. Therefore, the original data can be divided into the specified number of data shards, with each shard having the same size. For example, the data can be divided into 600 data shards, each containing 1000 bytes of data.

[0089] Step S208: In parallel, multiple cluster applications corresponding to the target cluster are invoked. The processing tasks are executed concurrently by the multiple cluster applications to convert the corresponding data shards into storage data suitable for the target cluster.

[0090] The target cluster refers to the cluster of storage data into which the original data will be imported. The target cluster can be a columnar storage database cluster, such as a ClickHouse cluster. ClickHouse is a columnar database management system. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters or mobile devices; or according to the query language used, such as SQL (Structured Query Language) or XQuery; or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages.

[0091] A cluster application is a lightweight application that can run independently to perform data queries without depending on a server-side program. The cluster application corresponding to the target cluster refers to the lightweight application for that target cluster, which can run independently and perform data queries without relying on the target cluster's server-side program.

[0092] For example, lickhouse-local can run most SQL queries independently without relying on any ClickHouse server programs. It is similar to a standalone microkernel version of the ClickHouse service and is a lightweight application.

[0093] Specifically, the computer device can identify the cluster application corresponding to the target cluster, and the computer device can call multiple cluster applications corresponding to the target cluster in parallel. Through multiple cluster applications, each processing task is executed concurrently. Executing the processing task means converting the data fragments corresponding to the processing task into storage data suitable for the target cluster.

[0094] In one embodiment, each cluster application can execute one processing task at a time. The computer device can allocate multiple processing tasks to the same cluster application, and the cluster application can execute each processing task sequentially to convert the data fragments corresponding to each processing task into the corresponding storage data. For example, if four cluster applications ABCD are called, with A and B each executing one processing task and C and D each executing two processing tasks, then A, B, C, and D can complete four processing tasks by executing them concurrently once. Then, C and D can complete the remaining two processing tasks by executing them concurrently once more.

[0095] In one embodiment, the computer device can determine the number of tasks for each processing task, and concurrently call a number of cluster applications corresponding to the target cluster that are equal to the number of tasks, so that each cluster application executes one processing task, and each cluster application concurrently processes its own processing task, so as to convert the data shards corresponding to each processing task into storage data suitable for the target cluster.

[0096] Step S210: By calling multiple cluster applications, the stored data is migrated to the online service of the target cluster.

[0097] Specifically, computer devices can obtain the stored data processed by each cluster application by concurrently calling multiple cluster applications to execute processing tasks. The computer devices can then migrate the stored data processed by each cluster application to the online service of the target cluster.

[0098] In one embodiment, computer devices can concurrently migrate the stored data processed by each cluster application to the online service of the target cluster.

[0099] In one embodiment, the target node in the target cluster corresponding to the data shard can be determined, and the stored data processed by each of the multiple cluster applications can be migrated to the online service of the target node in the target cluster. Further, all stored data can be written to each target node, or the online service of the target node corresponding to each data shard can be determined, and the stored data corresponding to each data shard can be written to the online service of its respective target node.

[0100] In this embodiment, migration information configured for the original data migration is obtained and split into multiple processing tasks that can be executed concurrently, allowing each processing task to handle one data shard. Based on the migration information, original data is retrieved from the original database, and the original data is sharded to obtain data shards corresponding to each processing task. Multiple cluster applications corresponding to the target cluster are invoked in parallel. These cluster applications concurrently execute each processing task to convert the corresponding data shards into storage data suitable for the target cluster. This allows for the simultaneous and rapid conversion of multiple data shards into their respective storage data, effectively improving the efficiency of data shard conversion. Through the invoked cluster applications, each storage data is migrated to the online service of the target cluster, thereby enabling the rapid migration of original data from the original database to the online service of the target cluster, improving the efficiency of data migration.

[0101] In one embodiment, obtaining migration information configured for migrating the original data includes: displaying a migration configuration interface in response to a migration operation triggered on the original data; and obtaining migration information generated by the editing operation in response to an editing operation on the migration configuration interface.

[0102] The migration configuration interface displays the attribute information required for data migration and shows the editing area corresponding to each attribute. Users can edit the specific text content corresponding to the attribute in the editing area.

[0103] Specifically, users can access the original database through an object identifier and query the original data within it. Users can trigger a migration operation based on this original data, and the computer device will display a migration configuration interface in response. The migration operation includes, but is not limited to, at least one of the following: touch operation, voice operation, operation via an input device such as a mouse, or gesture operation. For example, it can be any one of a click operation, double-click operation, long-press operation, left swipe operation, or right swipe operation, but is not limited to these.

[0104] In one embodiment, a user can access the original database via an object identifier. A migration control is displayed in the original database's interface, which the user can trigger to initiate a migration operation. The computer device detects the triggering of the migration control and displays a migration configuration interface.

[0105] The migration configuration interface displays attribute information and corresponding editing areas for each attribute. Attribute information includes at least one of the following: task type, task subtype, product to which the data belongs, application corresponding to the database, target cluster, database name, table name, partition, concurrent data, repartition optimization, data repartition rules, specified columns for data repartition, whether to perform SQL filtering, target cluster information, target cluster database name, target cluster table name, whether to force clear / remove directory, whether to force clear data in the corresponding partition of the target table, and field names of the target cluster, but is not limited to these. Users can trigger the editing areas corresponding to attribute information to edit the text content of that attribute. For example, editing "Data Outbound" in the editing area corresponding to the task type, and editing "rand()" or "Hash()" in the editing area corresponding to the data repartition rules. rand() indicates data repartitioning using random rules, and Hash() indicates data repartitioning using hash rules. The migration configuration interface can be viewed as follows: Figure 3 As shown, the interface provides the attribute information needed to migrate the original data, along with an editing area for each attribute. Users can enter the corresponding text content in the editing area as needed. It is understood that the attribute information and corresponding editing areas displayed in the migration configuration interface can be added or removed according to specific needs; no further restrictions are imposed here.

[0106] The editing operations performed in the editing area can specifically include touch operations, cursor operations, key operations, or voice operations. Touch operations can include touch clicks, touch presses, or touch swipes, and can be single-point or multi-point touches. Cursor operations can involve controlling the cursor to click, press, or swipe. Key operations can be virtual or physical key operations.

[0107] The computer device acquires the text content edited by the user in the editing area corresponding to each attribute information. After detecting that all attribute information has been configured with corresponding text content, it generates migration information corresponding to the original data based on each attribute information and the corresponding text content.

[0108] In one embodiment, when a computer device detects that at least one attribute is missing text content, it may prompt the user. Specifically, this can be done by providing a message to the user or by marking the editing area where no text content is configured.

[0109] In this embodiment, in response to a migration operation triggered on the original data, a migration configuration interface is displayed, allowing users to independently configure migration-related information for the original data, providing them with more options. In response to an editing operation on the migration configuration interface, the migration information generated by the editing operation is obtained. Based on the user-configured migration information, the migration task for the original data is automatically executed, improving the efficiency of data migration.

[0110] In one embodiment, migration information configured for the original data migration is obtained, and the migration information is split to generate multiple processing tasks that can be executed concurrently, including:

[0111] Obtain the migration information configured for the original data migration, and extract the concurrency information and the number of target nodes in the target cluster from the migration information; split the migration information according to the concurrency information and the number of target nodes to generate multiple processing tasks that can be executed concurrently.

[0112] Specifically, the computer device can obtain migration information configured for the original data migration, and from this migration information, it can obtain concurrency information and the number of target nodes in the target cluster. Target nodes are nodes in the target cluster that represent the target locations for the original data migration. Based on the concurrency information and the number of target nodes, the computer device determines the number of processing tasks. After obtaining the number of tasks, the computer device breaks down the migration information, generating a number of processing tasks that can be executed concurrently.

[0113] In one embodiment, the concurrency information can be the user-configured concurrency level, and the target node count refers to the number of target nodes to be written to in the target cluster.

[0114] In one embodiment, the computer device can obtain the configured concurrency level and the number of target nodes from the migration information, and use the product of the concurrency level and the number of target nodes as the number of processing tasks that need to be executed concurrently. For example, if the user-configured concurrency level is 32 and the number of target nodes is 5, then the number of tasks is 160. The migration information is then split to generate 160 processing tasks that can be executed concurrently. The processing logic and processing data of these 160 processing tasks are independent of each other, and the processing processes can be made independent of each other, thereby enabling concurrent processing.

[0115] In this embodiment, the migration information configured by the user for the original data migration is obtained. Concurrency information and the number of target nodes in the target cluster are obtained from the migration information. Based on the concurrency information and the number of target nodes, the migration information is automatically split to generate multiple processing tasks that can be executed concurrently. This ensures that the processing logic and data of each processing task are independent of each other, and the processing process can be made independent of each other. This enables multiple tasks to be processed concurrently at the same time, thereby improving data processing efficiency.

[0116] In one embodiment, obtaining raw data from the original database based on migration information includes: generating a configuration file based on the migration information, and obtaining raw data from the original database based on the configuration file; the configuration file includes transformation rules and partitioning rules.

[0117] The original data is sharded to obtain data shards corresponding to each processing task. This includes: transforming the original data according to transformation rules, and dividing the transformed data into multiple data shards according to partitioning rules; associating each data shard with each processing task, wherein different processing tasks are associated with different data shards.

[0118] Specifically, the computer equipment generates a configuration file based on filtering, transformation, repartitioning, and storage information from the migration information. This configuration file determines whether to perform filtering, transformation, repartitioning, and storage processing on the original data. The configuration file also includes information about the location of the original data in the original database, as well as transformation and partitioning rules for converting the original data. Transformation rules refer to the use of transformation functions to convert data of one data type to another. Transformation rules can be SQL transformation functions, CAST transformation functions, or CONVERT transformation functions, etc., and can be configured according to actual needs.

[0119] The computer device searches for the storage location corresponding to the location information in the original database based on the location information of the original data, and retrieves the original data stored at that storage location.

[0120] The computer equipment can perform data transformation processing on the original data according to the transformation rules in the configuration file to obtain data suitable for the target cluster. Next, the computer equipment can perform sharding processing on this data suitable for the target cluster according to the partitioning rules in the configuration file, resulting in multiple data shards. Further, the computer equipment performs sharding processing on this data suitable for the target cluster according to the partitioning rules, resulting in the same number of data shards as the number of processing tasks. Then, the computer equipment can associate each data shard with each processing task, so that different processing tasks are associated with different data shards.

[0121] In one embodiment, a computer device may associate data shards with processing tasks, such that each processing task is associated with a data shard.

[0122] In one embodiment, the computer device may associate the fragment identifier of each data fragment with the task identifier of each processing task, such that each processing task is associated with a data fragment.

[0123] In this embodiment, a configuration file is generated based on migration information. This configuration file determines whether to perform filtering, transformation, repartitioning, and storage processing on the original data. Original data is retrieved from the original database based on the configuration file. Data transformation is performed on the original data according to the transformation rules in the configuration file, converting the original data from its original database data type to data suitable for the target cluster. The transformed data is then divided into multiple data shards according to partitioning rules. Performing data transformation before sharding avoids inaccurate sharding due to differences in data volume and dependencies before and after transformation. Each data shard is associated with a processing task, with different processing tasks associated with different data shards. This ensures that each subsequent processing task performs an operation to convert a data shard into storage data suitable for the target cluster's storage format. Furthermore, the processing tasks can be executed concurrently, enabling the concurrent conversion of each data shard into its corresponding storage data, thus improving the efficiency of data transformation.

[0124] In one embodiment, generating a configuration file based on migration information and retrieving raw data from the original database based on the configuration file includes:

[0125] The scheduler generates a configuration file based on migration information and passes the configuration file to the computing engine; the computing engine then retrieves the original data from the original database based on the configuration file.

[0126] The original data is transformed according to transformation rules, and the transformed data is divided according to partitioning rules to obtain multiple data fragments. This includes: the original data is transformed by the computing engine according to transformation rules, and the transformed data is divided according to partitioning rules to obtain multiple data fragments.

[0127] The scheduler allocates system resources to running applications based on their resource requirements and the available resources on the cluster devices. The scheduler can be CapacityScheduler, FairScheduler, or other similar schedulers. The computing engine is the processing engine used to execute computational tasks; examples include Spark and MapReduce. Spark is a fast, general-purpose computing engine for Hadoop data. Spark completes task computations by performing an ETL (Extract-Transform-Load) process, which involves extracting, transforming, and loading data from the source to the destination.

[0128] Specifically, a scheduler and a computing engine are deployed on the computer equipment. The scheduler obtains the migration information configured by the user in the migration configuration interface. Based on the migration information, the scheduler generates a configuration file, which determines the processing required for the original data, such as whether to filter, transform, repartition, and store the original data. The scheduler starts the computing engine and sends the configuration file to the computing engine. This configuration file also includes the location information of the original data storage, the transformation rules for the original data, and the partitioning rules for the original data. After receiving the configuration file, the computing engine obtains the location information of the original data from the configuration file and retrieves the original data from the original database based on this location information.

[0129] The computing engine retrieves the transformation and partitioning rules for the original data from this configuration file. Following the processing logic of the transformation rules, it transforms the original data to obtain data of the data type corresponding to the target cluster. Next, the computing engine partitions the transformed data according to the partitioning rules, resulting in multiple data shards. Further, the computing engine partitions the transformed data according to the partitioning rules, obtaining the same number of data shards as the number of processing tasks. The computing engine can associate the shard information of each data shard with the task information of each processing task. The shard information can be a shard identifier, and the task information can be a task identifier.

[0130] In one embodiment, the scheduler generates a configuration file based on migration information, such as... Figure 4 As shown, the scheduler starts the Spark job and passes it to the Spark execution process. The Spark job reads the data from p_20210924 of the Hive table teg_db_01.teg_table_01, transforms the data according to the SQL rules in the configuration file, and then divides the data into 600 files of equal size using the rand() function. Each file is a data shard. These 600 files are written to a specified HDFS directory. The filenames of these 600 HDFS files contain shard information, which is associated with the task information of the corresponding processing task, thus linking data shards to processing tasks.

[0131] In this embodiment, the scheduler can generate a configuration file based on migration information. The scheduler then starts the computing engine and passes the configuration file to the computing engine. The computing engine can obtain the location information of the original data from the configuration file to retrieve the original data from the original database. The computing engine performs data transformation on the original data according to transformation rules and divides the data obtained from the data transformation according to partitioning rules, thus accurately obtaining multiple data shards that can be processed concurrently.

[0132] In one embodiment, after splitting the migration information to generate multiple processing tasks that can be executed concurrently, the method further includes: generating task information corresponding to each processing task, the task information including data sharding information and node information of the corresponding target node, the target node being a node in the target cluster;

[0133] Parallel invocation of multiple cluster applications corresponding to the target cluster, and concurrent execution of each processing task by multiple cluster applications to convert the corresponding data shards into storage data suitable for the target cluster, including: parallel invocation of multiple cluster applications corresponding to the target cluster, and concurrent execution of each processing task by multiple cluster applications based on each shard information to convert the corresponding data shards into storage data suitable for the target cluster.

[0134] By invoking multiple cluster applications, the stored data is migrated to the online service of the target cluster, including: by invoking multiple cluster applications based on the node information, the stored data is migrated to the online service of the target node corresponding to the corresponding node information.

[0135] The task information refers to the metadata of the processing task, including the source of the data to be transformed, the shard information of the specific data shards to be processed, and the node information of a specific node in the target cluster to which the data needs to be written. The target node refers to the node in the target cluster that stores the data corresponding to the original data, or it can be the node in the target cluster that stores the data corresponding to the data shards. The shard information is the information about the data shards, which can be the shard identifier. The node information indicates the location of the target node in the target cluster, which can be a node address.

[0136] Specifically, the computer device breaks down the migration information to generate multiple processing tasks that can be executed concurrently. The computer device generates task information for each processing task, which includes the source of the data to be processed, the data shards to be processed, and the node information of the target nodes in the target cluster to which the data shards need to be written.

[0137] The computer device can identify the cluster application corresponding to the target cluster, and simultaneously invoke multiple cluster applications corresponding to the target cluster, assigning each processing task to each cluster application. Further, the computer device assigns each processing task and its corresponding task information to each cluster application. Each cluster application determines the corresponding data shard based on the sharding information in the task information corresponding to the assigned processing task, and converts the corresponding data shard into storage data suitable for the target cluster.

[0138] Computer devices execute processing tasks concurrently through multiple cluster applications, enabling each cluster application to convert the data shards corresponding to its processing tasks into storage data suitable for the target cluster.

[0139] Each cluster application determines the target node based on the node information in the task information corresponding to the assigned processing task, and migrates the stored data processed by the processing task to the online service corresponding to the target node.

[0140] Computer devices can concurrently migrate stored data processed by multiple cluster applications to the online services of the corresponding target nodes.

[0141] like Figure 5 The diagram illustrates migration information and task information corresponding to multiple processing tasks generated from the splitting process in one embodiment. The scheduler obtains the job information Job0 entered by the user and splits it into processing tasks Task1, Task2, and Task3 that can be executed concurrently. The splitting logic is as follows: Figure 5 As shown: The product of the user-configured concurrency (shardParallel) and the number of nodes written to the ClickHouse cluster (clickhouseSink) is used as the number of processing tasks; a processing task can process one data shard, and the metadata information of the processing task includes the data source to be transformed (source), the specific data shard to be processed (shard), and the node information (sink) of a certain node in the target ClickHouse cluster to be written to.

[0142] In this embodiment, task information corresponding to each processing task is generated. This task information includes data shard information and corresponding target node information. The shard information determines the data shard to be processed by the task, and the node information determines the target node where the processed data from the data shards should be stored. By invoking multiple cluster applications corresponding to the target cluster in parallel, and having these applications concurrently execute each processing task based on the shard information, multiple data shards can be quickly converted into storage data suitable for the target cluster. Parallel processing of data shards improves the efficiency of converting data shards into storage data. Based on the node information, the invoked cluster applications migrate each storage data to the online service of the target node corresponding to the respective node information. This allows the original data to be migrated from the original database to the target node of the target cluster, improving the efficiency of data migration.

[0143] In one embodiment, such as Figure 6 As shown, task information corresponding to each processing task is generated, including:

[0144] Step S602: The scheduler generates task information corresponding to each processing task.

[0145] Specifically, the computer device deploys a scheduler, computing engine, application containers, and clustered applications. Application containers can be virtual machines or physical machines. For example, an application container can be Docker, an open-source application container engine that allows developers to package applications and their dependencies into a portable image and then deploy it to any Linux or Windows machine, thus achieving virtualization.

[0146] The scheduler retrieves the migration information generated in the migration configuration interface and splits it into multiple processing tasks that can be executed concurrently. For each processing task, the scheduler generates task information, including the source of the data to be processed, the data shards to be processed, and the node information of the target nodes in the target cluster to which the data shards need to be written.

[0147] The scheduler calls the computing engine, which will process the raw data into shards to obtain data shards corresponding to each processing task.

[0148] Parallel invocation of multiple cluster applications corresponding to the target cluster, with each cluster application concurrently executing processing tasks based on the shard information, to convert the corresponding data shards into storage data suitable for the target cluster, including steps S604-S606:

[0149] Step S604: The scheduler calls multiple application containers in parallel and sends the task information corresponding to each processing task to each application container.

[0150] Specifically, the scheduler can determine the cluster application corresponding to the target cluster, and the scheduler calls multiple application containers in parallel, running one cluster application through each application container.

[0151] The scheduler assigns each processing task to each application container. Further, the scheduler assigns each processing task and its corresponding task information to each application container.

[0152] Step S606: Each application container calls its respective cluster application to concurrently execute each processing task based on the sharding information in the received task information, so as to apply the corresponding data shards to the storage data of the target cluster and store them in the storage space of the corresponding application container.

[0153] Specifically, each application container determines the corresponding data shard based on the sharding information in the task information corresponding to the assigned processing task, and converts the corresponding data shard into storage data suitable for the target cluster through the running cluster application.

[0154] The scheduler executes processing tasks concurrently through cluster applications running in multiple application containers, enabling each cluster application to convert the data shards corresponding to its processing tasks into storage data suitable for the target cluster.

[0155] Each cluster application running in an application container converts the corresponding data shards into storage data suitable for the target cluster, and then stores the storage data in the storage space of the application container.

[0156] By invoking multiple cluster applications based on node information, the stored data is migrated to the online service of the target node corresponding to the corresponding node information, including step S608:

[0157] Step S608: Each application container calls its respective cluster application to determine the target node corresponding to each data shard from the node information it receives, and copies the stored data in its respective storage space to the online service of the target node corresponding to the data shard.

[0158] Specifically, each application container determines the target node based on the node information in the task information corresponding to the assigned processing task, and migrates the stored data stored in the storage space of its respective application container to the online service of the corresponding target node through the cluster application it is running.

[0159] In one embodiment, the scheduler can concurrently migrate stored data from the storage space of each application container to the online service of the corresponding target node by calling multiple application containers running cluster applications.

[0160] like Figure 7 The diagram illustrates a scheduler's method for concurrently executing processing tasks across application containers in one embodiment. The scheduler initiates Docker containers and passes in metadata information corresponding to the concurrently executable processing tasks. For example, it passes Task1 to Docker container-01, Task2 to Docker container-02, Task3 to Docker container-03, and Task4 to Docker container-03. Docker containers-01,-02, and-03 can execute processing tasks concurrently, with each application container executing one task at a time. Docker container-03 receives two processing tasks and can execute either Task3 or Task4 first, processing the next task only after the current task is completed.

[0161] The Docker internal process retrieves configuration information and begins executing the clickhouse-local tool. Based on the task information, it reads the file data of the corresponding data shards in the shared HDFS directory to generate ClickHouse storage data. The configuration information refers to Docker's preset processing rules, which are the processing logic of the Docker internal process.

[0162] like Figure 8 The diagram illustrates, in one embodiment, how an application container invokes a cluster application to execute a processing task. The Docker container determines the source of the data shard to be processed based on the task information received. It then executes the `clickhouse-local` tool to read the data shard associated with the processing task in the HDFS distributed file system directory, transforming the data shard into ClickHouse storage data. Once the storage data is generated, it is copied to the specified target node in the online ClickHouse cluster according to the node information of the processing task.

[0163] In this embodiment, a scheduler generates task information corresponding to each processing task. This task information includes data shard information and corresponding target node information. The shard information determines the data shard to be processed by the task, and the node information determines the target node where the processed data from the data shards will be stored. By concurrently calling multiple application containers and sending the task information corresponding to each processing task to each container, and by having each application container call its own cluster application to concurrently execute each processing task based on the shard information in the received task information, multiple data shards can be quickly converted into storage data suitable for the target cluster. Parallel processing of data shards improves the efficiency of converting data shards into storage data. Each application container calls its own cluster application to determine the target node corresponding to each data shard from the received node information, and then migrates the generated storage data to the online service of the target node. This enables the concurrent migration of original data from the original database to the target node of the target cluster, improving the efficiency of data migration.

[0164] In one embodiment, the method further includes: when the scheduler detects that all processing tasks have been completed, recycling each application container.

[0165] Specifically, the scheduler can detect whether each processing task has been completed. When the scheduler detects that each processing task has been completed, it can reclaim each application container to release the occupied resources and return the processing result to the user.

[0166] When the scheduler detects that there are unexecuted processing tasks, it can call the application container to process the unexecuted tasks and recycle the application container after the execution is completed.

[0167] In other embodiments, when the scheduler detects that all processing tasks have been completed, it returns the processing results to the user. Each application container can reside in the computer device for easy direct invocation next time.

[0168] In one embodiment, the source code logic for generating the target cluster's storage data can be migrated into the computing engine, allowing the target cluster's storage data to be generated directly within the computing engine while it performs data sharding processing. For example, the source code logic for generating ClickHouse's storage data can be directly migrated into Spark, allowing ClickHouse storage data to be generated directly within the Spark job while performing data preprocessing.

[0169] In one embodiment, generating task information corresponding to each processing task includes: generating task information corresponding to each processing task through a scheduler;

[0170] Multiple cluster applications corresponding to the target cluster are invoked in parallel. These applications concurrently execute processing tasks based on the shard information to convert the corresponding data shards into storage data suitable for the target cluster, including:

[0171] The scheduler sends the task information corresponding to each processing task to the computing engine; the computing engine calls multiple cluster applications corresponding to the target cluster in parallel; the multiple cluster applications execute each processing task concurrently based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster.

[0172] By invoking multiple cluster applications based on the node information, each stored data is migrated to the online service of the target node corresponding to the corresponding node information. This includes: determining the target node corresponding to each data shard from the node information of each task information through multiple cluster applications, and copying the stored data to the online service of the corresponding target node.

[0173] Specifically, a scheduler, computing engine, and cluster applications are deployed on the computer equipment. The scheduler obtains the migration information generated in the migration configuration interface, and splits the migration information to generate multiple processing tasks that can be executed concurrently. The scheduler generates task information for each processing task, which includes the source of the data to be processed, the data shards to be processed, and the node information of the target nodes in the target cluster to which the data shards need to be written.

[0174] The scheduler invokes the computing engine, which then segments the raw data into data shards, resulting in data shards corresponding to each processing task. The scheduler then sends each processing task and its corresponding task information to the computing engine.

[0175] The computing engine identifies the cluster applications corresponding to the target cluster. It then invokes multiple cluster applications in parallel, assigning each processing task to each application. Further, the computing engine assigns each processing task and its corresponding task information to each cluster application. Each cluster application determines the corresponding data shard based on the sharding information in the task information of its assigned processing task, transforming the data shard into storage data suitable for the target cluster.

[0176] The computing engine executes processing tasks concurrently through multiple cluster applications, enabling each cluster application to convert the data shards corresponding to its respective processing tasks into storage data suitable for the target cluster.

[0177] Each cluster application determines the target node based on the node information in the task information corresponding to the assigned processing task, and migrates the stored data processed by the processing task to the online service corresponding to the target node.

[0178] The computing engine can concurrently migrate the stored data processed by each cluster application to the online service of the corresponding target node by calling multiple cluster applications.

[0179] In this embodiment, task information corresponding to each processing task is generated through scheduling. This task information includes data shard information and corresponding target node information. The shard information determines the data shard to be processed by the task, and the node information determines the target node where the processed data from the data shards needs to be stored. The computing engine calls multiple cluster applications corresponding to the target cluster in parallel. These applications concurrently execute processing tasks based on the shard information, enabling the simultaneous and rapid conversion of multiple data shards into storage data suitable for the target cluster. This parallel processing of data shards improves the efficiency of converting data shards into storage data. The multiple cluster applications, based on the node information, migrate the storage data to the online service of the target node corresponding to the node information. This migrates the original data from the original database to the target node of the target cluster, improving the efficiency of data migration.

[0180] In one embodiment, such as Figure 9 As shown, after splitting the migration information to generate multiple processing tasks that can be executed concurrently, it also includes:

[0181] Step S902: The scheduler generates task information corresponding to each processing task; the task information includes data sharding information and the node information of the corresponding target node, which is a node in the target cluster.

[0182] Specifically, a scheduler, computing engine, and cluster applications are deployed on the computer equipment. The scheduler obtains the migration information generated in the migration configuration interface, and splits the migration information to generate multiple processing tasks that can be executed concurrently. The scheduler generates task information for each processing task, which includes the source of the data to be processed, the data shards to be processed, and the node information of the target nodes in the target cluster to which the data shards need to be written.

[0183] Step S904: The scheduler sends the task information corresponding to each processing task to the computing engine.

[0184] Specifically, the scheduler invokes the computing engine and sends each processing task and its corresponding task information to the computing engine.

[0185] The original data is split into data fragments to obtain data fragments corresponding to each processing task, including step S906:

[0186] Step S906: The original data is processed by the computing engine to obtain data fragments corresponding to each processing task.

[0187] Specifically, the scheduler invokes the computing engine, which then segments the raw data into data shards, resulting in data shards corresponding to each processing task. Further, the computing engine segments the raw data again, resulting in the same number of data shards as the number of processing tasks. Each processing task corresponds to one data shard.

[0188] In one embodiment, the scheduler generates a configuration file based on migration information and sends the configuration file to the computing engine. The computing engine performs data transformation processing on the original data according to the transformation rules in the configuration file to obtain data suitable for the target cluster. Then, the computing device partitions this data suitable for the target cluster according to the partitioning rules in the configuration file, obtaining a number of data partitions equal to the number of processing tasks. Next, the computing engine associates each data partition with a processing task, so that one processing task is associated with one data partition.

[0189] Parallel invocation of multiple cluster applications corresponding to the target cluster, with each cluster application concurrently executing its processing task to convert the corresponding data shards into storage data suitable for the target cluster, including steps S908-S910:

[0190] Step S908: The computing engine calls multiple cluster applications corresponding to the target cluster in parallel, and sends each data shard to the input pipeline of the corresponding cluster application through the output pipeline of the computing engine.

[0191] Specifically, the computing engine can identify the cluster application corresponding to the target cluster, and then invoke multiple cluster applications corresponding to that target cluster in parallel. Multiple output pipes of the computing engine are connected to the input pipes of each cluster application, and the computing engine sends each data slice to the corresponding cluster application's input pipe through its output pipe. Furthermore, the computing engine can send each data slice to the input pipe of a cluster application through an output pipe, ensuring that each cluster application receives the data slice corresponding to its processing task.

[0192] Step S910: Multiple cluster applications concurrently execute processing tasks based on each shard information to convert the corresponding data shards received from the input pipeline into storage data suitable for the target cluster.

[0193] Specifically, each cluster application receives the data shards corresponding to its respective processing task through an input pipeline. Based on the received shard information, the cluster applications concurrently execute each processing task, enabling each cluster application to convert the data shards corresponding to its own shard information into storage data suitable for the target cluster.

[0194] By invoking multiple cluster applications, the stored data is migrated to the online service of the target cluster, including step S912:

[0195] Step S912 involves using multiple cluster applications to determine the target node corresponding to each data shard from the node information, and then copying the stored data to the online service of the corresponding target node.

[0196] Specifically, each cluster application determines the target node based on the node information in the task information corresponding to its assigned processing task, and migrates the stored data processed by that task to the online service corresponding to the target node. Furthermore, multiple cluster applications can be invoked concurrently to migrate the stored data processed by each cluster application to the online service of the corresponding target node.

[0197] In this embodiment, task information corresponding to each processing task is generated through scheduling. This task information includes data shard information and corresponding target node information. The shard information determines the data shards to be processed by the task, and the node information determines the target node where the processed data needs to be stored. The scheduler sends the task information corresponding to each processing task to the computing engine, enabling the computing engine to perform sharding processing on the original data, dividing it into independent data shards that can be processed concurrently. The computing engine directly and in parallel calls multiple cluster applications corresponding to the target cluster. The input pipeline of the computing engine is connected to the input pipeline of each cluster application, enabling direct channel transmission of data, thus improving data transmission efficiency. By having multiple cluster applications concurrently execute each processing task based on the shard information, multiple data shards can be quickly converted into storage data suitable for the target cluster. Through parallel processing of data shards, the efficiency of converting data shards into storage data is improved. By calling multiple cluster applications based on the information of each node, the stored data is migrated to the online service of the target node corresponding to the corresponding node information. This enables the original data to be migrated from the original database to the target node of the target cluster, thus improving the efficiency of data migration.

[0198] In one embodiment, the original database is the database corresponding to the distributed computing framework, and the original data is the data corresponding to the distributed computing framework; the target cluster is a columnar storage database cluster, and the stored data is data that meets the storage format of the columnar storage database cluster.

[0199] Specifically, schedulers, computing engines, application containers, and cluster applications can be deployed on computer devices.

[0200] The computer device uses the data generated by the distributed computing framework as raw data and stores it in the raw database. When a user needs to migrate the raw data of this distributed computing framework to the online service of a columnar storage database cluster, the computer device can display a migration configuration interface on the object terminal, where the user can configure migration information. The object terminal is the terminal used by the user object. The scheduler obtains the migration information configured for the raw data migration, and from this information, it retrieves the concurrency information and the target number of nodes in the columnar storage database cluster. Based on the concurrency information and the target number of nodes, the scheduler splits the migration information to generate multiple processing tasks that can be executed concurrently. The scheduler generates task information corresponding to each processing task. The task information of a processing task includes the sharding information of the data shards to be processed by that processing task.

[0201] The scheduler generates a configuration file based on migration information and passes it to the computing engine. The computing engine retrieves raw data from the original database based on the configuration file; this configuration file includes transformation rules and partitioning rules; the computing engine transforms the raw data according to the transformation rules and partitions the transformed data according to the partitioning rules to obtain multiple data shards.

[0202] The scheduler invokes multiple application containers in parallel and sends each processing task and its corresponding task information to each application container. Each application container runs a cluster application, enabling each application container to execute a processing task.

[0203] Each application container invokes its respective cluster application to concurrently execute processing tasks based on the shard information received in the task information. This process converts the corresponding data shards into storage data suitable for columnar storage database clusters and stores them in the storage space of the corresponding application container. Each application container then invokes its respective cluster application to determine the target node corresponding to each data shard from the node information it receives, and copies the storage data from its own storage space to the target node corresponding to the corresponding data shard.

[0204] In this embodiment, the data migration method is applied to the scenario of migrating data from a distributed computing framework to an online service of a columnar storage database cluster. By using distributed scheduling, the user-configured migration information can be parsed into distributed processing tasks. The original data in the distributed computing framework is pre-processed and sharded by the computing engine. The sharded data is completely balanced. Each processing task processes one data shard, and each processing task is executed through a cluster application. At the same time, multiple application containers are called to start the cluster application to execute each processing task concurrently. This process converts each data shard into storage data in a columnar storage database and can automatically migrate the data to the online environment of the columnar storage database cluster.

[0205] In one embodiment, the distributed computing framework can be Hadoop, the column-oriented storage database cluster can be ClickHouse, the scheduler is Scheduler, the computing engine is Spark, the application container is Docker, and the cluster application is clickhouse-local. Figure 10 As shown, the core components of this data migration method are Scheduler, Spark, and Docker, and the functions of these core components are as follows:

[0206] Scheduler: Reads the job information entered by the user, breaks it down into concurrently executable processing tasks, manages the job and the lifecycle of processing tasks, and ensures data integrity. Scheduler generates metadata for each processing task, including the data source to be transformed, the specific data shard to be processed, and the information of the node in the ClickHouse cluster to be written to (sink).

[0207] Spark reads data from HDFS and preprocesses it. The preprocessing logic includes data loading, SQL filtering, data transformation, and data repartitioning (using random or hash rules) to create data shards. The files corresponding to each data shard are written to a specified HDFS directory. The HDFS filenames for each data shard will contain shard information to associate them with the corresponding processing task.

[0208] Docker: Converts a preprocessed Spark data shard into ClickHouse storage data. Docker uses the `clickhouse-local` tool to read the file data of the data shard in the shared HDFS directory, converts it into ClickHouse storage data, and then replicates and migrates the storage data to the online environment of the ClickHouse cluster based on node information.

[0209] By using distributed scheduling, the job information passed in by the user is parsed into a distributed task set. The raw data in Hadoop is automatically preprocessed and sharded by Spark. The sharded data is completely balanced. Each shard task corresponds to a clickhouse-local task. At the same time, multiple Docker containers are called to start the clickhouse-local task to generate ClickHouse storage data concurrently. The data can be automatically migrated to the ClickHouse production line.

[0210] For example, given 3.3 billion original data entries with 39 fields, totaling 85.8GB in compressed RCF format, which need to be migrated to six nodes in a ClickHouse cluster, the data migration method provided in this embodiment only takes 16 minutes, with minimal impact on the online cluster, and can be further accelerated by adding computing resources. In contrast, traditional data migration methods, such as writing via JDBC, can take up to 25 minutes in extreme cases. During the data import process, each node in the online cluster consumes six CPU cores, some memory, and disk I / O. Using a single write cluster to generate ClickHouse storage data takes one to two minutes longer than JDBC. Furthermore, the traditional ClickHouse-local tool cannot independently generate multi-sharded data and is limited by single-machine capabilities, processing a limited amount of data per second. Using the traditional ClickHouse-local tool to generate ClickHouse storage data takes over 1200 minutes. This embodiment, however, utilizes distributed scheduling to support the simultaneous execution of tens of thousands of data import jobs, resulting in higher resource utilization.

[0211] In one embodiment, a data migration method is provided, applied to a computer device, the computer device including an object terminal and having a scheduler, a computing engine, an application container, and a cluster application deployed on the computer device, the method comprising:

[0212] In response to migration operations triggered on the original data, the migration configuration interface is displayed on the object terminal.

[0213] In response to editing operations in the migration configuration interface, obtain the migration information generated by the editing operation.

[0214] The scheduler obtains the migration information configured for the original data migration, and extracts concurrency information and the target number of nodes in the target cluster from the migration information.

[0215] The scheduler splits the migration information based on concurrency information and the number of target nodes to generate multiple processing tasks that can be executed concurrently. The scheduler then generates task information corresponding to each processing task.

[0216] The scheduler generates a configuration file based on the migration information and then passes the configuration file to the computing engine.

[0217] The computing engine retrieves raw data from the original database based on the configuration file; the configuration file includes transformation rules and partitioning rules; the computing engine transforms the raw data according to the transformation rules, and partitions the data obtained from the data transformation according to the partitioning rules, resulting in multiple data fragments.

[0218] The scheduler invokes multiple application containers in parallel and sends the task information corresponding to each processing task to each application container. Each application container runs a cluster application.

[0219] Each application container calls its respective cluster application to concurrently execute each processing task based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster and store them in the storage space of the corresponding application container.

[0220] Each application container calls its respective cluster application to determine the target node corresponding to each data shard from the node information it receives, and copies the stored data in its respective storage space to the online service of the target node corresponding to the data shard.

[0221] Once the scheduler detects that all processing tasks have been completed, it will recycle each application container.

[0222] In this embodiment, in response to a migration operation triggered on the original data, a migration configuration interface is displayed, allowing users to independently configure migration-related information for the original data, providing them with more options. In response to an edit operation on the migration configuration interface, the migration information generated by the edit operation is obtained, enabling the scheduler to generate a configuration file based on the migration information. Furthermore, the migration information can be split to generate multiple processing tasks that can be executed concurrently, allowing each processing task to handle one data shard.

[0223] The scheduler starts the computing engine and sends the configuration file to it. The computing engine can obtain the location information of the raw data from the configuration file to retrieve the raw data from the original database. The computing engine transforms the raw data according to the transformation rules and divides the transformed data into multiple data shards that can be processed concurrently, according to the partitioning rules.

[0224] By using a scheduler to invoke multiple application containers in parallel and sending task information corresponding to each processing task to each container, and then having each application container invoke its own cluster application to concurrently execute each processing task based on the shard information in the received task information, multiple data shards can be quickly converted into storage data suitable for the target cluster. This parallel conversion of data shards improves the efficiency of data shard conversion into storage data. Furthermore, by having each application container invoke its own cluster application to determine the target node corresponding to each data shard from the received node information, and then having each application container invoke its own cluster application to migrate the generated storage data to the online service of the target node, the original data can be concurrently migrated from the original database to the target node of the target cluster, thus improving the efficiency of data migration.

[0225] It should be understood that, although Figures 2-10 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-10 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0226] In one embodiment, such as Figure 11 As shown, a data migration device 1100 is provided. This device can be a software module, a hardware module, or a combination of both, integrated into a computer device. Specifically, the device includes: a splitting module 1102, an acquisition module 1104, a processing module 1106, a concurrency module 1108, and a migration module 1110, wherein:

[0227] The splitting module 1102 is used to obtain migration information configured for the original data migration and split the migration information to generate multiple processing tasks that can be executed concurrently.

[0228] The acquisition module 1104 is used to retrieve the original data from the original database based on the migration information.

[0229] The processing module 1106 is used to perform data sharding on the raw data to obtain data shards corresponding to each processing task.

[0230] The concurrency module 1108 is used to call multiple cluster applications corresponding to the target cluster in parallel. By having multiple cluster applications execute each processing task concurrently, the corresponding data shards are converted into storage data suitable for the target cluster.

[0231] Migration module 1110 is used to migrate various stored data to the online service of the target cluster by calling multiple cluster applications.

[0232] In this embodiment, migration information configured for the original data migration is obtained and split into multiple processing tasks that can be executed concurrently, allowing each processing task to handle one data shard. Based on the migration information, original data is retrieved from the original database, and the original data is sharded to obtain data shards corresponding to each processing task. Multiple cluster applications corresponding to the target cluster are invoked in parallel. These cluster applications concurrently execute each processing task to convert the corresponding data shards into storage data suitable for the target cluster. This allows for the simultaneous and rapid conversion of multiple data shards into their respective storage data, effectively improving the efficiency of data shard conversion. Through the invoked cluster applications, each storage data is migrated to the online service of the target cluster, thereby enabling the rapid migration of original data from the original database to the online service of the target cluster, improving the efficiency of data migration.

[0233] In one embodiment, the splitting module 1102 is further configured to display a migration configuration interface in response to a migration operation triggered on the original data; and to obtain migration information generated by the editing operation in response to an editing operation on the migration configuration interface.

[0234] In this embodiment, in response to a migration operation triggered on the original data, a migration configuration interface is displayed, allowing users to independently configure migration-related information for the original data, providing them with more options. In response to an editing operation on the migration configuration interface, the migration information generated by the editing operation is obtained. Based on the user-configured migration information, the migration task for the original data is automatically executed, improving the efficiency of data migration.

[0235] In one embodiment, the splitting module 1102 is further configured to obtain migration information configured for the original data migration, obtain concurrency information and the number of target nodes in the target cluster from the migration information, and split the migration information according to the concurrency information and the number of target nodes to generate multiple processing tasks that can be executed concurrently.

[0236] In this embodiment, the migration information configured by the user for the original data migration is obtained. Concurrency information and the number of target nodes in the target cluster are obtained from the migration information. Based on the concurrency information and the number of target nodes, the migration information is automatically split to generate multiple processing tasks that can be executed concurrently. This ensures that the processing logic and data of each processing task are independent of each other, and the processing process can be made independent of each other. This enables multiple tasks to be processed concurrently at the same time, thereby improving data processing efficiency.

[0237] In one embodiment, the acquisition module 1104 is further configured to generate a configuration file based on migration information and acquire raw data from the original database based on the configuration file; the configuration file includes conversion rules and partitioning rules;

[0238] The processing module 1106 is also used to perform data transformation on the original data according to the transformation rules, and to divide the data obtained from the data transformation according to the partitioning rules to obtain multiple data fragments;

[0239] Each data shard is associated with a processing task, with different processing tasks associated with different data shards.

[0240] In this embodiment, a configuration file is generated based on migration information. This configuration file determines whether to perform filtering, transformation, repartitioning, and storage processing on the original data. Original data is retrieved from the original database based on the configuration file. Data transformation is performed on the original data according to the transformation rules in the configuration file, converting the original data from its original database data type to data suitable for the target cluster. The transformed data is then divided into multiple data shards according to partitioning rules. Performing data transformation before sharding avoids inaccurate sharding due to differences in data volume and dependencies before and after transformation. Each data shard is associated with a processing task, with different processing tasks associated with different data shards. This ensures that each subsequent processing task performs an operation to convert a data shard into storage data suitable for the target cluster's storage format. Furthermore, the processing tasks can be executed concurrently, enabling the concurrent conversion of each data shard into its corresponding storage data, thus improving the efficiency of data transformation.

[0241] In one embodiment, the acquisition module 1102 is further configured to generate a configuration file based on migration information via the scheduler and transmit the configuration file to the computing engine; and to obtain raw data from the original database via the computing engine based on the configuration file.

[0242] The processing module 1106 is also used to perform data transformation on the original data according to the transformation rules through the computing engine, and to divide the data obtained from the data transformation according to the partitioning rules to obtain multiple data fragments.

[0243] In this embodiment, the scheduler can generate a configuration file based on migration information. The scheduler then starts the computing engine and passes the configuration file to the computing engine. The computing engine can obtain the location information of the original data from the configuration file to retrieve the original data from the original database. The computing engine performs data transformation on the original data according to transformation rules and divides the data obtained from the data transformation according to partitioning rules, thus accurately obtaining multiple data shards that can be processed concurrently.

[0244] In one embodiment, the splitting module 1102 is further configured to generate task information corresponding to each processing task. The task information includes data sharding information and node information of the corresponding target node, wherein the target node is a node in the target cluster.

[0245] The concurrency module 1108 is also used to call multiple cluster applications corresponding to the target cluster in parallel. Multiple cluster applications concurrently execute each processing task based on each shard information to convert the corresponding data shards into storage data suitable for the target cluster.

[0246] The migration module 1110 is also used to migrate stored data to the online service of the target node corresponding to the corresponding node information by calling multiple cluster applications based on the node information.

[0247] In this embodiment, task information corresponding to each processing task is generated. This task information includes data shard information and corresponding target node information. The shard information determines the data shard to be processed by the task, and the node information determines the target node where the processed data from the data shards should be stored. By invoking multiple cluster applications corresponding to the target cluster in parallel, and having these applications concurrently execute each processing task based on the shard information, multiple data shards can be quickly converted into storage data suitable for the target cluster. Parallel processing of data shards improves the efficiency of converting data shards into storage data. Based on the node information, the invoked cluster applications migrate each storage data to the online service of the target node corresponding to the respective node information. This allows the original data to be migrated from the original database to the target node of the target cluster, improving the efficiency of data migration.

[0248] In one embodiment, the splitting module 1102 is further configured to generate task information corresponding to each processing task through the scheduler;

[0249] The concurrency module 1108 is also used to call multiple application containers in parallel through the scheduler and send the task information corresponding to each processing task to each application container; each application container calls its own cluster application to execute each processing task concurrently based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster and store them in the storage space of the corresponding application container.

[0250] The migration module 1110 is also used to call the respective cluster applications of each application container to determine the target node corresponding to each data shard from the node information received by each application, and to copy the stored data in their respective storage space to the online service of the target node corresponding to the data shard.

[0251] In this embodiment, a scheduler generates task information corresponding to each processing task. This task information includes data shard information and corresponding target node information. The shard information determines the data shard to be processed by the task, and the node information determines the target node where the processed data from the data shards will be stored. By concurrently calling multiple application containers and sending the task information corresponding to each processing task to each container, and by having each application container call its own cluster application to concurrently execute each processing task based on the shard information in the received task information, multiple data shards can be quickly converted into storage data suitable for the target cluster. Parallel processing of data shards improves the efficiency of converting data shards into storage data. Each application container calls its own cluster application to determine the target node corresponding to each data shard from the received node information, and then migrates the generated storage data to the online service of the target node. This enables the concurrent migration of original data from the original database to the target node of the target cluster, improving the efficiency of data migration.

[0252] In one embodiment, the apparatus further includes a recycling module, configured to recycle each application container when the scheduler detects that all processing tasks have been completed. The scheduler can detect whether each processing task has been completed, and when the scheduler detects that all processing tasks have been completed, it can recycle each application container to release the occupied resources.

[0253] In one embodiment, the splitting module 1102 is further configured to generate task information corresponding to each processing task through the scheduler;

[0254] The concurrency module 1108 is also used to send the task information corresponding to each processing task to the computing engine through the scheduler; to call multiple cluster applications corresponding to the target cluster in parallel through the computing engine; and to execute each processing task concurrently through multiple cluster applications based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster.

[0255] The migration module 1110 is also used to determine the target node corresponding to each data shard from the node information of each task information by calling multiple cluster applications, and to copy the stored data to the online service of the corresponding target node.

[0256] In this embodiment, task information corresponding to each processing task is generated through scheduling. This task information includes data shard information and corresponding target node information. The shard information determines the data shard to be processed by the task, and the node information determines the target node where the processed data from the data shards needs to be stored. The computing engine calls multiple cluster applications corresponding to the target cluster in parallel. These applications concurrently execute processing tasks based on the shard information, enabling the simultaneous and rapid conversion of multiple data shards into storage data suitable for the target cluster. This parallel processing of data shards improves the efficiency of converting data shards into storage data. The multiple cluster applications, based on the node information, migrate the storage data to the online service of the target node corresponding to the node information. This migrates the original data from the original database to the target node of the target cluster, improving the efficiency of data migration.

[0257] In one embodiment, the splitting module 1102 is further configured to generate task information corresponding to each processing task through the scheduler; the task information includes the data sharding information and the node information of the corresponding target node, the target node being a node in the target cluster; and the task information corresponding to each processing task is sent to the computing engine through the scheduler.

[0258] The processing module 1106 is also used to perform data sharding on the raw data through the computing engine to obtain data shards corresponding to each processing task.

[0259] The concurrency module 1108 is also used to call multiple cluster applications corresponding to the target cluster in parallel through the computing engine, send each data shard to the input pipe of the corresponding cluster application through the output pipe of the computing engine; and execute each processing task concurrently by multiple cluster applications based on the shard information to convert the corresponding data shards received from the input pipe into storage data suitable for the target cluster.

[0260] The migration module 1110 is also used to determine the target node corresponding to each data shard from the node information by calling multiple cluster applications, and to copy the stored data to the online service of the corresponding target node.

[0261] In this embodiment, task information corresponding to each processing task is generated through scheduling. This task information includes data shard information and corresponding target node information. The shard information determines the data shards to be processed by the task, and the node information determines the target node where the processed data needs to be stored. The scheduler sends the task information corresponding to each processing task to the computing engine, enabling the computing engine to perform sharding processing on the original data, dividing it into independent data shards that can be processed concurrently. The computing engine directly and in parallel calls multiple cluster applications corresponding to the target cluster. The input pipeline of the computing engine is connected to the input pipeline of each cluster application, enabling direct channel transmission of data, thus improving data transmission efficiency. By having multiple cluster applications concurrently execute each processing task based on the shard information, multiple data shards can be quickly converted into storage data suitable for the target cluster. Through parallel processing of data shards, the efficiency of converting data shards into storage data is improved. By calling multiple cluster applications based on the information of each node, the stored data is migrated to the online service of the target node corresponding to the corresponding node information. This enables the original data to be migrated from the original database to the target node of the target cluster, thus improving the efficiency of data migration.

[0262] In one embodiment, the original database is the database corresponding to the distributed computing framework, and the original data is the data corresponding to the distributed computing framework; the target cluster is a columnar storage database cluster, and the stored data is data that meets the storage format of the columnar storage database cluster.

[0263] In this embodiment, the data migration method is applied to the scenario of migrating data from a distributed computing framework to an online service of a columnar storage database cluster. By using distributed scheduling, the user-configured migration information can be parsed into distributed processing tasks. The original data in the distributed computing framework is pre-processed and sharded by the computing engine. The sharded data is completely balanced. Each processing task processes one data shard, and each processing task is executed through a cluster application. At the same time, multiple application containers are called to start the cluster application to execute each processing task concurrently. This process converts each data shard into storage data in a columnar storage database and can automatically migrate the data to the online environment of the columnar storage database cluster.

[0264] Specific limitations regarding the data migration device can be found in the limitations of the data migration method described above, and will not be repeated here. Each module in the aforementioned data migration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0265] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data migration data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data migration method.

[0266] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0267] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0268] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0269] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0270] It should be noted that the user information (including but not limited to user device information, user personal information, user configuration information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. Furthermore, users can refuse or conveniently refuse push notifications, etc.

[0271] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0272] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0273] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data migration method, characterized in that, The method includes: Obtain the migration information configured for the original data migration, and split the migration information to generate multiple processing tasks that can be executed concurrently; The scheduler generates task information corresponding to each processing task; the task information includes data sharding information and the node information of the corresponding target node, wherein the target node is a node in the target cluster; The original data is obtained from the original database based on the migration information; The original data is split into data fragments to obtain data fragments corresponding to each processing task. The scheduler invokes multiple application containers in parallel and sends the task information corresponding to each processing task to each application container. Each application container calls its respective cluster application to concurrently execute each processing task based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster and store them in the storage space of the corresponding application container. Each application container calls its respective cluster application to determine the target node corresponding to each data shard from the node information it receives, and copies the stored data in its respective storage space to the online service of the target node corresponding to the data shard.

2. The method according to claim 1, characterized in that, The acquisition of migration information configured for the original data migration includes: In response to migration operations triggered on the original data, display the migration configuration interface; In response to an edit operation on the migration configuration interface, the migration information generated by the edit operation is obtained.

3. The method according to claim 1, characterized in that, The step of acquiring migration information configured for the original data migration and splitting the migration information to generate multiple processing tasks that can be executed concurrently includes: Obtain the migration information configured for the original data migration, and extract the concurrency information and the target number of nodes in the target cluster from the migration information; Based on the concurrency information and the number of target nodes, the migration information is split to generate multiple processing tasks that can be executed concurrently.

4. The method according to claim 1, characterized in that, The step of obtaining the original data from the original database based on the migration information includes: A configuration file is generated based on the migration information, and the original data is retrieved from the original database based on the configuration file; the configuration file includes conversion rules and partitioning rules; The step of segmenting the original data to obtain data segments corresponding to each processing task includes: The original data is transformed using the transformation rules, and the transformed data is divided according to the partitioning rules to obtain multiple data fragments. Each data shard is associated with each processing task, wherein different processing tasks are associated with different data shards.

5. The method according to claim 4, characterized in that, The step of generating a configuration file based on the migration information and obtaining the original data from the original database based on the configuration file includes: The scheduler generates a configuration file based on the migration information and then transmits the configuration file to the computing engine. The computing engine retrieves the original data from the original database based on the configuration file. The original data is transformed using the transformation rules, and the transformed data is then divided according to the partitioning rules to obtain multiple data fragments, including: The computing engine performs data transformation on the original data according to the transformation rules, and divides the transformed data into multiple data fragments according to the partitioning rules.

6. The method according to claim 1, characterized in that, The method further includes: When the scheduler detects that all the processing tasks have been completed, it recycles each of the application containers.

7. The method according to claim 1, characterized in that, The method further includes: The scheduler sends the task information corresponding to each processing task to the computing engine. The original data is segmented by the computing engine to obtain data segments corresponding to each processing task. The computing engine calls multiple cluster applications corresponding to the target cluster in parallel, and sends each data shard to the input pipeline of the corresponding cluster application through the output pipeline of the computing engine. The multiple cluster applications concurrently execute each processing task based on the sharding information to convert the corresponding data shards received from the input pipeline into storage data suitable for the target cluster. The multiple cluster applications invoked determine the target node corresponding to each data shard from the node information, and copy the stored data to the online service of the corresponding target node.

8. The method according to any one of claims 1 to 7, characterized in that, The original database is the database corresponding to the distributed computing framework, and the original data is the data corresponding to the distributed computing framework; the target cluster is a columnar storage database cluster, and the stored data is the data that meets the storage format of the columnar storage database cluster.

9. A data migration method, characterized in that, The method includes: Obtain the migration information configured for the original data migration, and split the migration information to generate multiple processing tasks that can be executed concurrently; Generate task information corresponding to each of the processing tasks. The task information includes data sharding information and node information of the corresponding target node, wherein the target node is a node in the target cluster. The original data is obtained from the original database based on the migration information; The original data is split into data fragments to obtain data fragments corresponding to each processing task. The scheduler sends the task information corresponding to each of the processing tasks to the computing engine. The computing engine calls multiple cluster applications corresponding to the target cluster in parallel. The multiple cluster applications concurrently execute each processing task based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster. The multiple cluster applications invoked determine the target node corresponding to each data shard from the node information of each task information, and copy the stored data to the online service of the corresponding target node.

10. The method according to claim 9, characterized in that, The acquisition of migration information configured for the original data migration includes: In response to migration operations triggered on the original data, display the migration configuration interface; In response to an edit operation on the migration configuration interface, the migration information generated by the edit operation is obtained.

11. The method according to claim 9, characterized in that, The step of acquiring migration information configured for the original data migration and splitting the migration information to generate multiple processing tasks that can be executed concurrently includes: Obtain the migration information configured for the original data migration, and extract the concurrency information and the target number of nodes in the target cluster from the migration information; Based on the concurrency information and the number of target nodes, the migration information is split to generate multiple processing tasks that can be executed concurrently.

12. The method according to claim 9, characterized in that, The step of obtaining the original data from the original database based on the migration information includes: A configuration file is generated based on the migration information, and the original data is retrieved from the original database based on the configuration file; the configuration file includes conversion rules and partitioning rules; The step of segmenting the original data to obtain data segments corresponding to each processing task includes: The original data is transformed using the transformation rules, and the transformed data is divided according to the partitioning rules to obtain multiple data fragments. Each data shard is associated with each processing task, wherein different processing tasks are associated with different data shards.

13. The method according to claim 12, characterized in that, The step of generating a configuration file based on the migration information and obtaining the original data from the original database based on the configuration file includes: The scheduler generates a configuration file based on the migration information and then transmits the configuration file to the computing engine. The computing engine retrieves the original data from the original database based on the configuration file. The original data is transformed using the transformation rules, and the transformed data is then divided according to the partitioning rules to obtain multiple data fragments, including: The computing engine performs data transformation on the original data according to the transformation rules, and divides the transformed data into multiple data fragments according to the partitioning rules.

14. The method according to claim 9, characterized in that, The method further includes: The scheduler sends the task information corresponding to each processing task to the computing engine. The original data is segmented by the computing engine to obtain data segments corresponding to each processing task. The computing engine calls multiple cluster applications corresponding to the target cluster in parallel, and sends each data shard to the input pipeline of the corresponding cluster application through the output pipeline of the computing engine. The multiple cluster applications concurrently execute each processing task based on the sharding information to convert the corresponding data shards received from the input pipeline into storage data suitable for the target cluster. The multiple cluster applications invoked determine the target node corresponding to each data shard from the node information, and copy the stored data to the online service of the corresponding target node.

15. The method according to any one of claims 9 to 14, characterized in that, The original database is the database corresponding to the distributed computing framework, and the original data is the data corresponding to the distributed computing framework; the target cluster is a columnar storage database cluster, and the stored data is the data that meets the storage format of the columnar storage database cluster.

16. A data migration device, characterized in that, The device includes: The splitting module is used to obtain migration information configured for the original data migration and split the migration information to generate multiple processing tasks that can be executed concurrently. The scheduling module is used to generate task information corresponding to each of the processing tasks through a scheduler; the task information includes data sharding information and node information of the corresponding target node, wherein the target node is a node in the target cluster; The acquisition module is used to acquire the original data from the original database based on the migration information; The processing module is used to perform data sharding on the original data to obtain data shards corresponding to each processing task. The scheduling module is also used to call multiple application containers in parallel through the scheduler and send the task information corresponding to each processing task to each application container. The concurrency module is used to call the respective cluster application of each application container to concurrently execute each processing task based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster and store them in the storage space of the corresponding application container. The migration module is used to call the respective cluster application of each application container to determine the target node corresponding to each data shard from the node information received by each application container, and to copy the stored data in the storage space of each application container to the online service of the target node corresponding to the data shard.

17. The data migration apparatus according to claim 16, characterized in that, The splitting module is also used to display a migration configuration interface in response to a migration operation triggered on the original data; and to obtain migration information generated by the editing operation in response to an editing operation on the migration configuration interface.

18. The data migration apparatus according to claim 16, characterized in that, The splitting module is also used to obtain migration information configured for the original data migration, and to obtain concurrency information and the number of target nodes in the target cluster from the migration information; Based on the concurrency information and the number of target nodes, the migration information is split to generate multiple processing tasks that can be executed concurrently.

19. The data migration apparatus according to claim 16, characterized in that, The acquisition module is also used to generate a configuration file based on the migration information, and to obtain the original data from the original database based on the configuration file; the configuration file includes conversion rules and partitioning rules; The processing module is further configured to perform data transformation on the original data according to the transformation rules, and divide the data obtained from the data transformation according to the division rules to obtain multiple data fragments; and associate each data fragment with each processing task, wherein different processing tasks are associated with different data fragments.

20. The data migration apparatus according to claim 19, characterized in that, The acquisition module is also used to generate a configuration file based on the migration information through the scheduler, and transmit the configuration file to the computing engine; and to obtain the original data from the original database through the computing engine based on the configuration file. The processing module is also used to perform data transformation on the original data according to the transformation rules through the computing engine, and to divide the data obtained from the data transformation according to the partitioning rules to obtain multiple data fragments.

21. The data migration apparatus according to claim 16, characterized in that, The device further includes a recycling module, which is used to recycle each application container when the scheduler detects that all the processing tasks have been completed.

22. The data migration apparatus according to claim 16, characterized in that, The scheduling module is also used to send the task information corresponding to each of the processing tasks to the computing engine through the scheduler. The processing module is also used to perform sharding processing on the original data through a computing engine to obtain data shards corresponding to each processing task. The concurrency module is also used to call multiple cluster applications corresponding to the target cluster in parallel through the computing engine, and send each data shard to the input pipe of the corresponding cluster application through the output pipe of the computing engine; and to execute each processing task concurrently by the multiple cluster applications based on the shard information, so as to convert the corresponding data shards received from the input pipe into storage data suitable for the target cluster. The migration module is also used to determine the target node corresponding to each data shard from the node information by calling the multiple cluster applications, and to copy the stored data to the online service of the corresponding target node.

23. The data migration apparatus according to any one of claims 16 to 22, characterized in that, The original database is the database corresponding to the distributed computing framework, and the original data is the data corresponding to the distributed computing framework; The target cluster is a columnar storage database cluster, and the stored data is data that meets the storage format of the columnar storage database cluster.

24. A data migration device, characterized in that, The device includes: The splitting module is used to obtain migration information configured for the original data migration and split the migration information to generate multiple processing tasks that can be executed concurrently. The scheduling module is used to generate task information corresponding to each of the processing tasks. The task information includes data sharding information and node information of the corresponding target node, wherein the target node is a node in the target cluster. The acquisition module is used to acquire the original data from the original database based on the migration information; The processing module is used to perform data sharding on the original data to obtain data shards corresponding to each processing task. The scheduling module is also used to send the task information corresponding to each processing task to the computing engine through the scheduler; The concurrency module is used to call multiple cluster applications corresponding to the target cluster in parallel through the computing engine. The multiple cluster applications concurrently execute each processing task based on the sharding information in the received task information, so as to convert the corresponding data shards into storage data suitable for the target cluster. The migration module is used to determine the target node corresponding to each data shard from the node information of each task information by calling the multiple cluster applications, and to copy the stored data to the online service of the corresponding target node.

25. The data migration apparatus according to claim 24, characterized in that, The splitting module is also used to display a migration configuration interface in response to a migration operation triggered on the original data; and to obtain migration information generated by the editing operation in response to an editing operation on the migration configuration interface.

26. The data migration apparatus according to claim 24, characterized in that, The splitting module is also used to obtain migration information configured for the original data migration, and to obtain concurrency information and the number of target nodes in the target cluster from the migration information; Based on the concurrency information and the number of target nodes, the migration information is split to generate multiple processing tasks that can be executed concurrently.

27. The data migration apparatus according to claim 24, characterized in that, The acquisition module is also used to generate a configuration file based on the migration information, and to obtain the original data from the original database based on the configuration file; the configuration file includes conversion rules and partitioning rules; The processing module is further configured to perform data transformation on the original data according to the transformation rules, and divide the data obtained from the data transformation according to the division rules to obtain multiple data fragments; and associate each data fragment with each processing task, wherein different processing tasks are associated with different data fragments.

28. The data migration apparatus according to claim 27, characterized in that, The acquisition module is also used to generate a configuration file based on the migration information through the scheduler, and transmit the configuration file to the computing engine; and to obtain the original data from the original database through the computing engine based on the configuration file. The processing module is also used to perform data transformation on the original data according to the transformation rules through the computing engine, and to divide the data obtained from the data transformation according to the partitioning rules to obtain multiple data fragments.

29. The data migration apparatus according to claim 24, characterized in that, The scheduling module is also used to send the task information corresponding to each of the processing tasks to the computing engine through the scheduler. The processing module is also used to perform sharding processing on the original data through a computing engine to obtain data shards corresponding to each processing task. The concurrency module is also used to call multiple cluster applications corresponding to the target cluster in parallel through the computing engine, and send each data shard to the input pipe of the corresponding cluster application through the output pipe of the computing engine; and to execute each processing task concurrently by the multiple cluster applications based on the shard information, so as to convert the corresponding data shards received from the input pipe into storage data suitable for the target cluster. The migration module is also used to determine the target node corresponding to each data shard from the node information by calling the multiple cluster applications, and to copy the stored data to the online service of the corresponding target node.

30. The data migration apparatus according to any one of claims 24 to 29, characterized in that, The original database is the database corresponding to the distributed computing framework, and the original data is the data corresponding to the distributed computing framework; The target cluster is a columnar storage database cluster, and the stored data is data that meets the storage format of the columnar storage database cluster.

31. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 15.

32. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 15.

33. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 15.

Images