Data migration method and computing device

Abstract

The embodiment of the application discloses a data migration method and a computing device, and relates to the technical field of computing devices. The method comprises the following steps: displaying a visual interface, wherein the visual interface comprises visual configuration elements of configuration items; receiving configuration information input by a user based on the visual configuration elements; the configuration information is used for describing migration of to-be-migrated data from a source end to a target end, and the to-be-migrated data comprises multiple types of offline data, real-time streaming data, cache data or file data; based on the configuration information, a scheduling instruction recognizable by a second computing device is determined, the scheduling instruction is used for scheduling the second computing device to execute a target task; the target task is used for indicating the migration of the to-be-migrated data from the source end to the target end; and the scheduling instruction is sent to the second computing device, and the second computing device is used for executing the target task in response to the scheduling instruction. In this way, different types of data migration tasks are uniformly arranged and processed in the same migration period.

Description

Technical Field

[0001] This application relates to the field of computing device technology, and in particular to a data migration method and computing device. Background Technology

[0002] In data migration scenarios, the data to be migrated typically includes multiple formats, such as offline data, real-time streaming data, cached data, or various types of file data. These different types of data differ in storage media, synchronization timelines, or transmission methods. Therefore, during the data migration process, different types of data cannot be uniformly orchestrated and processed within the same migration cycle, resulting in fragmented migration links for different data types and low migration efficiency.

[0003] Therefore, there is an urgent need for an efficient data migration method that can uniformly orchestrate and process different types of data migration tasks within the same migration cycle. Summary of the Invention

[0004] This application provides a data migration method and computing device, which enables unified orchestration and processing of different types of data migration tasks within the same migration cycle.

[0005] In a first aspect, embodiments of this application provide a data migration method applied to a first computing device. The method includes: displaying a visual interface, the visual interface including visual configuration elements of configuration items; receiving configuration information input by a user based on the visual configuration elements; the configuration information includes a source and a target of the data to be migrated, the configuration information describing the migration of the data to be migrated from the source to the target, the data to be migrated including various types such as offline data, real-time streaming data, cached data, or file data; based on the configuration information, determining a scheduling instruction recognizable by a second computing device, the scheduling instruction being used to schedule the second computing device to execute a target task; the target task being used to instruct the migration of the data to be migrated from the source to the target; and sending the scheduling instruction to the second computing device, the second computing device executing the target task in response to the scheduling instruction.

[0006] In this way, a unified configuration entry point is provided to users through a visual interface, supporting the configuration of migration information for various heterogeneous data types such as offline data, real-time streaming data, cached data, and file data. Migration configuration can be completed without requiring specialized data technology knowledge, lowering the barrier to entry and fundamentally simplifying the configuration operation for migrating multiple types of heterogeneous data. This solves the problems of scattered configuration and cumbersome operation in migrating multiple types of data. Based on this, standardized scheduling instructions recognizable by a second computing device are automatically generated based on the unified configuration information. This achieves unified configuration, classification, adaptation, and centralized scheduling of multi-form data migration tasks within the same migration lifecycle. It avoids the fragmentation of migration links for different types of data and decouples the configuration layer from the execution layer. It enables centralized and unified orchestration and collaborative processing of multiple different types of data migration tasks within the same migration cycle, improving the ease of operation of data migration configuration and scheduling, the management efficiency and system compatibility of multi-form data migration, and optimizing the overall migration process execution efficiency.

[0007] In one possible implementation, the visual interface includes graphical or form-based visual configuration elements, which are structured encapsulations of configuration items. Furthermore, the visual interface employs a guided configuration approach, which may include one or more of the following: wizard-style configuration prompts, configuration templates, or default value configuration methods.

[0008] In this way, by encapsulating configuration items into structured graphical and form-based visual configuration elements, and using guided configuration methods such as wizard prompts, configuration templates, and default values ​​to intuitively guide non-professionals, the exposure of required parameters is reduced, the learning threshold for non-professionals is lowered, and guided migration task construction is achieved, enabling non-professionals to complete the migration independently.

[0009] In one possible implementation, multiple types of data migration tasks are carried out through a process model. These multiple types of data migration tasks include offline migration tasks for migrating offline data, real-time migration tasks for migrating real-time streaming data, cache migration tasks for migrating cached data, and file migration tasks for migrating file data. Different types of data migration tasks correspond to different semantic nodes in the process model. The visualization interface includes visual configuration elements for semantic nodes and their configuration items. Multiple nodes include multiple offline task nodes, real-time task nodes, cache task nodes, or file task nodes. The visual configuration elements for semantic nodes are used to configure the configuration information for the corresponding type of data migration task. And / or, one or more of the offline migration tasks, real-time migration tasks, or cache migration tasks correspond to an integrated task container in the process model. The visualization interface includes an integrated task container and visual configuration elements for its configuration items. The visual configuration elements for the integrated task container are used to configure the data processing method for the corresponding data migration task. The data processing method includes one or more of the following: Structured Query Language (SQL) conversion, field mapping, field renaming, or field splitting.

[0010] In this way, by using a unified process model to support multiple types of data migration tasks, and expressing migration semantics through dedicated semantic nodes such as offline, real-time, cached, and file-based methods and integrated task containers, the parameter differences between different data sources can be shielded. In the visual interface, only the migration semantics need to be described; at runtime, it can be automatically mapped to scheduling instructions, achieving a "what you see is what you get" configuration-to-execution mapping. This lowers the configuration threshold, facilitates the development of standardized migration templates, and significantly improves the automated implementation capability of task configuration.

[0011] In one possible implementation, the method further includes: receiving operation instructions triggered by the user on a visual interface. In response to the operation instructions, directed connections are established between task elements corresponding to different data migration tasks to configure dependencies between different data migration tasks. Task elements include semantic nodes and / or integrated task containers. Dependencies include one or more of the following: task execution order, process branching and merging relationships, conditional routing execution relationships, multi-level transformation link relationships, or sub-process call relationships. Based on the configuration information, scheduling instructions that the second computing device can recognize are determined, including: determining scheduling instructions based on the configuration information and the dependencies between different data migration tasks. These scheduling instructions are specifically used to schedule the second computing device to execute the target task based on the dependencies.

[0012] Thus, by configuring dependencies through directed connections between semantic nodes and / or integrated task containers in a visual interface, complex migration chains such as multi-level transformations, branch merging, conditional routing, and sub-process calls can be intuitively expressed. This supports the construction of reusable process fragments and task templates in a component-based manner, effectively avoiding the verbose nesting and repetitive pasting of plain text configurations, significantly improving the maintainability and configuration efficiency of complex migration processes. Furthermore, it enables the second computing device to execute target tasks including multiple data migration tasks according to these dependencies, improving the accuracy of process orchestration and the controllability of execution.

[0013] In one possible implementation, the method further includes: storing configuration items corresponding to semantic nodes and integration task containers; receiving user update commands; and, in response to user update commands, adding, deleting, or modifying one or more of the stored configuration items through visual configuration elements; and / or providing preset interfaces for different data migration tasks, whereby the preset interfaces are used by data migration tasks of the corresponding type to reference the stored configuration items.

[0014] In this way, by uniformly storing and visually updating the configuration items of semantic nodes and / or integrated task containers, and supporting the reuse of stored configuration items through preset interfaces, centralized management and reuse of various configuration information can be achieved, effectively reducing the cost of repetitive configuration and changes, and lowering the configuration complexity and parameter maintenance cost of migration tasks. At the same time, modularization of configuration is achieved, eliminating the need for extensive manual writing of configuration files and copy-pasting configurations, improving the readability, maintainability, and reusability of migration tasks, and facilitating multi-person collaboration and configuration auditing.

[0015] In one possible implementation, the configuration information further includes data migration methods and / or data processing methods. The data migration method indicates the migration method used during the migration process of the data to be migrated. Data migration methods include one or more of the following: data acquisition strategy, concurrent sharding strategy, batch size strategy, parallel write strategy, breakpoint resume strategy, retry strategy, rate limiting strategy, or idempotent write strategy. The data processing method indicates the processing method used on the data to be migrated during the migration process; data processing methods include one or more of the following: Structured Query Language (SQL) conversion, field mapping, field renaming, or field splitting. The scheduling instruction is specifically used to schedule the second computing device to execute the target task based on the data migration method and / or data processing method.

[0016] In this way, by visually configuring data migration methods and / or data processing methods, complex strategies such as concurrent sharding, breakpoint resumption, idempotent writing, SQL conversion, and field mapping can be uniformly incorporated into the configuration process and automatically converted into scheduling instructions for execution. There is no need to manually write underlying configurations and scripts, which realizes the integrated configuration and standardized execution of migration rules and processing logic, improving the completeness of task configuration and execution accuracy.

[0017] In one possible implementation, the method further includes: receiving verification information input by a user based on a visual configuration element, the verification information indicating the data to be verified after migration from the source to the target. The verification includes one or more of the following: quantity consistency verification, content integrity verification, data security verification, format validity verification, or data accuracy verification. Based on the verification information, a verification instruction that a second computing device can recognize is determined. The verification instruction is sent to the second computing device, which is further configured to execute a corresponding verification task in response to the verification instruction.

[0018] In this way, by visually configuring verification information and automatically generating and executing verification commands, the migration data indicated by the target task can be automatically verified in depth across multiple dimensions, including consistency, integrity, accuracy, security, and legality, after the migration is completed, thus building a closed loop for data consistency assurance across platforms and modalities. Simultaneously, replacing plain text configuration with a visual approach avoids configuration errors and difficulties in troubleshooting under complex processes. It can automatically complete data comparison, difference location, and result solidification, replacing manual sampling and improving the reliability of migration results, data quality, and acceptance efficiency.

[0019] In one possible implementation, determining the scheduling instructions that the second computing device can recognize based on the configuration information includes: performing syntax and semantic verification on the configuration information to obtain a first verification result, the first verification result indicating whether the configuration information conforms to preset syntax and semantic rules. If the first verification result indicates that the configuration information conforms to the preset syntax and semantic rules, the scheduling instructions that the second computing device can recognize are determined based on the configuration information. The method further includes: if the first verification result indicates that the configuration information does not conform to the preset syntax and semantic rules, issuing an error message, the error message indicating that the configuration information does not conform to the preset syntax and semantic rules.

[0020] Thus, by performing both syntactic and semantic validation on the configuration information before generating scheduling instructions, illegal configurations can be intercepted before the target task is released, preventing errors from being exposed during runtime. Simultaneously, it accurately identifies configuration problems, eliminating the need for reliance on underlying log checks and multiple rounds of trial and error, effectively reducing manual debugging, shortening the error location and repair cycle, and improving the reliability and efficiency of configuration migration.

[0021] In one possible implementation, the method further includes: pre-running the target task; extracting sampled data generated during the pre-running process; and based on the sampled data, predicting the execution result of the target task performed by the second computing device, the execution result indicating whether the target task was successfully executed.

[0022] In this way, by pre-running the target task and extracting sample data to predict the execution results, the feasibility of the task can be verified before formal execution. Combined with the sample data, anomalies can be quickly discovered, effectively improving the efficiency of problem discovery and troubleshooting, and further ensuring the stable execution of the migration task.

[0023] In one possible implementation, the method further includes: acquiring execution log information generated by the second computing device executing the target task, and pre-run log information generated by the first computing device pre-running the target task. The execution log information and the pre-run log information are then aggregated to obtain aggregated log information, which is used to determine errors during the execution of the target task by the second computing device and / or the pre-running of the target task by the first computing device.

[0024] In this way, by aggregating the pre-run log information and the execution log information generated by the formal execution of the target task, errors throughout the migration process can be uniformly located, data traceability throughout the entire process can be achieved, and the observability, controllability and operational efficiency of the entire migration process can be improved.

[0025] In one possible implementation, based on configuration information, determining scheduling instructions that the second computing device can recognize includes: determining a task type based on the configuration information, where the task type includes multiple types such as offline migration task, real-time migration task, cache migration task, or file migration task; and determining a scheduling instruction based on the configuration information and the task type, whereby the scheduling instruction is used to invoke a preset component to implement the target task corresponding to the task type, wherein the preset component includes a first component and a second component, the first component being used to implement the target task of the offline migration task and / or real-time migration task type, and the second component being used to implement the target task of the cache migration task and / or file migration task type.

[0026] In this way, by automatically matching task types based on configuration information and calling corresponding preset components to generate scheduling instructions, it is possible to simultaneously schedule different task execution components to achieve parallel execution and integrated scheduling of various types of data migration tasks, such as offline, real-time, cached, or file-based tasks, within the same migration cycle, ensuring efficient, stable, and reliable multi-form data migration collaboration.

[0027] In one possible implementation, determining scheduling instructions based on configuration information and task type includes: obtaining a code template corresponding to the task type; filling the code template with configuration information to generate a target configuration file corresponding to the task type, which instructs the second computing device to execute the target task corresponding to the task type; and converting the target configuration file into scheduling instructions in a format recognizable by the second computing device.

[0028] In this way, by matching the code template corresponding to the task type, the configuration information is automatically filled into the target configuration file and converted into standardized scheduling instructions that can be recognized by the second computing device. This decouples the business logic layer from the execution layer, eliminates the need for manual writing of underlying execution code, improves the standardization, automation and reusability of scheduling instruction generation, and reduces the development and maintenance costs of complex migration tasks.

[0029] Secondly, embodiments of this application provide another data migration method applied to a second computing device. The method includes: receiving a scheduling instruction sent by a first computing device; and executing a target task based on the scheduling instruction.

[0030] In this way, the second computing device can separate the scheduling and execution of migration tasks by receiving and executing scheduling instructions, and can stably undertake multiple types of data migration tasks, ensuring the standardization and collaborative efficiency of task execution within the same migration cycle.

[0031] Thirdly, embodiments of this application provide a computing device, including: a memory and a processor. The memory is used to store program instructions. The processor is used to execute the program instructions, causing the computing device to perform the data migration method as described in the first and / or second aspects above.

[0032] Fourthly, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed on a computing device, cause the computing device to perform the data migration method as described in the first and / or second aspects above.

[0033] Fifthly, a computer program product is provided, the computer program product including computer execution instructions, which, when executed on a computing device, cause the computing device to perform the data migration method as described in the first and / or second aspects above.

[0034] The technical effects of any of the implementation methods in the third to fifth aspects can be found in the technical effects of the first and / or second aspects, and will not be repeated here.

[0035] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0036] Figure 1 A schematic diagram of a system architecture provided in an embodiment of this application; Figure 2 This is a schematic diagram of another system architecture provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application; Figure 4 A schematic diagram of a visual interface provided for an embodiment of this application; Figure 5 A schematic diagram of a configuration interface corresponding to an offline task node provided in an embodiment of this application; Figure 6 A schematic diagram of a configuration interface corresponding to a real-time task node provided in an embodiment of this application; Figure 7 This is a schematic diagram of a configuration interface corresponding to a cache task node provided in an embodiment of this application; Figure 8 A schematic diagram of a configuration interface corresponding to a file task node provided in an embodiment of this application; Figure 9 A schematic diagram of another visual interface provided for an embodiment of this application; Figure 10 A schematic diagram of a physical execution layer provided in an embodiment of this application; Figure 11 A schematic diagram illustrating the interaction between a visual editing layer and a business conversion logic layer, provided as an embodiment of this application; Figure 12 A flowchart illustrating a data migration method provided in an embodiment of this application; Figure 13 This is a flowchart illustrating another data migration method provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0038] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

[0039] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0040] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0041] The following provides an exemplary description of the application scenarios of the embodiments of this application.

[0042] This application primarily applies to data migration scenarios, where data migration can be understood as moving data from a source to a target. Both the source and target can be data storage carriers. For example, a data storage carrier may include one or more of a database, message queue, caching system, or object storage. The type of data to be migrated may include offline data, real-time streaming data, cached data, or file data. For example, offline data refers to relatively static, non-real-time changing existing data within a certain period, such as static structured offline data stored in a database, and / or historically accumulated offline existing data stored in a message queue. Real-time streaming data refers to streaming incremental data generated in real-time and continuously generated during business operations, such as real-time incremental data stored and carried by a database through a log capture mechanism (such as a Change Data Capture (CDC) tool), and / or real-time streaming data stored in a message queue, which can carry incremental data capture and real-time data distribution. Cached data refers to cached data generated during system operation, such as data stored in a caching system. File data refers to unstructured file-type data, such as file data stored in object storage.

[0043] The source and target ends read and write data through a data source. The data source is a configuration entity used to access the corresponding data storage medium during the data migration process, providing the entry point for accessing the data storage medium. Data sources include source data sources and target data sources. The source data source is used to access the source data storage medium to obtain the data to be migrated. The target data source is used to access the target data storage medium to receive and write the migrated data. For example, the data source corresponding to the database is a configuration entity that accesses the database and is used to read and write static structured offline data. The data source corresponding to the message queue is a configuration entity that accesses the message queue and is used to read and write real-time streaming data and historical offline stored data. The data source corresponding to the caching system is a configuration entity that accesses the caching system and is used to read and write cached data. The data source corresponding to object storage is a configuration entity that accesses object storage and is used to read and write unstructured file data.

[0044] Different types of data migration correspond to different types of migration tasks, such as offline migration tasks for migrating offline data, real-time migration tasks for migrating real-time streaming data, cache migration tasks for migrating cached data, or file migration tasks for migrating file data.

[0045] The following example illustrates the migration scenario of an Enterprise Resource Planning (ERP) system: During business operations, ERP generates and accumulates various types of data. Offline data specifically includes structured static data corresponding to core ERP business processes such as production, finance, sales, procurement, and inventory, or historical data of business events triggered after order creation, such as inventory deduction records, financial voucher generation notifications, and approval process status change logs. Real-time streaming data specifically includes incremental change data triggered by business processes such as order status changes, inventory fluctuations, and approval process progression. Cached data specifically includes temporary cached data during runtime, such as user sessions, high-frequency business status, and system configurations. File data specifically includes unstructured file assets such as contracts, images, electronic documents, and business attachments. ERP writes and stores the above data into corresponding data storage media through various data sources.

[0046] The system architecture of the embodiments of this application will be described below as an example.

[0047] like Figure 1 As shown in the illustration, this application provides a first computing device. The first computing device is capable of communicating with a second computing device.

[0048] A first computing device is used to display a visual interface, which includes visual configuration elements for configuration items. It receives configuration information input by the user based on these visual configuration elements. The configuration information includes the source and destination of the data to be migrated, describing the migration of the data from the source to the destination. The data to be migrated includes various types such as offline data, real-time streaming data, cached data, or file data. Based on the configuration information, a first computing device determines scheduling instructions that a second computing device can recognize. These scheduling instructions are used to schedule the second computing device to execute a target task. The target task instructs the migration of the data from the source to the target. Furthermore, the first computing device also sends the scheduling instructions to the second computing device.

[0049] The second computing device is used to execute target tasks in response to scheduling instructions.

[0050] For example, the target task may include multiple data migration tasks.

[0051] In one implementation, the second computing device and the first computing device can be the same device or different devices, as described above. Figure 1 As shown.

[0052] In some embodiments, such as Figure 2 As shown, the first computing device includes a front-end visualization editing layer. The first computing device displays a visualization interface through the front-end visualization editing layer and receives configuration information input by the user based on the visualization configuration elements.

[0053] Optionally, the first computing device is also used to establish directed connections between multiple data migration tasks in response to user operation commands, in order to configure the dependencies between different data migration tasks. Details regarding these dependencies will be provided in the subsequent visualization interface and will not be elaborated upon here. In one implementation, the first computing device configures the dependencies between different data migration tasks through a front-end visual editing layer.

[0054] In some embodiments, such as Figure 2 As shown, the first computing device also includes a business logic transformation layer. The business logic transformation layer includes a node metadata parser, a migration logic mapper, and a configuration file generator. The first computing device uses the node metadata parser to convert configuration information into a format recognizable by the first computing device, uses the migration logic mapper to map the converted configuration information to corresponding code templates, and uses the configuration file generator to generate a target configuration file based on the converted configuration information and its corresponding code template. In one implementation, the first computing device generates scheduling instructions based on the target configuration file.

[0055] Optional, continue as follows Figure 2As shown, the business logic transformation layer also includes a task dependency builder. The first computing device further transforms dependencies into a format recognizable by itself using a node metadata parser, and maps the transformed dependencies to their corresponding dependency description templates using a migration logic mapper. These dependency description templates define a standardized description format for dependencies. The task dependency builder generates a directed acyclic graph (DAG) based on the transformed dependencies and their corresponding dependency description templates. The DAG describes the dependencies between data migration tasks. In one implementation, the first computing device generates scheduling instructions based on the target configuration file and the DAG. These scheduling instructions are specifically used to schedule the second computing device to execute the target task based on the dependencies.

[0056] In some embodiments, continue as follows Figure 2 As shown, the second computing device includes a scheduling management layer. The scheduling management layer is equipped with a scheduling system. The second computing device responds to scheduling instructions through the scheduling system by invoking task execution components to perform data migration tasks. The task execution components include a first component and a second component. The first component is used to implement offline migration tasks and / or real-time migration tasks. The first component, such as a batch-stream integrated distributed data integration platform, is specifically used to achieve efficient migration, synchronization, and transformation of massive, multi-source heterogeneous data (such as offline data and / or real-time streaming data), supporting reading data from the source and writing it to the target. The second component is used to implement cache migration tasks and / or file migration tasks. The second component, such as a shell executor, is used to execute shell scripts.

[0057] It should be noted that the task execution component can be deployed on a second computing device or a third computing device. The third computing device can be the same device as the first computing device or a different device. This embodiment of the application uses the deployment of the task component on a second computing device as an example for illustration.

[0058] In some embodiments, continue as follows Figure 2 As shown, the second computing device also includes a physical execution layer, on which the first component is deployed.

[0059] In one implementation, continue as follows Figure 2As shown, the second computing device also includes backend services and specialized tools, including cache migration tools such as Redis-shake. Redis-shake, as a dedicated cache migration tool, actually performs the migration and synchronization of Redis cached data between the source and target endpoints. A shell script is created, containing commands to start the cache migration tool. Executing the shell script automatically invokes the cache migration tool to begin the migration. This achieves shell-triggered cache migration, enabling the cache migration task to be completed through the tool.

[0060] In one implementation, continue as follows Figure 2 As shown, backend services and specialized tools include file data migration tools, such as Rclone. Rclone is a command-line tool used for file / object storage data migration. A shell script is created, containing commands to start the file data migration tool. Executing the shell script automatically invokes the file data migration tool to begin the migration. This achieves shell-triggered file data migration, enabling the tool to complete cache migration tasks.

[0061] In some embodiments, the first computing device is further configured to receive verification information input by a user based on visual configuration elements, determine a verification instruction that the second computing device can recognize based on the verification information, and send the verification instruction to the second computing device.

[0062] Accordingly, the second computing device is also used to execute the verification task corresponding to the verification instruction in response to the verification instruction. The specific contents of the verification information, verification instruction, and verification task will be described later and will not be repeated here.

[0063] In some embodiments, continue as follows Figure 2 As shown, the task execution component also includes a third component, such as a Hypertext Transfer Protocol (HTTP) executor, which is used to send HTTP requests, such as calling external APIs. Correspondingly, in response to the verification command, the second computing device initiates an HTTP request through the HTTP executor, calls the data query interfaces provided by the source and target data storage carriers, and obtains the data to be migrated from the source and the migrated data from the target to execute the verification task. For example, the backend services and specialized tools include the third component.

[0064] In one implementation, continue as follows Figure 2 As shown, the scheduling system includes worker nodes, which load and start the corresponding task execution components, and finally the corresponding task execution components complete the actual data migration task.

[0065] Accordingly, the second computing device and the first computing device being the same device can be understood as the aforementioned scheduling system and visualization interface being on the same computing device.

[0066] In this embodiment, the computing device (such as the first computing device and / or the second computing device) can be a server. The server can be a single physical server or logical server, or it can be composed of two or more physical servers or logical servers that share different responsibilities, working together to achieve various server functions such as data processing and service provision.

[0067] In terms of hardware form, servers can be blade servers, high-density servers, rack servers, or tower servers, which are suitable for different application scenarios such as high-density cluster deployment in data centers and small enterprise server rooms.

[0068] like Figure 3 As shown, this application embodiment provides a computing device 500. The computing device 500 includes a processor 510 and a memory 520 for storing processor-executable instructions. When the processor 510 is configured to execute instructions, the computing device 500 performs the various functions described above.

[0069] Figure 3 The computing device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0070] The computing device 500 is manifested in the form of a general-purpose computing device. The components of the computing device 500 may include, but are not limited to: one or more processors 510, memory 520, communication bus 540 connecting different system components (including memory 520 and processor 510), and communication interface 530.

[0071] Communication bus 540 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnection (PCI) bus.

[0072] Memory 520 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computing device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Although Figure 3 Not shown, a disk drive may be provided for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and a removable non-volatile optical disk (e.g., a compact disc read-only memory, CD). ROM, Digital Video Disc Read-Only Memory (DVD) An optical disc drive that reads and writes to ROM or other optical media. In these cases, each drive can be connected to the communication bus 540 through one or more data media interfaces. The memory 520 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0073] A program / utility having a set (at least one) of program modules can be stored in memory 520. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0074] The computing device 500 can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), and with one or more devices that enable a user to interact with the computing device, and / or with any device that enables the computing device to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can be performed through the communication interface 530. Furthermore, the computing device 500 can also communicate through a network adapter (… Figure 3 (Not shown) communicates with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet). The aforementioned network adapter can communicate with other modules of the computing device via the communication bus 540. It should be understood that, although... Figure 3As not shown, the computing device 500 may be used with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (RAID) systems, tape drives, and data backup storage systems.

[0075] Processor 510 executes various functional applications and data processing by running programs stored in memory 520, such as implementing the functions provided in the embodiments of this application as follows. Figure 12 and Figure 13 The data migration method shown.

[0076] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the computing device 500. In other embodiments of this application, the computing device 500 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0077] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0078] For ease of understanding, the data migration method provided in the embodiments of this application will be described exemplarily below with reference to the first computing device and the accompanying drawings.

[0079] It is understood that the embodiments of this application are based on a visual interface, and the data migration method is completed through interface interaction. Therefore, the embodiments of this application will be divided into two parts to exemplarily describe the data migration method.

[0080] Part One: Combination Figures 4-10 This section introduces the specific implementation methods of the visual interface.

[0081] Part Two: Combination Figures 12-13 This section introduces the specific implementation methods of data migration.

[0082] The following sections will provide detailed explanations of these two parts.

[0083] Part One, such as Figure 4 As shown, this application embodiment provides a visual interface. The visual interface includes visual configuration elements for configuration items.

[0084] In one implementation, the visual configuration elements are presented in a graphical or form-based format.

[0085] In one implementation, visual configuration elements are obtained by structurally encapsulating configuration items.

[0086] In one implementation, the visual interface employs a guided configuration approach, which includes one or more of the following: wizard-style configuration prompts, configuration templates, or default value configuration. Wizard-style configuration prompts respond to user actions, guiding the user step-by-step through a preset configuration process to input and confirm relevant configuration information. Configuration templates are pre-configured, standardized information templates containing required configuration items, optional configuration items, and default configuration values ​​for the corresponding data migration task. Default value configuration means that when the user does not manually input or modify configuration item information, the preset default value is automatically used as the actual configuration parameter for that item.

[0087] In this embodiment of the application, graphical, form-based, and guided configuration methods are used to lower the configuration threshold for non-professionals, reduce the cost of manual input and parameter understanding, and improve configuration efficiency and accuracy.

[0088] like Figure 4 As shown, the visualization interface includes a node area, which contains workflow nodes. For example, a workflow node can be understood as a task orchestration component, providing the ability to visually configure the configuration information for different types of data migration tasks. Workflow nodes include semantic nodes. Semantic nodes include multiple nodes selected from offline task nodes, real-time task nodes, cached task nodes, or file task nodes. Different semantic nodes correspond to different data migration task types. Different task types include one or more of offline migration tasks, real-time migration tasks, cached migration tasks, or file migration tasks. Specifically, offline task nodes correspond to offline migration tasks. Real-time task nodes correspond to real-time migration tasks. Optionally, real-time task nodes can also correspond to offline migration tasks for historically accumulated offline data in the message queue. Cache task nodes correspond to cached migration tasks. File task nodes correspond to file migration tasks.

[0089] In this application embodiment, a unified modeling and representation of data in various forms is achieved through dedicated semantic nodes. This enables unified orchestration and compatibility of the migration of one or more types of data, including offline data, real-time streaming data, cached data, or file data, within the same migration cycle, avoiding fragmentation of the migration link.

[0090] The visualization interface also includes visualization configuration elements corresponding to each semantic node. These elements are used to configure the data migration task information for the corresponding type. The configuration information includes the source and target of the data to be migrated. Understandably, when determining the source and target of the data to be migrated through the visualization configuration elements, it is necessary to first determine the source and target data sources that connect to the source and target. Then, based on the determined source and target data sources, the data objects to be migrated in the source data storage medium and the data objects to be written in the target data storage medium are selected respectively, thus ultimately clarifying the source and target of the data to be migrated. Correspondingly, the configuration information also includes the data sources corresponding to the source and target of the data to be migrated.

[0091] For example, the data source may include basic connection information, such as the data source address (Internet Protocol (IP) / domain name), connection port number, connection protocol, etc. Optionally, the data source may also include one or more of the following: authentication information, data resource location information, or extended connection parameters. Authentication information includes access username, login password, key, or token, etc. Data resource location information includes information about the data storage medium. For example, if the data storage medium is a database, the information includes the database name, table name, and schema information; if the data storage medium is object storage, the information includes the file storage path and bucket name. If the data storage medium is a caching system, the information includes the cache library number. Extended connection parameters include one or more of the following: connection timeout, character set encoding, or number of parallel connections. The connection timeout refers to the maximum waiting time for the first / second computing device to attempt to connect to the data source. For example, if the connection to the database / Redis / file storage cannot be established within this time, the connection will automatically stop and the system will be considered to have failed, preventing the program from waiting indefinitely and freezing. Character set encoding refers to the encoding format of text / symbols used for data storage and transmission. For example, the source and destination must use the same encoding; otherwise, problems such as garbled Chinese characters and loss of special symbols may occur after migration, ensuring the integrity and accuracy of the data content. The number of parallel connections refers to the number of connection channels established simultaneously with the data source. For example, setting it to 10 means opening 10 channels simultaneously to migrate data.

[0092] Optionally, the configuration information may also include data migration method and / or data processing method.

[0093] The data migration method indicates the migration method used during the data migration process. For example, data migration methods include one or more of the following: data acquisition strategy, concurrent sharding strategy, batch size strategy, parallel write strategy, breakpoint resumption strategy, retry strategy, rate limiting strategy, or idempotent write strategy. Specifically, the data acquisition strategy indicates the rules for acquiring the full or incremental data to be migrated, or acquiring it at different granularities, such as single-table acquisition or database acquisition. The concurrent sharding strategy indicates the sharding rules and the number of parallel processing operations for the data to be migrated. The batch size strategy indicates the single batch processing size for reading and writing the data to be migrated. The parallel write strategy indicates the number of concurrent write threads to the target end. The breakpoint resumption strategy indicates the breakpoint recording method and resumption rules for interrupted migration. The retry strategy indicates the retry rules after a migration failure. The rate limiting strategy indicates the upper limit of the read / write rate for the data to be migrated. The idempotent write strategy configures the rules for handling duplicate data and ensuring uniqueness of writes on the target end.

[0094] Data processing methods indicate how the data to be migrated will be processed during the migration process. For example, data processing methods may include one or more of the following: Structured Query Language (SQL) conversion, field mapping, field renaming, or field splitting. SQL conversion means converting source query statements and data manipulation statements into SQL statements adapted for execution on the target side, according to the target side's syntax specifications, to ensure that data read / write logic can be executed correctly on the target side. Field mapping means establishing a correspondence between source and target data fields, writing source field data into corresponding fields on the target side according to a preset correspondence. This preset correspondence refers to the one-to-one correspondence rules between source and target data fields pre-configured by the user in the visual interface, determined based on the user's actual settings. Field renaming means modifying source field names during the migration process to adapt them to the target side's field naming conventions or business requirements. Field splitting means splitting composite data within a single source field into multiple independent fields based on delimiters, lengths, or rules, and writing each independent field into its corresponding field on the target side.

[0095] In this embodiment, by configuring one or more of the above migration methods, the migration process is ensured to be efficient, stable, and reliable, supporting breakpoint resumption, abnormal retry, idempotent security, and rate control. By configuring the data processing method, automatic adaptation of fields, syntax, and formats between heterogeneous systems is achieved, and data conversion can be completed without writing complex scripts.

[0096] Alternatively, the configuration information may also include task metadata and / or extended information.

[0097] Task metadata is used to identify, manage, and track different data migration tasks. It may include the task name of each data migration task. Optionally, task metadata may also include one or more of the following: task description, task identity ID, task creator, creation time, task status, task priority, or the business group to which it belongs.

[0098] Extended information includes task scheduling parameters and / or log and alarm parameters. Task scheduling parameters are used to specify the time for scheduled execution, such as single / periodic execution, specifically single execution. Log and alarm parameters include one or more of the following: log level, alarm recipients, or alarm thresholds, specifically default log level and default alarm rules.

[0099] In this embodiment, the migration task is managed throughout its entire lifecycle, making it traceable, schedulable, and capable of issuing alerts, thereby improving the convenience of operation and maintenance.

[0100] In some embodiments, the visual configuration elements corresponding to each semantic node are displayed through the configuration interface corresponding to each semantic node. For example, after a user selects and triggers the corresponding semantic node in the visual interface, the configuration interface specific to that semantic node is then displayed in the visual interface. The following, in conjunction with... Figure 5 , Figure 6 , Figure 7 and Figure 8 The document provides an exemplary description of the visual configuration elements for each semantic node, as well as the configuration items (i.e., configuration information) of these visual configuration elements: like Figure 5The diagram illustrates a configuration interface for an offline task node. This interface includes a data source area, which configures information about the source of the data to be migrated. This includes the following visual configuration elements: data source selection, database selection, schema selection, and table selection, with corresponding configuration information of "POSTGRESQL", "TEST_PG164", "postgres", "public schema", and "test". These source-side information indicate the configuration parameters for the POSTGRESQL data source named TEST_PG164, used to read the static structured offline data to be migrated from the "test" table in the "public schema" of the "postgres" database. The interface also includes a data target area, which configures information about the target of the data to be migrated. This includes the following visual configuration elements: data source selection, database selection, schema selection, and target selection, with corresponding configuration information of "POSTGRESQL", "15 target PG", "postgres", "public", and "test_copy". The relevant information about the target end of this data to be migrated indicates the configuration parameters of the target end data source PostgreSQL with the access name 15 target PG, which is used to write the migrated offline data to the test_copy data table in the public schema of the postgres database.

[0101] Understandable. Figure 5 The diagram illustrates a configuration example for offline data in a database. In another scenario, the configuration interface can also be used to configure the migration of historical data in a message queue. In this scenario, the data objects to be migrated / written in the source and / or target data storage carriers are message queue topics.

[0102] Continue as Figure 5As shown, the configuration interface can also include the following visual configuration elements: task name input field, task description input field, selection options for single-table synchronization or database synchronization to indicate the data acquisition strategy (e.g., located in the data source area), "whether to concurrent read switch, shard key selection option" to indicate the concurrent sharding strategy (e.g., located in the source object synchronization setting area of ​​the configuration interface), "batch pull size input field (e.g., located in the source object synchronization setting area of ​​the configuration interface), batch processing size input field (e.g., located in the target object synchronization setting area of ​​the configuration interface)" to indicate the batch size strategy, and write parallelism input field to indicate the write parallelism strategy (e.g., located in the target object synchronization setting area of ​​the configuration interface). The corresponding configuration information is as follows: offline migration task, empty, single table acquisition, "enable concurrent read, shard key is id field", "10000, 5000", 10. For example, the configuration interface can also display the mapping of the data to be migrated from the source to the target. Specifically, the source table `test` contains the field `id`, which is of data type `int8`, has no configurable length, and is a primary key. It also has no configurable primary key, not null, default value, or comment. The corresponding operation element indicates whether to delete or modify this field. The target table `test_copy` contains the same field `id`, which is of data type `numeric`, has a length of 38, is a primary key, not null, has no configurable default value, and no configurable comment. It includes field mapping options (setting the conversion rules from the `id` field in `test` to the `id` field in `test_copy`) and an audit option (if selected, the `id` fields in both `test` and `test_copy` are validated; otherwise, no validation is required). This configuration interface also includes visual elements such as "OK," "Design," "Test," and "Cancel."

[0103] like Figure 6The diagram illustrates a configuration interface for a real-time task node. The data source area of ​​this interface includes the following visual configuration elements: data source selection, database selection, schema selection, and table selection to be synchronized. The corresponding configuration information includes: "ORACLE", "20253771**", ORCL, LOGMINER_USER, and TEST_NOT_KEY. TEST_NOT_KEY is selected from a list of multiple data tables, which may also include other business data tables such as TEST2, CUSTOMERS003, CUSTOMERS004, CUSTOMERS001, and CUSTOMERS002. This information from the source of the data to be migrated indicates the configuration parameters for the ORACLE data source with the name 20253771, reading real-time incremental data from the TEST_NOT_KEY table under the LOGMINER_USER schema. The data target area of ​​this configuration interface includes the following visual configuration elements: data source selection, database selection, schema selection, and target mapping. The corresponding configuration information includes "GAUSSDB, target gs", postgres, czz, and test_copy / test_not_key. This information about the target end of the data to be migrated indicates the configuration parameters for the target data source GAUSSDB named target gs, which will write the real-time incremental data synchronized from the source end into the corresponding target data table under czz mode.

[0104] Understandable. Figure 6 The diagram illustrates a configuration example for real-time streaming data from a database. In another scenario, the configuration interface can also be configured for real-time streaming data migration from a message queue, where the data objects to be migrated / written in the source and / or target data storage carriers are message queue topics.

[0105] Continue as Figure 6 As shown, the configuration interface can also include the following visual configuration elements: task name input field, task description input field, real-time task mode selection field for indicating data acquisition strategy (selection fields include "existing + incremental" and "incremental only"), batch size input field for indicating batch size strategy, and write parallelism input field for indicating write parallelism strategy. The corresponding configuration information is as follows: gauss single table incremental task 001, "Target end gaussdb; incremental only, single table synchronization - target end has table, no duplicate data (indicates that the target end of this real-time migration task is the GAUSSDB database, executed with single table synchronization scope, the core is to synchronize incremental changes of data on the source end, and the target end has pre-created the corresponding table and there is no duplicate data, which can avoid data conflicts)", **, 1.

[0106] like Figure 7 The diagram illustrates a configuration interface for a cache task node. This interface includes the following visual configuration elements: source system selection, source database selection, target system selection, and target database selection. The corresponding configuration information is: test fusion mapping source system, source Redis1, test fusion destination system, and target Redis. This configuration information represents the configuration parameters for accessing the source and target cache systems. It reads the cache data to be migrated from source Redis1 and writes the cache data to the target Redis, achieving the migration and synchronization of cache data between different cache nodes.

[0107] Continue as Figure 7 As shown, the configuration interface also includes the following visual configuration elements: task name input field, "whether to synchronize the entire database, whether to synchronize all keys, database selection drop-down menu, prefix matching input field, and key tree display area and value data display area" used to indicate the data acquisition strategy, and the corresponding configuration information is as follows: cache migration task, no (disable full database synchronization), no (disable full key synchronization), database 0 (quantity: 0), test, test:now / test:json (key tree), and cached value data in JSON format (value area). The "Select Library" dropdown and "Prefix Matching" input fields are part of the scanning and filtering rules for source-side cached data. The "Select Library" dropdown is used to limit the database index (e.g., library 0) of the data to be migrated in the source-side cache system, and only read cached data from that library. The "Prefix Matching" input field is used to further filter the cached keys to be migrated by the key prefix (e.g., "test"), and only synchronize cached data that starts with the specified prefix. The Key Tree display area and Value data display area are previews of the results of the above scanning and filtering rules. The Key Tree displays a list of source-side cached keys that meet the rules (e.g., "test:now", "test:json"), and the Value area displays the specific cached data content corresponding to the selected key, which helps users confirm the scope and content of the cached data to be migrated.

[0108] like Figure 8 The diagram illustrates a configuration interface for a file task node. This interface includes the following visual configuration elements: source database selection, source bucket selection, target database selection, and target bucket selection, with corresponding configuration information of 15Minio, test, 1111, and **. These configuration parameters represent the access parameters for the source and target object storage systems. The system reads unstructured file data to be migrated from the 'test' bucket of the source 15Minio object storage system and writes the migrated file data to the corresponding bucket in the target object storage system.

[0109] Continue as Figure 8 As shown, the configuration interface also includes the following visual configuration elements: task name input field, "whether to switch for full bucket synchronization", "current path input field", "keyword input field", and "file list selection field" used to indicate the data acquisition strategy. The corresponding configuration information is as follows: file synchronization task, no (full bucket synchronization is off), test, empty, and files 000c8b95477d2715449b3ff6a00a34f9a62b1e90a443a420612a88k and 10913.log are selected. The input fields are as follows: Current Path Input: This is a scanning and filtering rule for source file data, used to limit the source path prefix (e.g., test) of the files to be migrated, and only read file data from this path; Keyword Input: This is a scanning and filtering rule for source file data, used to further filter the files to be migrated by filename keywords; File List Selection: This is a preview and selection configuration of the results of the above scanning and filtering rules, displaying a list of source files that match the source bucket, current path, and keyword rules. Users can select specific files to be migrated from the list (e.g., 000c8b95477d2715449b3ff6a00a34f9a62b1e90a443a420612a88k, 10913.log) to precisely limit the range of files to be migrated.

[0110] For example, when the user triggers such as Figure 5 , Figure 6 , Figure 7 and Figure 8 When the "OK" visual element is selected, the corresponding data migration tasks for each of the aforementioned semantic nodes are generated. In this way, each data migration task takes effect.

[0111] For example, when the user triggers such as Figure 5 , Figure 6 , Figure 7 and Figure 8 When the "Test" visualization element is accessed, it triggers semantic and syntax validation of the configuration information in the corresponding configuration interface. Details of semantic and syntax validation are provided in Part Two and will not be elaborated upon here.

[0112] For example, for offline migration tasks corresponding to offline task nodes and / or real-time migration tasks corresponding to real-time task nodes, the second computing device, upon receiving a scheduling instruction, executes the offline migration task and / or real-time migration task by calling the first component through system calls. For cache migration tasks corresponding to cache task nodes and / or file migration tasks corresponding to file task nodes, the second computing device, upon receiving a scheduling instruction, executes the cache migration task and / or file migration task by calling the second component through system calls.

[0113] In one implementation, after configuring one or more data migration tasks for the visual configuration elements of each semantic node, task elements corresponding to each data migration task can be generated based on the configuration information. These task elements are used to carry the configured data migration task logic for scheduling and execution by the scheduling system.

[0114] For example, when the user triggers such as Figure 5 , Figure 6 , Figure 7 and Figure 8 When viewing the "Design" visualization elements, it can generate task elements corresponding to each data migration task.

[0115] In some embodiments, continue as follows Figure 4 As shown, the visual interface also includes a process design area. The process design area is used to visually construct the processes between data migration tasks.

[0116] In one implementation, the process design area includes task elements corresponding to each data migration task. For example... Figure 9 As shown, the process design area includes task element 1 corresponding to the offline migration task configured in the offline task node above, and task element 2 corresponding to the cache migration task configured in the cache task node above.

[0117] In one implementation, when a user triggers an operation command, the process design area includes directed connections between task elements corresponding to different data migration tasks to indicate the dependencies between different data migration tasks. The dependencies include one or more of the following: task execution order, process branching and merging relationship, conditional routing execution relationship, multi-level transformation link relationship, or sub-process call relationship.

[0118] The task execution order refers to the linear execution constraint between multiple data migration tasks. That is, a subsequent data migration task can only start after the preceding data migration task has been completed. For example, an offline data migration task must be executed before a real-time incremental data migration task to ensure that full static data synchronization is completed first, and then incremental change data capture is initiated. Continuing... Figure 9 As shown, the process design area includes: a directed connection from task element 1 to task element 2, indicating that there is a dependency relationship between the offline data migration task corresponding to task element 1 and the cache data migration task corresponding to task element 2, that is, the cache data migration task corresponding to task element 2 can only be started after the offline data migration task corresponding to task element 1 has been completed.

[0119] The process branching and merging relationship refers to the fact that the master data migration process can be broken down into multiple parallel sub-process branches. After the data migration tasks of all branches are completed, they are merged into a single process to continue execution. For example, the main process is split into an offline data migration task branch and a real-time streaming data migration task branch. The two branches are executed in parallel, and after the migration tasks of both branches are completed, they are merged for execution.

[0120] Conditional routing execution refers to dynamically selecting the execution path of a data migration task based on preset conditions (such as data volume and task execution results). Different conditions correspond to different data migration task branches. If the source data volume is greater than the threshold, a full data migration task is executed; if the data volume is less than the threshold, an incremental data migration task is executed.

[0121] Multi-level transformation chain relationships refer to the multiple stages and levels of data processing and transformation that a data migration task needs to go through, forming a multi-level processing chain that depends on the output of the preceding steps. For example, the internal chain of a real-time incremental database migration task is: the CDC tool captures incremental events, field mapping transformation, standard syntax adaptation, and writing to the target database. Each transformation step depends on the output of the previous step.

[0122] Sub-process call relationship refers to the main data migration process calling one or more fully encapsulated sub-processes, each containing independent data migration task logic. For example, the main process "ERP system full migration" calls the sub-processes "financial data offline migration sub-process" and "order data real-time migration sub-process," with each sub-process encapsulating the corresponding data migration task configuration.

[0123] In one implementation, the task elements corresponding to different data migration tasks and the directed connections between them are used to form a DAG.

[0124] In this embodiment of the application, the logic of task execution order, process branching and merging, conditional routing, multi-level transformation links or sub-process calls is intuitively expressed through visual process orchestration, avoiding the problems of text configuration being prone to errors and difficult to maintain.

[0125] In some embodiments, continue as follows Figure 4 As shown, the node area also includes process control nodes. For example, a process control node can be understood as a process scheduling component, capable of defining dependencies between data migration tasks. A process control node includes one or more of the following: task execution nodes, branch nodes, condition nodes, transformation link nodes, or sub-process nodes.

[0126] Among them, task execution nodes are used to configure the dependencies of task execution order. Branch nodes are used to configure process branching and merging relationships. Condition nodes are used to configure conditional routing execution relationships. Transformation link nodes are used to configure multi-level transformation link relationships. Sub-process nodes are used to configure sub-process calling relationships. For example, user operation instructions may include: the user selecting one or more of the task execution nodes, branch nodes, condition nodes, transformation link nodes, or sub-process nodes from the process control nodes, and adding the selected process control nodes to the process design area; connecting the selected process control nodes with the task elements corresponding to upstream and downstream data migration tasks with directed lines, so that the selected process control nodes and task elements form a complete process link; defining dependency relationship types through process control nodes, determining the execution flow through directed lines, and configuring the dependencies between different data migration tasks by combining nodes and lines.

[0127] In some embodiments, continue as follows Figure 4 As shown, the workflow node also includes an integrated task container.

[0128] In one implementation, continue as follows Figure 4 As shown, the node area also includes a first type of data source and a second type of data source. The first type of data source represents the source data source that the integration task container can configure, and the second type of data source represents the target data source that the integration task container can configure. For example, both the first and second types of data sources can include one or more of the following data storage systems: database type, message queue type, object storage type, or structured text type, such as POSTGRESQL, ORACLE, HANA, GAUSDB, KAFKA, and S3FILE. Correspondingly, the data types corresponding to the first and second types of data sources can be one or more of the following: structured offline data, real-time streaming data, or cached data.

[0129] In some embodiments, continue as follows Figure 9 As shown, the process design area includes visual configuration elements for integrated task container configuration items. These visual configuration elements are used to configure the data processing method for the corresponding data migration task (such as one or more of offline migration tasks, real-time migration tasks, or cached migration tasks). Other relevant configuration information for offline migration tasks, real-time migration tasks, and cached migration tasks (excluding data processing methods) can be found in the descriptions of the corresponding visual configuration elements for offline task nodes, real-time task nodes, and cached task nodes above. For example, the visual configuration elements for the integrated task container are integrated within the internal orchestration box of the integrated task container.

[0130] For example, users can identify the source and target data sources of the data to be migrated from the first and second types of data sources, respectively, and drag them into the internal orchestration box of the integrated task container. Clicking on the source and / or target data sources in the internal orchestration box can trigger visual configuration elements for other relevant configuration information (other than data processing methods) for the data migration task.

[0131] In some embodiments, continue as follows Figure 4 As shown, the node area also includes Transform operators, which are used to configure data processing methods. Transform operators include components corresponding to one or more of the following: SQL transformation, field mapping, field renaming, or field splitting. For example, the user selects the component corresponding to SQL transformation, field mapping, field renaming, or field splitting from the transformation operators and drags it into the internal orchestration box of the integrated task container. Continuing as... Figure 9 As shown, the internal orchestration box of the integrated task container includes the source data source, the target data source, and one or more visual configuration elements corresponding to SQL transformation, field mapping, field renaming, or field splitting.

[0132] In one implementation, such as Figure 10 As shown, the first computing device executes the target task based on the data processing method through the aforementioned physical execution layer. Specifically, in response to the scheduling command, the first computing device loads the configuration and instantiates the selected first type of data source (such as Oracle CDC, JDBC Source, Kafka Source). After injecting the first type of data source into the integrated task container, it calls the corresponding transformation operators in the transformation operator library according to the data processing method configured by the user (such as SQL transformation, field mapping, field renaming, field splitting, etc.) to complete data cleaning and business rule implementation. Finally, it writes the processed data to the target end through operators such as the second type of data source (such as JDBC Sink, Kafka Sink), realizing the fully automated execution of the multi-source heterogeneous data migration process.

[0133] In this embodiment of the application, complex transformation rules are encapsulated into a single node by integrating a task container, which reduces canvas complexity and facilitates reuse and maintenance.

[0134] In some embodiments, the visual configuration elements of the integration task container may also include directed connections, which are used to indicate the data processing methods performed during migration from the source to the target in the data migration task configured by the integration task container. For example, within the integration task container of an offline migration task, the source data source POSTGRESQL, SQL transformation, field renaming, and the target data source GAUSSDB are sequentially connected by directed connections, intuitively presenting the complete data processing chain of "reading source data, performing SQL transformation, renaming and mapping fields, and writing to the target database," clearly demonstrating the multi-level data processing flow from the source to the target.

[0135] In one implementation, the data migration task configured in the integration task container can migrate data from one source to one target, from one source to multiple target locations, from multiple sources to one target location, or from multiple sources to multiple target locations. For example, by dragging and dropping one or more data sources from the first type of data source and one or more data sources from the second type of data source into the internal orchestration box of the integration task container, and establishing data flow relationships, data processing links, and execution sequences between one or more source locations and one or more target locations through directed connections, users can achieve flexible one-to-one, one-to-many, many-to-one, and many-to-many data migration topology configurations.

[0136] The process design area can include internal orchestration frames for one or more integrated task containers. Each integrated task container independently encapsulates and hosts a data migration task within its internal orchestration frame. Continuing... Figure 9 As shown, the process design area includes internal orchestration boxes for multiple integrated task containers.

[0137] For example, after completing all the configurations for the corresponding data migration task in the visual configuration element of the integrated task container, the user can obtain the task element corresponding to the data migration task by triggering a confirmation operation.

[0138] In some embodiments, the task elements corresponding to the data migration tasks configured in the integrated task container can be connected in a directed manner with the task elements corresponding to the semantic nodes to configure the execution dependency between the data migration tasks corresponding to the two, thereby achieving unified scheduling and sequential execution of the overall workflow.

[0139] Optionally, the workflow nodes also include a data audit node. The data audit node is used to configure verification information. Verification information is used to indicate whether data has been migrated from the source to the target. Verification includes one or more of the following: quantity consistency verification, content integrity verification, data security verification, format validity verification, or data accuracy verification. Quantity consistency verification checks whether the total number of records and files in the data to be migrated from the source end is equal to the total number of records in the migrated data from the target end. For example, if the data to be migrated from the source end has 1000 records, the migrated data from the target end must also have 1000 records. Content integrity verification checks whether fields in the migrated data are missing, truncated, or have abnormal null values. For example, if each data record in the source end contains 10 fields, the target end must also contain all 10 fields. Data security verification can be implemented using hash fingerprint verification, which calculates and compares the hash values ​​of the data to be migrated and the migrated data to ensure that the migration process has not been tampered with, lost, or damaged. For example, the MD5 value of the source file must match the MD5 value of the target file. Format validity checks verify whether the type, length, encoding, and value range of migrated data conform to the target database's specifications. For example, date fields must be in yyyy-MM-dd format, and numeric fields must not contain letters. Data security checks verify whether the target database's primary key, unique key, and business identifier are duplicated or conflicting. Example: User ID and order number must be unique and not duplicated in the target database.

[0140] For example, the first computing device determines a verification command based on the verification information configured in the data audit node and sends it to the second computing device. The scheduling system of the second computing device responds to the verification command by invoking a third component to execute the verification task.

[0141] In this embodiment, the built-in data auditing capability enables automatic verification of migration quantity consistency, content integrity, data security, format legality, or data accuracy, forming a closed loop between data migration and data auditing, thereby improving the reliability of data migration results.

[0142] Optionally, the workflow nodes also include the library-to-API node. The library-to-API node is used to extract data from the source (such as a database) and push it to an external API service via API calls. It is mainly used in data distribution and push scenarios from the database to the application interface to complete the API request delivery of data.

[0143] In this application embodiment, the data distribution capability from database to API will be extended to enrich migration scenarios and meet the interface-based data push requirements of business systems.

[0144] like Figure 11As shown, the first computing device displays multiple offline task nodes, real-time task nodes, cached task nodes, file task nodes, integrated task containers, or data audit nodes through a visual interface in the front-end visual editing layer. Based on the user's visual configuration elements for each node in the visual interface, the user submits node configurations (such as the configuration information mentioned above) and transmits each node configuration to the business logic transformation layer for further processing.

[0145] In some embodiments, the various types of data migration tasks described above are carried out through a process model. That is, each semantic node and / or integrated task container encapsulates the configuration information and execution logic of the corresponding migration task in the form of task elements, and incorporates them into the process model. The execution order and dependencies between tasks are defined by directed connections in a visual interface, forming a schedulable end-to-end data migration workflow.

[0146] In this embodiment of the application, the orchestration and scheduling of multiple types of tasks are realized through the process model, which can simplify the configuration and management of the overall migration process, improve the orderliness and controllability of task execution, ensure the stable and efficient operation of the migration process, and improve the convenience of migration implementation and operation and maintenance management.

[0147] In some embodiments, the first computing device stores configuration items corresponding to semantic nodes and integrated task containers. For example, these items are uniformly stored in a public configuration center or public resource repository, allowing different migration tasks and users to reference, reuse, and quickly load them in a visual interface, avoiding redundant configuration and improving configuration efficiency and consistency. For example, users can uniformly manage these stored configuration items, such as adding, deleting, or modifying one or more of them.

[0148] In some embodiments, the first computing device provides preset interfaces for different types of data migration tasks. The preset interfaces are used for corresponding types of data migration tasks to reference stored configuration items, so that various data migration tasks can directly call and reuse the saved standardized configuration information during the visual configuration process, without repeated configuration, thereby improving configuration efficiency and configuration consistency.

[0149] The first part above introduces the visual interface, which provides graphical, form-based, and guided task configuration capabilities. Based on semantic nodes, process control nodes, integrated task containers, data audit nodes, or one or more nodes from the library to the API node in the node area, it enables the visual configuration and orchestration of various data migration tasks, including offline data, real-time streaming data, cached data, and file data. Each semantic node and integrated task container completes the setting of information such as source and target ends, migration strategies, and data processing through visual configuration elements. Execution dependencies are built in the process design area in the form of task elements through directed connections. Finally, a unified process model carries all tasks, realizing the integrated orchestration, scheduling, and operation and maintenance of multiple types of migration tasks. Based on the above visual interface, configuration information can be transformed into actual executable migration logic, thereby completing the data migration.

[0150] The specific implementation method of the data migration method is described below: Part Two, such as Figure 12 and Figure 13 As shown in the embodiments of this application, a specific implementation of the data migration method is provided.

[0151] Among them, such as Figure 12 As shown, this application embodiment provides a data migration method, which includes, for example, the following steps: S1201-S1204, which can be executed by the processor of the first computing device, such as the CPU.

[0152] S1201 displays a visual interface, which includes visual configuration elements for configuration items.

[0153] For details on the visual interface, please refer to Part 1, which will not be repeated here.

[0154] S1202 receives configuration information input by the user based on visual configuration elements.

[0155] The configuration information describes how to migrate the data from the source to the target.

[0156] S1203, based on the configuration information, determine the scheduling instructions that the second computing device can recognize.

[0157] Scheduling instructions are used to schedule the second computing device to execute a target task. The target task instructs the migration of data from the source to the target; for example, the target task may include multiple types such as offline migration, real-time migration, cached migration, or file migration. It is understood that scheduling instructions are task management instructions generated by the first computing device based on configuration information from the visual interface and recognizable by the scheduling system deployed on the second computing device. The scheduling system will be described in detail below, in conjunction with the aforementioned second computing device: The scheduling instruction specifically instructs the scheduling system on how to schedule, execute, and manage the corresponding target task. For example, the scheduling instruction includes a task execution method, a task start command, and a task execution node. The task execution method indicates the execution type of the target task, such as the command execution type corresponding to the first component and / or the command execution type corresponding to the second component. The task execution node instructs the scheduling system to distribute the target task to a designated worker node, which will then execute the task start command and run the corresponding task execution component to complete the target task. It should be noted that the scheduling system can simultaneously schedule the first component and the second component to perform different types of data migration tasks within the same migration cycle. Optionally, the scheduling instruction may also include one or more of the following: task identifier, task trigger time, and failure retry rules derived from the aforementioned retry strategy.

[0158] For example, when the configuration information includes data migration method and / or data processing method, the scheduling instruction is specifically used to schedule the second computing device to execute the target task based on the data migration method and / or data processing method.

[0159] The second computing device can recognize, which can be understood as standardized scheduling control instructions that the scheduling system can recognize.

[0160] In an implementation method that determines scheduling instructions recognizable by a second computing device based on configuration information, the task type is determined based on the configuration information. The scheduling instruction is then determined based on the configuration information and the task type. Specifically, the scheduling instruction invokes a preset component to implement the target task corresponding to the task type. Here, the preset component is the aforementioned first component and / or second component.

[0161] In this embodiment of the application, by binding task types with preset components, execution components for different migration scenarios such as offline synchronization, cache synchronization, file synchronization, and data auditing can be automatically matched, thereby improving scheduling adaptability and execution stability.

[0162] In one possible implementation, determining the task type based on configuration information includes: determining the task type according to the type of data to be migrated indicated by the source and target ends in the configuration information. For example, if the source and target ends are databases and the data to be migrated is structured offline data, then the task type is determined to be an offline synchronization task; or, if the source and target ends are caching systems such as Redis and the data to be migrated is business cache data, then the task type is determined to be a cache synchronization task.

[0163] In this embodiment, the task type is automatically determined by the data types of the source and target ends, eliminating the need for manual configuration by the user. This can accurately adapt to various data migration scenarios, reduce the configuration threshold, and improve the accuracy of task type matching.

[0164] In one possible implementation, determining the scheduling instruction based on configuration information and task type includes: converting the configuration information into a format recognizable by a first computing device, and determining the scheduling instruction based on the converted configuration information and task type.

[0165] In this embodiment, unstructured semantic configuration information is converted into a standardized format recognizable by the first computing device. That is, the unstructured semantic configuration information is converted into a standardized format configuration information recognizable by the first computing device, which facilitates the device to parse and process the configuration information. Combined with the task type, scheduling instructions can be automatically and accurately generated, improving the standardization and reliability of the scheduling instruction generation process.

[0166] In one possible implementation, determining the scheduling instruction based on configuration information and task type includes: obtaining the code template corresponding to the task type; filling the configuration information into the code template corresponding to the task type to generate a target configuration file corresponding to the task type; the target configuration file instructing the second computing device to execute the target task corresponding to the task type; and converting the target configuration file into scheduling instructions in a format recognizable by the second computing device.

[0167] The code template is a predefined configuration framework for different task types. It solidifies the fixed syntax, field structure, and parameter format required by the first / second component. The template contains configuration placeholders to be filled, without specific business data. It can be reused for different task types such as offline synchronization and cache synchronization, and serves as the basic carrier for generating the target configuration file.

[0168] The target configuration file is a complete, proprietary file generated by populating the specific business information that the user has visually configured into the code template. It contains the core parameter file for all execution rules of the target task, including source and target information, data to be migrated, data migration methods, and other specific content. It can be directly recognized and parsed by the task execution component and is the direct basis for the execution component to complete the data migration.

[0169] For example, during the generation of scheduling instructions based on the target configuration file, the target configuration file is standardized and encapsulated according to a format recognizable by the scheduling system on the second computing device. Specifically, key information such as the storage path of the target configuration file, associated task execution components, task execution methods, and task execution nodes are integrated to generate scheduling instructions that the scheduling system can parse. Correspondingly, the scheduling instructions, through a built-in startup command, associate with and invoke the target configuration file, ultimately instructing the scheduling system to drive the execution components to complete the migration task based on the target configuration file.

[0170] For example, the configuration information in the above implementation can specifically be the converted configuration information.

[0171] In this embodiment, the target configuration file is generated by code template and then converted into scheduling instructions, which realizes the standardization and automation of configuration information. This not only ensures the syntactic standardization of the executable file, but also simplifies the generation process of scheduling instructions, thereby improving the deployment efficiency and execution reliability of data migration tasks.

[0172] In another implementation method for determining scheduling instructions recognizable by a second computing device based on configuration information, the configuration information undergoes syntactic and semantic verification to obtain a first verification result. The first verification result indicates whether the configuration information conforms to preset syntactic and semantic rules. When the first verification result indicates that the configuration information conforms to the preset syntactic and semantic rules, the scheduling instructions recognizable by the second computing device are determined based on the configuration information.

[0173] Syntax rules are used to validate one or more of the following: the format validity of configuration information, the correctness of data types, the validity of parameter value ranges, or the completeness of required parameters. For example, the connection port number of a data source must be an integer in the range of 1-65535, the data source address must conform to the IP / domain name format, required parameters such as username and password cannot be empty, and the number of parallel connections must be a positive integer.

[0174] Semantic rules are used to verify one or more of the following: the rationality of the business logic of the configuration information, the connectivity of the data source, the matching of parameters between the source and target ends, or the feasibility of task execution. For example, the rules may include: successfully establishing a connection with the source / target data source; the data types of the source and target ends being compatible; the configured database tables actually existing; and the access permissions of the source and target ends being legal and valid.

[0175] The fields here refer to the configuration items configured by the user in the visual interface. The nodes here refer to the aforementioned semantic nodes and / or integration task containers.

[0176] For example, the error message is: the port number field format is invalid, the offline synchronization node is missing target configuration parameters, the source and target data sources cannot establish a connection, and there are mismatches in the field mapping configuration.

[0177] For example, the configuration information in the above implementation can specifically be the converted configuration information.

[0178] In this embodiment of the application, by performing syntactic and semantic dual verification on the configuration information before generating scheduling instructions, erroneous configurations can be screened and intercepted in advance, avoiding scheduling instruction execution failure and task operation abnormalities due to invalid configurations, effectively improving the standardization of scheduling instruction generation and the execution stability of data migration tasks.

[0179] In some embodiments, the method further includes: issuing an error message when the first verification result indicates that the configuration information does not conform to preset syntax and semantic rules. The error message is used to indicate fields and / or nodes in the configuration information that do not conform to preset syntax and semantic rules.

[0180] Optionally, after issuing an error message, in response to the user's reconfiguration operation, the corrected configuration information is obtained and its syntax and semantics are validated again.

[0181] In this embodiment, by performing syntax and semantic checks on the configuration information and indicating the location of errors, illegal configurations can be avoided in advance, scheduling command execution anomalies can be prevented, and the stability of data migration tasks can be improved. Furthermore, by supporting user reconfiguration and secondary verification, configuration errors can be corrected promptly, ensuring that the configuration information used to generate scheduling commands is legal and valid, and reducing the probability of migration task execution failure.

[0182] S1204, the scheduling instruction is sent to the second computing device, which is used to execute the target task in response to the scheduling instruction.

[0183] Combining S1201-S1204 above, a unified configuration entry point is provided to users through a visual interface. This allows users to configure migration settings for various heterogeneous data types, including offline data, real-time streaming data, cached data, and file data. Migration configuration can be completed without requiring specialized data technology knowledge, lowering the barrier to entry and fundamentally simplifying the configuration operations for migrating multiple types of heterogeneous data. This solves the problems of scattered configuration and cumbersome operations associated with multi-type data migration. Furthermore, based on the unified configuration information, standardized scheduling instructions recognizable by a second computing device are automatically generated. This enables unified configuration, classification, adaptation, and centralized scheduling of multi-form data migration tasks within the same migration lifecycle. It avoids fragmentation of migration links for different types of data and decouples the configuration layer from the execution layer. It allows for centralized orchestration and collaborative processing of multiple different types of data migration tasks within the same migration cycle, improving the ease of data migration configuration and scheduling, the management efficiency of multi-form data migration, system compatibility, and optimizing the overall migration process execution efficiency.

[0184] In some embodiments, the method further includes: a first computing device receiving an operation command triggered by a user on a visual interface. In response to the operation command, directed connections are established between task elements corresponding to different data migration tasks to configure dependencies between the different data migration tasks.

[0185] In this embodiment of the application, by configuring the dependency relationship between each semantic node and / or integrated task container through directed connections in the visual interface, complex migration links such as multi-level transformation, branch merging, conditional routing, and sub-process calls can be intuitively expressed. It supports the construction of reusable process fragments and task templates in a component-based manner, effectively avoiding the lengthy nesting and repeated pasting of plain text configuration, and significantly improving the maintainability and configuration efficiency of complex migration processes.

[0186] In one implementation, determining the scheduling instructions that the second computing device can recognize based on the configuration information includes: determining the scheduling instructions based on the configuration information and the dependencies between different data migration tasks, wherein the scheduling instructions are specifically used to schedule the second computing device to execute the target task based on the dependencies.

[0187] Understandably, the scheduling instructions also include the upstream and downstream dependencies of the tasks derived from the dependencies between the aforementioned different data migration tasks.

[0188] In this embodiment of the application, the second computing device can execute a target task including multiple data migration tasks according to the dependency relationship, thereby improving the accuracy of process orchestration and the controllability of execution.

[0189] In one possible implementation, determining the scheduling instructions based on configuration information and the dependencies between different data migration tasks includes: determining the target configuration file corresponding to the configuration information; and converting the target configuration file and dependencies into scheduling instructions in a format recognizable by the second computing device. Thus, the second computing device can respond to the scheduling instructions upon receiving them.

[0190] In some embodiments, the method further includes: a first computing device receiving verification information input by a user based on visual configuration elements; determining a verification instruction that a second computing device can recognize based on the verification information, the verification instruction being used to verify data migrated from the source end to the target end; and sending the verification instruction to the second computing device, the second computing device further being used to execute a corresponding verification task in response to the verification instruction.

[0191] Verification tasks include verifying one or more of the following: data quantity consistency, content integrity, data security, format validity, or data accuracy.

[0192] In this embodiment, by visually configuring verification information and automatically generating and executing verification commands, the migration data indicated by the target task can be automatically verified in multiple dimensions, including consistency, integrity, accuracy, security, and legality, after the migration is completed, thus constructing a closed loop for data consistency assurance across platforms and modalities. Simultaneously, replacing plain text configuration with a visual approach avoids configuration errors and difficulties in troubleshooting under complex processes. It can automatically complete data comparison, difference location, and result solidification, replacing manual sampling and improving the reliability of migration results, data quality, and acceptance efficiency.

[0193] In one possible implementation, determining the verification instructions that the second computing device can recognize based on the verification information includes: filling the verification information into a verification code template to generate a target verification file. The target verification file is used to instruct the second computing device to verify the consistency of the quantity, integrity of the content, data security, format validity, or data accuracy of the data to be migrated from the source to the target. The target verification file is then converted into verification instructions according to a format that the second computing device can recognize.

[0194] In this embodiment, the target verification file is generated by verifying the code template and converted into standardized verification instructions, thereby realizing the standardized encapsulation and generation of data verification rules. There is no need to manually write verification logic code, which not only ensures the standardization of the verification instructions, but also simplifies the configuration and distribution process of verification tasks, and improves the efficiency and reliability of multi-dimensional verification execution after data migration.

[0195] In some embodiments, the method further includes: a first computing device receiving an update instruction from a user, and in response to the user's update instruction, adding, deleting, or modifying one or more of the stored configuration items through visual configuration elements.

[0196] Update commands are used to instruct one or more of the following operations to be performed on stored configuration items: add, delete, or modify.

[0197] In this embodiment of the application, by uniformly storing and visually updating the configuration items of semantic nodes and / or integrated task containers, centralized management of various configuration information can be achieved, effectively reducing the cost of repeated configuration and changes, and lowering the configuration complexity and parameter maintenance cost of migration tasks.

[0198] In some embodiments, the method further includes: a first computing device pre-running the target task; extracting sampled data generated during the pre-running process; and based on the sampled data, predicting the execution result of the target task executed by a second computing device, the execution result indicating whether the target task was successfully executed.

[0199] For example, the first computing device constructs a simulation environment and pre-runs the target task based on this simulation environment. The simulation environment is a runtime environment built by the first computing device to simulate the execution of the target task by the second computing device. The simulation environment is constructed in two scenarios based on the deployment relationship between the first and second computing devices: When the first and second computing devices are the same device, the actual runtime environment, dependent components, and configuration parameters of the target task are directly reused to locally simulate the task execution process. When the first and second computing devices are different devices, the first computing device completely replicates the runtime parameters, dependent environment, data source connection rules, and task execution logic of the second computing device to construct a simulation environment consistent with the actual execution environment, thereby simulating the task execution process of the second computing device.

[0200] In this embodiment of the application, by pre-running the target task and extracting sampled data to predict the execution result, the feasibility of the task can be verified before formal execution. Combined with the sampled data, anomalies can be quickly discovered, effectively improving the efficiency of problem discovery and troubleshooting, and further ensuring the stable execution of the migration task.

[0201] In an implementation method for predicting the execution result of a target task by a second computing device based on sampled data, if the amount of sampled data is within a preset data range, the execution result is determined to indicate that the second computing device can successfully execute the target task; if the amount of sampled data is not within the preset data range, the execution result is determined to indicate that the second computing device cannot successfully execute the target task.

[0202] For example, the preset data volume can be based on a user-defined data volume.

[0203] In this embodiment of the application, by comparing the amount of sampled data with a preset data range, the execution result of the target task can be predicted simply and efficiently without complex data analysis and processing. While ensuring the accuracy of the judgment, the efficiency of the pre-run result prediction is improved, making it easier to quickly identify the execution risks of the migration task.

[0204] In some embodiments, the method further includes: a first computing device acquiring execution log information of a target task executed by a second computing device, and pre-run log information of a target task to be pre-run by the first computing device. The execution log information and the pre-run log information are aggregated to obtain aggregated log information, which is used to determine errors during the execution of the target task and / or the pre-run target task by the second computing device.

[0205] For example, if the aggregated log information contains records such as data source connection timeout, data format mismatch, or permission verification failure, it can pinpoint the errors that occurred during the execution of the target task, such as data source connectivity issues, incorrect data migration format, or insufficient access permissions.

[0206] In this embodiment of the application, by aggregating the execution logs of the second computing device and the pre-running logs of the first computing device, it is possible to centrally display and compare the entire process of task operation information, quickly identify and locate various problems such as abnormal data source connectivity, incorrect data format, and failed permission verification, and effectively improve the efficiency and accuracy of troubleshooting and locating faults in migration tasks and pre-running processes.

[0207] The above content, using the first computing device as the execution entity, introduced the unified orchestration and scheduling of multiple types of data migration tasks. The following section, using the second computing device as the execution entity, introduces the unified orchestration and scheduling of multiple types of data migration tasks.

[0208] like Figure 13 As shown, this application embodiment provides another data migration method, which includes, for example, the following steps: S1301-S1302, which can be executed by the processor of the second computing device, such as the CPU.

[0209] S1301, Receive scheduling instructions sent by the first computing device.

[0210] S1302, responding to scheduling instructions, executes the target task.

[0211] In one possible implementation, as described above, the second computing device responds to scheduling instructions through a scheduling system, distributes the target task to the designated working node, and calls the corresponding preset component to execute the target task.

[0212] In conjunction with steps S1301-S1302, the second computing device receives scheduling instructions, distributes the target task to the designated working node, and calls the corresponding preset components to execute it. This decouples the scheduling and execution of migration tasks, enabling unified orchestration and processing of different types of data migration tasks within the same migration cycle. This effectively improves the coordination, standardization, and overall efficiency of the execution of multiple types of migration tasks.

[0213] In some embodiments, the method further includes: a second computing device receiving a verification instruction and performing a verification task based on the verification instruction.

[0214] In this embodiment of the application, the second computing device can automatically execute the corresponding verification task according to the verification instruction to verify the consistency, integrity, security and other dimensions of the migrated data, so as to realize the integrated execution of data migration and data verification, and improve the integrity and reliability of the overall data migration process while ensuring the quality of the migrated data.

[0215] This application embodiment can divide the above-mentioned computing device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0216] This application also provides a storage medium storing computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the method described above.

[0217] This application also provides a computer program product, which includes a computer program that, when at least one processor executes the computer program, causes the at least one processor to perform the methods described above in this application.

[0218] The computing device, storage medium, or computer program product provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0219] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0220] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0221] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0222] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data migration method, characterized in that, Applied to a first computing device; the method includes: Display a visual interface, which includes visual configuration elements for configuration items; Receive configuration information input by the user based on the visual configuration element; the configuration information includes the source end and the target end of the data to be migrated, and the configuration information is used to describe the migration of the data to be migrated from the source end to the target end, and the data to be migrated includes multiple types of offline data, real-time streaming data, cached data or file data; Based on the configuration information, a scheduling instruction that the second computing device can recognize is determined. The scheduling instruction is used to schedule the second computing device to execute a target task. The target task is used to instruct the data to be migrated from the source end to the target end. The scheduling instruction is sent to a second computing device, which executes the target task in response to the scheduling instruction.

2. The method according to claim 1, characterized in that, The visual interface includes graphical or form-based visual configuration elements, which are structured encapsulations of the configuration items; and the visual interface adopts a guided configuration method, which includes one or more of the following: wizard-style configuration prompts, configuration templates, or default value configuration methods.

3. The method according to claim 1 or 2, characterized in that, Multiple types of data migration tasks are carried out through a process model; the multiple types of data migration tasks include offline migration tasks for migrating offline data, real-time migration tasks for migrating real-time streaming data, cache migration tasks for migrating cached data, or file migration tasks for migrating file data. Different types of data migration tasks correspond to different semantic nodes in the process model. The visualization interface includes the semantic nodes and visualization configuration elements of the semantic node configuration items. The multiple nodes include multiple offline task nodes, real-time task nodes, cache task nodes, or file task nodes. The visualization configuration elements of the semantic nodes are used to configure the configuration information of the corresponding type of data migration task. And / or, One or more of the offline migration task, the real-time migration task, or the cached migration task correspond to the integrated task container in the process model. The visualization interface includes the integrated task container and the visualization configuration elements of the integrated task container configuration items. The visualization configuration elements of the integrated task container are used to configure the data processing method of the corresponding data migration task. The data processing method includes one or more of the following: Structured Query Language (SQL) conversion, field mapping, field renaming, or field splitting.

4. The method according to claim 3, characterized in that, The method further includes: Receive operation commands triggered by the user in the visual interface; In response to the operation command, directed connections are made between the task elements corresponding to the different data migration tasks to configure the dependencies between the different data migration tasks. The task elements include the semantic node and / or the integrated task container. The dependencies include one or more of the following: task execution order, process branching and merging relationship, conditional routing execution relationship, multi-level transformation link relationship, or sub-process call relationship. The step of determining the scheduling instructions that the second computing device can recognize based on the configuration information includes: Based on the configuration information and the dependencies between the different data migration tasks, the scheduling instruction is determined. The scheduling instruction is specifically used to schedule the second computing device to execute the target task based on the dependencies.

5. The method according to claim 3 or 4, characterized in that, The method further includes: Store the configuration items corresponding to the semantic nodes and the integrated task container; The system receives an update command from the user and, in response to the user's update command, adds, deletes, or modifies one or more of the stored configuration items through the visual configuration element; and / or provides a preset interface for the different data migration tasks, the preset interface being used by the corresponding type of data migration task to reference the stored configuration items.

6. The method according to any one of claims 1-5, characterized in that, The configuration information also includes data migration methods and / or data processing methods; The data migration method is used to indicate the migration method in the migration process of the data to be migrated. The data migration method includes one or more of the following: data acquisition strategy, concurrent sharding strategy, batch size strategy, parallel writing strategy, breakpoint resume strategy, retry strategy, rate limiting strategy, or idempotent writing strategy. The data processing method is used to indicate the processing method of the data to be migrated during the migration process; the data processing method includes one or more of the following: Structured Query Language (SQL) conversion, field mapping, field renaming, or field splitting; The scheduling instruction is specifically used to schedule the second computing device to execute the target task based on the data migration method and / or the data processing method.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The system receives verification information input by the user based on the visual configuration element. The verification information is used to indicate the verification of data after it has been migrated from the source end to the target end. The verification includes one or more of the following: quantity consistency verification, content integrity verification, data security verification, format legality verification, or data accuracy verification. Based on the verification information, a verification instruction that the second computing device can recognize is determined; The verification instruction is sent to a second computing device, which is further configured to execute a corresponding verification task in response to the verification instruction.

8. The method according to any one of claims 1-3, characterized in that, The step of determining the scheduling instructions that the second computing device can recognize based on the configuration information includes: The configuration information is subjected to syntax and semantic verification to obtain a first verification result, which is used to indicate whether the configuration information conforms to preset syntax and semantic rules; When the first verification result indicates that the configuration information conforms to the preset syntax and semantic rules, the scheduling instructions that the second computing device can recognize are determined based on the configuration information. The method further includes: If the first verification result indicates that the configuration information does not conform to the preset syntax and semantic rules, an error message is issued, which indicates that the configuration information does not conform to the preset syntax and semantic rules.

9. The method according to any one of claims 1-3, characterized in that, The step of determining the scheduling instructions that the second computing device can recognize based on the configuration information includes: Based on the configuration information, the task type is determined, which includes multiple types such as offline migration task, real-time migration task, cache migration task, or file migration task; Based on the configuration information and the task type, a scheduling instruction is determined. The scheduling instruction is used to call a preset component to implement the target task corresponding to the task type. The preset component includes a first component and a second component. The first component is used to implement the target task of the offline migration task and / or real-time migration task type, and the second component is used to implement the target task of the cache migration task and / or file migration task type.

10. The method according to claim 9, characterized in that, The step of determining the scheduling instruction based on the configuration information and the task type includes: Obtain the code template corresponding to the task type; The configuration information is filled into the code template corresponding to the task type to generate a target configuration file corresponding to the task type. The target configuration file is used to instruct the second computing device to execute the target task corresponding to the task type. The target configuration file corresponding to the task type is converted into the scheduling instruction in a format that the second computing device can recognize.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: Pre-run the target task; Extract the sampling data generated during the pre-run process; Based on the sampled data, the execution result of the second computing device in executing the target task is predicted, and the execution result is used to indicate whether the target task was successfully executed.

12. The method according to claim 11, characterized in that, The method further includes: Obtain execution log information generated by the second computing device executing the target task, and pre-run log information generated by the first computing device pre-running the target task; The execution log information and the pre-run log information are aggregated to obtain aggregated log information, which is used to determine errors in the process of the second computing device executing the target task and / or the first computing device pre-running the target task.

13. A computing device, characterized in that, It includes a processor and a memory; the processor is coupled to the memory; The memory is used to store instructions; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-12.

Images