Task migration method and device, electronic equipment and storage medium
Through automated batch migration methods and open source real-time stream processing platform, the problems of low manual operation efficiency and high technical complexity in Apache Yarn to Kubernetes migration are solved, and efficient and low-cost task migration is achieved.
Patent Information
- Application Number
- CN202411110313.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-26
AI Technical Summary
When migrating data stream processing tasks from Apache Yarn to Kubernetes, there are a lot of manual operations, which are inefficient and cannot be standardized, resulting in inefficiency and high complexity. The underlying technology differences between the two lead to high migration complexity and cost.
By collecting multi-task information running on the first resource management platform, determining multi-task migration metadata information, configuring the task migration interface of the open source real-time stream processing platform, realizing automated migration, using the open source real-time stream processing platform to block underlying technical differences, and using tasks Basic mirroring and initialization containers automatically pull task resource compression packages to realize automated batch migration.
It greatly reduces manual intervention, improves migration efficiency, reduces complexity and cost, solves the problems of mirror construction time and mirror warehouse expansion, and realizes automated batch migration from the first resource management platform to the second resource management platform.
Smart Images

Figure CN119025240A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a task migration method, device, electronic device and storage medium. Background Art
[0002] In related technologies, migrating data stream processing tasks from Apache Yarn to Kubernetes often involves a large amount of manual operations, which not only consumes human resources but also is inefficient. At the same time, the migration process cannot be standardized, resulting in many human problems. The differences in the underlying technologies of Apache Yarn and Kubernetes also lead to inefficiency and high complexity of the data migration process. Summary of the invention
[0003] In order to overcome the problems existing in the related art, the present disclosure provides a task migration method, device, electronic device and storage medium.
[0004] According to a first aspect of an embodiment of the present disclosure, a task migration method is provided, comprising: collecting multi-task information running on a first resource management platform, wherein the multi-task information includes single-task information corresponding to each task, and the single-task information includes multiple metadata parameters for task migration; based on the multi-task information, determining multi-task migration metadata information, wherein the multi-task migration metadata information includes task migration status information, and the multi-task migration metadata information is stored in a migration tool; based on the multi-task migration metadata information in the migration tool, determining a task migration interface corresponding to an open source real-time stream processing platform; based on the task migration interface, migrating the multi-task migration metadata information to a second resource management platform, and obtaining migration task information after migration, wherein the performance of the second resource management platform is better than that of the first resource management platform.
[0005] In one embodiment, determining the multi-task migration metadata information based on the multi-task information includes: determining multiple metadata parameters of a single task information in the multi-task information based on the multi-task information; configuring migration metadata parameters of the single task information corresponding to the open source real-time stream processing platform based on the multiple metadata parameters, the migration metadata parameters including a user identifier to be migrated, a department identifier to be migrated, interface parameters to be migrated, and a compressed package of task resources to be migrated; determining the single task migration metadata information based on the migration metadata parameters; and repeating the above steps until the task migration metadata information corresponding to the multi-task information running on the first resource management platform is obtained.
[0006] In one embodiment, the determining multiple metadata parameters of a single task information in the multi-task information based on the multi-task information includes: designing multi-tenant task management based on the multi-task information, determining the user identifier to be migrated, the department identifier to be migrated and the interface parameters to be migrated corresponding to the single task information in the multi-task information, the multi-tenant task management is designed based on the task department identifier, and the interface parameters to be migrated correspond to the department identifier to be migrated; updating the single task information based on the user identifier to be migrated, the department identifier to be migrated and the interface parameters to be migrated; based on the updated single task information, collecting the task resource compression package in the single task information in real time, and storing the task resource compression package with the task name as the prefix to a remote file server; determining multiple metadata parameters of the single task information in the multi-task information based on the updated single task information and the task resource compression package.
[0007] In one embodiment, the real-time collection of the task resource compression package includes: based on the multi-task information, determining a multi-task resource compression package corresponding to the multi-task information; based on the multi-task resource compression package, reading the task name corresponding to each task resource compression package in the multi-task resource compression package; based on the task name corresponding to each task resource compression package, reusing the task basic image, and pulling each task resource compression package into a single task information environment in real time to obtain the task resource compression package.
[0008] In one embodiment, based on the task migration interface, the multi-task migration metadata information is migrated to the second resource management platform, and the migration task information after the migration is obtained, including: based on the task migration interface, determining the task migration status; in response to the task migration status being a successful migration, migrating the multi-task migration metadata information to the second resource management platform, and obtaining the migration task information after the migration.
[0009] In one implementation, the method further includes: in response to the data migration status being migration failure, a one-click operation is performed to roll back the single task information of the multi-task information that has failed to migrate to the first resource management platform for execution.
[0010] According to a second aspect of an embodiment of the present disclosure, a task migration device is provided, comprising: a collecting unit, for collecting multi-task information running on a first resource management platform, wherein the multi-task information includes single-task information corresponding to each task, and the single-task information includes multiple metadata parameters for task migration; a determining unit, for determining multi-task migration metadata information based on the multi-task information, wherein the multi-task migration metadata information includes task migration status information, and the multi-task migration metadata information is stored in a migration tool; based on the multi-task migration metadata information in the migration tool, determining a task migration interface corresponding to an open source real-time stream processing platform; a processing unit, for migrating the multi-task migration metadata information to a second resource management platform based on the task migration interface, and obtaining migration task information after migration, wherein the performance of the second resource management platform is better than that of the first resource management platform.
[0011] In one embodiment, the determination unit determines the multi-task migration metadata information based on the multi-task information in the following manner: based on the multi-task information, determine multiple metadata parameters of the single task information in the multi-task information; based on the multiple metadata parameters, configure the migration metadata parameters of the single task information corresponding to the open source real-time stream processing platform, the migration metadata parameters including the user identifier to be migrated, the department identifier to be migrated, the interface parameters to be migrated and the task resource compression package to be migrated; based on the migration metadata parameters, determine the single task migration metadata information; repeat the above steps until the multi-task migration metadata information corresponding to the multi-task information running on the first resource management platform is obtained.
[0012] In one embodiment, the determination unit determines multiple metadata parameters of a single task information in the multi-task information based on the multi-task information in the following manner: based on the multi-task information, design multi-tenant task management, determine the user identifier to be migrated, the department identifier to be migrated and the interface parameters to be migrated corresponding to the single task information in the multi-task information, the multi-tenant task management is designed based on the task department identifier, and the interface parameters to be migrated correspond to the department identifier to be migrated; based on the user identifier to be migrated, the department identifier to be migrated and the interface parameters to be migrated, update the single task information; based on the updated single task information, collect the task resource compression package in the single task information in real time, and store the task resource compression package with the task name as the prefix to a remote file server; based on the updated single task information and the task resource compression package, determine multiple metadata parameters of the single task information in the multi-task information.
[0013] In one embodiment, the determination unit collects the task resource compression package in real time in the following manner: based on the multi-task information, determine the multi-task resource compression package corresponding to the multi-task information; based on the multi-task resource compression package, read the task name corresponding to each task resource compression package in the multi-task resource compression package; based on the task name corresponding to each task resource compression package, reuse the task basic image, pull each task resource compression package into the single task information environment in real time, and obtain the task resource compression package.
[0014] In one embodiment, the processing unit migrates the multi-task migration metadata information to the second resource management platform based on the task migration interface and obtains the migration task information after the migration in the following manner: based on the task migration interface, determines the task migration status; in response to the task migration status being a successful migration, migrates the multi-task migration metadata information to the second resource management platform and obtains the migration task information after the migration.
[0015] In one implementation, the processing unit is further configured to: in response to the data migration status being migration failure, perform a one-click operation to roll back the single task information of the multi-task information that has failed to migrate to the first resource management platform for execution.
[0016] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to: execute the task migration method described in the first aspect or any one of the implementations of the first aspect.
[0017] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the method described in the first aspect or any one of the implementations of the first aspect can be implemented.
[0018] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: through the multi-task information running on the first resource management platform by the mobile phone, it is ensured that all necessary task information is collected for subsequent migration. According to the open source real-time stream processing platform interface, it is ensured that the migration tool can correctly interact with the second resource management platform to realize automated migration, and realize automated batch migration from the first resource management platform to the second resource management platform, which greatly reduces the need for manual intervention and improves migration efficiency. In addition, the underlying technical differences are shielded through the characteristics of the open source real-time stream processing platform itself, reducing the complexity and cost of migration.
[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0021] Figure 1 The figure is a flowchart of a task migration method according to an exemplary embodiment.
[0022] Figure 2 The invention is a flowchart showing a method of determining multiple metadata parameters of single-task information according to an exemplary embodiment.
[0023] Figure 3 The present invention is a flowchart showing a method of collecting a task resource compression package according to an exemplary embodiment.
[0024] Figure 4 The present invention is a flowchart showing a method for determining migration metadata information according to an exemplary embodiment.
[0025] Figure 5 The present invention is a flowchart of obtaining migration task information according to an exemplary embodiment.
[0026] Figure 6 The diagram is a schematic diagram showing a task migration according to an exemplary embodiment.
[0027] Figure 7 The figure is a block diagram of a task migration device according to an exemplary embodiment.
[0028] Figure 8 The figure is a task migration block diagram according to an exemplary embodiment.
[0029] Fig. 9 The figure is a schematic diagram of a task migration structure according to an exemplary embodiment. DETAILED DESCRIPTION
[0030] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0031] The task migration method provided in the embodiment of the present disclosure is applied to the scenario of task migration. The task migration method involved in the embodiment of the present disclosure is mainly applied to the scenario of migrating from the current task platform to another task platform.
[0032] In the related technology, migrating tasks from the first resource management platform to the second resource management platform often involves a large amount of manual operations, which not only consumes human resources but also is inefficient. At the same time, the migration process cannot be standardized, resulting in many human problems. The underlying technical differences between the first resource management platform and the second resource management platform also lead to inefficiency and high complexity of the data migration process.
[0033] In response to these problems, the present disclosure provides an automated batch migration method. By connecting the step processes in a systematic way to achieve batch automated migration of tasks, the reliance on manual intervention is greatly reduced, thereby improving the efficiency of migration. At the same time, the technical differences between the underlying first resource management platform and the second resource management platform are shielded by the open source real-time stream processing platform (Apache StreamPark), reducing the complexity and cost of migration. In particular, the present disclosure adopts a unique method to implement a method in which all tasks reuse a task base image and automatically pull task resource compression packages based on the initial container of the second resource management platform at runtime. It solves the problem that some current open source solutions directly package task-related task resource compression packages into exclusive images, which is very time-consuming and the image warehouse is bloated. It also solves the problem that each task migration and each task release requires the construction of a new image, which improves the migration efficiency and reduces the space cost of the image warehouse.
[0034] In view of this, the present invention provides a task migration method, which is used to, during the task migration process, use a migration tool to determine the task migration interface of the corresponding open source real-time stream processing platform for multiple task information on the first resource management platform, and migrate each task information in the multiple task information to the second resource management platform. This method improves the efficiency of task migration.
[0035] In the embodiments of the present disclosure, the first resource management platform involved can be understood as a platform mainly used for distributed resource management and task scheduling. Allow resources (such as CPU, memory) in the Hadoop cluster to be shared and used by multiple applications, by separating the functions of resource management and job scheduling / monitoring, such as Apache Yarn. The second resource management platform can be understood as an open source container orchestration platform for automating the deployment, expansion and management of containerized applications. It aims to provide a "production-ready" environment that supports declarative configuration and automation, making the deployment and management of containerized applications simpler and more efficient. For example, Kubernetes. Among them, the performance of the second resource management platform is better than that of the first resource management platform. The task resource compression package involved in the embodiments of the present disclosure can be understood as a Jar package, which is used interchangeably in the present disclosure, but the consistency of its meaning should be understood. Among them, in Apache Flink, tasks are usually submitted and run by programs packaged into Jar files. Jar files contain the business logic code of the task, dependent libraries, and other resources required for the Flink task. Jar is called JavaArchive, which is a format that packages multiple Java class files and related resources (such as images, text files, configuration files, etc.) into a compressed file for easy distribution and use.
[0036] Figure 1 is a flowchart of a task migration method according to an exemplary embodiment. Figure 1 As shown, the following steps are included.
[0037] In step S11, multi-task information running on the first resource management platform is collected.
[0038] In the disclosed embodiment, relevant information of all active tasks is collected from the Flink tasks currently running on the first resource management platform (such as Apache Yarn).
[0039] The multi-task information includes the single-task information corresponding to each task, and the single-task information includes a plurality of metadata parameters used for task migration;
[0040] In step S12, based on the multi-task information, multi-task migration metadata information is determined, where the multi-task migration metadata includes task migration state information and is stored in the migration tool.
[0041] In the disclosed embodiment, a data structure including metadata information required for task migration is constructed based on the collected task information, and the information is stored in a migration tool.
[0042] The migration tool can be understood as a data structure containing metadata information required for task migration.
[0043] In step S13, based on the multi-task migration metadata information in the migration tool, a task migration interface corresponding to the open source real-time stream processing platform is determined.
[0044] In the disclosed embodiment, a task migration interface that matches a migration tool with an open source real-time stream processing platform is determined based on multi-task migration metadata information.
[0045] In step S14, based on the task migration interface, the multi-task migration metadata information is migrated to the second resource management platform to obtain the migrated migration task information.
[0046] In an embodiment of the present disclosure, a determined task migration interface is used to transmit the multi-task migration metadata information stored in the migration tool to an open source real-time stream processing platform, and the multi-task migration metadata information is migrated to a second resource management platform (such as Kubernetes) using the open source real-time stream processing platform, and the migrated task information is obtained.
[0047] According to the exemplary embodiment of the present disclosure, the need for manual intervention is reduced by automating the batch migration process, thereby greatly improving the efficiency of migration. With the help of an open source real-time stream processing platform, such as Apache StreamPark, the technical differences between the first resource management platform and the second resource management platform are shielded, reducing the complexity and cost of migration. Reducing manual operations and improving the degree of automation allow users to focus more on business logic rather than underlying technical details.
[0048] In the disclosed embodiment, before executing task migration, all task metadata information of the task migration need to be obtained by the mobile phone.
[0049] In an exemplary embodiment, all active Flink task information is collected from the task management platform, including task type, version, main class name, jar package path, etc. Example information includes task name, type, Flink version, main class name, etc.
[0050] In the disclosed embodiment, metadata information of Flink tasks deployed on Apache Yarn is collected.
[0051] In one implementation, since the Flink task is deployed on Apache Yarn, an open source or self-developed task management platform is used for ease of management. All valid and active Flink task information is collected from the management platform for migration.
[0052] Among them, Flink can be understood as an open source framework for processing unbounded and bounded data streams, providing a highly scalable way to process large-scale data streams. Apache Yarn can be understood as a general resource management and task scheduling framework. Apache Yarn provides a unified resource management platform for applications, allowing multiple computing frameworks to coexist and run in parallel on the same cluster. Apache Yarn provides an execution environment for Flink, which is responsible for managing computing resources (such as CPU, memory) and allocating them to Flink's JobManager and TaskManagers. ApacheYarn manages the resources of the entire cluster through a combination of ResourceManager and NodeManagers.
[0053] In an exemplary embodiment, in order to more conveniently manage Flink tasks, a task management platform is usually used. The platform can be open source, such as Apache StreamPark, or it can be developed internally by the company.
[0054] Among them, the functions of the management platform include: Centralized management: Centrally manage the deployment, monitoring and maintenance of Flink tasks. Simplified operations: Simplify daily operation processes, such as starting, stopping, and restarting tasks. Enhanced functions: Provide additional functions, such as task visualization, log viewing, and alarm notification. Collect task information: In order to migrate Flink tasks from Apache Yarn to Kubernetes, you first need to collect information about all valid and active Flink tasks from the task management platform. Each Flink task information usually includes but is not limited to: task name, task type (SQL task or Jar task), Flink version, main class name, user jar package name and path, dependent jar package name and path, UDF (user defined function) jar package name and path, parallelism, program parameters, Flink parameters, business department, and other related information. Based on the collected information, ensure that the Flink task can be correctly redeployed and configured during the migration process. This information will be used to rebuild the deployment configuration of the Flink task in the Kubernetes environment. The task collection information introduction and example of single task information are shown in Table 1.
[0055] Table 1
[0056]
[0057]
[0058] In an exemplary embodiment of the present disclosure, based on the multi-task information, determining multiple metadata parameters of the single-task information in the multi-task information may be performed as follows: Figure 2 The method shown. Figure 2 The invention is a flowchart showing a method for determining multiple metadata parameters of single-task information according to an exemplary embodiment, and includes the following steps.
[0059] In step S21, based on the multi-task information, multi-tenant task management is designed to determine the user identifier to be migrated, the department identifier to be migrated, and the interface parameters to be migrated corresponding to the single task information in the multi-task information.
[0060] In the disclosed embodiment, relevant information of all active tasks is collected from the Flink tasks currently running on the first resource management platform (such as Apache Yarn), and a multi-tenant task management strategy is designed to determine the user identifier to be migrated, the department identifier to be migrated, and the interface parameters to be migrated for each task.
[0061] Among them, the multi-tenant task management involved in the embodiments of the present disclosure can be understood as multi-tenant task management performed according to the department account corresponding to the business department, or it can be understood as multi-tenant task management performed according to the member accounts corresponding to members with different permissions in the business department. This disclosure does not make any limitations.
[0062] In an exemplary embodiment, multi-tenant task management is designed. It can be understood that many business lines are developing real-time tasks. In order to ensure effective isolation between various businesses, the concept of tenant management is introduced, and a virtual organization ID is created for each business department. Each virtual organization ID has the authority to manage the business department.
[0063] Among them, tenant management can be understood as a mechanism that allows different business departments to have independent resources and permissions. The virtual organization identifier can be understood as the identifier of the user to be migrated, the identifier of the department to be migrated, and the interface parameters to be migrated, or the identifier of the corresponding business department entity.
[0064] Among them, the functions of the virtual organization identification include task management, business independence and security. Among them, task management can be understood as the virtual organization identification corresponding to the business department entity, and the members of the entity business department can manage their internal tasks, including the addition, deletion, modification and query functions of tasks. Business independence can be understood as the tasks between different business lines are isolated from each other. Each team can only see and manage its own tasks, but cannot access the tasks of other teams. In this way, business independence is achieved. Security can be understood as providing security guarantees for tenant management. By restricting access rights, it is ensured that only authorized team members can access and manage specific resources, thereby enhancing the security of the system.
[0065] In step S22, the single task information is updated based on the user identifier to be migrated, the department identifier to be migrated, and the interface parameters to be migrated.
[0066] In the disclosed embodiment, the single task information is updated according to the determined user identifier to be migrated, department identifier to be migrated, and interface parameters to be migrated, ensuring that the information is compatible with the target resource management platform (such as Kubernetes).
[0067] In step S23, based on the updated single task information, the task resource compression package in the single task information is collected in real time, and the task resource compression package is stored in the remote file server with the task name as a prefix.
[0068] In the disclosed embodiment, task resource compression packages related to single task information, such as user jar packages, dependent jar packages, etc., are collected, and these task resource compression packages are stored in a remote file server with the task name as a prefix.
[0069] In step S24, multiple metadata parameters of the single task information in the multi-task information are determined based on the updated single task information and the task resource compression package.
[0070] In the disclosed embodiment, multiple metadata parameters of each task are determined based on the updated single task information and the task resource compression package stored on the remote file server.
[0071] According to the exemplary embodiment of the present disclosure, the concept of multi-tenant management is introduced to provide independent migration environments and configurations for different business lines, ensuring the standardization and consistency of the migration process. At the same time, through unified metadata parameter management, confusion and errors in the migration process are avoided.
[0072] In an exemplary embodiment, based on the user identifier to be migrated, the department identifier to be migrated and the interface parameter to be migrated, the single task information is updated, and the updated single task information table is shown in Table 2.
[0073] Table 2
[0074]
[0075]
[0076] In the exemplary embodiment of the present disclosure, the real-time collection of task resource compression packages can be performed as follows: Figure 3 The method shown. Figure 3 The present invention is a flowchart of collecting a task resource compression package according to an exemplary embodiment, and includes the following steps.
[0077] In step S31, based on the multi-task information, a multi-task resource compression package corresponding to the multi-task information is determined.
[0078] In the disclosed embodiment, relevant information of all active tasks is collected from the Flink tasks currently running on the first resource management platform (such as Apache Yarn), and the task resource compression package corresponding to each task is determined. The task resource compression package includes user jar packages, dependent jar packages, etc.
[0079] In step S32, based on the multi-task resource compression package, the task name corresponding to each task resource compression package in the multi-task resource compression package is read.
[0080] In the disclosed embodiment, the task name corresponding to each task resource compression package is read from the collected multi-task resource compression packages.
[0081] In step S33, based on the task name corresponding to each task resource compression package, the task basic image is reused, and each task resource compression package is pulled into the single task information environment in real time to obtain the task resource compression package.
[0082] In the disclosed embodiment, based on the task name of each task resource compression package, a shared basic image is used, and the Kubernetes initialization container (Init Container) mechanism is used to pull each task resource compression package from the remote file server to the single task information environment in real time to complete the acquisition of the task resource compression package.
[0083] According to an exemplary embodiment of the present disclosure, by implementing the reuse of one task base image for all tasks and automatically pulling jar packages at runtime through the Kubernetes initialization container, the problems of time-consuming image construction and image warehouse expansion are solved, the migration efficiency is improved and the space cost of the image warehouse is reduced.
[0084] In the exemplary embodiment of the present disclosure, the multi-task migration metadata information can be determined according to the multi-task information by using the following method: Figure 4 The method shown. Figure 4 The flowchart of determining migration metadata information according to an exemplary embodiment includes the following steps.
[0085] In step S41 , based on the multi-task information, a plurality of metadata parameters of the single-task information in the multi-task information are determined.
[0086] In the disclosed embodiment, relevant information of all active tasks is collected from the Flink tasks currently running on the first resource management platform (such as Apache Yarn). Based on the collected multi-task information, multiple metadata parameters in the single-task information corresponding to each task are determined.
[0087] The metadata parameters may be understood as all parameters of the corresponding task required to execute the migration task.
[0088] In step S42, based on the multiple metadata parameters, migration metadata parameters corresponding to the open source real-time stream processing platform in the single task information are configured.
[0089] In the embodiment of the present disclosure, migration metadata parameters matching the open source real-time stream processing platform are configured according to the metadata parameters in the single task information. These parameters include but are not limited to the user ID to be migrated, the department ID to be migrated, the interface parameters to be migrated, and the compressed package of the task resources to be migrated.
[0090] The migration metadata parameters include the user ID to be migrated, the department ID to be migrated, the interface parameters to be migrated, and the compressed package of the task resources to be migrated.
[0091] In step S43, single-task migration metadata information is determined based on the migration metadata parameters.
[0092] In the disclosed embodiment, migration metadata information of each task is determined based on configured migration metadata parameters.
[0093] In step S44, the above steps are repeatedly performed until the multi-task migration metadata information corresponding to the multi-task information running on the first resource management platform is obtained.
[0094] In the disclosed embodiment, the above steps are repeated until the migration metadata information of all tasks is obtained and summarized as multi-task migration metadata information.
[0095] According to an exemplary embodiment of the present disclosure, by extracting the metadata parameters of each task in detail, an accurate data foundation is provided for subsequent steps, ensuring the accuracy and reliability of the migration process. By configuring the migration metadata parameters for each task, Apache StreamPark can identify and manage these tasks, providing necessary support for subsequent automated migration. The determination of single-task migration metadata information makes the migration process more systematic and standardized, improving the efficiency and accuracy of migration. By batch processing the migration metadata information of all tasks, automated control of the entire migration process is achieved, which greatly reduces manual intervention and improves the efficiency and repeatability of migration.
[0096] In the disclosed embodiment, according to the multiple single-task information lists corresponding to the collected multi-task information, the task version corresponding to the single task in the single-task information list is identified, and the task multi-version configuration is performed in the open source real-time stream processing platform to determine the task version corresponding to the open source real-time stream processing platform. According to the task department identifiers in the multiple single-task information lists and the task-to-be-migrated business department, the user identifier to be migrated and the department identifier to be migrated corresponding to the open source real-time stream processing platform are created, and the interface parameters to be migrated are determined for the created department identifier to be migrated, and multiple updated single-task information are obtained.
[0097] In an exemplary embodiment, the used index Flink version is identified based on all collected task information and Flink multi-version configuration is performed in Apache StreamPark. Teams and team accounts are created based on the business department and multi-tenant design of the collected single task information, and the task collection information userId and teamId are completed. A token required to use the Apache StreamPark API is created for each team account. The created userId, teamId, and token are updated to the single task information.
[0098] In the disclosed embodiment, the jar packages are found according to udfJarPaths, dependencyJarPath, and dependencyJarPaths in the task collection information list and uploaded to the remote file server on the path prefixed by the task name. When each task is started on the second resource management platform, the second resource management container running the task can automatically pull the task resource compression package required by the task from the remote file server to the running environment of each task before execution, realizing a dynamic runtime acquisition method.
[0099] In an exemplary embodiment, based on the updated single task information, the jar package that the task depends on is uploaded to the remote storage on the remote file server path prefixed with the respective task JobName. Customize the Flink Pod Template, and define the initial container (Flink Init Container) in the Pod Template. The initial container obtains the task-related jar package list and task name from the configuration file of the corresponding Flink task in the container through the command command, and splices the remote storage path that can obtain the jar package from the remote file server according to the jar name and task name. Based on the remote storage path of the jar package, the initial container downloads the Jar package from the remote file server through the command command, and mounts the Jar package to the corresponding task directory in the main container (Flink Main Container) through the emptyDir volume, such as local: / / / job-name / userjar.jar. At this point, the environment preparation for the task-related jar package is completed.
[0100] In an exemplary embodiment, Flink task types are divided into Flink SQL and Jar package tasks. For Flink SQL tasks, two types of jar packages are required for task running, flink connector jar and udfJar. In order to improve user experience and development efficiency, the most frequently used connector jar is pre-packaged into the Flink base image, so no additional processing is required during the migration process. UDF Jar, user Jar, and dependency jar can be treated as the same type of jar package in the process processing and processed in the above steps.
[0101] In the exemplary embodiment of the present disclosure, based on the task migration interface, the multi-task migration metadata information is migrated to the second resource management platform, and the migration task information after migration is obtained. Figure 5 The method shown. Figure 5 A flowchart of obtaining migration task information is shown according to an exemplary embodiment, including the following steps.
[0102] In step S51, the task migration state is determined based on the task migration interface.
[0103] In the disclosed embodiment, during the task migration process, the migration status of each task is determined through interaction with the task migration interface.
[0104] The migration status includes: not migrated, migration in progress, migration successful, and migration failed.
[0105] In step S52, in response to the task migration status being migration success, the multi-task migration metadata information is migrated to the second resource management platform, and the migration task information after the migration is obtained.
[0106] In the disclosed embodiment, when it is monitored that the task migration status is successful, the multi-task migration metadata information is migrated to the second resource management platform (such as Kubernetes), and the migrated task information is obtained.
[0107] According to the exemplary embodiment of the present disclosure, by real-time monitoring of the migration status, problems that may occur during the migration process can be discovered in a timely manner, and corresponding measures can be taken to adjust or repair them to ensure the success rate of the migration. It is ensured that only tasks that have successfully migrated will be formally deployed to the second resource management platform, avoiding resource waste and management confusion caused by migration failure.
[0108] In an exemplary embodiment of the present disclosure, when the data migration status is migration failure, a one-click operation is performed to roll back the single task information of the multi-task information that has failed to migrate to the first resource management platform for execution.
[0109] In an exemplary embodiment, a migration tool is constructed in a systematic way, and the updated single-task information migration task collection information is used as the internal task migration metadata information of the migration tool, and the task migration status and other information are supplemented, and the supplemented information is used as the multi-task migration metadata information. Apache StreamPark provides an API for application management, including publishing applications ( / flink / pipe / build), starting applications ( / flink / app / start), and obtaining application status ( / flink / app / get), etc. Each Flink task corresponds to an Apache StreamPark application, and Apache StreamPark completes the management of Flink tasks by controlling applications. The migration tool batch creates applications in Apache StreamPark based on the internal metadata information of each migration task, including userId, teamId, api token, jobName, etc. through the Apache StreamPark API. Secondly, the migration of Flink tasks from Apache Yarn to Kubernetes is completed through the following process. The old tasks in Flink on Apache Yarn mode are stopped by controlling the stop of old tasks on other management platforms. Through the Apache StreamPark API and the migration task metadata information, we complete the application release verification, application startup, and application status acquisition processes, and complete the startup of the new task in the Flink on Kubernetes mode. The migration tool updates the status of each step in the task metadata information, and finally connects the above processes in a systematic way to automate the entire migration process. At the same time, for tasks that fail to migrate, we support one-click operations to roll back to Apache Yarn for execution, and ensure that the task can continue to execute normally through rollback.
[0110] According to the exemplary embodiments of the present disclosure, batch migration of Flink tasks is achieved through automated tools, reducing manual participation and improving migration efficiency. The Apache StreamPark platform is used to shield the technical differences between Apache Yarn and Kubernetes, reducing the complexity of migration. In addition, all Flink tasks share a basic image, and the required jar packages are automatically pulled at runtime through the Kubernetes initialization container, solving the problems of time-consuming image construction and image warehouse expansion.
[0111] In an exemplary embodiment, the Figure 6 The method shown in the figure illustrates the process of migrating multiple Flink tasks running on another task management platform Apache Yarn to Kubernetes. Figure 6 The diagram is a schematic diagram showing a task migration according to an exemplary embodiment.
[0112] In one example, the migration tool collects Flink task information from Apache Yarn, including task configuration, status, and other related metadata. Based on the collected information, the migration tool generates migration metadata suitable for the new platform. The migration tool creates, publishes, and starts new Flink applications by calling the Apache StreamPark API. After receiving the request, Apache StreamPark starts to manage the new Flink application, including starting and stopping operations. Kubernetes creates, publishes, and starts new Flink applications based on the request from Apache StreamPark.
[0113] In the embodiments of the present disclosure, the Flink applications and Flink tasks involved have the same meaning and both contain task information that needs to be migrated. However, due to the different corresponding resource processing platforms, the task information corresponding to different resource processing platforms will be expressed in the form of the corresponding platforms, but the consistency of their meanings should be understood.
[0114] According to an exemplary embodiment of the present disclosure, an automated solution is provided for efficiently and cost-effectively migrating Flink tasks running on Apache Yarn to Kubernetes. By implementing an automated batch migration process method, the need for manual intervention is significantly reduced, thereby greatly improving the efficiency of the entire migration process. With the help of Apache StreamPark, the differences between the underlying Apache Yarn and Kubernetes are shielded, reducing the complexity and cost of migration. A method is implemented for all Flink tasks to reuse a Flink base image and automatically pull the jar package based on the Kubernetes initial container at startup, which solves the problem that some current open source solutions directly package task-related Jars into exclusive images, which is very time-consuming and the image repository is bloated. At the same time, it also solves the problem that each task migration and each task release requires the construction of a new image, which improves the migration efficiency and reduces the space cost of the image repository.
[0115] Based on the same concept, the embodiment of the present disclosure also provides a task migration device.
[0116] It is understandable that the task migration device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.
[0117] Figure 7 is a block diagram of a task migration device according to an exemplary embodiment. Figure 7 The device includes a collecting unit 101, a determining unit 102 and a processing unit 103.
[0118] The collecting unit 101 is used to collect multi-task information running on the first resource management platform, wherein the multi-task information includes single-task information corresponding to each task, and the single-task information includes multiple metadata parameters for task migration.
[0119] The determination unit 102 is used to determine multi-task migration metadata information based on the multi-task information, where the multi-task migration metadata information includes task migration status information, and the multi-task migration metadata information is stored in the migration tool; based on the multi-task migration metadata information in the migration tool, determine the task migration interface corresponding to the open source real-time stream processing platform.
[0120] The processing unit 103 is used to migrate the multi-task migration metadata information to the second resource management platform based on the task migration interface, and obtain the migration task information after migration, wherein the performance of the second resource management platform is better than that of the first resource management platform.
[0121] In one embodiment, the determination unit 102 determines the multi-task migration metadata information based on the multi-task information in the following manner: based on the multi-task information, determine multiple metadata parameters of the single task information in the multi-task information; based on the multiple metadata parameters, configure the migration metadata parameters of the single task information corresponding to the open source real-time stream processing platform, the migration metadata parameters including the user identifier to be migrated, the department identifier to be migrated, the interface parameters to be migrated, and the compressed package of the task resources to be migrated; based on the migration metadata parameters, determine the single task migration metadata information; repeat the above steps until the multi-task migration metadata information corresponding to the multi-task information running on the first resource management platform is obtained.
[0122] In one implementation, the determination unit 102 determines multiple metadata parameters of single task information in the multi-task information based on the multi-task information in the following manner: based on the multi-task information, design multi-tenant task management, determine the user identifier to be migrated, the department identifier to be migrated and the interface parameters to be migrated corresponding to the single task information in the multi-task information, the multi-tenant task management is designed based on the task department identifier, and the interface parameters to be migrated correspond to the department identifier to be migrated; based on the user identifier to be migrated, the department identifier to be migrated and the interface parameters to be migrated, update the single task information; based on the updated single task information, collect the task resource compression package in the single task information in real time, and store the task resource compression package with the task name as the prefix to the remote file server; based on the updated single task information and the task resource compression package, determine multiple metadata parameters of the single task information in the multi-task information.
[0123] In one embodiment, the determination unit 102 collects task resource compression packages in real time in the following manner: based on multi-task information, determine the multi-task resource compression package corresponding to the multi-task information; based on the multi-task resource compression package, read the task name corresponding to each task resource compression package in the multi-task resource compression package; based on the task name corresponding to each task resource compression package, reuse the task basic image, and pull each task resource compression package into the single task information environment in real time to obtain the task resource compression package.
[0124] In one embodiment, the processing unit 103 adopts the following method to migrate the multi-task migration metadata information to the second resource management platform based on the task migration interface, and obtain the migration task information after the migration: based on the task migration interface, determine the task migration status; in response to the task migration status being a successful migration, migrate the multi-task migration metadata information to the second resource management platform, and obtain the migration task information after the migration.
[0125] In one implementation, the processing unit 103 is further configured to: in response to the data migration status being migration failure, perform a one-click operation to roll back the single task information of the multi-task information that has failed to migrate to the first resource management platform for execution.
[0126] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0127] Figure 8 FIG. 1 is a task migration block diagram according to an exemplary embodiment. Figure 8 The architecture shown is only an optional implementation method. In the specific practice, the above can be modified according to actual needs. Figure 7The number and type of components can be selected, deleted, added or replaced; different functional components can be set up by separate settings or integrated settings, for example, the GPU and CPU can be set up separately or the GPU can be integrated on the CPU, the communication unit can be set up separately or integrated on the CPU or GPU, etc. These replaceable implementations all fall within the protection scope disclosed in the present invention.
[0128] Fig. 9 The figure is a schematic diagram of a task migration structure according to an exemplary embodiment. Fig. 9 FIG. 1 is a schematic diagram showing a structure of an embodiment of an electronic device disclosed in the present invention. Fig. 9 , which shows a schematic diagram of the structure of an electronic device suitable for implementing a terminal device or a server in an embodiment of the present application. Fig. 9 As shown, the electronic device includes a processor and a memory. The electronic device may also include an input and output device. The memory and the input and output device are connected to the processor through a bus. The memory is used to store instructions executed by the processor; the processor is used to call the instructions stored in the memory and execute the task migration method involved in the above embodiment.
[0129] The embodiments of the present disclosure also provide a computer program product including instructions. When the computer program product including instructions is run on a computer, the computer is enabled to execute the task migration method involved in the above embodiments.
[0130] In one or more optional implementations, the embodiment of the present disclosure further provides a computer-readable storage medium for storing computer-readable instructions, which, when executed, causes a computer to execute the image processing method in any of the above possible implementations. In another optional example, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.
[0131] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous.
[0132] The methods and apparatus of the present disclosure can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "apparatus" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.
[0133] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.
[0134] The foregoing description of the implementation of the present disclosure has been given for the purpose of illustration and description. The foregoing description is not exhaustive nor is it intended to limit the present disclosure to the exact form disclosed, and various variations and modifications may exist in accordance with the above teachings or may be obtained from the practice of the present disclosure. These embodiments are selected and described in order to illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can utilize the present disclosure in various embodiments and various modifications suitable for the specific purpose contemplated.
[0135] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.
[0136] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.
[0137] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.
[0138] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.
[0139] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.
[0140] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.
Claims
1. A task migration method, characterized in that: include: Collecting multi-task information running on the first resource management platform, wherein the multi-task information includes single-task information corresponding to each task, and the single-task information includes a plurality of metadata parameters for task migration; Based on the multi-task information, determining multi-task migration metadata information, the multi-task migration metadata information including task migration state information, and the multi-task migration metadata information is stored in a migration tool; Based on the multi-task migration metadata information in the migration tool, determine the task migration interface corresponding to the open source real-time stream processing platform; Based on the task migration interface, the multi-task migration metadata information is migrated to the second resource management platform to obtain the migrated migration task information, wherein the performance of the second resource management platform is better than that of the first resource management platform.
2. The method according to claim 1, characterized in that The determining the multi-task migration metadata information based on the multi-task information includes: Based on the multi-task information, determining a plurality of metadata parameters of single-task information in the multi-task information; Based on the multiple metadata parameters, configure the migration metadata parameters corresponding to the open source real-time stream processing platform in the single task information, wherein the migration metadata parameters include the user identifier to be migrated, the department identifier to be migrated, the interface parameters to be migrated, and the compressed package of the task resources to be migrated; Based on the migration metadata parameters, determining single-task migration metadata information; The above steps are repeatedly performed until the multi-task migration metadata information corresponding to the multi-task information running on the first resource management platform is obtained.
3. The method according to claim 2, characterized in that The determining, based on the multi-task information, a plurality of metadata parameters of single-task information in the multi-task information includes: Based on the multi-task information, design multi-tenant task management, determine the user identifier to be migrated, the department identifier to be migrated and the interface parameters to be migrated corresponding to the single task information in the multi-task information, the multi-tenant task management is designed based on the task department identifier, and the interface parameters to be migrated correspond to the department identifier to be migrated; Based on the user identifier to be migrated, the department identifier to be migrated and the interface parameter to be migrated, updating the single task information; Based on the updated single task information, the task resource compression package in the single task information is collected in real time, and the task resource compression package is stored in a remote file server with the task name as a prefix; Based on the updated single-task information and the task resource compression package, a plurality of metadata parameters of the single-task information in the multi-task information are determined.
4. The method according to claim 3, characterized in that The real-time collection of the task resource compression package includes: Based on the multi-task information, determining a multi-task resource compression package corresponding to the multi-task information; Based on the multi-task resource compression package, reading the task name corresponding to each task resource compression package in the multi-task resource compression package; Based on the task name corresponding to each task resource compression package, the task basic image is reused, and each task resource compression package is pulled into the single task information environment in real time to obtain the task resource compression package.
5. The method according to claim 1, characterized in that The step of migrating the multi-task migration metadata information to the second resource management platform based on the task migration interface and obtaining the migration task information after the migration includes: Based on the task migration interface, determining a task migration state; In response to the task migration status being migration success, the multi-task migration metadata information is migrated to the second resource management platform, and the migration task information after the migration is obtained.
6. The method according to claim 1, characterized in that The method further comprises: In response to the data migration status being migration failure, a one-click operation is performed to roll back the single task information of the multi-task information that has failed to migrate to the first resource management platform for execution.
7. A task migration device, characterized in that: include: A collecting unit, used to collect multi-task information running on the first resource management platform, wherein the multi-task information includes single-task information corresponding to each task, and the single-task information includes multiple metadata parameters for task migration; A determination unit is used to determine multi-task migration metadata information based on the multi-task information, wherein the multi-task migration metadata information includes task migration state information and the multi-task migration metadata information is stored in a migration tool; based on the multi-task migration metadata information in the migration tool, determine a task migration interface corresponding to an open source real-time stream processing platform; The processing unit is used to migrate the multi-task migration metadata information to the second resource management platform based on the task migration interface, and obtain the migration task information after migration, wherein the performance of the second resource management platform is better than that of the first resource management platform.
8. The device according to claim 7, characterized in that The determining unit determines the multi-task migration metadata information based on the multi-task information in the following manner: Based on the multi-task information, determining a plurality of metadata parameters of single-task information in the multi-task information; Based on the multiple metadata parameters, configure the migration metadata parameters corresponding to the open source real-time stream processing platform in the single task information, wherein the migration metadata parameters include the user identifier to be migrated, the department identifier to be migrated, the interface parameters to be migrated, and the compressed package of the task resources to be migrated; Based on the migration metadata parameters, determining single-task migration metadata information; The above steps are repeatedly performed until the multi-task migration metadata information corresponding to the multi-task information running on the first resource management platform is obtained.
9. The device according to claim 8, characterized in that The determining unit determines, based on the multi-task information, a plurality of metadata parameters of the single-task information in the multi-task information in the following manner: Based on the multi-task information, design multi-tenant task management, determine the user identifier to be migrated, the department identifier to be migrated and the interface parameters to be migrated corresponding to the single task information in the multi-task information, the multi-tenant task management is designed based on the task department identifier, and the interface parameters to be migrated correspond to the department identifier to be migrated; Based on the user identifier to be migrated, the department identifier to be migrated and the interface parameter to be migrated, updating the single task information; Based on the updated single task information, the task resource compression package in the single task information is collected in real time, and the task resource compression package is stored in a remote file server with the task name as a prefix; Based on the updated single-task information and the task resource compression package, a plurality of metadata parameters of the single-task information in the multi-task information are determined.
10. The device according to claim 9, characterized in that The determining unit collects the task resource compression package in real time in the following manner: Based on the multi-task information, determining a multi-task resource compression package corresponding to the multi-task information; Based on the multi-task resource compression package, reading the task name corresponding to each task resource compression package in the multi-task resource compression package; Based on the task name corresponding to each task resource compression package, the task basic image is reused, and each task resource compression package is pulled into the single task information environment in real time to obtain the task resource compression package.
11. The device according to claim 7, characterized in that The processing unit migrates the multi-task migration metadata information to the second resource management platform based on the task migration interface and obtains the migration task information after the migration in the following manner: Based on the task migration interface, determining a task migration state; In response to the task migration status being migration success, the multi-task migration metadata information is migrated to the second resource management platform, and the migration task information after the migration is obtained.
12. The device according to claim 7, characterized in that The processing unit is also used for: In response to the data migration status being migration failure, a one-click operation is performed to roll back the single task information of the multi-task information that has failed to migrate to the first resource management platform for execution.
13. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method according to any one of claims 1 to 6.
14. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the method according to any one of claims 1 to 6 can be executed.
