Data processing method and device, equipment, medium and product

Abstract

The invention relates to a data processing method and device, equipment, a medium and a product. The method comprises the steps of obtaining task description data of a to-be-processed task; determining a target component adapter corresponding to the to-be-processed task according to the task description data; wherein the data architectures supported by the component adapters corresponding to different tasks are different; according to the target component adapter, performing data analysis on the task description data to obtain task synchronization configuration and task operation configuration of the to-be-processed task; according to the task synchronization configuration, determining a target storage path of a target component package corresponding to the to-be-processed task; and according to the task operation configuration and the target storage path of the target component package, generating a target job package of the to-be-processed task. By adopting the method, cooperative work of different data frameworks can be realized.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, medium and product. Background Technology

[0002] In the field of modern data processing and integration, Apache Flink (a distributed computing framework) is widely used as a high-performance stream processing engine for real-time data synchronization and batch processing tasks.

[0003] FlinkCDC and SeaTunnel, both based on Flink, are two excellent data synchronization frameworks. FlinkCDC (Change Data Capture) excels in real-time data synchronization scenarios, automatically handling table structure changes and supporting seamless integration of concurrent full and incremental synchronization of historical data, making it suitable for scenarios with high real-time requirements and dynamic data structures. SeaTunnel, on the other hand, offers flexibility and versatility in batch data synchronization, supporting flexible combinations of multiple data sources and target endpoints through a rich set of connectors, thus meeting complex data integration needs.

[0004] However, FlinkCDC and SeaTunnel are currently difficult to combine in practical applications, making it impossible for them to leverage their respective strengths to provide comprehensive data synchronization capabilities. Summary of the Invention

[0005] Therefore, it is necessary to provide a data processing method, apparatus, device, medium, and product that can overcome the shortcomings of existing technologies in which different data frameworks are difficult to work together.

[0006] In a first aspect, this application provides a data processing method, including:

[0007] Retrieve task description data for tasks to be processed;

[0008] Based on the task description data, determine the target component adapter corresponding to the task to be processed; the component adapters for different tasks support different data architectures.

[0009] Based on the target component adapter, the task description data is parsed to obtain the task synchronization configuration and task execution configuration of the task to be processed.

[0010] Based on the task synchronization configuration, determine the target storage path of the target component package corresponding to the task to be processed;

[0011] Based on the task execution configuration and the target storage path of the target component package, generate the target job package for the task to be processed.

[0012] In one embodiment, the task description data includes adaptation reference parameters for the task to be processed; accordingly, based on the task description data, a target component adapter corresponding to the task to be processed is determined, including:

[0013] Use the component adapter corresponding to the adaptation reference parameters as the target component adapter; different component adapters corresponding to different adaptation reference parameters support different data architectures.

[0014] In one embodiment, the task description data is encoded based on unified interface parameters corresponding to different data architectures.

[0015] In one embodiment, the method further includes: uploading the target job package to the target cluster environment based on a preset interface, so as to process the task to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0016] Secondly, this application provides a data processing method, including:

[0017] Obtain the target storage path and task execution configuration of the target job package to be processed by parsing the target job package; wherein the target job package is obtained based on any of the methods in the first aspect;

[0018] Based on the task execution configuration, resources are allocated for the tasks to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0019] Based on the allocated resources, the target component package under the target storage path is invoked to process the task to be processed in the target cluster environment.

[0020] Thirdly, this application also provides a data processing apparatus, comprising:

[0021] The data acquisition module is used to acquire task description data for the tasks to be processed.

[0022] The adapter selection module is used to determine the target component adapter corresponding to the task to be processed based on the task description data; the component adapters for different tasks support different data architectures.

[0023] The data parsing module is used to parse the task description data according to the target component adapter to obtain the task synchronization configuration and task execution configuration of the task to be processed.

[0024] The path selection module is used to determine the target storage path of the target component package corresponding to the task to be processed based on the task synchronization configuration.

[0025] The job package generation module is used to generate target job packages for tasks to be processed based on the task execution configuration and the target storage path of the target component package.

[0026] Fourthly, this application also provides a data processing apparatus, comprising:

[0027] The information acquisition module is used to acquire the target storage path and the task execution configuration of the task to be processed, obtained by parsing the target job package of the task to be processed; wherein, the target job package is obtained based on any of the methods in the first aspect;

[0028] The resource allocation module is used to allocate resources to the tasks to be processed in the target cluster environment according to the task running configuration; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0029] The component package invocation module is used to invoke the target component package under the target storage path based on the allocated resources, so as to process the pending tasks in the target cluster environment.

[0030] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0031] Retrieve task description data for tasks to be processed;

[0032] Based on the task description data, determine the target component adapter corresponding to the task to be processed; the component adapters for different tasks support different data architectures.

[0033] Based on the target component adapter, the task description data is parsed to obtain the task synchronization configuration and task execution configuration of the task to be processed.

[0034] Based on the task synchronization configuration, determine the target storage path of the target component package corresponding to the task to be processed;

[0035] Based on the task execution configuration and the target storage path of the target component package, generate the target job package for the task to be processed.

[0036] Sixthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0037] Obtain the target storage path and task execution configuration of the target job package to be processed by parsing the target job package; wherein the target job package is obtained based on any of the methods in the first aspect;

[0038] Based on the task execution configuration, resources are allocated for the tasks to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0039] Based on the allocated resources, the target component package under the target storage path is invoked to process the task to be processed in the target cluster environment.

[0040] In a seventh aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0041] Retrieve task description data for tasks to be processed;

[0042] Based on the task description data, determine the target component adapter corresponding to the task to be processed; the component adapters for different tasks support different data architectures.

[0043] Based on the target component adapter, the task description data is parsed to obtain the task synchronization configuration and task execution configuration of the task to be processed.

[0044] Based on the task synchronization configuration, determine the target storage path of the target component package corresponding to the task to be processed;

[0045] Based on the task execution configuration and the target storage path of the target component package, generate the target job package for the task to be processed.

[0046] Eighthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0047] Obtain the target storage path and task execution configuration of the target job package to be processed by parsing the target job package; wherein the target job package is obtained based on any of the methods in the first aspect;

[0048] Based on the task execution configuration, resources are allocated for the tasks to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0049] Based on the allocated resources, the target component package under the target storage path is invoked to process the task to be processed in the target cluster environment.

[0050] Ninthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0051] Retrieve task description data for tasks to be processed;

[0052] Based on the task description data, determine the target component adapter corresponding to the task to be processed; the component adapters for different tasks support different data architectures.

[0053] Based on the target component adapter, the task description data is parsed to obtain the task synchronization configuration and task execution configuration of the task to be processed.

[0054] Based on the task synchronization configuration, determine the target storage path of the target component package corresponding to the task to be processed;

[0055] Based on the task execution configuration and the target storage path of the target component package, generate the target job package for the task to be processed.

[0056] In a tenth aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0057] Obtain the target storage path and task execution configuration of the target job package to be processed by parsing the target job package; wherein the target job package is obtained based on any of the methods in the first aspect;

[0058] Based on the task execution configuration, resources are allocated for the tasks to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0059] Based on the allocated resources, the target component package under the target storage path is invoked to process the task to be processed in the target cluster environment.

[0060] The aforementioned data processing methods, apparatus, computer equipment, computer-readable storage media, and computer program products, by matching task description data with corresponding component adapters, can adapt to different data architectures (such as FlinkCDC and SeaTunnel), achieving unified parsing of different types of tasks. By combining task synchronization configuration to determine the component package storage path, it can accurately call dedicated components for different data architectures (such as FlinkCDC or SeaTunnel). Finally, based on the running configuration and component path, a target job package is generated, which can uniformly submit and run different types of tasks, thereby solving the problem of the difficulty in combining different data architectures, realizing the synergistic effect of both, providing users with comprehensive data synchronization capabilities that combine real-time performance and flexibility, and improving the efficiency and adaptability of data processing. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0063] Figure 2A This is a flowchart illustrating a data processing method in one embodiment;

[0064] Figure 2B This is a schematic diagram of the design of unified interface parameters in one embodiment;

[0065] Figure 3 This is a flowchart illustrating the steps for determining the target component adapter in one embodiment;

[0066] Figure 4 This is a flowchart illustrating the processing steps of a task to be processed in one embodiment;

[0067] Figure 5 This is a flowchart illustrating the data processing method in another embodiment;

[0068] Figure 6 This is a flowchart illustrating the data processing method in another embodiment;

[0069] Figure 7 This is a structural block diagram of a data processing device in one embodiment;

[0070] Figure 8 This is a structural block diagram of the data processing apparatus in another embodiment;

[0071] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0073] For ease of understanding, this application will use FlinkCDC (Change Data Capture, an extension of the Apache Flink ecosystem for capturing and processing changing data in databases) and SeaTunnel (an open-source distributed data integration platform focused on efficient transmission and processing of real-time and batch data) data architectures as examples to illustrate data collaborative processing methods under different data architectures. It is worth noting that the above data architectures can also be extended to other data architectures according to actual needs, and this application does not impose any limitations on this.

[0074] like Figure 1 The data processing system shown can be deployed using Flink (a stream processing framework for distributed data streaming and batch processing) in Hadoop Yarn (a key component in the Hadoop ecosystem responsible for cluster resource management and job scheduling) mode, enabling joint submission and collaborative execution of FlinkCDC and SeaTunnel. The data processing system integrates the real-time synchronization capabilities of FlinkCDC with the batch processing flexibility of SeaTunnel through a unified interface, providing a unified task entry point for upper-layer business systems. The data processing system supports deployment on a single terminal or on multiple terminal devices (one terminal device provides the interface for task submission, while other terminal devices handle task processing). After the business system submits tasks through the unified interface, the data processing system can automatically schedule resources in the Yarn Flink environment (i.e., running Flink applications in the Yarn environment) and call either FlinkCDC or SeaTunnel components to execute real-time or batch processing tasks, thereby simplifying the data processing flow and enabling flexible switching between the two types of tasks and unified resource scheduling.

[0075] FlinkCDC can monitor database changes (such as insert, update, and delete operations) in real time and stream these changes to Flink for processing. FlinkCDC supports various databases, such as MySQL and PostgreSQL, and achieves seamless integration with databases through connectors. FlinkCDC's main advantage lies in its low-latency and high-throughput real-time data processing capabilities, enabling users to analyze and respond promptly after data changes occur. This is extremely useful for applications requiring real-time data synchronization, data replication, and streaming data analysis, such as real-time monitoring, data warehouse updates, and event-driven applications. With FlinkCDC, data assets can be utilized more efficiently, enabling faster business decisions. SeaTunnel supports multiple data sources and target systems, providing flexible configuration and scalability through a pluggable architecture. SeaTunnel is known for its high performance and ease of use, allowing users to define complex data processing flows through simple configuration files. It is suitable for scenarios such as data synchronization, cleaning, and real-time analysis, making it an ideal tool for data-driven decision-making. SeaTunnel can run on Flink, Spark, and Zeta engines; this application primarily focuses on its operation on the Flink engine.

[0076] In one exemplary embodiment, such as Figure 2A As shown, a data processing method is provided, which can be applied to... Figure 1 The following explanation uses data processing equipment as an example:

[0077] S201, Obtain the task description data of the task to be processed.

[0078] The task to be processed can be a specific data processing task that needs to be executed through a target cluster environment (such as a YARN-Flink cluster environment) in a distributed data synchronization and processing scenario, and can be submitted through the business system; the task description data can be a collective term for all the configuration and parameter information required to fully define the task to be processed, including but not limited to the basic parameters of the task, the synchronization task description configuration, and the task running configuration.

[0079] It's worth noting that when the target cluster environment is a YARN-Flink cluster environment, Flink can refer to Apache Flink. Apache Flink is an open-source stream processing framework designed for distributed data streams and batch processing. It provides high-throughput, low-latency real-time data processing capabilities and supports event-driven application development. Flink's core features include state management, fault tolerance mechanisms, and flexible window operations, enabling it to handle complex data stream analysis tasks. Its powerful stream processing capabilities and integration with multiple data sources make Flink an ideal choice for processing real-time data in the big data ecosystem. Yarn, on the other hand, can refer to Apache Hadoop YARN (Yet Another Resource Negotiator). Apache Hadoop YARN is a key component in the Hadoop ecosystem, responsible for cluster resource management and job scheduling. YARN is designed to improve the scalability and flexibility of Hadoop by separating resource management and job scheduling, allowing different types of applications to run on the same cluster. The YARN architecture includes a global ResourceManager, a NodeManager for each node, and an application-specific ApplicationMaster. This architecture allows various data processing frameworks (such as MapReduce, Spark, Flink, etc.) to run on YARN, making full use of cluster resources and improving computational efficiency. The target cluster environment described above, a YARN-Flink cluster environment, is only an example; other cluster environments can also be used, and this embodiment is not limited to them.

[0080] Optionally, the process of receiving structured parameters submitted by the caller to define all elements of the task through a standardized unified interface parameter can obtain the task description data of the task to be processed.

[0081] The design of unified interface parameters is as follows: Figure 2B As shown, it includes basic information about the synchronization task (such as task name, task description, etc.), a Pipeline file (data flow configuration file) describing the execution of the synchronization task, and task runtime configuration (such as various parameters required for Flink to run on Yarn). Through unified interface parameters, different data architectures (such as FlinkCDC and SeaTunnel) can submit tasks to the target cluster environment in a unified form and through a unified entry point.

[0082] S202, Based on the task description data, determine the target component adapter corresponding to the task to be processed; wherein, the component adapters corresponding to different tasks support different data architectures.

[0083] The target component adapter can be an intermediate component used to adapt to specific data processing tools (such as FlinkCDC and SeaTunnel), that is, to convert general task description data into execution configurations that can be recognized by the corresponding tools.

[0084] Optionally, the task type corresponding to the task to be processed can be determined by identifying the field describing the task type in the task description data, and then the corresponding target component adapter can be called based on the task type. For example, when the task type corresponding to the task to be processed is a FlinkCDC task type, the adapter corresponding to the FlinkCDC component can be used as the target component adapter.

[0085] For example, corresponding adapters can be designed for different pipeline files to meet the submission requirements of different jobs with different data architectures. When creating a component adapter, it is necessary to know the specific versions of FlinkCDC Jars and SeaTunnel Jars in the workspace, as well as their compatibility with the Flink version. The adapter needs to automatically check version compatibility when submitting tasks based on this information to avoid task failure due to version conflicts. The above method of creating component adapters is only an example. It is also possible to create component adapters for all versions without paying attention to the specific versions of components in the workspace. Therefore, this embodiment is not limited.

[0086] S203, based on the target component adapter, perform data parsing on the task description data to obtain the task synchronization configuration and task execution configuration of the task to be processed.

[0087] Among them, task synchronization configuration can be a business rule configuration that describes how data flows from the data source to the target end, and task execution configuration can be an environment and resource configuration that describes how the task runs in the cluster.

[0088] Optionally, the target component adapter will identify and read the Pipeline file in the task description data. For example, the SeaTunnel adapter will parse the configuration blocks such as env, source, transform, and sink in the file, extract all parameters related to the data synchronization logic, and finally generate a structured task synchronization configuration. The target component adapter will also directly extract Flink running parameters (such as memory and parallelism) from the task description data to obtain the task running configuration.

[0089] S204, Based on the task synchronization configuration, determine the target storage path of the target component package corresponding to the task to be processed.

[0090] The target component package can be a collection of dedicated dependency components corresponding to the task adaptation tool that support the execution of the task to be processed. It can include at least one of the following: component packages and data source driver files. For example, if the task adapts to the FlinkCDC tool, the target component package includes the FlinkCDC core JAR file. The target storage path can be a specific location in the preset workspace where the target component package is pre-stored.

[0091] It is worth noting that, to jointly run FlinkCDC and SeaTunnel on Yarn Flink, this embodiment can pre-build a unified workspace. The workspace is mainly used to store a series of runtime component packages required to create FlinkCDC or SeaTunnel jobs on Yarn. The workspace directory design may include the following: Flink App Jars: Java JAR packages of the main Flink program running on Yarn; Datasource Drivers: Data source driver files required to run FlinkCDC and SeaTunnel synchronization jobs; Datasource Config Files: Data source configuration files required to run FlinkCDC and SeaTunnel synchronization jobs; FlinkCDC Jars: Java JAR packages required to run FlinkCDC on Flink; SeaTunnel Jars: Java JAR packages required to run SeaTunnel on Flink. After designing and building the workspace, a unified runtime package can be provided for jointly running FlinkCDC and SeaTunnel on Flink Yarn.

[0092] Optionally, by obtaining the task synchronization configuration, the storage location of the component packages required to process the task can be located in the workspace. The storage path of the target component package can then be determined through the directories in the workspace. For example, if the core component the task depends on is the FlinkCDC component, this can be determined by the component identifier field in the task synchronization configuration (e.g., tool_type:flink-cdc indicates a dependency on the FlinkCDC component). Then, by locating the FlinkCDC component, its storage path can be obtained.

[0093] S205, generate the target job package for the task to be processed based on the task execution configuration and the target storage path of the target component package.

[0094] The target job package can be a structured unit containing complete execution information of the task to be processed, which is submitted to the distributed cluster for execution. In essence, it is an integration of the task running configuration and the target storage path of the target component package. It serves as a complete instruction set for cluster execution of tasks, ensuring that the cluster can allocate resources, load dependent components and execute data processing logic according to the configuration.

[0095] Optionally, the task execution configuration and dependent component paths can be integrated into a target job package containing complete execution information. This job package serves as the final instruction carrier for the tasks to be processed in the cluster, ensuring that the cluster can allocate resources according to the configuration, load dependent components, and execute data processing logic, thereby achieving automated deployment and operation of tasks.

[0096] This embodiment matches task description data with corresponding component adapters, enabling adaptation to different data architectures (such as FlinkCDC and SeaTunnel), and achieving unified parsing of different types of tasks. By combining task synchronization configuration to determine the component package storage path, it can accurately call the exclusive components of different data architectures (such as FlinkCDC or SeaTunnel). Finally, based on the running configuration and component path, a target job package is generated, which can uniformly submit and run different types of tasks, thereby solving the problem of difficulty in combining different data architectures, realizing the synergistic effect of both, and providing users with comprehensive data synchronization capabilities that combine real-time performance and flexibility, improving the efficiency and adaptability of data processing.

[0097] Based on the technical solutions of the above embodiments, this application also provides an optional embodiment. In this optional embodiment, the task description data is further refined to include adaptation reference parameters of the task to be processed; correspondingly, S202 is further refined. See also Figure 3 The steps for determining the target component adapter, as shown, include:

[0098] S301, the component adapter corresponding to the adaptation reference parameter is used as the target component adapter; different component adapters corresponding to different adaptation reference parameters support different data architectures.

[0099] The adaptation reference parameters include, but are not limited to, task type and adapter version information.

[0100] Optionally, the corresponding component adapter can be invoked through adaptation reference parameters such as task type and adapter version information. For example, when the task type is FlinkCDC, the adapter corresponding to the FlinkCDC component is invoked as the target component adapter. This embodiment can also support running different versions of component adapters. For example, if adapter version information exists in the adaptation reference parameters, the component adapter corresponding to the adapter version information can be directly invoked as the target component adapter; otherwise, the default version of the component adapter is invoked as the target component adapter.

[0101] This embodiment sets adaptation reference parameters in the task description data and uses the corresponding component adapter as the target component adapter. It can directly and accurately match the component adapters corresponding to FlinkCDC or SeaTunnel based on the adaptation reference parameters, which solves the problem of difficulty in accurately matching adapters. This provides an accurate adaptation basis for the two to work together and improves the adaptation efficiency and accuracy of tasks and components.

[0102] In one embodiment, the task description data is encoded based on unified interface parameters corresponding to different data architectures.

[0103] Optionally, the task description data for FlinkCDC is different from that for SeaTunnel. That is, both FlinkCDC and SeaTunnel task description data include basic task information, synchronous task description configuration, and task execution configuration, but the basic task information for FlinkCDC differs from that for SeaTunnel. For example, the basic task information for FlinkCDC is the description information for the FlinkCDC task, while the basic task information for SeaTunnel is the description information for the SeaTunnel task.

[0104] In this implementation, the task description data is encoded based on the unified interface parameters corresponding to different data architectures. This allows the same set of interface parameters to be compatible with the different data architectures of FlinkCDC and SeaTunnel, solving the problem that the task description data cannot uniformly express the tasks of both. This ensures the consistency and compatibility of task information during the startup phase, provides a unified data foundation for subsequent collaborative parsing and component adaptation, and reduces the complexity of task description.

[0105] Based on the technical solutions of the above embodiments, this application also provides an optional embodiment in which, after S205, a processing step for the task to be processed is further added.

[0106] See Figure 4 The data processing method shown includes:

[0107] S401, based on a preset interface, uploads the target job package to the target cluster environment for processing the task to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0108] The preset interface can be a wrapper around the Flink-Yarn interface (i.e., a component package provided by Flink that allows Flink tasks to be submitted to Yarn via a JavaJar package); the target cluster environment can be the Yarn Flink environment, i.e., the environment in which Flink tasks run on Yarn; and the common runtime environment can be the unified underlying support provided by the target cluster environment for components with different data architectures.

[0109] Optionally, the data processing system initiates a target job package upload request to the target cluster environment via a preset interface call method. Upon receiving the request, the preset interface converts the target job package into a transmission format supported by the target cluster environment and encapsulates transmission instructions according to the target cluster's communication protocol to ensure the cluster can correctly identify the uploaded content. After the target job package transmission is complete, the interface sends a transmission completion signal to the cluster, waiting for the cluster to confirm successful reception of the job package. The target cluster environment then executes the pending tasks based on the parsed target job package.

[0110] In this embodiment, the target job package is uploaded to a shared runtime environment (such as a Yarn Flink cluster) of different data architecture components through a preset interface. This solves the problem that FlinkCDC and SeaTunnel job packages lack a unified upload path and a shared runtime environment, and realizes centralized management and collaborative scheduling of job packages in the same cluster environment.

[0111] The above embodiments have provided a detailed description of the target job package generation process. The task processing process based on the target job package will be described in detail below after its generation. It is worth noting that the execution device for the data processing method corresponding to the aforementioned target job package generation process can be the same as or different from the execution device for the data processing method corresponding to the task processing process of the task to be processed described below; this application does not impose any limitations on this.

[0112] In one exemplary embodiment, such as Figure 5 As shown, a data processing method is provided, including:

[0113] S501, obtain the target storage path and task execution configuration of the target job package to be processed, obtained by data parsing.

[0114] In the data processing system, the target cluster environment triggers the data parsing process of the target job package. According to the format of the target job package, the key files can be located and read, the target storage path of the target component package and the task execution configuration of the task to be processed can be extracted from the target job package, and the target storage path and task execution configuration can be temporarily stored in the target cluster environment.

[0115] S502 allocates resources for the tasks to be processed in the target cluster environment according to the task execution configuration; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0116] The resource manager of the target cluster environment makes scheduling decisions in its global resource pool based on task runtime configuration and current cluster load, and allocates corresponding resources to the tasks to be processed, including but not limited to the computing resources, storage resources, and node resources required by the tasks. It is worth noting that this target cluster environment is a shared runtime environment and may simultaneously run tasks based on different data architectures such as FlinkCDC and SeaTunnel.

[0117] S503, based on the allocated resources, calls the target component package under the target storage path to process the task to be processed in the target cluster environment.

[0118] Optionally, after successful resource allocation, the resource manager of the target cluster environment starts the application process based on the resources allocated for the tasks to be processed. During initialization, this process automatically downloads (or caches) all dependent files under the target storage path from the workspace, including the target component package, according to the target storage path. The target cluster environment loads the target component package and executes the main program logic in the target job package, thereby starting to process the data stream of the tasks to be processed.

[0119] This embodiment obtains the storage path and running configuration by parsing the target job package, and allocates resources and calls the corresponding component packages in a shared cluster environment. This solves the problems of disordered resource allocation and chaotic component calling in a shared environment, ensuring that FlinkCDC and SeaTunnel tasks can be executed efficiently in a unified cluster according to the configuration. It achieves precise collaboration between the two during the running phase, and improves the coordination of task processing and resource utilization.

[0120] Based on the technical solutions of the above embodiments, this application also provides an optional embodiment, in which the complete task processing process of the task to be processed under different data structures is described in detail.

[0121] See Figure 6 The data processing method shown includes:

[0122] S601, Obtain the task description data of the task to be processed. The task description data is encoded based on the unified interface parameters corresponding to different data architectures. The task description data includes the adaptation reference parameters of the task to be processed.

[0123] S602, The task description data includes the adaptation reference parameters of the task to be processed; the component adapter corresponding to the adaptation reference parameters is used as the target component adapter; the component adapters corresponding to different adaptation reference parameters support different data architectures.

[0124] S603, based on the target component adapter, performs data parsing on the task description data to obtain the task synchronization configuration and task execution configuration of the task to be processed.

[0125] S604, based on the task synchronization configuration, determine the target storage path of the target component package corresponding to the task to be processed.

[0126] S605 generates the target job package for the task to be processed based on the task execution configuration and the target storage path of the target component package.

[0127] S606, based on a preset interface, uploads the target job package to the target cluster environment for processing the tasks to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0128] S607, obtain the target storage path and task execution configuration of the target job package to be processed, obtained by data parsing.

[0129] S608 allocates resources for tasks to be processed in the target cluster environment according to the task execution configuration; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0130] S609, based on the allocated resources, calls the target component package under the target storage path to process the task to be processed in the target cluster environment.

[0131] The specific processes of S601 to S609 described above can be found in the description of the above method embodiments. Their implementation principles and technical effects are similar, and will not be repeated here.

[0132] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0133] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0134] In one exemplary embodiment, such as Figure 7 As shown, a data processing device is provided, including: a data acquisition module 71, an adapter selection module 72, a data parsing module 73, a path selection module 74, and a job package generation module 75, wherein:

[0135] The data acquisition module 71 is used to acquire the task description data of the task to be processed.

[0136] The adapter selection module 72 is used to determine the target component adapter corresponding to the task to be processed based on the task description data; wherein, the component adapters corresponding to different tasks support different data architectures.

[0137] The data parsing module 73 is used to parse the task description data according to the target component adapter to obtain the task synchronization configuration and task execution configuration of the task to be processed.

[0138] The path selection module 74 is used to determine the target storage path of the target component package corresponding to the task to be processed based on the task synchronization configuration.

[0139] The job package generation module 75 is used to generate the target job package for the task to be processed based on the task running configuration and the target storage path of the target component package.

[0140] In one embodiment, the adapter selection module 72 is further configured to:

[0141] Use the component adapter corresponding to the adaptation reference parameters as the target component adapter; different component adapters corresponding to different adaptation reference parameters support different data architectures.

[0142] In one embodiment, the job package generation module 75 is further configured to:

[0143] Based on the preset interface, the target job package is uploaded to the target cluster environment for processing of the tasks to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0144] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0145] In one exemplary embodiment, such as Figure 8 As shown, a data processing device is provided, including: an information acquisition module 81, a resource allocation module 82, and a component package invocation module 83, wherein:

[0146] The information acquisition module 81 is used to acquire the target storage path and the task execution configuration of the task to be processed, obtained by parsing the target job package of the task to be processed.

[0147] Resource allocation module 82 is used to allocate resources to the tasks to be processed in the target cluster environment according to the task running configuration; the target cluster environment is a common running environment for components corresponding to different data architectures.

[0148] The component package invocation module 83 is used to invoke the target component package under the target storage path based on the allocated resources, so as to process the task to be processed in the target cluster environment.

[0149] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0150] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0151] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0152] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0153] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0154] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0157] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, include: Retrieve task description data for tasks to be processed; Based on the task description data, a target component adapter corresponding to the task to be processed is determined; wherein, the component adapters corresponding to different tasks support different data architectures; Based on the target component adapter, the task description data is parsed to obtain the task synchronization configuration and task execution configuration of the task to be processed; Based on the task synchronization configuration, determine the target storage path of the target component package corresponding to the task to be processed; Based on the task execution configuration and the target storage path of the target component package, the target job package of the task to be processed is generated.

2. The method according to claim 1, characterized in that, The task description data includes adaptation reference parameters for the task to be processed; correspondingly, determining the target component adapter corresponding to the task to be processed based on the task description data includes: The component adapter corresponding to the adaptation reference parameters is used as the target component adapter; the component adapters corresponding to different adaptation reference parameters support different data architectures.

3. The method according to claim 1, characterized in that, The task description data is encoded based on unified interface parameters corresponding to different data architectures.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Based on a preset interface, the target job package is uploaded to the target cluster environment for processing the task to be processed in the target cluster environment; the target cluster environment is a common operating environment for components corresponding to different data architectures.

5. A data processing method, characterized in that, include: The target storage path and task execution configuration of the target job package to be processed are obtained by parsing the target job package of the task to be processed; wherein the target job package is obtained based on the method according to any one of claims 1-4; Based on the task execution configuration, resources are allocated for the task to be processed in the target cluster environment; the target cluster environment is a common running environment for components corresponding to different data architectures. Based on the allocated resources, the target component package under the target storage path is invoked to process the task to be processed in the target cluster environment.

6. A data processing apparatus, characterized in that, include: The data acquisition module is used to acquire task description data for the tasks to be processed. The adapter selection module is used to determine the target component adapter corresponding to the task to be processed based on the task description data; wherein, the component adapters corresponding to different tasks support different data architectures; The data parsing module is used to parse the task description data according to the target component adapter to obtain the task synchronization configuration and task execution configuration of the task to be processed. The path selection module is used to determine the target storage path of the target component package corresponding to the task to be processed based on the task synchronization configuration. The job package generation module is used to generate the target job package for the task to be processed based on the task running configuration and the target storage path of the target component package.

7. A data processing apparatus, characterized in that, include: The information acquisition module is used to acquire the target storage path obtained by parsing the target job package of the task to be processed and the task execution configuration of the task to be processed; wherein, the target job package is obtained based on the apparatus of claim 6; The resource allocation module is used to allocate resources to the task to be processed in the target cluster environment according to the task running configuration; the target cluster environment is a common running environment for components corresponding to different data architectures. The component package invocation module is used to invoke the target component package under the target storage path based on the allocated resources, so as to process the task to be processed in the target cluster environment.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-5.

Images