Stream-batch integrated data processing method, system and computer program product

Abstract

This application discloses a method, system, and computer program product for integrated stream and batch data processing, belonging to the field of data processing technology. The method includes: setting up data sources and data assets, and determining unified metadata; receiving data stream configuration instructions and setting the operating mode according to these instructions; dynamically filtering and displaying supported operator nodes in a graphical orchestration interface based on the operating mode, dragging and dropping operator nodes onto the canvas, and connecting and configuring them to obtain a data processing stream; automatically converting the logic of the graphical orchestration interface into the execution logic corresponding to the underlying computing engine based on the operating mode and the data processing stream, and deploying the execution logic to the corresponding engine cluster; and executing the data processing stream on the engine cluster to complete task execution and resource allocation. This application's method fully utilizes a shared graphical orchestration configuration and shared operator node conversion logic, reducing the difficulty of data processing for development and configuration personnel and improving real-time stream processing capabilities.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a batch processing method, system, and computer program product for integrated batch processing. Background Technology

[0002] In the current data processing field, especially in data integration, transformation, and loading scenarios, it is often necessary to simultaneously handle offline processing of historical batch data and streaming processing of real-time incremental data. Traditional technical solutions generally adopt a stream-batch separation architecture, that is, building independent data processing pipelines for batch processing and stream processing. These two systems rely on different computing engines, development interfaces, and metadata management methods, resulting in a disconnect at the data processing logic level. For example, the same business logic, such as data joining or field transformation, requires two separate development and implementations for batch jobs and stream jobs. This separation architecture not only causes repetitive investment in development and maintenance work, significantly increasing technical complexity and human resource costs, but more importantly, it makes it difficult to guarantee that the computing logic for the same business needs has completely consistent behavior and results in both stream and batch modes, thus affecting the accuracy and reliability of data processing results. Summary of the Invention

[0003] The purpose of this application is to overcome the shortcomings of the prior art and provide a batch and stream integrated data processing method, system and computer program product to achieve operational consistency between batch processing and stream processing of data exchange.

[0004] Firstly, this application provides a method for integrated batch and stream data processing, comprising the following steps:

[0005] Set up data sources and data assets, and define a unified metadata (Schema);

[0006] Receive data stream configuration instructions, and set the operating mode according to the data stream configuration instructions. The operating mode includes batch mode and stream mode.

[0007] In the graphical orchestration interface, the supported operator nodes are dynamically filtered and displayed according to the running mode. The operator nodes are dragged to the canvas and connected and configured to obtain the data processing stream.

[0008] Based on the operating mode and the data processing flow, the logic of the graphical orchestration interface is automatically converted into the execution logic corresponding to the underlying computing engine, and the execution logic is deployed to the corresponding engine cluster.

[0009] The data processing stream is executed on the corresponding engine cluster to complete task execution and resource allocation.

[0010] Optionally, data sources and data assets are set up, and uniform metadata is defined, including:

[0011] Set the data source;

[0012] For data sources that support Change Data Capture (CDC), enable the archive log option;

[0013] Obtain the table structure from the data source to obtain offline data assets;

[0014] Manually add real-time data structures, define real-time data assets, and obtain unified metadata.

[0015] Optionally, receiving a data stream configuration instruction and setting a running mode according to the data stream configuration instruction, the running mode including batch mode and stream mode includes: receiving a data stream configuration instruction issued by the user, creating a new data stream through a graphical orchestration interface, and setting basic attributes in a new data stream pop-up form according to the data stream configuration instruction, the basic attributes including data stream name, description, running mode, and execution engine.

[0016] Optionally, the operating modes include batch mode and streaming mode, which share a common set of data flow graphical orchestration interface; the execution engine is deployed in Standalone mode.

[0017] Optionally, the node types include: stream-batch shared type, stream processing type, and batch processing type. The stream-batch shared type uses shared operator node conversion logic and has the same conversion logic.

[0018] Optionally, operator nodes include: input, output, basic transformation, join, aggregation, mathematical function, data quality, and advanced operators.

[0019] Optionally, it also includes the following: when it is necessary to reuse the existing stream processing conversion logic for batch processing, if the stream processing conversion logic is consistent with the batch processing conversion logic, then by saving a new process, the stream mode is converted to the batch mode.

[0020] Optionally, a new process can be saved, including: opening the data stream list, selecting the stream mode record to be converted, changing the stream mode to batch mode in the pop-up basic properties dialog box, and obtaining a new batch processing task graphical orchestration interface; deleting the original real-time input source and real-time output source in the new batch processing task graphical orchestration interface, dragging and dropping the offline input source and offline output source onto the canvas, and configuring the offline input source and offline output source; confirming whether the conversion logic of all operator nodes is applicable in batch mode. If the conversion logic is consistent with the stream mode, there is no need to configure the intermediate operator nodes in the process. If the conversion logic is inconsistent with the stream mode, update the operator nodes to batch processing type, connect and configure them to complete the stream-to-batch conversion.

[0021] Secondly, this application also provides a batch processing system, comprising: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the batch processing method as described in any one aspect.

[0022] Thirdly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the batch processing method as described in any one of the first aspects.

[0023] This application provides a streaming and batch processing method and system that greatly improves development efficiency and reduces maintenance costs by sharing the transformation logic of operator nodes. It ensures that streaming and batch processing execute the exact same transformation logic, effectively solving the long-standing problem of inconsistent data standards in the data field, significantly lowering the technical threshold and optimizing the development experience. Using a graphical orchestration configuration, streaming and batch modes and different data processing engines all share the same orchestration page, reducing the difficulty of data processing for developers and improving real-time streaming processing capabilities. It provides users with an efficient, unified, and reliable data processing method.

[0024] To make the above-mentioned features and advantages of the invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a batch processing method for integrated batch processing provided in one embodiment of this application.

[0027] Figure 2 This is a flowchart of step S1 in a batch processing method for integrated batch processing provided in one embodiment of this application.

[0028] Figure 3 This is a blank state diagram for the batch mode of the graphical layout interface.

[0029] Figure 4 Edit the state diagram for the graphical layout interface flow mode. Detailed Implementation

[0030] To make the objectives and technical solutions of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the described embodiments of this application without creative effort are within the scope of protection of this application.

[0031] In one embodiment, see Figure 1 This application provides a batch processing method for integrated data processing, which may include the following steps: steps S1 to S5.

[0032] Step S1: Set up the data source and data assets, and determine the unified metadata.

[0033] Step S2: Receive data stream configuration instructions, and set the operating mode according to the data stream configuration instructions. The operating mode includes batch mode and stream mode.

[0034] Step S3: In the graphical orchestration interface, dynamically filter and display the supported operator nodes according to the running mode, drag the operator nodes to the canvas, and connect and configure them to obtain the data processing stream.

[0035] Step S4: Based on the operating mode and the data processing flow, automatically convert the logic of the graphical orchestration interface into the execution logic corresponding to the underlying computing engine, and deploy the execution logic to the corresponding engine cluster.

[0036] Step S5: Execute the data processing stream on the corresponding engine cluster to complete task execution and resource allocation.

[0037] In the integrated streaming and batch data processing method of this application, offline and real-time data assets are integrated by setting unified metadata, and streaming and batch data are made from the same source by using CDC technology, which ensures the consistency of data processing standards from the source. By receiving the running mode instructions and dynamically filtering and displaying the corresponding operator nodes in the shared streaming and batch orchestration interface, users can configure tasks in a unified graphical way, which significantly reduces the development complexity and learning cost caused by different modes, and ensures a high degree of consistency of core business logic in streaming and batch modes. The orchestration logic is automatically converted into underlying engine code according to the running mode and submitted to the corresponding cluster for execution, which simplifies operation and maintenance and optimizes the utilization of cluster resources.

[0038] In step S1, please refer to Figure 1 In step S1, set up the data source and data assets, and determine the unified metadata.

[0039] For example, please refer to Figure 2Step S1 may include the following steps: Step S11 to Step S14.

[0040] Step S11: Set the data source.

[0041] Step S12: For data sources that support Change Data Capture (CDC), enable the archive log option.

[0042] Step S13: Obtain the table structure from the data source to obtain offline data assets.

[0043] Step S14: Manually add real-time data structures, determine real-time data assets, and obtain unified metadata.

[0044] Specifically, in step S11, the user determines the data source as needed in the system management interface. The data source may include batch processing data sources and stream processing data sources.

[0045] As an example, batch processing data sources can include relational databases such as MySQL and PostgreSQL.

[0046] As an example, stream processing data sources can include message queues such as Kafka and Pulsar.

[0047] Furthermore, in step S12, for data sources that support Change Data Capture (CDC), the option to enable archived logs is set, and archived logs are configured for the data source. After enabling archived logs, the corresponding data source has streaming mode data capture capabilities and can be used as a data source for both streaming and batch modes, thereby achieving a unified streaming and batch data asset structure.

[0048] As an example, archived logs can be MySQL's binlog.

[0049] Furthermore, in step S13, the data assets may include offline data assets (bounded data assets) and real-time data assets (unbounded data assets). By connecting to the data source through a standard interface, performing a metadata query, and automatically obtaining the table structure in the data source, the bounded data assets are obtained.

[0050] As an example, standard interfaces could be JDBC / ODBC, etc.

[0051] As an example, the table structure may include field names, data type of each field, length / precision, whether it is nullable, primary key / foreign key constraints, etc.

[0052] As an example, you can set up streaming mode and batch mode to share offline data assets.

[0053] Furthermore, in step S14, for data from message queues or real-time log streams, users need to manually set field mapping relationships, data types, parsing rules, etc., to obtain real-time data assets.

[0054] Furthermore, by integrating offline and real-time data assets to obtain unified metadata, stream processing and batch processing can be made to share the same source and data structure. Whether it is real-time stream processing or offline batch processing, the same metadata is used, thus eliminating the problem of inconsistent data retrieval standards from the source.

[0055] As an example, the unified metadata can share a set of asset names, identifiers, field names, data types, constraints, business attributes, storage indexes, etc. The unified metadata also has its own unique attributes, but is stored in a unified manner.

[0056] As an example, only real-time input sources or real-time output sources can be selected for real-time data assets.

[0057] In one example, when the data source is MySQL, if archived logs are enabled and the data assets defined by the data source are referenced, the data assets can be selected in both real-time and offline input on the switch's graphical orchestration interface, thus achieving the goal of sharing the data asset structure between batch and stream operations.

[0058] In step S2, please refer to Figure 1 In step S2, a data stream configuration instruction is received, and the operating mode is set according to the data stream configuration instruction. The operating mode includes batch mode and stream mode.

[0059] Specifically, the system receives data stream configuration instructions from the user, creates a new data stream through a graphical orchestration interface, and sets basic attributes in the new data stream pop-up form according to the data stream configuration instructions. The basic attributes may include data stream name, description, running mode, execution engine, etc.

[0060] As an example, the running modes can include batch mode and streaming mode, which share a common graphical orchestration interface.

[0061] As an example, batch mode can choose either the Spark or Flink engine.

[0062] As an example, the streaming mode can select the Flink engine.

[0063] In step S3, please refer to Figure 1 In step S3, in the graphical arrangement interface, the supported operator nodes are dynamically filtered and displayed according to the running mode. The operator nodes are dragged to the canvas and connected and configured to obtain the data processing stream.

[0064] Specifically, the graphical orchestration interface automatically filters based on the operating mode, displaying only the operator nodes supported in that operating mode. Users drag and drop the desired operator nodes from the filtered list onto the canvas, with the input source as the start node, the output source as the end node, and other operator nodes as intermediate nodes. A directed acyclic data processing flow is constructed by connecting these nodes, and a corresponding record is generated in the data flow list.

[0065] As an example, the graphical orchestration interface can be built based on an open-source node framework, where front-end and back-end developers can set up integrated stream and batch conversion operator nodes and all orchestration logic. Compared to existing graphical orchestration interfaces that only provide an entry point for advanced SQL writing, requiring users to manually write complex SQL statements, this application encapsulates all commonly used basic transformations, joins, aggregations, mathematical functions, data quality operations, etc., into visual nodes, providing them all to users for visual configuration, thus lowering the development threshold.

[0066] Figure 3 This is a blank state diagram for the batch mode of the graphical layout interface. Figure 4 This is a diagram showing the editing status of the graphical orchestration interface in flow mode. As you can see, the left side of the graphical orchestration interface displays a node model that can automatically filter nodes based on the running mode; the top side displays the title, showing the current data flow's running mode in real time, and allows operations such as copying, sharing, viewing historical version records, viewing execution logs, renaming, and deleting; the center is the canvas, where you can arrange node layouts and design data flows.

[0067] As an example, the graphical orchestration interface also allows users to configure variables, perform one-click automatic layout, set parameters, and perform one-click debugging. One-click automatic layout provides a clean and intuitive data flow diagram, while one-click debugging allows for quick verification of the correctness of the entire processing logic, thus achieving efficient development and seamless operation and maintenance.

[0068] As an example, configure the input and output sources by selecting the corresponding data assets for each.

[0069] As an example, operator nodes can be abstracted as interfaces. When a task is running, the system will dynamically select and call the corresponding underlying implementation code based on preset running modes and execution engine parameters.

[0070] As an example, the execution engine could include Spark or Flink, etc.

[0071] As an example, operator nodes can include: input, output, basic transformation, join, aggregation, mathematical functions, data quality, and advanced operators; input includes offline input sources and real-time input sources, output includes offline output sources and real-time output sources, basic transformation includes field selection, adding constants, and deduplication, join includes join, dimension table association, and union, aggregation includes statistics and window statistics, mathematical functions include calculations and functions, and advanced operators include JSON parsing, advanced SQL, and repartitioning.

[0072] As an example, node types can include: stream-batch shared type, stream processing type, and batch processing type. Among them, the stream-batch shared type uses the same operator node conversion logic, and can run in both stream and batch modes.

[0073] As an example, you can set the offline input source to be shared by both stream and batch, the real-time input source to stream processing, the offline output source to be shared by both stream and batch, the real-time output source to stream processing, field selection to be shared by both stream and batch, adding constants to be shared by both stream and batch, deduplication to be batch processing, joins to be shared by both stream and batch, dimension table joins to be batch processing, unions to be shared by both stream and batch, statistics to be shared by both stream and batch, window statistics to be stream processing, calculations to be shared by both stream and batch, functions to be shared by both stream and batch, data quality to be batch processing, JSON parsing to be batch processing, advanced SQL to be shared by both stream and batch, and repartitioning to be batch processing. See Table 1 for details on shared operator nodes. As shown in Table 1, 70% of the transformation processing operators are completely shared in both stream and batch modes, achieving a unified development experience and greatly reducing the complexity of development and maintenance.

[0074] Table 1. Explanation of shared operator nodes

[0075]

[0076] In step S4, please refer to Figure 1 In step S4, based on the operating mode and the data processing flow, the logic of the graphical orchestration interface is automatically converted into the execution logic corresponding to the computing engine, and the execution logic is submitted to the corresponding engine cluster.

[0077] Specifically, based on the operating mode and the data processing flow, the system automatically converts the logic of the graphical orchestration interface into the execution logic corresponding to the underlying computing engine and submits it to the corresponding engine cluster.

[0078] As an example, depending on the running mode, streaming tasks can be converted into the corresponding Flink Table API and deployed to the corresponding Flink cluster; batch tasks can be converted into Flink Table API or Spark Dataset API and deployed to the corresponding Flink cluster or Spark cluster.

[0079] In some embodiments, the logic of the graphical orchestration interface is automatically converted into the execution logic corresponding to the computing engine according to the operating mode and the data processing flow. It may also include converting the streaming mode into the batch mode by saving a new process when it is necessary to reuse the existing streaming conversion logic for batch processing. If the streaming conversion logic is consistent with the batch conversion logic, the streaming mode is converted into the batch mode by saving a new process.

[0080] As an example, to create a new process, you only need to readjust the input and output sources. If the transformation logic of the intermediate nodes remains unchanged, there is no need to reconfigure the transformation logic of the intermediate operator nodes.

[0081] As an example, a new process is created, including: opening the data stream list, selecting the stream mode record to be converted, changing the stream mode to batch mode in the pop-up basic properties dialog box, and obtaining a new batch processing task graphical orchestration interface; deleting the original real-time input source and real-time output source in the new batch processing task graphical orchestration interface, dragging and dropping the offline input source and offline output source onto the canvas, and configuring the offline input source and offline output source; confirming whether the conversion logic of all operator nodes is applicable in batch mode. If the conversion logic is consistent with the stream mode, there is no need to configure the intermediate operator nodes in the process. If the conversion logic is inconsistent with the stream mode, update the operator nodes to batch processing type, connect and configure them to complete the stream-to-batch conversion.

[0082] Specifically, open the data stream list, select the stream mode record to be converted, select the copy option, and a dialog box will pop up displaying the data stream name, description, running mode, and execution engine basic properties. You can modify the corresponding basic properties and change the stream mode to batch mode. Click OK to generate a new batch processing task graphical orchestration interface. In the new batch processing task graphical orchestration interface, delete the original real-time input source and real-time output source, drag and drop the offline input source and offline output source onto the canvas, and configure the offline input source and offline output source. The user needs to confirm whether the conversion logic of all other intermediate operator nodes is applicable in batch mode. If the conversion logic is consistent with the stream mode, there is no need to configure intermediate operator nodes. If the conversion logic is inconsistent with the stream mode, update the operator nodes to batch processing type, connect and configure them, and finally complete the stream-to-batch conversion, realizing a fast stream-to-batch conversion.

[0083] In step S5, please refer to Figure 1 In step S5, the data processing stream is executed on the corresponding engine cluster to complete task execution and resource allocation.

[0084] Specifically, if the running mode is streaming mode, a persistent task is generated on the Flink cluster and runs continuously, and fault tolerance is achieved through the fault tolerance mechanism; if the running mode is batch mode, the scheduler triggers the process according to a preset period, and then the configured data processing stream is submitted to the Spark or Flink cluster for execution once, and the computing resources are released after the processing is completed.

[0085] As an example, the fault tolerance mechanism can be a Flink Savepoint or Checkpoint mechanism. If a task fails, it can be recovered through a checkpoint or checkpoint, ensuring that real-time tasks can be quickly recovered and continue to be processed when a failure occurs, thus guaranteeing processing continuity.

[0086] As an example, the preset period can be daily, hourly, or 1 a.m. daily, etc.

[0087] As an example, the scheduler could use the Quartz scheduling framework.

[0088] Furthermore, for batch processing tasks, the cluster dynamically requests and allocates a certain number of computing execution units (Executors) for the task during execution and releases them after the task is completed, returning them to the cluster for use by other tasks, thereby improving the overall utilization of cluster resources. For stream processing tasks, a fixed resource allocation strategy is adopted, which permanently occupies a certain number of slots when the stream task starts. As long as the task does not stop, these resources are occupied to ensure low latency and high availability of stream processing tasks and avoid processing delays caused by resource contention.

[0089] As an example, a slot is the smallest unit of resource for task execution in a Flink cluster.

[0090] As an example, the execution engine can be configured to be deployed in standalone mode.

[0091] This application's integrated stream and batch data processing method utilizes a unified graphical orchestration configuration. Stream and batch modes, as well as different data processing engines, all share the same orchestration page. Over 70% of the transformation processing operator nodes are shared across both stream and batch processes, reducing the difficulty of data processing for developers and the complexity of user learning and configuration. It also ensures consistency of the same business logic across stream and batch processes, significantly improving development efficiency and maintainability. By constructing unified metadata, it ensures consistency in data definitions between stream and batch processing, eliminating discrepancies in processing results caused by different data sources or structures. Furthermore, by automatically converting the graphical orchestration logic into the underlying computing engine's executable code and performing intelligent resource scheduling and task execution based on the running mode, it simplifies operation and maintenance and optimizes cluster resource utilization. This method enhances real-time stream processing capabilities and effectively addresses the core pain points of traditional solutions, such as separated stream and batch architectures, high development and maintenance costs, and inconsistent data results, providing users with an efficient, unified, and reliable data processing method.

[0092] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0093] In another embodiment, this application also provides a batch processing system, the batch processing system comprising: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the batch processing method described in the above embodiment.

[0094] In another embodiment, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the various steps of the batch processing method provided in the above embodiments.

[0095] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The computer programs may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0096] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data electronic devices), or computing systems that include middleware components (e.g., application electronic devices), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0097] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] Although this application has been disclosed above with reference to embodiments, it is not intended to limit this application. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of this application.

Claims

1. A method for integrated batch and stream data processing, characterized in that, Includes the following steps: Set up data sources and data assets, and define unified metadata; Receive data stream configuration instructions, and set the operating mode according to the data stream configuration instructions. The operating mode includes batch mode and stream mode. In the graphical orchestration interface, the supported operator nodes are dynamically filtered and displayed according to the running mode. The operator nodes are dragged to the canvas and connected and configured to obtain the data processing stream. Node types include: stream-batch shared type, stream processing type, and batch processing type. The stream-batch shared type uses the same operator node conversion logic and has the same conversion logic. Based on the operating mode and the data processing flow, the logic of the graphical orchestration interface is automatically converted into the execution logic corresponding to the underlying computing engine, and the execution logic is deployed to the corresponding engine cluster. The data processing stream is executed on the corresponding engine cluster to complete task execution and resource allocation.

2. The batch processing method for integrated data processing according to claim 1, characterized in that, Configure data sources and data assets, and define unified metadata, including: Set the data source; For data sources that support change data capture, enable the archive log option; Obtain the table structure from the data source to obtain offline data assets; Manually add real-time data structures, define real-time data assets, and obtain unified metadata.

3. The batch processing method for integrated data processing according to claim 2, characterized in that, Receive data stream configuration instructions and set the running mode according to the data stream configuration instructions. The running mode includes batch mode and stream mode. The process includes: receiving data stream configuration instructions from the user, creating a new data stream through a graphical orchestration interface, and setting basic attributes in the new data stream pop-up form according to the data stream configuration instructions. The basic attributes include data stream name, description, running mode, and execution engine.

4. The batch processing method for integrated data processing according to claim 3, characterized in that, The operating modes include batch mode and streaming mode, which share a common set of graphical orchestration interfaces for data flow; the execution engine is deployed in Standalone mode.

5. The batch processing method for integrated streaming data processing according to claim 4, characterized in that, Operator nodes include: input, output, basic transformation, join, aggregation, mathematical function, data quality, and advanced operators.

6. The batch processing method for integrated streaming data processing according to claim 5, characterized in that, It also includes the ability to convert existing stream processing conversion logic to batch processing when the stream processing conversion logic is consistent with the batch processing conversion logic, by saving a new process to convert the stream mode to the batch mode.

7. The batch processing method for integrated streaming data processing according to claim 6, characterized in that, Save a new process, including: opening the data stream list, selecting the stream mode record to be converted, changing the stream mode to batch mode in the pop-up basic properties dialog box, and obtaining a new batch processing task graphical orchestration interface; deleting the original real-time input source and real-time output source in the new batch processing task graphical orchestration interface, dragging and dropping the offline input source and offline output source onto the canvas, and configuring the offline input source and offline output source; confirming whether the conversion logic of all operator nodes is applicable in batch mode. If the conversion logic is consistent with the stream mode, there is no need to configure the intermediate operator nodes in the process. If the conversion logic is inconsistent with the stream mode, update the operator nodes to batch processing type, connect and configure them to complete the stream-to-batch conversion.

8. A batch processing system for integrated batch and stream data processing, characterized in that, The integrated batch and stream data processing system includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the integrated batch and stream data processing method as described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the batch processing method according to any one of claims 1 to 7.

Images