Real-time data stream processing system

Abstract

The invention discloses a real-time data stream processing system and a real-time data stream processing method. According to the system, a standalone version deployment structure or a cluster version deployment structure is formed by a component factory, a canvas arrangement and a scheduling executor; the component factory is used for producing, constructing and publishing components; the method has the advantages that complex large-scale parallel tasks are supported, resources are effectively managed, the method is suitable for embedded edge computing equipment, hardware horizontal transverse extension, clustering deployment and multi-language plug-in extension are supported, any language is supported to compile a self-defined node assembly and a memory DAG model, C + + / Python algorithm ecology is opened, real-time stream processing is oriented, and the method is suitable for the embedded edge computing equipment. The invention relates to low-code visual programming dragging.

Description

Technical Field

[0001] The present invention relates to the field of real-time data stream processing. Background Art

[0002] At a time when AI, reasoning engines, autonomous driving, the Internet of Things, financial decision-making, robotics and industrial control applications are emerging, data stream processing is an important task. Using DAG, we can clearly express the dependencies between tasks, so that we can optimize and execute tasks in parallel more efficiently during task scheduling. It helps organize and manage complex tasks and logic, and improves the scalability, maintainability and performance of the system.

[0003] For example, the public document with publication number CN118012586A, publication date 2024-05-10, and patent name "A Scheduling and Orchestration Method for Industrial Warehousing Internet Data Streams" discloses the field of artificial intelligence Internet of Things. The specific scheme includes the following steps: Step 1. Use an image recognition algorithm to analyze the acquired material images, and use Ai rflow to decompose the steps of the image recognition algorithm into a series of ordered tasks, where each task represents a step in the image recognition algorithm; Step 2. Use DAG to define the dependencies and execution order between tasks; Step 3. In the visual user interface, monitor the execution status of the task, view logs and perform debugging.

[0004] by Fig.12 Taking the industrial control process as an example, a complete loop control strategy needs to go through a series of steps such as industrial data access, data preprocessing, algorithm calculation, and output reverse writing equipment. Among them, the areas that can be refined are: first, DCS data, time series database, local file or third-party data source input, involving many types of equipment and protocols. Second, different algorithms are used in different scenarios, including data filtering, vision, machine learning, MPC, PID, AI large models and other algorithms, with different parameters. The implementation of various algorithm combinations is complex.

[0005] by Fig.13 Take streaming media processing as an example, a complete audio and video processing process includes media data access collection, decoding, various bitstream data processing, including image beauty and sharpening effects, sound recognition and replacement, subtitle generation and synchronization, video encoding, streaming and sending, and other complex processes.

[0006] by Fig.14 Taking autonomous driving as an example, a vehicle's decision-making process will include point cloud image processing and simultaneous and parallel detection of multiple targets such as obstacles, pedestrians, traffic lights, lane lines, etc., and ultimately make an autonomous driving decision.

[0007] Common processing methods are Kettle, Node-RED, AI CS and Taier products

[0008] Kettle, also known as Pentaho Data Integration (PTI), is an open source ETL (Extract, Transform, Load) tool developed and maintained by Pentaho, a provider of business intelligence and data integration solutions. Kettle is designed to help users easily process and transform data to support various data integration, data warehouse and business intelligence needs. All three products implement visual data task orchestration, parallel task scheduling,

[0009] Node-RED is a process orchestration tool designed to simplify the development of IoT (Internet of Things) applications. It provides a web-based visual interface that allows users to create processes by dragging and dropping. These processes can include tasks ranging from collecting data from sensors to performing various processing operations to controlling output devices.

[0010] AI CS is a control and optimization product of Alibaba Cloud Industrial Brain's intelligent manufacturing platform. It supports visual, drag-and-drop rapid construction of real-time control processes. It uses the control components and feature processing components provided by the platform, as well as the Server-less function development model to develop customized computing logic, and uses Python microservices to integrate into the entire real-time control process.

[0011] Taier is one of Kangaroo Cloud's open source projects and is a distributed visual DAG task scheduling system. It aims to reduce ETL development costs and improve the stability of the big data platform. Big data developers can directly develop business logic in Taier without having to worry about the complex dependencies of tasks and the architecture implementation of the underlying big data platform, and focus more on the business. Based on the plug-in architecture design, users develop tasks in the interface and submit them for operation. The submission and operation plug-ins are divided into worker-plugins and datasource-plugins. It adopts the decentralized distributed mode of Zookeeper, supports the scheduling of big data jobs Spark and Flink, and visualizes the workflow configuration.

[0012] The above processing method has the following problems:

[0013] 1. Kettle was originally designed to simplify the data processing process and provide an easy-to-use graphical interface to design and execute ETL tasks. Kettle's shortcomings are obvious.

[0014] Limited parallelism support: It is more suitable for processing linear tasks rather than parallel tasks. Although it has a certain degree of parallel execution capabilities, its parallel performance is relatively weak. For complex tasks that require large-scale parallel processing, Kettle's capabilities may be insufficient.

[0015] Inadequate resource management: Kettle may encounter challenges in resource management when scheduling parallel tasks. Especially when processing large-scale data, it is not easy to scale horizontally. The computing resources of a single machine may become a bottleneck, and it is impossible to utilize distributed computing resources.

[0016] 2. Node-RED is a process orchestration tool used to connect hardware devices, APIs, and online services, and to create applications through simple visual programming. Although Node-RED has advantages in many aspects, it also has some disadvantages.

[0017] Extensibility limitations: Although Node-RED provides many built-in nodes and community-contributed nodes, you may find that the functionality is insufficient when dealing with specific needs. In this case, you need to write custom nodes, but for some complex needs, writing custom nodes may not be simple.

[0018] Performance issues: Node-RED is implemented using the NodeJs weakly typed interpreted language. For algorithm applications that require complex logic, a large number of control flows, or high-concurrency real-time application scenarios, traditional strongly typed programming languages ​​are more suitable for complexity and real-time performance.

[0019] 3. Alibaba AI CS uses Python microservices to implement server less function development mode to implement component algorithm services. K8S POD is used to manage the life cycle of task nodes. Java is used to develop DAG to schedule each algorithm microservice to implement concurrent task execution. Although AI CS is excellent, it still has shortcomings in the field of real-time processing.

[0020] Limited by Python's scalability: Although AI CS implements the server-side function development model, users can only write custom nodes in Python scripts. It is impossible to complete multi-language or other programming methods in other environments.

[0021] Waste of hardware resources: Starting Python microservices in the cloud POD management mode is cumbersome. When the number of tasks reaches a certain scale, the number of containers will become huge. When the hardware resources at the edge are fixed, it will cause a huge waste of resources, requiring a higher-configuration server, and cannot be deployed in some embedded hardware machines.

[0022] Microservice performance bottleneck: AICS uses microservice calls to communicate between Java and Python, and the content of the communication must be serialized and transmitted on the network. This causes Alibaba AI CS to suffer performance loss due to network transmission when transmitting complex types of data such as pictures, videos, or large matrices between components. It cannot meet scenarios with high real-time requirements.

[0023] 4. Taier is mainly used in data ETL application scenarios, supports the scheduling of big data jobs Spark and Flink, and visualizes workflow configuration. Its disadvantages are obvious in embedded applications.

[0024] 1. Single data ETL scenario: It can only be used in data ETL scenarios to process data from various data sources, but cannot handle complex real-time data type scenarios such as docking with existing AI algorithms and media streams.

[0025] 2. Cloud technology: It takes up a lot of resources and cannot be applied in the field of embedded edge hardware.

[0026] 3. Multi-language expansion: Single Java programming cannot be connected to other programming languages ​​except network interface calls, and there are limitations in processing real-time streams. Summary of the invention

[0027] The technical problem to be solved by the present invention is to realize a new data stream processing method, which is a universal low-code visual orchestration method in business applications, can reuse various algorithm application fields, and achieve the most efficient task processing method and computer hardware resource utilization in terms of performance.

[0028] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a real-time data stream processing system and method, the system is composed of a component factory, a canvas orchestration, and a scheduling executor to form a stand-alone version deployment structure or a cluster version deployment structure;

[0029] The component factory is used to produce, build and publish components;

[0030] The canvas arrangement allows users to drag components in the canvas to configure connections and build parallel tasks;

[0031] The scheduling executor is a unit responsible for running the canvas process;

[0032] The scheduling executor of the stand-alone deployment structure outputs signals to multiple canvases, and the component factory and canvases are connected to the scheduling executor;

[0033] The cluster version deployment structure is provided with a master node connected to a plurality of slave nodes respectively, the master node is composed of a component factory and a canvas arrangement, and the master node is composed of a scheduling executor and a canvas.

[0034] The canvas is a complete task flow, and the component is a single task node in the flow.

[0035] The functional modules of the stand-alone version deployment structure are deployed on the same server;

[0036] When the number of canvas processes exceeds the resource load limit of a single machine, the scheduling executor is deployed on other slave nodes for cluster expansion.

[0037] A control list tree is constructed by using a component factory and building components. The control list tree includes input boxes, selection boxes, Python or basic input and output controls. A component control module is constructed based on the control list tree. The process of each control is selectively configured. The process configuration includes the control's title, style, and action trigger interface information. The control after the process configuration is provided as a specified version for canvas layout.

[0038] The components in the orchestration canvas are connected through ports, and the port output of the previous node is used as the port input of the next node to trigger the canvas cycle adjustment through a timer.

[0039] The canvas is provided with an executor for canvas operation, which is a parallel scheduling engine based on the DAG directed graph. The producer-consumer mode is used to perform breadth-first traversal on the scheduled tasks, and the running tasks of each node in the graph are produced and sent to a pipeline queue by a message mechanism, so that the scheduling engine acts as a consumer to call back the task methods registered in each node in the task engine.

[0040] The executor of the canvas operation adopts the DAG mode to decouple the components from the canvas task scheduling. The system provides a plug-in mechanism and extension points, and the scheduling engine provides a general interface and abstract layer.

[0041] The scheduling engine is loaded with a graph node plug-in loader, which is bridged with other programming languages ​​through C++ to store all component information in the same canvas in the same process memory.

[0042] This system supports mixed programming with other languages, including but not limited to Python, Java, Lua, and Go.

[0043] The advantages of the present invention are that it supports complex large-scale parallel tasks, effectively manages resources, is suitable for embedded edge computing devices, supports hardware-level horizontal expansion, clustered deployment, multi-language plug-in expansion, supports custom node components written in any language, memory DAG model, opens up the C++ / Python algorithm ecosystem, is oriented to real-time stream processing, and uses low-code visual programming and drag-and-drop. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The following is a brief description of the contents expressed in each figure in the specification of the present invention:

[0045] Figure 1 This is a stand-alone version deployment diagram;

[0046] Figure 2 This is a diagram of cluster version deployment;

[0047] Figure 3 Edit the build schematic for the component;

[0048] Figure 4 This is a schematic diagram of the component factory;

[0049] Figure 5 Arrange schematics for the canvas;

[0050] Figure 6 This is a schematic diagram of the DAG canvas scheduling engine;

[0051] Figure 7 It is a schematic diagram of the component plug-in node;

[0052] Figure 8 Calling the logical diagram for the business component plug-in;

[0053] Fig. 9 This is a schematic diagram of the plugin loader;

[0054] Fig.10 Schematic diagram for heterogeneous language plug-in expansion;

[0055] Fig.11 This is a diagram of a custom Python script component;

[0056] Fig.12 It is a schematic diagram of industrial control process;

[0057] Fig.13 It is a schematic diagram of the streaming media processing process;

[0058] Fig.14 Schematic diagram of the autonomous driving process. DETAILED DESCRIPTION

[0059] The following is a further detailed description of the specific implementation methods of the present invention, such as the shape, structure, relative position and connection relationship between the various components involved, the function and working principle of each part, the manufacturing process and operation method, etc., through the description of the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0060] The present invention decomposes the complex data processing process into independent, splittable and reusable component operators, organizes the task processing process through a visual arrangement method, and finally processes the task calculation logic in parallel through the DAG task scheduling engine. In business applications, a more general low-code visual arrangement method is implemented, which reuses various algorithm application fields; in terms of performance, the most efficient task processing method and computer hardware resource utilization are achieved.

[0061] The present invention adopts DAG directed graph as the basic data structure, designs a parallel task scheduling engine, and stipulates a set of component rendering DSL for component visualization. The dynamic link library technology is used to realize the dynamic loading and calling of each component. The core technologies used include: DAG parallel task scheduling, dynamic library plug-in technology, C++ / Python hybrid programming, multi-process distribution and other key technologies.

[0062] This method is described in the UML unified modeling language. Depending on whether the system needs cluster deployment, two implementation plans are provided: stand-alone version and cluster version. Figure 1 , 2 As shown. The system mainly includes three core functional components: component factory, canvas layout, and scheduling executor. The present invention refers to a complete task process as a canvas, a single task node in the process as a component, and the unit responsible for the operation of the canvas process is called a scheduling executor. The component factory is mainly used for production-building-releasing components. Canvas layout allows users to drag components in the canvas for wiring configuration and build parallel tasks. The scheduling executor is responsible for the canvas operation life cycle.

[0063] The system mainly includes three core functional components: component factory, canvas orchestration, and scheduling executor. In this article, a complete task process is called a canvas, a single task node in the process is called a component, and the unit responsible for the operation of the canvas process is called a scheduling executor. The component factory is mainly used for production-building-releasing components. Canvas orchestration allows users to drag components in the canvas for wiring configuration and build parallel tasks. The scheduling executor is responsible for the canvas operation life cycle.

[0064] The stand-alone version deploys the three functional components on the same server. If the number of canvas processes exceeds the resource load limit of a single machine, the scheduling executor can be deployed on other slave nodes to achieve cluster expansion. The following will introduce the component factory, canvas orchestration, and scheduling executor in detail.

[0065] When users use the component factory to edit and build components, they will enter Figure 3 In the page shown, Figure 3 The control list tree at the mark 1 contains input boxes, selection boxes, Python or basic input and output controls. Drag the control at the mark 1 to the center area to form the following Figure 3 In the component controls at mark 4, mark 5, mark 6, mark 7, mark 8, mark 9, and then through Figure 3 At mark 10, each control process is configured including the control's title, style, action trigger interface, and other information. Figure 3 Edit the input and output port information of the component at mark 2 and mark 3, and associate the specified control with the port at the control configuration. The edited component can be published in a specified version for canvas arrangement. Figure 4 , 5 shown.

[0066] exist Figure 5 Each component in the orchestrated canvas is connected through ports, and the port output of the previous node is used as the port input of the next node to trigger the canvas cycle scheduling through a timer. This system designs a parallel scheduling engine based on a DAG directed graph as the executor of the canvas operation. The internal implementation logic is as follows Figure 6 shown.

[0067] This system opens up an independent process for the running canvas, uses the producer-consumer model to perform breadth-first traversal of the scheduled tasks, and produces the running tasks of each node in the graph and sends them to a pipeline queue using a message mechanism, so that the scheduling engine can act as a consumer to call back the task methods registered in each node in the task engine. The DAG model decouples components from canvas task scheduling. The platform provides plug-in mechanisms and extension points, so that components can be more flexibly integrated into the canvas parallel scheduling system. The scheduling engine provides a common interface and abstraction layer to achieve seamless interaction between components and scheduling, reducing the impact of components on the operation of the existing platform when they are put on the shelf. Developers can expand system functions more efficiently, reduce code redundancy and repeated development, and improve development efficiency.

[0068] The system's plug-in standard interface implementation solution divides the components on the shelf into two abstract concepts: business components and graph node components. Figure 7 As shown. The business component mainly handles the construction logic of the canvas when the editing state is not running. The system is designed in a restful style, such as the custom interface standard in the format of / ap i / canvas / cmpt / {name} / {version} / {method}, to extract the component name, version, and method information. Through the dynamic link library, the dynamic link library is named by splicing the component name and the component version. To load a plug-in, you only need to call the get_plugin_lib_name() method through Plug inUtil to obtain the path of the dynamic library, construct a Plug in Handler object, and call the load method to load the plug-in. The business component plug-in scheduling logic is as follows: Figure 8shown.

[0069] Generally, microservice platforms use network requests to call algorithm services. The disadvantage of this is that the real-time operation of the algorithm cannot be guaranteed, and there are network serialization and deserialization overheads. Or when the number of algorithms increases dynamically, it causes an explosion in the number of services or waste of resources. When other data types such as pictures and videos need to be transmitted between components, a large amount of system network bandwidth is occupied, the scheduling delay is increased, and it is not suitable for the field of real-time stream processing.

[0070] To solve the above difficulties, this system designs a graph node plug-in loader to load the algorithm into the DAG graph scheduling engine. The design interface is as follows Fig. 9 As shown. The plug-in loader of this system supports the use of algorithms in any programming language, and the main method used is the mode of bridging with other programming languages ​​through C++. The advantage of using C++ is that all component information in the same canvas is stored in the same process memory. Information is transmitted between components, and data is transmitted directly through memory to avoid network overhead and time delay. When processing complex data such as images and videos, there is no need to go through serialization operations, and pointer objects are transmitted between them, which is conducive to the application of this system in the field of real-time stream processing and embedded fields.

[0071] In addition, this system supports mixed programming with other languages, such as Python, Java, Lua, Go and other languages, which greatly improves the functional scalability of the platform. Fig.10 As shown. In today's Python artificial intelligence field, C++ can open up the Python ecosystem. And it can make full use of the hardware CUDA parallel programming capabilities to maximize the efficiency of DAG scheduling computing performance. In addition to expanding other programming languages ​​on the code interface, this system implements low-code online writing for weakly typed scripting languages ​​in the canvas to improve the flexibility of system applications, such as Fig.11 shown.

[0072] The present invention is described above by way of example in conjunction with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A real-time data stream processing system and method, characterized in that: The system consists of component factories, canvas orchestration, and scheduling executors to form a stand-alone version deployment structure or a cluster version deployment structure; The component factory is used to produce, build and publish components; The canvas arrangement allows users to drag components in the canvas to configure connections and build parallel tasks; The scheduling executor is a unit responsible for running the canvas process; The scheduling executor of the stand-alone deployment structure outputs signals to multiple canvases, and the component factory and canvases are connected to the scheduling executor; The cluster version deployment structure is provided with a master node connected to a plurality of slave nodes respectively, the master node is composed of a component factory and a canvas arrangement, and the master node is composed of a scheduling executor and a canvas.

2. The real-time data stream processing system according to claim 1, characterized in that: The canvas is a complete task flow, and the component is a single task node in the flow.

3. The real-time data stream processing system according to claim 1 or 2, characterized in that: The functional modules of the stand-alone version deployment structure are deployed on the same server; When the number of canvas processes exceeds the resource load limit of a single machine, the scheduling executor is deployed on other slave nodes for cluster expansion.

4. The real-time data stream processing system according to claim 3, characterized in that: A control list tree is constructed by using a component factory and building components. The control list tree includes input boxes, selection boxes, Python or basic input and output controls. A component control module is constructed based on the control list tree. The process of each control is selectively configured. The process configuration includes the control's title, style, and action trigger interface information. The control after the process configuration is provided as a specified version for canvas layout.

5. The real-time data stream processing system according to claim 4, characterized in that: The components in the orchestration canvas are connected through ports, and the port output of the previous node is used as the port input of the next node to trigger the canvas cycle adjustment through a timer.

6. The real-time data stream processing system according to claim 5, characterized in that: The canvas is provided with an executor for canvas operation, which is a parallel scheduling engine based on the DAG directed graph. The producer-consumer mode is used to perform breadth-first traversal on the scheduled tasks, and the running tasks of each node in the graph are produced and sent to a pipeline queue by a message mechanism, so that the scheduling engine acts as a consumer to call back the task methods registered in each node in the task engine.

7. The real-time data stream processing system according to claim 6, characterized in that: The executor of the canvas operation adopts the DAG mode to decouple the components from the canvas task scheduling. The system provides a plug-in mechanism and extension points, and the scheduling engine provides a general interface and abstract layer.

8. The real-time data stream processing system according to claim 7, characterized in that: The scheduling engine is loaded with a graph node plug-in loader, which is bridged with other programming languages ​​through C++ to store all component information in the same canvas in the same process memory.

9. The real-time data stream processing system according to claim 8, characterized in that: This system supports mixed programming with other languages, including but not limited to Python, Java, Lua, and Go.

Images