A stream processing method and device
By configuring and optimizing the DAG diagram of streaming processing tasks in the graphical interface, the inconvenience caused by DAG changes caused by SQL changes in streaming computing to debug and problem positioning is solved, and user-friendly streaming processing configuration and monitoring are realized. Improves flexibility and version compatibility for stream processing.
Patent Information
- Application Number
- CN202010676621.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-14
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-07-14
AI Technical Summary
In the existing streaming calculation, the SQL calculation part is abstracted into DAG. The changes in SQL lead to changes in DAG, which users cannot perceive, causing inconvenience to debugging and problem positioning.
By receiving configuration instructions for configuring stream processing tasks in the graphical interface, configuring the initial directed acyclic graph DAG diagram, and optimizing the initial DAG diagram by matching higher-order operators, obtaining the target DAG diagram and monitoring the flow processing The task progress, the abstract basic operator is a higher-order operator or a custom operator, and optimizes processing, including error processing, log processing and signaling control.
Users can complete stream processing task configuration without encoding, monitor the status and progress of each node on the DAG graph, simplify debugging and problem positioning, improve the flexibility and version compatibility of the stream processing process, and reduce the amount of code development.
Smart Images

Figure CN111897693B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a stream processing method and device. Background Art
[0002] With the widespread application of big data technology in all walks of life, the demand for real-time processing of massive data is increasing. At the same time, the business logic of data processing is becoming more and more complex. Traditional batch processing methods and early streaming processing frameworks are becoming increasingly difficult to meet the increasingly stringent business requirements in terms of latency, throughput, fault tolerance, and ease of use.
[0003] There are many application scenarios for big data streaming computing. According to the three different dimensions of data generation method, data scale and technology maturity, there are financial banking applications, Internet applications and Internet of Things applications. At present, there are many mature real-time streaming processing systems in the industry. There are mature commercial products, as well as open source products from companies, organizations or individuals. Some of these products have been verified in production, while some are just prototypes, and there are too many interfaces open for calling. This product status has led to the fact that in the face of so many dazzling products, as front-line code developers, they face many difficulties and pressures in the early product selection and trial, and later product learning and use, which increases the learning curve of developers, and the calling process is too complicated. Therefore, the industry is faced with the problem of how to simplify existing streaming processing products.
[0004] Stream processing: Business system real-time operations generate events (messages / calls). After a series of processing and analysis, this process is completed in parallel on multiple computers. It looks like the events continuously flow through multiple computing nodes for processing, forming a streaming computing process.
[0005] A data processing method is proposed in the related art, which abstracts the SQL processing of data processing into DAG, and modifies the DAG graph as the SQL changes to meet the computing requirements of streaming computing applications that may change over time. The SQL computing part of streaming computing is abstracted into DAG, and the DAG changes caused by SQL changes are not perceived by users, which brings inconvenience to debugging and problem location.
[0006] Regarding the problem that the SQL computing part in the streaming computing is abstracted into DAG in the related technology, the DAG changes caused by the SQL changes are not perceived by users, which brings inconvenience to debugging and problem location. No solution has been proposed yet. Summary of the invention
[0007] The embodiments of the present invention provide a streaming processing method and device to at least solve the problem in the related art that the SQL computing part in the streaming computing is abstracted into DAG, and the DAG changes caused by the SQL changes are not perceived by the user, which brings inconvenience to debugging and problem location.
[0008] According to an embodiment of the present invention, a stream processing method is provided, comprising:
[0009] Configure an initial directed acyclic graph DAG according to the configuration instructions for configuring the stream processing task received in the graphical interface;
[0010] Optimizing the initial DAG graph by matching high-order operators to obtain a target DAG graph, wherein the target DAG graph has the same connection relationship as the initial DAG graph;
[0011] The progress of the stream processing task is monitored through the target DAG graph.
[0012] Optionally, the initial DAG graph is optimized by matching a high-order operator to obtain a target DAG graph including:
[0013] Abstracting the nodes in the initial DAG graph into basic operators;
[0014] By abstracting the extended functions on the basic operators obtained in the graphical interface into high-order operators, the initial DAG graph is optimized to obtain the target DAG graph; or
[0015] The initial DAG graph is optimized by using the high-order operator or the custom operator obtained in the graphical interface to obtain the target DAG graph.
[0016] Optionally, the initial DAG graph is optimized by abstracting the extended functions on the basic operator obtained in the graphical interface into a high-order operator to obtain the target DAG graph, including:
[0017] The initial DAG graph is optimized by abstracting the extended functions on the basic operator into an error handling operator and / or a general operator to obtain the target DAG graph;
[0018] The initial DAG graph is optimized by abstracting the extended functions on the basic operator into a log processing operator, a remote call operator and / or a signaling operator to obtain the target DAG graph, wherein the signaling operator is used to control the start and stop of the stream processing task.
[0019] Optionally, after optimizing the initial DAG graph by abstracting the extended functions on the basic operator into an error handling operator and / or a general operator to obtain the target DAG graph, the method further includes:
[0020] Calculate each node in the target DAG graph to obtain a calculation result;
[0021] Performing error processing on the calculation result according to the error processing operator to obtain a processing result; and / or
[0022] Information statistics are performed according to the statistical calculator.
[0023] Optionally, after performing error processing on the calculation result according to the error processing operator to obtain the processing result, the method further includes:
[0024] Determine the message type of the processing result;
[0025] Perform error message classification processing on the processing result according to the message type of the processing result;
[0026] The processing result is reported.
[0027] Optionally, performing error message classification processing on the processing result according to the message type of the processing result includes:
[0028] If the processing result is a recoverable error message, the processing result is delivered to the first node of the target DAG graph for re-flow calculation;
[0029] If the processing result is an unrecoverable error message, a message log of the processing result is stored, wherein the message log includes message content, error information, error operator and time information.
[0030] According to another embodiment of the present invention, there is further provided a stream processing device, comprising:
[0031] A configuration module, configured to configure an initial directed acyclic graph DAG according to a configuration instruction for configuring a stream processing task received in a graphical interface;
[0032] An optimization module, configured to optimize the initial DAG graph by matching high-order operators to obtain a target DAG graph, wherein the target DAG graph has the same connection relationship as the initial DAG graph;
[0033] A monitoring module is used to monitor the progress of the stream processing task through the target DAG graph.
[0034] Optionally, the optimization module includes:
[0035] An abstract submodule, used to abstract the nodes in the initial DAG graph into basic operators;
[0036] a first optimization submodule, configured to optimize the initial DAG graph by abstracting the extended functions on the basic operators obtained in the graphical interface into high-order operators to obtain the target DAG graph; or
[0037] The second optimization submodule is used to optimize the initial DAG graph through the high-order operator or custom operator obtained in the graphical interface to obtain the target DAG graph.
[0038] Optionally, the first optimization submodule includes:
[0039] An abstraction unit is used to optimize the initial DAG graph by abstracting the extended functions on the basic operators into error handling operators and / or system operators to obtain the target DAG graph;
[0040] An optimization unit is used to optimize the initial DAG graph by abstracting the extended functions on the basic operator into a log processing operator, a remote call operator and / or a signaling operator to obtain the target DAG graph, wherein the signaling operator is used to control the start and stop of the stream processing task.
[0041] Optionally, the device further comprises:
[0042] A calculation unit, used to calculate each node in the target DAG graph to obtain a calculation result;
[0043] an error processing unit, used to perform error processing on the calculation result according to the error processing operator to obtain a processing result; and / or
[0044] A statistical unit is used to perform information statistics according to the statistical calculator.
[0045] Optionally, the device further comprises:
[0046] A determination unit, used to determine the message type of the processing result;
[0047] A classification unit, configured to classify error messages of the processing results according to the message types of the processing results;
[0048] A reporting unit is used to report the processing result.
[0049] Optionally, the classification unit is also used for
[0050] If the processing result is a recoverable error message, the processing result is delivered to the first node of the target DAG graph for re-flow calculation;
[0051] If the processing result is an unrecoverable error message, a message log of the processing result is stored, wherein the message log includes message content, error information, error operator and time information.
[0052] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.
[0053] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0054] Through the present invention, an initial directed acyclic graph DAG diagram is configured according to a configuration instruction for configuring a stream processing task received in a graphical interface; the initial DAG diagram is optimized by matching high-order operators to obtain a target DAG diagram, wherein the target DAG diagram has the same connection relationship as the initial DAG diagram; the progress of the stream processing task is monitored through the target DAG diagram, which can solve the problem in the related art that the SQL calculation part in the streaming calculation is abstracted into DAG, and the DAG changes caused by the SQL changes cannot be perceived by the user, which brings inconvenience to debugging and problem location. By adopting a graphical stream processing configuration, the user can complete the configuration of the stream processing task without coding and learning costs, and can monitor the status and progress of each node on the DAG diagram during the stream processing process in terms of operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0056] Figure 1 is a hardware structure block diagram of a mobile terminal of a streaming processing method according to an embodiment of the present invention;
[0057] Figure 2 is a flow chart of a stream processing method according to an embodiment of the present invention;
[0058] Figure 3 is a schematic diagram of DAG transformation according to an embodiment of the present invention;
[0059] Figure 4is a block diagram of a stream processing device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0060] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.
[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0062] Example 1
[0063] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal of a streaming processing method according to an embodiment of the present invention, such as Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0064] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the streaming processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0065] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0066] In this embodiment, a streaming processing method running on the above mobile terminal or network architecture is provided. Figure 2 is a flow chart of a stream processing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0067] Step S202, configuring an initial directed acyclic graph DAG diagram according to the configuration instruction for configuring the stream processing task received in the graphical interface;
[0068] Step S204, optimizing the initial DAG graph by matching high-order operators to obtain a target DAG graph, wherein the target DAG graph has the same connection relationship as the initial DAG graph;
[0069] Step S206: monitor the progress of the stream processing task through the target DAG graph.
[0070] Through the above steps S202 to S206, the problem in the related technology that the SQL computing part in the streaming computing is abstracted into DAG, the DAG changes caused by the SQL changes cannot be perceived by the user, and it brings inconvenience to debugging and problem location can be solved. By adopting the graphical stream processing configuration, the user can complete the configuration of the stream processing task without coding and learning costs, and the status and progress of each node on the DAG graph during the stream processing process can be monitored in operation and maintenance.
[0071] Optionally, the above step S204 may specifically include:
[0072] The nodes in the initial DAG graph are abstracted into basic operators; the initial DAG graph is optimized by abstracting the extended functions on the basic operators obtained in the graphical interface into high-order operators to obtain the target DAG graph, and further, the initial DAG graph is optimized by abstracting the extended functions on the basic operators into error handling operators and / or system operators to obtain the target DAG graph; the initial DAG graph is optimized by abstracting the extended functions on the basic operators into log processing operators, remote call operators and / or signaling operators to obtain the target DAG graph, wherein the signaling An operator is used to control the start and stop of the stream processing task. Further, each node in the target DAG graph is calculated to obtain a calculation result; the calculation result is error processed according to the error processing operator to obtain a processing result. Specifically, if the processing result is a recoverable error message, the processing result is delivered to the first node of the target DAG graph for re-flow calculation; if the processing result is an unrecoverable error message, a message log of the processing result is stored, wherein the message log includes message content, error information, error operator and time information; and / or information statistics are performed according to the statistical operator; or
[0073] The initial DAG graph is optimized by using the high-order operator or the custom operator obtained in the graphical interface to obtain the target DAG graph.
[0074] In an optional embodiment, after performing error processing on the calculation result according to the error processing operator and obtaining the processing result, determining the message type of the processing result; performing error message classification processing on the processing result according to the message type of the processing result; and reporting the processing result.
[0075] This embodiment uses the category theory method to perform homomorphic mapping on the original DAG. After the transformation, the user input DAG connection relationship remains unchanged. This homomorphic mapping is simple to implement relative to the graph transformation algorithm, and is completely orthogonal to the graph transformation algorithm; it is more conducive to users to understand the flow processing logic in the system, and can locate problems more quickly during the debugging and task processing stages. Compared with the completely user-defined DAG, the method of optimizing DAG by superposition is more flexible and easier to ensure compatibility between versions; the start and stop operation of the operator is controlled by signaling, and there is no central management node during the operation of the operator to avoid single point failure; the flow of signaling messages is consistent with the flow of ordinary messages, and the processing logic of ordinary messages is fully reused, reducing the amount of code development. The different functions of the stream processing process are abstracted into operators, and the development of new functions corresponds to the development of an independent operator. The development is relatively independent, scalable, and the version compatibility is controllable. Users can configure and select a flexible combination of different extended functions (in the system or customized), and can quickly develop specific operators as needed to complete stream processing tasks in complex scenarios, including:
[0076] Supports task configuration, operation and maintenance subscription, and custom operator development and upload. Users configure the data flow of the task through the graphical interface, and select the required high-order operators and superposition order to complete the task flow processing logic. Operation and maintenance subscription, users can choose to subscribe to single or multiple task operation and maintenance information as needed, and push it to users through http / https / rmq / kafka and other methods. Custom operator development, this is to meet the specific needs of users in complex scenarios, and can be used in task configuration after uploading.
[0077] Users only need to configure the DAG graph of stream processing on the graphical interface to define their own stream processing tasks. The learning cost is low and the modification is convenient. The system changes the DAG according to the user's configuration, and the connection relationship of the transformed DAG remains unchanged. In terms of operation and maintenance, users can monitor the operating status of each DAG node in real time and understand the progress of real-time stream processing tasks. In order to meet the needs of specific scenarios, the system also provides users with custom operator templates. Users can modify the operator templates to quickly develop custom high-order operators to meet the needs of specific scenarios.
[0078] Operator management, including internal operators and external operators.
[0079] Each independent function in the system is abstracted into operators, including basic operators, high-order operators, and signaling operators. Basic operators include input and output, and SQL processing operators; high-order operators include error handling, statistical processing, retry processing, log processing, remote call RPC, signaling operators, etc. Another way to understand it is to abstract the nodes in the DAG graph into basic operators, and the extended functions on the operators are abstracted into high-order operators; signaling operators control the start and stop of tasks. Ordinary operators serve as the skeleton of stream processing to complete basic functions. Users can choose different high-order operators to enrich and optimize the entire stream processing link according to their needs.
[0080] Error handling operator: (1) Error message classification processing: Recoverable error messages are delivered to the first operator for re-circulation calculation, and irrecoverable errors store detailed message logs (message content, error information, error operator, time, etc.); (2) Error reporting: Push the message content and number of retries of failed processing.
[0081] Statistics calculator, used to summarize message statistics (arrival time, processing time, time hotspot, processing results), consumption progress, task processing re-information, etc.
[0082] Retry processing (different retry strategies can be selected), log processing (push message processing log, etc.).
[0083] External operators are used to meet user needs in complex scenarios. When the operators in the system do not meet user needs, users can define high-level operators with characteristics according to the system template. The system provides a basic debugging and verification interface. After debugging and uploading, the system includes the high-level operator, and users can select the high-level operator to complete specific stream processing logic when configuring tasks.
[0084] Independent functions are abstracted into operators. Operators are independent of each other, and users can select different high-order operators for combination according to their needs. Extended functions are abstracted into operators to ensure that the common processing logic of each operator is consistent and greatly reduce the development workload.
[0085] Task parsing and DAG transformation, receiving task configuration (json format), checking the validity of the task, and converting it into an internal data structure. According to the high-order operators and superposition order configured by the user, the DAG graph is transformed using the category theory method. The following uses the error handling operator and the statistics operator as examples to illustrate the DAG transformation process.
[0086] Figure 3 is a schematic diagram of a DAG transformation according to an embodiment of the present invention, such as Figure 3As shown in the figure, the transformation process first superimposes error processing and then statistical processing, that is, after the calculation of each node in the DAG is completed, the result is first processed for error (processing logic see operator description) and then information statistics are performed (logic see operator description). After the transformation, the entire stream processing process adds message error processing and statistical functions.
[0087] The original DAG is changed using the category theory method. After the transformation, the user input DAG connection relationship remains unchanged. Compared with the graph transformation algorithm, this homomorphic mapping is simple to implement and is completely orthogonal to the graph transformation algorithm. The homomorphic mapping of DAG is more conducive to users' understanding of the stream processing logic within the system, and can locate problems more quickly during debugging and task processing. Compared with completely user-defined DAG, optimizing DAG by superposition is more flexible and compatibility between versions is easier to ensure.
[0088] Task processing, including task starting and task stopping.
[0089] Task start: When starting a task, the signaling operator is called to send a start signaling message to the first node of the DAG to drive all operators to start running.
[0090] When a task stops, including when the user stops it actively or due to internal logic, the signaling operator is called to send a stop signaling message to the first node, driving all operators to stop running.
[0091] There is no central node in the entire task processing process. The boxing method of the operator is parsed layer by layer on a single node and finally delivered to the next node. After receiving the message, the node performs the unboxing action layer by layer. The processing logic on each node is consistent. There is no need to pay attention to which specific operator is used during the boxing and unboxing process. You only need to call the corresponding method. This method delegates the logic to specific operators, centralizes complex logic, simplifies the operator calling logic, and has strong testability.
[0092] The user configuration is simple. Graphical stream processing configuration is adopted. Users can complete the configuration of stream processing tasks without coding and learning costs. In terms of operation and maintenance, the status of each node on the DAG during the stream processing and the real-time consumption progress can be monitored. The homomorphic mapping method is adopted to keep the connection relationship of the original DAG graph unchanged, so that users can better understand the processing logic on the entire link, and can quickly locate the position of the erroneous operator in debugging and log analysis. During the task processing process, the operator is controlled by signaling. There is no central management node during the operation of the operator to avoid single point failure. The flow of signaling messages is consistent with the flow of ordinary messages, which fully serves the processing logic of ordinary messages and reduces the amount of code development. A single function is abstracted into an operator, and the operators are not coupled. The functions can be flexibly combined and have strong scalability. Error handling, statistics, retries, etc. are abstracted into operators to make the processing logic of all nodes consistent.
[0093] Example 2
[0094] According to another embodiment of the present invention, a stream processing device is provided. Figure 4 is a block diagram of a stream processing device according to an embodiment of the present invention. Figure 4 As shown, including:
[0095] A configuration module 42, configured to configure an initial directed acyclic graph DAG according to a configuration instruction for configuring a stream processing task received in a graphical interface;
[0096] An optimization module 44 is used to optimize the initial DAG graph by matching high-order operators to obtain a target DAG graph, wherein the target DAG graph has the same connection relationship as the initial DAG graph;
[0097] The monitoring module 46 is used to monitor the progress of the stream processing task through the target DAG graph.
[0098] Optionally, the optimization module 44 includes:
[0099] An abstract submodule, used to abstract the nodes in the initial DAG graph into basic operators;
[0100] a first optimization submodule, configured to optimize the initial DAG graph by abstracting the extended functions on the basic operators obtained in the graphical interface into high-order operators to obtain the target DAG graph; or
[0101] The second optimization submodule is used to optimize the initial DAG graph through the high-order operator or custom operator obtained in the graphical interface to obtain the target DAG graph.
[0102] Optionally, the first optimization submodule includes:
[0103] An abstraction unit is used to optimize the initial DAG graph by abstracting the extended functions on the basic operators into error handling operators and / or system operators to obtain the target DAG graph;
[0104] An optimization unit is used to optimize the initial DAG graph by abstracting the extended functions on the basic operator into a log processing operator, a remote call operator and / or a signaling operator to obtain the target DAG graph, wherein the signaling operator is used to control the start and stop of the stream processing task.
[0105] Optionally, the device further comprises:
[0106] A calculation unit, used to calculate each node in the target DAG graph to obtain a calculation result;
[0107] an error processing unit, used to perform error processing on the calculation result according to the error processing operator to obtain a processing result; and / or
[0108] A statistical unit is used to perform information statistics according to the statistical calculator.
[0109] Optionally, the device further comprises:
[0110] A determination unit, used to determine the message type of the processing result;
[0111] A classification unit, configured to classify error messages of the processing results according to the message types of the processing results;
[0112] A reporting unit is used to report the processing result.
[0113] Optionally, the classification unit is also used for
[0114] If the processing result is a recoverable error message, the processing result is delivered to the first node of the target DAG graph for re-flow calculation;
[0115] If the processing result is an unrecoverable error message, a message log of the processing result is stored, wherein the message log includes message content, error information, error operator and time information.
[0116] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0117] Example 3
[0118] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0119] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:
[0120] S1, configuring an initial directed acyclic graph DAG according to the configuration instructions for configuring the stream processing task received in the graphical interface;
[0121] S2, optimizing the initial DAG graph by matching high-order operators to obtain a target DAG graph, wherein the target DAG graph has the same connection relationship as the initial DAG graph;
[0122] S3, monitoring the progress of the stream processing task through the target DAG graph.
[0123] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.
[0124] Example 4
[0125] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0126] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0127] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0128] S1, configuring an initial directed acyclic graph DAG according to the configuration instructions for configuring the stream processing task received in the graphical interface;
[0129] S2, optimizing the initial DAG graph by matching high-order operators to obtain a target DAG graph, wherein the target DAG graph has the same connection relationship as the initial DAG graph;
[0130] S3, monitoring the progress of the stream processing task through the target DAG graph.
[0131] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0132] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A streaming processing method, It is characterized in that include: Configure an initial directed acyclic graph DAG according to the configuration instructions for configuring the stream processing task received in the graphical interface; The initial DAG graph is optimized by matching high-order operators, and the initial DAG graph is transformed by category theory according to the high-order operators and superposition order configured by the user to obtain a target DAG graph, and the connection relationship of the user input DAG remains unchanged after the transformation, wherein the connection relationship of the target DAG graph is the same as that of the initial DAG graph; The progress of the stream processing task is monitored through the target DAG graph, and the start and stop operation of the operator is controlled by signaling; The initial DAG graph is optimized by matching high-order operators to obtain a target DAG graph including: Abstracting the nodes in the initial DAG graph into basic operators; By abstracting the extended functions on the basic operators obtained in the graphical interface into high-order operators, the initial DAG graph is optimized to obtain the target DAG graph; or The initial DAG graph is optimized by using the high-order operator or the custom operator obtained in the graphical interface to obtain the target DAG graph.
2. The method according to claim 1, It is characterized in that By abstracting the extended functions on the basic operators obtained in the graphical interface into high-order operators, the initial DAG graph is optimized to obtain the target DAG graph including: The initial DAG graph is optimized by abstracting the extended functions on the basic operator into an error handling operator and / or a general operator to obtain the target DAG graph; The initial DAG graph is optimized by abstracting the extended functions on the basic operator into a log processing operator, a remote call operator and / or a signaling operator to obtain the target DAG graph, wherein the signaling operator is used to control the start and stop of the stream processing task.
3. The method according to claim 2, It is characterized in that After optimizing the initial DAG graph by abstracting the extended functions on the basic operators into error handling operators and / or general operators to obtain the target DAG graph, the method further includes: Calculate each node in the target DAG graph to obtain a calculation result; Performing error processing on the calculation result according to the error processing operator to obtain a processing result; and / or Information statistics are performed according to the statistical calculator.
4. The method according to claim 3, It is characterized in that After performing error processing on the calculation result according to the error processing operator to obtain the processing result, the method further includes: Determine the message type of the processing result; Perform error message classification processing on the processing result according to the message type of the processing result; The processing result is reported.
5. The method according to claim 4, It is characterized in that Performing error message classification processing on the processing result according to the message type of the processing result includes: If the processing result is a recoverable error message, the processing result is delivered to the first node of the target DAG graph for re-flow calculation; If the processing result is an unrecoverable error message, a message log of the processing result is stored, wherein the message log includes message content, error information, error operator and time information.
6. A stream processing device, It is characterized in that include: A configuration module, configured to configure an initial directed acyclic graph DAG according to a configuration instruction for configuring a stream processing task received in a graphical interface; An optimization module is used to optimize the initial DAG graph by matching high-order operators, and transform the initial DAG graph by category theory according to the high-order operators and superposition order configured by the user to obtain a target DAG graph. After the transformation, the connection relationship of the user input DAG remains unchanged, wherein the connection relationship of the target DAG graph is the same as that of the initial DAG graph; A monitoring module, used to monitor the progress of the stream processing task through the target DAG graph and control the start and stop operation of the operator by signaling; Wherein, the optimization module includes: An abstract submodule, used to abstract the nodes in the initial DAG graph into basic operators; a first optimization submodule, configured to optimize the initial DAG graph by abstracting the extended functions on the basic operators obtained in the graphical interface into high-order operators to obtain the target DAG graph; or The second optimization submodule is used to optimize the initial DAG graph through the high-order operator or custom operator obtained in the graphical interface to obtain the target DAG graph.
7. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 5 when executed.
8. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Distributed stream calculation system, method and device
CN106293892A
Workflow processing method for distributed system and workflow engine system
CN106980530A
A data processing method and system based on a directed acyclic graph
CN109726216A
Stream computing platform and method based on visualization, equipment and storage medium
CN110908641A
Data processing method, device and system
CN110941467A
Cited By
Stream processing method and system based on graph structure and SQL (Structured Query Language) statement
CN120849417A
A graph structure and SQL statement based stream processing method and system
CN120849417B
