Big data flow batch integrated processing architecture method based on responsibility chain quick constructor

Abstract

The application discloses a big data flow batch integrated processing architecture method based on a responsibility chain fast constructor, first, the architecture needs to apply for reading data flow and file flow, and initialization of an execution environment is carried out; secondly, mapping and key value grouping operations of data are carried out in a data transformation process; thirdly, a responsibility chain is executed, a responsibility chain object is newly created in each working node, and batch data in a rolling window is subjected to batch processing; fourthly, data subjected to the responsibility chain is subjected to aggregation calculation, and statistical data is output; finally, data generated in the conversion process is output to a sink end and output to a message queue system. The application simplifies objects, reduces the coupling degree between modules, enhances the flexibility of object responsibility assignment, can quickly construct a general distributed software architecture, simultaneously constructs a running entrance in different modes, and simplifies the work flow of developers and engineering deployment personnel.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology, specifically relating to a big data stream-batch integrated processing architecture method based on a chain of responsibility fast constructor. Background Technology

[0002] Stream-batch processing technology primarily involves two main processing modes in the field of big data processing: stream processing and batch processing, and their integration. This technological concept aims to improve data processing efficiency and data consistency by supporting both modes simultaneously through a unified technical framework and tools. Stream processing primarily handles real-time data streams, while batch processing handles historical data. By integrating these two processing modes, stream-batch processing technology enables enterprises to process and analyze data more efficiently while maintaining data consistency and reliability.

[0003] The concept and practical background of stream-batch processing technology can be traced back to the development of big data technology. As enterprises increase their demands for data processing speed and real-time performance, stream processing technology has gradually become mainstream. However, batch processing technology still plays an irreplaceable role, especially in the analysis and processing of historical data. Therefore, the proposed stream-batch processing technology aims to solve the problem of stream processing and batch processing operating independently, achieving seamless integration and efficient collaboration between the two processing modes through a unified computing engine and development framework.

[0004] In terms of technical implementation, the integrated stream and batch processing technology ensures data consistency and processing efficiency by adopting a unified computing engine, a unified semantic development layer, and unified data storage and processing logic.

[0005] Chain of Responsibility is a design pattern that allows a request to be passed between multiple objects until an object capable of handling the request is found.

[0006] 1. Dynamic Responsibility Assignment: The Chain of Responsibility pattern allows the system to dynamically reorganize and assign responsibilities at runtime, meaning that request handlers can be replaced or added without changing client code. This flexibility makes the system more scalable and maintainable.

[0007] 2. Request Passing: In a chain of responsibility, a request is passed from one object to the next until an object capable of handling it is found. This passing mechanism ensures that every request is processed, preventing some requests from being ignored.

[0008] 3. Decoupling: By separating the request processing logic from the client code, the Chain of Responsibility pattern reduces the coupling between the client and the processing object. This means the client doesn't need to know which specific object handled the request, thus improving the system's maintainability and scalability.

[0009] 4. Suitable for scenarios involving multiple objects: The Chain of Responsibility pattern is very useful when a problem or request needs to be processed by multiple objects in a certain order. For example, in a leave approval process, different management levels can approve leave applications of different levels according to their respective permissions.

[0010] 5. Abstract and Concrete Processors: The Chain of Responsibility pattern includes abstract and concrete processor roles. The abstract processor defines an interface for handling requests, while the concrete processor implements this interface and decides whether to process the request or pass it on to the next processor.

[0011] In "An Optimized Computation Method for Big Data Batch Computation Based on Stream-Batch Integration," field combinations are determined according to business requirements. Data streams are grouped according to these field combinations to form multiple different data streams. These multiple different data streams are then processed in the KeyedProcessFunction according to a stepped decreasing time window to handle data logic. When the computation is complete, a subset of data that meets the stated business requirements is output.

[0012] The disadvantages of existing technologies are as follows:

[0013] 1. High architecture and business maintenance costs: The implementation of integrated stream and batch processing may involve different technologies and tools, which increases the architecture and business maintenance costs.

[0014] 2. Strong coupling of data transformation: There are multiple transformation operators and processes in the data processing, and the coupling between these operators is strong, which is not conducive to the construction of a general architecture.

[0015] 3. Debugging difficulties: The above solutions involve a long processing procedure with multiple steps, and the debugging process may become very complicated.

[0016] 4. Low application build speed: The multiple processes involved in the claims of this technology make it difficult to flexibly and quickly build production application software according to user needs;

[0017] 5. This technology mainly optimizes batch computing scenarios, but does not optimize the architecture of the main coding body of the stream-batch integrated framework.

[0018] Currently, in the process of data transformation using multiple transformers in the unified batch and stream architecture, multiple transform methods are often manually stacked in the code. This can cause code implementation conflicts and unclear architecture in large-scale parallel development scenarios, and the original code needs to be modified when the order of transforms needs to be adjusted. Summary of the Invention

[0019] To overcome the shortcomings of existing technologies, this invention provides a big data stream batch processing architecture method based on a chain of responsibility fast constructor. First, the architecture needs to request and read data streams and file streams to initialize the execution environment. Second, during data transformation, operations such as data mapping and key value grouping are performed. Third, the chain of responsibility is executed by creating a new chain of responsibility object at each worker node and batch processing the batch data in the scrolling window. Fourth, the data processed by the chain of responsibility is aggregated and calculated to output statistical data. Finally, the data generated during the transformation process is output to the sink end and then to the message queue system. This invention simplifies objects, reduces coupling between modules, enhances the flexibility of assigning responsibilities to objects, and enables the rapid construction of a general distributed software architecture. Simultaneously, it constructs runtime entry points in different modes, simplifying the workflow for developers and engineering deployment personnel.

[0020] The technical solution adopted by this invention to solve its technical problem is as follows:

[0021] Step 1: Initialize the engine environment;

[0022] It includes four steps: connecting to the data source, initializing the runtime environment, initializing the chain of responsibility handler, and starting the stream and batch processing engine.

[0023] Step 1-1: Connect to the data source;

[0024] The execution environment of the stream and batch processing engine is initialized by setting the abstract function StreamExecutionEnvironment, and the data source is loaded into the execution environment by the addSource operation;

[0025] Step 1-2: Initialize the runtime environment;

[0026] By defining FlashTransform(T dataStream, U factory) as the engine variable, the first parameter is the input data stream and the second is the chain of responsibility object; at the same time, the engine is initialized by inheriting the general engine interface AbstractEngine in the engine, creating debugging mode, cluster mode and file mode, and setting the corresponding working mode for developers and engineers in different roles in the software ecosystem, and the engine creates the input stream when the execution mode is selected.

[0027] Steps 1-3: Chain of Responsibility Initialization;

[0028] The chain of responsibility is initialized before the engine is executed, implementing the createHandler method in the MessageHandlerFactory chain of responsibility interface, and outputting logs when the chain of responsibility is constructed, so that operation and maintenance personnel can monitor in real time whether each link in the chain is running normally.

[0029] Steps 1-4: Start the integrated batch processing engine;

[0030] The engine is started by calling the engine variables from steps 1-2 and inheriting the start method of the Engine interface.

[0031] Step 2: Perform transformer operations;

[0032] Load balancing is performed on the incoming data streams. Load balancing allocates all parallel subtasks in a round-robin fashion and shuffles and regroups the hashed data read by different threads, helping the software achieve fairness and fault tolerance in a distributed environment.

[0033] Step 3: Perform a map loop to map the data and set the key value;

[0034] First, define the variable Random. Then, call the map method to set a random number using random.next(100) on the data hashed in the worker nodes.

[0035] Step 4: Set the timestamp and generate the watermark;

[0036] Event timestamps can be set for later events, but the framework is only allowed to know the timestamp of each element and the progress of event times in order to determine which elements have been processed, provided that the difference between these events and the current processing time does not exceed a specified tolerance.

[0037] Step 5: According to the framework requirements, set the key value and group the data as in Step 3 to implement key-based aggregation operations. By performing a hash modulo operation on the key value, perform a hash partitioning operation on the data to generate a keyedStream data stream object.

[0038] Step 6: Request a rolling processing time window (TumblingProcessingTimeWindows);

[0039] Based on the window type of data processing time, the data is divided into fixed-size panes, each of which is non-overlapping; the start time of the pane is determined according to the data processing time, which controls the data size of the batch data, and accumulates the data for the user-defined time during the streaming process as the data source for batch step 8.

[0040] Step 7: Initialize the chain of responsibility;

[0041] In step 6, during the time window processing, the `process` function is used to call the chain of responsibility initialization method, with `Context` and `Iterable` as parameters. <t>Collector <Tuple2<String,Long> The `MessageHandlerFactory` interface is defined during chain initialization, containing the `createHandler()` method. The basic object of the chain is `ApacheChainHandler`. The chain's execution functionality is configured through settings in the `chain-config.xml` file. These functionalities are added to the chain object during initialization using the `addCommand()` method. Adding the object simultaneously initializes the executor objects in the chain, performing rapid initialization of the defined class objects. The executor's constructor initializes necessary user objects and some static objects, eliminating the need for re-initialization in subsequent calls. This speeds up chain construction. The chain is then executed using the `execute` method. The chain is regenerated when data objects are created; a new chain is generated for each batch of data, including temporary objects that need initialization. Finally, the chain is returned through the defined `Collector` data structure. <Tuple2<String,Long> Return the execution result;

[0042] Step 8: Organize the batch data;

[0043] Set up a receiving list for the data from the scrolling window in step 6, and put this batch of data into the list as data objects when the chain of responsibility is executed. These data objects will be passed in the chain by reference, including data deletion, copying, and modification.

[0044] Step 9: Triggering the execution of the chain of responsibility;

[0045] During execution, multiple processing modules inherit the same processing interface CommandBase, which defines two main methods: execute and close. Each module receives batch data objects and processes them accordingly. It is important to note that the data received by the next chain references the data objects updated by the previous chain. This ensures the order of data processing and the uniqueness of data objects. After execution, the close() method is used to reclaim other resources requested by the responsibility chain and destroy the chain.

[0046] Step 10: The data after Step 8 is regrouped by key value. The key values ​​in this step are used to aggregate and statistically analyze the processed data.

[0047] Step 11: Apply for a rolling execution time window again to set the statistical output frequency and control the aggregation time of the output data stream;

[0048] Step 12: Perform an aggregation reduce operation on the output data generated in Step 11 according to the key value, and process the tuple data (Tuple2) output in Step 7.<String,Long> ), group the data into pairs according to element 1, count the elements of 2, and output the data of pairs;

[0049] Step 13: Print the tuple data output in Step 12. After this step is completed, the framework will also end the transformer operation.

[0050] Step 14: The sink operation outputs data to the Kafka message stream: By inheriting the SinkFunction interface KafkaSink, setting the Kafka connection parameter setKafkaProducerConfig and the serialization parameter setRecordSerializer, and finally setting the Kafka address information setBootstrapServers, the data is continuously output to Kafka by implementing the invoke method of the sink interface.

[0051] Preferably, the input streaming data in step 2 is Kafka real-time streaming message data or offline batch data files.

[0052] The beneficial effects of this invention are as follows:

[0053] This invention overcomes the pain points of single working modes and inflexible disassembly and assembly between different software modules in big data processing architectures, achieving: 1) the integration of stream-batch integrated architecture and the chain of responsibility design pattern; 2) flexible disassembly and assembly of functional modules, a plug-and-play processing architecture; breaking through the bottleneck of deep integration of stream-batch integrated architecture and chain of responsibility design pattern, realizing the rapid construction of the big data processing field by introducing the chain of responsibility pattern into the streaming processing architecture, providing an open, agile, efficient, and unified architecture development method for the rapid deployment of big data applications; 3) rapid construction of the chain of responsibility pattern, overcoming the additional resource occupation caused by frequent consumption chains in the basic chain of responsibility pattern; 4) the application of multiple operating modes, making it a general framework that is suitable for both developers and operations personnel. Attached Figure Description

[0054] Figure 1 This is a flowchart of the method of the present invention;

[0055] Figure 2 This is a flowchart of the initialization process of the method of the present invention. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0057] The purpose of this invention is to utilize the flexible reconfiguration pattern in the Chain of Responsibility design pattern for general software architecture design, enabling agile software development. Parallel development of multiple processes and operators can decouple senders and receivers, thereby reducing system complexity. Isolating the dependency relationships between chains through the chain of responsibility can accelerate software development. Furthermore, constructing a fast constructor during the construction of the chain of responsibility accelerates the construction speed of the chain of responsibility, thus solving the performance problems caused by the chain of responsibility.

[0058] To address the complex processing logic required when building application software in a batch processing architecture, and the need to handle complex processing logic when adding or modifying processing modules, this invention simplifies objects, reduces coupling between modules, and enhances the flexibility of assigning responsibilities to objects. It enables the rapid construction of a general distributed software architecture. At the same time, it constructs runtime entry points in different modes (debugging mode, production mode), simplifying the workflow of developers and engineering deployment personnel.

[0059] A responsibility chain processing architecture based on stream and batch processing is proposed. Its key step, during the transformation process, applies the responsibility chain design pattern to achieve deep integration of the stream and batch processing architecture and the responsibility chain. First, the architecture requests and reads data streams and file streams to initialize the execution environment. Second, during data transformation, operations such as data mapping and key grouping are performed. Third, the responsibility chain is executed by creating a new responsibility chain object at each worker node and batch processing the batch data in the scrolling window. Fourth, the data processed by the responsibility chain is aggregated and calculated to output statistical data. Finally, the data generated during the transformation process is output to the sink end and then to the message queue system.

[0060] Step 101: Initialize the engine environment. Initialization mainly consists of four steps: connecting to the data source, initializing the runtime environment, initializing the chain of responsibility Handler, and starting the stream and batch engine.

[0061] Step 201: Connect to the data source. Initialize the execution environment of the stream and batch processing engine by setting the abstract function StreamExecutionEnvironment. The input stream data can be Kafka real-time stream message data or file stream. Load the data source into the execution environment through the addSource operation.

[0062] Step 202: Initialize the runtime environment by defining FlashTransform(T dataStream, Ufactory), where the first parameter is the input data stream and the second is the chain of responsibility object; at the same time, the engine is initialized by inheriting the general engine interface AbstractEngine in the engine. It can create various execution modes such as debugging mode, cluster mode, and file mode. In the software ecosystem with different roles, the corresponding working mode can be set for developers and engineers. When the execution mode is selected, the engine will create the input stream.

[0063] Step 203: Chain of Responsibility Initialization. The chain of responsibility is initialized before the engine is executed. Its main process is to implement the createHandler method in the MessageHandlerFactory responsibility chain interface and output logs when the chain of responsibility is constructed, so that the operation and maintenance personnel can monitor whether each link in the chain is running normally in real time.

[0064] Step 204: Start the engine by calling the engine variable from step 202 and inheriting the start method of the Engine interface;

[0065] Step 102: This step begins the transformer operation, which rebalances the incoming data stream. It uses a round-robin approach to allocate data to all parallel subtasks and shuffles and regroups the hashed data read by different threads, helping the software achieve fairness and fault tolerance in a distributed environment.

[0066] Step 103: Set random numbers by performing a map loop mapping on the data. First, define the variable Random, and then call the map method to set random numbers by calling random.next(100) on the data hashed on the worker nodes, in order to prepare for the data uniqueness in the key value grouping process in the subsequent step 105.

[0067] Step 104: Set timestamps and generate watermarks. To properly handle event times, this method allows event times to be set to later timestamps for later events, but only allows the framework to know the timestamp of each element and the progress of event times to determine which elements have been processed, provided that the difference between these events and the current processing time does not exceed a specified tolerance.

[0068] Step 105: According to the framework requirements, set the key value and group the data as in step 103 to implement key-based aggregation operations. By performing a hash modulo operation on the key value, perform a hash partitioning operation on the data to generate a keyedStream data stream object.

[0069] Step 106: Request a TumblingProcessingTimeWindows, a window type based on data processing time. It divides the data into fixed-size panes (or buckets), each pane being non-overlapping. The start time of the panes is determined based on the data processing time, controlling the data size of the batch data. During streaming processing, data accumulated over a user-defined time period serves as the data source for the batch step 108.

[0070] Step 107: Initialize the chain of responsibility. During the time window processing in step 106, the initialization method of the chain of responsibility is called through the process function, with the function parameters being Context and Iterable. <t>Collector <Tuple2<String,Long> The `MessageHandlerFactory` interface is defined during chain initialization, containing the `createHandler()` method. The basic object of the chain is `ApacheChainHandler`. The chain's execution functionality is configured through settings in the `chain-config.xml` file. These functionalities are added to the chain object during initialization using the `addCommand()` method. Adding the object simultaneously initializes the executor objects in the chain, performing rapid initialization of the defined class objects. The executor's constructor initializes necessary user objects and some static objects, eliminating the need for re-initialization in subsequent calls. This speeds up chain construction. The chain is then executed using the `execute` method. The chain is regenerated when data objects are created; a new chain is generated for each batch of data, including temporary objects that need initialization. Finally, the chain is returned through the defined `Collector` data structure. <Tuple2<String,Long> Return the execution result;

[0071] Step 108: Organize batch data, set up a receiving list for the data from the scrolling window in step 106, and put this batch of data into the list as data objects when the chain of responsibility is executed. These data objects will be passed in the chain by reference, including data deletion, copying, and modification.

[0072] Step 109: Triggering the execution of the chain of responsibility. During the execution process, multiple processing modules inherit the same processing interface CommandBase. The interface defines two main methods: execute and close. Each module receives batch data objects and processes them accordingly. It is important to note that the data received by the next chain references the data objects updated by the previous chain. This ensures the order of data processing and the uniqueness of data objects. After execution, the close() method is used to reclaim other resources requested by the chain of responsibility and destroy the chain.

[0073] Step 110: The data after step 108 is regrouped by key value. The key value election in this step is different from that in step 105. It mainly focuses on the aggregation and statistics of the processed data.

[0074] Step 111: Apply for a rolling execution time window again. This window time is unrelated to the time in step 106. This event window is set for statistical output frequency and can control the aggregation time of the output data stream.

[0075] Step 112: Perform a reduce (aggregate) operation on the output data generated in Step 111 according to the key value, and process the tuple data (Tuple2) output in Step 107.<String,Long> ), count the pairs of _1 and _2, and output the pairs of data;

[0076] Step 113: Print the tuple data output in step 112. After this step is completed, the framework will also end the transformer operation.

[0077] Step 114: The sink operation outputs data to the Kafka message stream: by inheriting the SinkFunction.

[0078] The interface uses KafkaSink, which sets the Kafka connection parameters (setKafkaProducerConfig) and serialization parameters (setRecordSerializer). Finally, it sets the Kafka address information (setBootstrapServers) and then continuously outputs data to Kafka by implementing the invoke method of the sink interface.

[0079] The key point of this invention is how to integrate the batch processing framework and the chain of responsibility design pattern, and use the distributed processing capabilities provided by the batch big data processing framework to build a general processing framework in combination with the chain of responsibility.

[0080] The points to be protected by this invention:

[0081] 1. The entire method flow in the initialization environment;

[0082] 2. The entire methodology integrating batch processing framework and chain of responsibility design pattern;

[0083] 3. Methods for constructing and destroying the chain of responsibility during data processing;

[0084] Existing distributed stream-batch processing frameworks frequently modify existing processing logic during data transformation, and adding processing operations to the existing codebase requires further modifications, posing a significant challenge to code maintainability. The implementation scheme of this invention unifies and integrates the stream-batch processing framework with the chain of responsibility design pattern, combining the advantages of various solutions. It allows for the rapid construction of application software with different responsibilities for different distributed processing frameworks. This architecture makes software architecture design more flexible and versatile.< / t> < / t>

Claims

1. A method for a big data stream batch processing architecture based on a chain of responsibility fast constructor, characterized in that, Includes the following steps: Step 1: Initialize the engine environment; It includes four steps: connecting to the data source, initializing the runtime environment, initializing the chain of responsibility handler, and starting the stream and batch processing engine. Step 1-1: Connect to the data source; The execution environment of the stream and batch processing engine is initialized by setting the abstract function StreamExecutionEnvironment, and the data source is loaded into the execution environment by the addSource operation; Step 1-2: Initialize the runtime environment; By defining FlashTransform(T dataStream, U factory) as the engine variable, the first parameter is the input data stream and the second is the chain of responsibility object; at the same time, the engine is initialized by inheriting the general engine interface AbstractEngine in the engine, creating debugging mode, cluster mode and file mode, and setting the corresponding working mode for developers and engineers in different roles in the software ecosystem, and the engine creates the input stream when the execution mode is selected. Steps 1-3: Chain of Responsibility Initialization; The chain of responsibility is initialized before the engine is executed, implementing the createHandler method in the MessageHandlerFactory chain of responsibility interface, and outputting logs when the chain of responsibility is constructed, so that operation and maintenance personnel can monitor in real time whether each link in the chain is running normally. Steps 1-4: Start the integrated batch processing engine; The engine is started by calling the engine variables from steps 1-2 and inheriting the start method of the Engine interface. Step 2: Perform transformer operations; Perform load balancing on the incoming data streams; Load balancing distributes all parallel subtasks in a round-robin fashion, and shuffles and regroups the hashed data read by different threads, helping the software achieve fairness and fault tolerance in a distributed environment. Step 3: Perform a map loop to map the data and set the key value; First, define the variable Random. Then, call the map method to set a random number using random.next(100) on the data hashed in the worker nodes. Step 4: Set the timestamp and generate the watermark; Event timestamps can be set for later events, but the framework is only allowed to know the timestamp of each element and the progress of event times in order to determine which elements have been processed, provided that the difference between these events and the current processing time does not exceed a specified tolerance. Step 5: According to the framework requirements, set the key value and group the data as in Step 3 to implement key-based aggregation operations. By performing a hash modulo operation on the key value, perform a hash partitioning operation on the data to generate a keyedStream data stream object. Step 6: Request a rolling processing time window (TumblingProcessingTimeWindows); Based on the window type of data processing time, the data is divided into fixed-size panes, each of which is non-overlapping; the start time of the pane is determined according to the data processing time, which controls the data size of the batch data, and accumulates the data for the user-defined time during the streaming process as the data source for batch step 8. Step 7: Initialize the chain of responsibility; In step 6, during the time window processing, the `process` function is used to call the chain of responsibility initialization method, with `Context` and `Iterable` as parameters. <t> Collector <Tuple2<String,Long> The `MessageHandlerFactory` interface is defined during chain initialization, containing the `createHandler()` method. The basic object of the chain is `ApacheChainHandler`. The chain's execution functionality is configured through settings in the `chain-config.xml` file. These functionalities are added to the chain object during initialization using the `addCommand()` method. Adding the object simultaneously initializes the executor objects in the chain, performing rapid initialization of the defined class objects. The executor's constructor initializes necessary user objects and some static objects, eliminating the need for re-initialization in subsequent calls. This speeds up chain construction. The chain is then executed using the `execute` method. The chain is regenerated when data objects are created; a new chain is generated for each batch of data, including temporary objects that need initialization. Finally, the chain is returned through the defined `Collector` data structure. <Tuple2<String,Long> Return the execution result;< / t> Step 8: Organize the batch data; Set up a receiving list for the data from the scrolling window in step 6, and put this batch of data into the list as data objects when the chain of responsibility is executed. These data objects will be passed in the chain by reference, including data deletion, copying, and modification. Step 9: Triggering the execution of the chain of responsibility; During execution, multiple processing modules inherit the same processing interface CommandBase, which defines two main methods: execute and close. Each module receives batch data objects and processes them accordingly. It is important to note that the data received by the next chain references the data objects updated by the previous chain. This ensures the order of data processing and the uniqueness of data objects. After execution, the close() method is used to reclaim other resources requested by the responsibility chain and destroy the chain. Step 10: The data after Step 8 is regrouped by key value. The key values ​​in this step are used to aggregate and statistically analyze the processed data. Step 11: Apply for a rolling execution time window again to set the statistical output frequency and control the aggregation time of the output data stream; Step 12: Perform an aggregation reduce operation on the output data generated in Step 11 according to the key value, and process the tuple data (Tuple2) output in Step 7.<String,Long> ), group the data into pairs according to element 1, count the elements of 2, and output the data of pairs; Step 13: Print the tuple data output in Step 12. After this step is completed, the framework will also end the transformer operation. Step 14: The sink operation outputs data to the Kafka message stream: By inheriting the SinkFunction interface KafkaSink, setting the Kafka connection parameter setKafkaProducerConfig and the serialization parameter setRecordSerializer, and finally setting the Kafka address information setBootstrapServers, the data is continuously output to Kafka by implementing the invoke method of the sink interface.

2. The big data stream batch processing architecture method based on a fast chain of responsibility builder as described in claim 1, characterized in that, The input streaming data in step 2 consists of Kafka real-time streaming message data and offline batch data files.

Images