Big data flow batch integrated processing architecture method based on responsibility chain rapid constructor

Abstract

The invention discloses a big data stream batch integrated processing architecture method based on a chain-of-responsibility rapid constructor, which comprises the following steps: firstly, an architecture needs to apply for reading a data stream and a file stream to initialize an execution environment; 2, performing data mapping, key value grouping and other operations in the data conversion process; 3, executing a responsibility chain, newly establishing a responsibility chain object at each working node, and performing batch processing on batch data in the rolling window; 4, performing aggregation calculation on the data passing through the responsibility chain, and outputting statistical data; and finally, outputting data generated in the conversion process to a sink end and outputting the data to a message queue system. According to the method, the object is simplified, the coupling degree between the modules is reduced, the flexibility of object assignment responsibilities is enhanced, a universal distributed software architecture can be quickly constructed, meanwhile, operation entrances in different modes are constructed, and the workflow of developers and project deployment personnel is simplified.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to a big data stream-batch integrated processing architecture method based on a chain of responsibility fast constructor. Background Art

[0002] Stream-batch integration technology mainly involves two main processing modes in the field of big data processing: stream processing and batch processing, and the integration between them. This technical concept aims to support both processing modes through a unified technical framework and tools to improve data processing efficiency and data consistency. Stream processing mainly processes real-time data streams, while batch processing processes historical data. By integrating these two processing modes, stream-batch integration technology enables enterprises to process and analyze data more efficiently while maintaining data consistency and reliability.

[0003] The concept and practice of stream-batch integration technology can be traced back to the development of big data technology. As enterprises have higher requirements for data processing speed and real-time performance, stream processing technology has gradually become mainstream, but at the same time, batch processing technology still has its irreplaceable role, especially in the analysis and processing of historical data. Therefore, the introduction of stream-batch integration technology aims to solve the problem of stream processing and batch processing being independent of each other, and to achieve seamless connection and efficient collaboration between the two processing modes through a unified computing engine and development framework.

[0004] In terms of technical implementation, the stream-batch integration technology ensures data consistency and processing efficiency by adopting a unified computing engine, a unified semantic development layer, and unified data storage and processing logic.

[0005] Chain of Responsibility is a design pattern that allows a request to be passed between multiple objects until an object that can handle the request is found:

[0006] 1. Dynamically assign responsibilities: The chain of responsibility pattern allows the system to dynamically reorganize and assign responsibilities at runtime, which means that request handlers can be replaced or added without changing the client code. This flexibility makes the system more scalable and maintainable.

[0007] 2. Request passing: In the chain of responsibility, the request is passed from one object to the next until an object that can handle the request is found. This transmission mechanism ensures that each request can be processed and avoids the situation where some requests are ignored.

[0008] 3. Decoupling: By separating the request processing logic from the client code, the responsibility chain pattern reduces the coupling between the client and the processing object. This makes it unnecessary for the client to know which specific object handles the request, thereby improving the maintainability and scalability of the system.

[0009] 4. Applicable to scenarios where multiple objects are processed: When a problem or request needs to be processed by multiple objects in a certain order, the responsibility chain pattern is very useful. For example, in the leave approval process, different management levels can approve leave applications of different levels according to their respective permissions.

[0010] 5. Abstract and concrete handlers: The responsibility chain pattern includes abstract handler roles and concrete handler roles. The abstract handler defines an interface for processing requests, while the concrete handler implements this interface and decides whether to process the request or pass it to the next handler.

[0011] In "An Optimized Calculation Method for Batch Calculation of Big Data Based on Stream-Batch Integration", the field combination is determined according to the business requirements, and the data stream is grouped according to the field combination to form multiple groups of different data streams. The multiple groups of different data streams are calculated in KeyedProcessFunction according to the step-by-step decreasing time window to process the data logic. When the calculation is completed, the data subset that meets the business requirements is output.

[0012] The disadvantages of the prior art are as follows:

[0013] 1. High architecture and business maintenance costs: The implementation of stream-batch integration may involve different technologies and tools, which increases the maintenance costs of architecture and business.

[0014] 2. Strong coupling of data conversion: There are multiple conversion operators and processes in the data processing process, and the coupling between multiple operators is strong, which is not conducive to the construction of a general architecture.

[0015] 3. Difficulty in debugging: The above solution involves a long and multi-step processing process, and the debugging process may become very complicated.

[0016] 4. Low application building speed: The multiple processes involved in the technical claims make it difficult to flexibly and quickly build application software for production according to user needs;

[0017] 5. This technology mainly optimizes batch computing scenarios, and does not perform architectural optimization on the encoding body of the stream-batch integrated framework.

[0018] Currently, in the process of multiple transformers for data conversion in the integrated stream-batch architecture, multiple transform methods are mostly manually superimposed in the code. This will cause code implementation conflicts and unclear architecture in large-scale parallel development scenarios, and the original code needs to be modified when the order of transforms needs to be adjusted. Summary of the invention

[0019] In order to overcome the deficiencies of the prior art, the present invention provides a large data stream batch integrated processing architecture method based on a fast chain of responsibility constructor. First, the architecture needs to apply for reading data streams and file streams to initialize the execution environment; the second step is to perform operations such as data mapping and key value grouping during the data transformation process; the third step is to execute the chain of responsibility by creating a new chain of responsibility object in each working node and batch processing the batch data in the rolling window; the fourth step is to perform aggregate calculations on the data passing through the chain of responsibility and output statistical data; finally, the data generated in the conversion process is output to the sink end and output to the message queue system. The present invention simplifies objects, reduces the coupling degree between modules, enhances the flexibility of object assignment responsibilities, can quickly build a general distributed software architecture, and at the same time, constructs operation entrances in different modes, simplifying the workflow of developers and engineering deployment personnel.

[0020] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0021] Step 1: Initialize the engine environment;

[0022] It includes four steps: connecting to the data source, initializing the operating environment, initializing the responsibility chain handler, and starting the batch-stream integration engine.

[0023] Step 1-1: Link to data source;

[0024] Initialize the execution environment of the stream-batch engine by setting the abstract function StreamExecutionEnvironment, and load the data source into the execution environment through the addSource operation;

[0025] Step 1-2: Initialize the operating environment;

[0026] By defining FlashTransform (T dataStream, U factory) as an engine variable, the first parameter is the input data stream, and the second is the responsibility chain object; at the same time, the general engine interface AbstractEngine in the engine is inherited to initialize the engine, create debugging mode, cluster mode, and file mode, and set corresponding working modes for developers and engineers in the software ecosystem with different roles. The engine creates an input stream while selecting the execution mode;

[0027] Step 1-3: Initialize the chain of responsibility;

[0028] The responsibility chain is initialized before the engine is executed, the createHandler method in the MessageHandlerFactory responsibility chain is implemented, and the construction time of the responsibility chain is output to the log, which is convenient for operation and maintenance personnel to monitor whether each link in the chain is running normally in real time;

[0029] Step 1-4: Start the stream-batch integration engine;

[0030] Start the engine by calling the engine variable in step 1-2 and inheriting the start method of the Engine interface;

[0031] Step 2: Perform transformer operation;

[0032] Rebalance the incoming data streams; perform round-robin allocation on all parallel subtasks, scatter and regroup the hashed data read by different threads, and help the software achieve fairness and fault tolerance in a distributed environment;

[0033] Step 3: Map the data in a loop and set the key value;

[0034] First, define the variable Random, then call the map method to call random.next(100) to set the random number for the data hashed on the working node;

[0035] Step 4: Set timestamp and generate watermark;

[0036] Allow event time to have later timestamps for later events, but only if those events do not differ from the current processing time by more than a specified tolerance. The framework needs to know the timestamp of each element and understand the progress of event time to determine which elements have been processed.

[0037] Step 5: According to the framework requirements, set and group the key values ​​in step 3 for the data to implement key-based aggregation operations. Perform Hash modulo operations on the key values ​​and perform Hash partition operations on the data to generate a keyedStream data stream object.

[0038] Step 6: Apply for Tumbling Processing Time Windows.

[0039] The window type based on the data processing time groups the data into fixed-size panes, each of which is non-overlapping. The start time of the pane is determined based on the data processing time to control the size of the batch data. During the streaming process, the data of the user-set time is accumulated as the data source for batch step 8.

[0040] Step 7: Initialize the chain of responsibility;

[0041] In the time window processing of step 6, the initialization method of the responsibility chain is called through the process function operation, and the function parameters are Context, Iterable <t>、Collector <Tuple2<String,Long> >, define the MessageHandlerFactory interface when initializing the chain, define the createHandler() method in the interface, the basic object of the chain is ApacheChainHandler, set the execution function of the chain by defining the configuration items in the chain-config.xml configuration file, and add these functions to the initialized chain object through the addCommand() method. When adding the object, the executor object in the chain will be initialized to quickly initialize the defined class object. In the construction of the executor, the necessary objects to be used and some static objects will be initialized once. These objects will no longer need to be initialized again in subsequent calls. This operation will speed up the construction of the chain, and then execute the chain through the execute method. The responsibility chain is regenerated when the data object is generated. Each batch of data will generate a new chain, including temporary objects that need to be initialized in the chain, and finally return the data structure Collector through the defined <Tuple2<String,Long> >Return the execution result;

[0042] Step 8: Organize batch data;

[0043] Set up a receiving list for the data in the scroll window in step 6, and put this batch of data into the list as the data objects when the responsibility chain is executed. These data objects will be passed in the chain by reference, including data deletion, copying, and modification;

[0044] Step 9: Trigger execution of the responsibility chain;

[0045] During the execution process, multiple processing modules inherit the same processing interface CommandBase, which defines two main methods, execute and close. Each module receives batch data objects for corresponding processing. It should be noted that the data received by the next chain refers to the data object updated by the previous chain, which can ensure the order of data processing and the uniqueness of data objects. After execution, the close() method is used to recycle other resources applied by the responsibility chain resources and destroy the chain at the same time;

[0046] Step 10: The data after step 8 is re-grouped by key value, and the key value of this step is aggregated and counted for the processed data;

[0047] Step 11: Apply for the rolling execution time window again to set the statistical output frequency and control the aggregation time of the output data stream;

[0048] Step 12: Perform an aggregation reduce operation on the output data generated in step 11 according to the key value, and perform an aggregation reduce operation on the tuple data (Tuple2<String,Long> ), count the element 2 according to the element 1 of the binary data, and output the binary data;

[0049] Step 13: Print the binary data output from step 12. After this step is completed, the framework ends the transformer operation at the same time.

[0050] Step 14: The sink operation outputs data to the Kafka message stream: through the KafkaSink that inherits the SinkFunction interface, and then by setting the Kafka connection parameter setKafkaProducerConfig and the serialization parameter setRecordSerializer, and finally setting the Kafka address information setBootstrapServers, finally by implementing the invoke method of the sink interface to continuously output data to Kafka.

[0051] Preferably, the input stream data in step 2 is Kafka real-time stream message data or offline batch data files.

[0052] The beneficial effects of the present invention are as follows:

[0053] The present invention overcomes the pain points of a single working mode in a big data processing architecture and the inability to flexibly disassemble and assemble modules of the same software, and achieves: 1) the integration of a stream-batch integrated architecture and a responsibility chain design pattern; 2) flexible assembly and disassembly of functional modules, and a plug-and-play processing architecture; it breaks the bottleneck of the deep integration of the stream-batch integrated architecture and the responsibility chain design pattern, and achieves rapid construction of the big data processing field by introducing the responsibility chain pattern into the streaming processing architecture, providing an open, agile, efficient, and unified architecture development method for the rapid launch of big data applications; 3) rapid construction of the responsibility chain pattern, overcoming the additional resource occupation caused by frequent consumption of the chain in the basic responsibility chain pattern; 4) the application of multi-scenario operation modes, making it a universal framework that is suitable for both developers and operation and maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of the method of the present invention;

[0055] Figure 2 It is a flow chart of the initialization process of the method of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0057] The purpose of the present invention is to use the flexible reorganization pattern in the responsibility chain design pattern to design a general software architecture, so as to enable agile software development. The parallel development of multiple processing processes and operators can decouple senders and receivers, thereby reducing system complexity. The subordinate relationship between chains can be isolated by the responsibility chain to accelerate the software development speed. In addition, a fast constructor design is constructed in the process of building the responsibility chain to accelerate the construction speed of the responsibility chain, thereby solving the performance problems caused by the responsibility chain.

[0058] In view of the construction of complex processing logic when building application software in the integrated stream-batch architecture, and the need to process complex processing logic when adding or modifying processing modules, the present invention simplifies objects, reduces the coupling between modules, and enhances the flexibility of object assignment responsibilities. It can quickly build a general distributed software architecture. At the same time, it constructs operation entries in different modes (debugging mode, production mode), simplifying the workflow of developers and engineering deployment personnel.

[0059] A responsibility chain processing architecture method based on stream-batch integration, the key step of which is to apply the responsibility chain design pattern in the transform process to complete the deep integration of the stream-batch integration architecture and the responsibility chain. First, the architecture needs to apply to read the data stream and file stream to initialize the execution environment; the second step is to perform operations such as data mapping and key value grouping during the data transformation process; the third step is to execute the responsibility chain by creating a new responsibility chain object in each working node and batch processing the batch data in the rolling window; the fourth step is to aggregate and calculate the data that has passed the responsibility chain and output statistical data; finally, the data generated during the transformation process is output to the sink end and output to the message queue system.

[0060] Step 101: Initialize the engine environment. The initialization mainly includes four steps: connecting to the data source, initializing the operating environment, initializing the responsibility chain Handler, and starting the stream-batch integrated engine;

[0061] Step 201: Connect the data source and initialize the execution environment of the stream-batch engine by setting the abstract function StreamExecutionEnvironment. The input stream data can be Kafka real-time stream message data or file stream. The data source is loaded into the execution environment through the addSource operation.

[0062] Step 202: Initialize the operating environment by defining FlashTransform (T dataStream, Ufactory), where the first parameter is the input data stream and the second is the responsibility chain object; at the same time, the general engine interface AbstractEngine in the engine is inherited to initialize the engine, and multiple execution modes such as debugging mode, cluster mode, and file mode can be created. In the software ecosystem with different roles, corresponding working modes can be set for developers and engineers, and the engine will create an input stream when selecting the execution mode;

[0063] Step 203: Initialize the responsibility chain. The responsibility chain is initialized before the engine is executed. The main process is to implement the createHandler method in the MessageHandlerFactory responsibility chain interface and output the log of the construction time of the responsibility chain to facilitate the operation and maintenance personnel to monitor in real time whether each link in the chain is running normally;

[0064] Step 204: Start the engine by calling the engine variable of step 202 and inheriting the start method of the Engine interface to start the engine;

[0065] Step 102: This step starts the transformer operation to rebalance the incoming data stream. It will poll and allocate all parallel subtasks in a round-robin manner, and scatter and regroup the hashed data read by different threads to help the software achieve fairness and fault tolerance in a distributed environment.

[0066] Step 103: Map the data in a loop to set random numbers. First, define the variable Random. Then call the map method to call random.next(100) to set random numbers for the data hashed in the working node, in preparation for the data uniqueness in the key value grouping process in the subsequent step 105.

[0067] Step 104: Setting timestamps and generating watermarks. To correctly process event time, this method allows event time to set later timestamps for later events, but only when the difference between these events and the current processing time does not exceed the specified tolerance. The framework needs to know the timestamp of each element and understand the progress of event time to determine which elements have been processed;

[0068] Step 105: According to the framework requirements, the key value of step 103 is set and grouped for the data to implement the key-based aggregation operation, and the keyedStream data stream object is generated by performing a Hash modulo operation on the key value and performing a Hash partition operation on the data;

[0069] Step 106: Apply for Tumbling Processing Time Windows, a window type based on data processing time, which groups data into fixed-size panes (or buckets), each of which is non-overlapping. The start time of the pane is determined based on the data processing time. Here, the data size of the batch data is controlled. During the streaming process, the data of the user-set time is accumulated as the data source for the batch step 108;

[0070] Step 107: Initialize the chain of responsibility. In the time window processing of step 106, the initialization method of the chain of responsibility is called through the process function operation. The function parameters are Context, Iterable <t>、Collector <Tuple2<String,Long> >, define the MessageHandlerFactory interface when initializing the chain, define the createHandler() method in the interface, the basic object of the chain is ApacheChainHandler, set the execution function of the chain by defining the configuration items in the chain-config.xml configuration file, and add these functions to the initialized chain object through the addCommand() method. When adding the object, the executor object in the chain will be initialized to quickly initialize the defined class object. In the construction of the executor, the necessary objects to be used and some static objects will be initialized once. These objects will no longer need to be initialized again in subsequent calls. This operation will speed up the construction of the chain, and then execute the chain through the execute method. The responsibility chain is regenerated when the data object is generated. Each batch of data will generate a new chain, including temporary objects that need to be initialized in the chain, and finally return the data structure Collector through the defined <Tuple2<String,Long> >Return the execution result;

[0071] Step 108: Organize batch data, set a receiving list for the data in the scroll window in step 106, and insert this batch of data into the list as data objects when the responsibility chain is executed. These data objects will be passed in the chain by reference, including data deletion, copying, and modification;

[0072] Step 109: triggering execution of the responsibility chain. During the execution, multiple processing modules inherit the same processing interface CommandBase, which defines two main methods: execute and close. Each module receives batch data objects for corresponding processing. It should be noted that the data received by the next chain refers to the data object updated by the previous chain, so as to ensure the order of data processing and the uniqueness of the data object. After execution, the other resources applied by the responsibility chain resources are recycled through the close() method, and the chain is destroyed at the same time.

[0073] Step 110: re-group the data after step 108 by key value. The key value selection in this step is different from that in step 105, and mainly aggregates and counts the processed data.

[0074] Step 111: Apply for a rolling execution time window again. The window time is independent of the time of step 106. This event window is set for statistical output frequency and can control the aggregation time of the output data stream.

[0075] Step 112: Perform a reduce (aggregation) operation on the output data generated in step 111 according to the key value, and perform a reduction operation on the tuple data (Tuple2<String,Long> ), count the pairs of _1 and _2, and output the binary data;

[0076] Step 113: Print the binary data outputted in step 112. After this step is completed, the framework ends the transformer operation at the same time.

[0077] Step 114: The sink operation outputs data to the Kafka message stream: by inheriting SinkFunction

[0078] The KafkaSink of the interface, then set the kafka connection parameter setKafkaProducerConfig and the serialization parameter setRecordSerializer, and finally set the kafka address information setBootstrapServers, and finally implement the invoke method of the sink interface to continuously output data to kafka.

[0079] The key point of the present invention is how to integrate the integrated stream-batch processing framework and the responsibility chain design pattern, and use the distributed processing capabilities provided by the integrated stream-batch big data processing framework combined with the responsibility chain to build a general processing framework.

[0080] The invention is intended to protect:

[0081] 1. Initialize the entire method flow in the environment;

[0082] 2. The entire method flow of integrating the stream-batch integrated processing framework and the responsibility chain design pattern;

[0083] 3. How to build and destroy the chain of responsibility during data processing;

[0084] The existing distributed batch-stream integrated processing framework will frequently modify the existing processing logic during the data conversion process, and adding processing operations on the basis of the existing code will modify the existing code, which is a great challenge to the maintainability of the code. The implementation scheme of the present invention integrates the batch-stream integrated framework and the responsibility chain design pattern, combines the advantages of each scheme, and can quickly build application software with different responsibilities for different distributed processing frameworks. The use of this architecture makes the software architecture design more flexible and more versatile.< / t> < / t>

Claims

1. A big data stream batch integrated processing architecture method based on a fast responsibility chain constructor, characterized in that: The steps include: Step 1: Initialize the engine environment; It includes four steps: connecting to the data source, initializing the operating environment, initializing the responsibility chain handler, and starting the batch-stream integration engine. Step 1-1: Link to data source; Initialize the execution environment of the stream-batch engine by setting the abstract function StreamExecutionEnvironment, and load the data source into the execution environment through the addSource operation; Step 1-2: Initialize the operating environment; By defining FlashTransform (T dataStream, U factory) as an engine variable, the first parameter is the input data stream, and the second is the responsibility chain object; at the same time, the general engine interface AbstractEngine in the engine is inherited to initialize the engine, create debugging mode, cluster mode, and file mode, and set corresponding working modes for developers and engineers in the software ecosystem with different roles. The engine creates an input stream while selecting the execution mode; Step 1-3: Initialize the chain of responsibility; The responsibility chain is initialized before the engine is executed, the createHandler method in the MessageHandlerFactory responsibility chain is implemented, and the construction time of the responsibility chain is output to the log, which is convenient for operation and maintenance personnel to monitor whether each link in the chain is running normally in real time; Step 1-4: Start the stream-batch integration engine; Start the engine by calling the engine variable in step 1-2 and inheriting the start method of the Engine interface; Step 2: Perform transformer operation; Rebalance the incoming data streams; Load balancing performs round-robin allocation for all parallel subtasks and scatters and regroups the hashed data read by different threads, helping the software achieve fairness and fault tolerance in a distributed environment. Step 3: Map the data in a loop and set the key value; First, define the variable Random, then call the map method to call random.next(100) to set the random number for the data hashed on the working node; Step 4: Set timestamp and generate watermark; Allow event time to have later timestamps for later events, but only if those events do not differ from the current processing time by more than a specified tolerance. The framework needs to know the timestamp of each element and understand the progress of event time to determine which elements have been processed. Step 5: According to the framework requirements, set and group the key values ​​in step 3 for the data to implement key-based aggregation operations. Perform Hash modulo operations on the key values ​​and perform Hash partition operations on the data to generate a keyedStream data stream object. Step 6: Apply for Tumbling Processing Time Windows. The window type based on the data processing time groups the data into fixed-size panes, each of which is non-overlapping. The start time of the pane is determined based on the data processing time to control the size of the batch data. During the streaming process, the data of the user-set time is accumulated as the data source for batch step 8. Step 7: Initialize the chain of responsibility; In the time window processing of step 6, the initialization method of the responsibility chain is called through the process function operation, and the function parameters are Context, Iterable <t> 、Collector <Tuple2<String,Long> >, define the MessageHandlerFactory interface when initializing the chain, define the createHandler() method in the interface, the basic object of the chain is ApacheChainHandler, set the execution function of the chain by defining the configuration items in the chain-config.xml configuration file, and add these functions to the initialized chain object through the addCommand() method. When adding the object, the executor object in the chain will be initialized to quickly initialize the defined class object. In the construction of the executor, the necessary objects to be used and some static objects will be initialized once. These objects will no longer need to be initialized again in subsequent calls. This operation will speed up the construction of the chain, and then execute the chain through the execute method. The responsibility chain is regenerated when the data object is generated. Each batch of data will generate a new chain, including temporary objects that need to be initialized in the chain, and finally return the data structure Collector through the defined <Tuple2<String,Long> >Return the execution result;< / t> Step 8: Organize batch data; Set up a receiving list for the data in the scroll window in step 6, and put this batch of data into the list as the data objects when the responsibility chain is executed. These data objects will be passed in the chain by reference, including data deletion, copying, and modification; Step 9: Trigger execution of the responsibility chain; During the execution process, multiple processing modules inherit the same processing interface CommandBase, which defines two main methods, execute and close. Each module receives batch data objects for corresponding processing. It should be noted that the data received by the next chain refers to the data object updated by the previous chain, which can ensure the order of data processing and the uniqueness of data objects. After execution, the close() method is used to recycle other resources applied by the responsibility chain resources and destroy the chain at the same time; Step 10: The data after step 8 is re-grouped by key value, and the key value of this step is aggregated and counted for the processed data; Step 11: Apply for the rolling execution time window again to set the statistical output frequency and control the aggregation time of the output data stream; Step 12: Perform an aggregation reduce operation on the output data generated in step 11 according to the key value, and perform an aggregation reduce operation on the tuple data (Tuple2<String,Long> ), count the element 2 according to the element 1 of the binary data, and output the binary data; Step 13: Print the binary data output by step 12. After this step is completed, the framework ends the transformer operation at the same time. Step 14: The sink operation outputs data to the Kafka message stream: through the KafkaSink that inherits the SinkFunction interface, and then by setting the Kafka connection parameter setKafkaProducerConfig and the serialization parameter setRecordSerializer, and finally setting the Kafka address information setBootstrapServers, finally by implementing the invoke method of the sink interface to continuously output data to Kafka.

2. According to the method of claim 1, a large data stream batch integrated processing architecture method based on a fast responsibility chain constructor is characterized in that: The input stream data in step 2 is Kafka real-time stream message data and offline batch data files.

Images