Big data integration and distribution unit component of hierarchical architecture and use method

Abstract

The invention provides a big data integration and distribution unit component of a hierarchical architecture and a use method, relates to the technical field of data integration and processing, and solves the problems of real-time integration, unified processing and safe distribution of multi-source heterogeneous data. The device adopts a five-layer modular architecture; a data access layer supports multi-source data access and is buffered through Kafka; the cleaning conversion layer performs data cleaning and standardization by using a FlinkSQL, an ETL engine and a rule engine; the intelligent distribution layer judges a data distribution path in combination with a rule engine and a machine learning model; the storage calculation layer adopts a cold and hot separation storage strategy and supports parallel operation of Spark batch processing and Flink stream processing; and the service and security layer provides an interface through an API gateway, and guarantees data security in combination with AES-256 encryption and RBAC access control. Besides, dynamic resource scheduling is realized through Kubernetes, cross-domain model cooperative training is supported by utilizing a federated learning technology, the intelligent shunting decision-making capability is improved, various industrial protocols such as MQTT and Modbus are compatible, and VXLAN tunnel transmission is supported.

Description

Technical Field

[0001] The present invention relates to the technical field of data integration and processing, and in particular to a big data integration and diversion unit component with a layered architecture and a method for using the same. Background Art

[0002] Modern enterprises and IoT systems often face the challenge of integrating and processing massive amounts of data from diverse sources. Due to the diversity of data sources and formats, traditional big data integration approaches face numerous challenges. This is particularly true in the Industrial Internet of Things, where devices from different vendors use varying bus standards, network protocols (such as MQTT and Modbus), and data formats, preventing data sources from being directly connected to a unified platform. Within enterprises, multiple, isolated data collection systems often lack interoperability, creating information silos. Due to protocol incompatibilities and data format differences, traditional solutions require custom development, resulting in high investment and difficulty in scalability.

[0003] Furthermore, efficient processing of both real-time and batch data is a key challenge facing big data architectures. Traditional Lambda architectures separate data into real-time hot paths and offline cold paths for processing, but this requires maintaining two processing chains, resulting in complex systems and high development and maintenance costs. With the advancement of real-time computing technology, the Kappa architecture attempts to unify data processing using a single stream processing engine, but this presents limitations when dealing with heterogeneous data sources, multi-model decision making, and cross-domain collaboration.

[0004] The use of machine learning models has become a trend in big data analytics, but centralized model training often leads to data privacy leaks. Federated learning technology protects data privacy by training models locally and sharing only model parameters. However, combining it with real-time data offload still faces challenges in local feature extraction, model updates, and real-time decision making.

[0005] With the maturity of microservices and containerization technologies, enterprises are increasingly adopting distributed architectures, using Kubernetes for container orchestration and automated scheduling to achieve elastic resource scaling and ensure high system availability and resource utilization. Existing big data integration systems that lack intelligent scheduling mechanisms may struggle to cope with traffic fluctuations and resource competition, leading to performance bottlenecks or wasted resources. Summary of the Invention

[0006] A hierarchical big data integration and distribution unit, including the following modules: The data access module includes a multi-source adapter unit and a protocol parsing engine to access structured, semi-structured, and unstructured data sources; parses industrial IoT protocols (including MQTT and Modbus) and network protocols (including VXLAN and GTP) and converts them into a unified internal format; and buffers output data streams through a message queue (such as Kafka). The cleansing and conversion module connects to the downstream of the data access module and includes: an ETL engine for filtering abnormal data, correcting formats, and standardizing to a predetermined pattern; a rule engine (such as Drools) for generating data labels and diversion preprocessing instructions based on preset business rules; The intelligent traffic diversion module, connected downstream of the cleaning and conversion module, includes: a dual engine of rule engines and machine learning models, which integrate data labels and model prediction results (such as LSTM priority scores) to generate diversion decisions; a dynamic routing unit, which distributes data to real-time processing channels, batch processing channels, or cold / hot storage areas; and a backpressure control unit, which monitors downstream loads and provides feedback to adjust data flow rate. The storage and computing module includes: hot and cold separation storage units, storing hot data in an in-memory database (such as Redis) and cold data in a distributed file system (such as HDFS); stream processing engines (such as Flink) and batch processing engines (such as Spark), supporting real-time computing and offline model training; state snapshot and fault recovery units, ensuring exactly-once semantics of data processing; The service security module includes: a dynamic encryption gateway that uses AES-256 to encrypt data transmission; an RBAC access control unit that provides a multi-tenant permission-isolated data service interface through an API gateway; The device also includes: a dynamic resource scheduler that implements elastic scaling of containerized modules based on Kubernetes; a federated learning unit that is deployed on edge nodes and generates a global diversion decision model by aggregating local model parameters.

[0007] Furthermore, the protocol parsing engine of the data access module also includes: a dynamic protocol loading plug-in that supports hot-swapping of industrial protocols (OPC-UA) and Internet protocols (HTTP / 2); and a VXLAN tunnel encapsulation module to achieve secure transmission of data across networks.

[0008] Furthermore, the dynamic resource scheduler performs the following operations: triggering a photoelectric signal alarm based on the backlog of the Kafka queue and adjusting the parallelism of the Flink operator; and allocating computing resources using three-level priority tags (urgent / high / normal).

[0009] A big data integration and distribution method according to the device includes the following steps: S1. Multi-data collection access and protocol parsing: Debezium is used to capture database change logs, Filebeat is used to collect file logs, and these logs are parsed into a unified format; Debezium is used to capture database change logs, Filebeat is used to collect file logs, and these logs are parsed into a unified format; data is written to Kafka partition queues, and topics are divided by protocol type; S2: Data cleaning and standard conversion, writing data into Kafka partition queues, dividing topics by protocol type and calling the rule engine to add business tags to the data; S3, diversion decision-making and execution: the rule engine matches label routing data; the federated learning model predicts data priority and dynamically adjusts routing weights; S4: Hierarchical storage and computational processing: real-time aggregation and statistics of hot data, with the results stored in the OLAP database; cold data is periodically trained on the LSTM model to update the diversion decision parameters; S5: Federated learning collaborative optimization, periodically training the LSTM model with cold data, and updating the diversion decision parameters; the central server aggregates and generates a global model and sends it to the diversion module.

[0010] Furthermore, the S3 diversion decision and execution also includes: when the Kafka backlog exceeds the threshold, triggering a red light signal and reducing the data intake rate; using idempotence design to avoid repeated consumption and resulting in erroneous results.

[0011] Compared with the prior art, the present invention has the following advantages: The present invention utilizes a layered architecture for big data integration and distribution. This layer supports dynamic access to heterogeneous data from multiple sources, including structured, semi-structured, and unstructured data, through a data access layer. This layer utilizes a multi-source adapter module and a protocol parsing engine (e.g., supporting industrial protocols such as MQTT and Modbus) to achieve standardized processing of heterogeneous data. High-throughput data transfer is achieved through the Kafka message queue, avoiding data silos. After data is accessed from different sources, it is uniformly buffered through Kafka, ensuring the system's flexible access to multiple data sources and avoiding integration difficulties caused by protocol incompatibilities and format differences.

[0012] This invention adopts a hot-cold separation storage strategy, processing real-time data and batch data separately through the storage and computing layer. Hot data is stored in high-performance storage (such as Redis or Druid), supporting low-latency real-time queries and analysis; cold data is stored in low-cost archival storage (such as HDFS or HBase), suitable for batch processing and long-term storage. In terms of processing, a real-time computing engine (such as Flink) is responsible for real-time processing of streaming data, while a batch processing engine (such as Spark) handles offline analysis of cold data. This allows real-time and batch data to be efficiently processed by dedicated processing engines, avoiding the complexity and high maintenance costs of the dual-link architecture in the traditional Lambda architecture.

[0013] This invention addresses the problem of improving model decision-making capabilities while protecting data privacy by introducing federated learning technology. Within the federated learning framework, each data source (such as internal enterprise departments or edge devices) can perform model training locally, sharing only model parameters rather than raw data, thus preventing data privacy leaks. Through global model updates, all parties can collaboratively train a more efficient model. This technology effectively addresses the privacy concerns associated with centralized data aggregation, ensuring that data remains within the domain while still enhancing the decision-making capabilities of intelligent traffic diversion through a global model.

[0014] The present invention adopts a containerized deployment based on Kubernetes and utilizes the automated resource scheduling and elastic scaling functions of Kubernetes to manage the various modules in the big data processing platform. Kubernetes dynamically adjusts the resource allocation of each module according to the actual load, achieving on-demand expansion and contraction. For example, under high load conditions, Kubernetes can automatically increase Flink and Spark instances to ensure the real-time performance and computing performance of data stream processing; under low load conditions, it automatically reduces resources to avoid resource waste. Through this dynamic scheduling mechanism, the high availability, elastic scaling and optimal utilization of resources of the system can be ensured, thereby solving the resource competition and performance bottleneck problems under the microservice architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the method steps of the present invention; Figure 2 is a schematic diagram of the device architecture of the present invention; DETAILED DESCRIPTION

[0016] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is intended only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not intended to limit the scope of protection, applicability, or examples set forth in the claims. The functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described relative to some examples may also be combined in other examples.

[0017] As used herein, the term "including" and its variations are open terms meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification. Example

[0018] A hierarchical big data integration and distribution unit, including the following modules: The data access module includes a multi-source adapter unit and a protocol parsing engine to access structured, semi-structured, and unstructured data sources; parses industrial IoT protocols (including MQTT and Modbus) and network protocols (including VXLAN and GTP) and converts them into a unified internal format; and buffers output data streams through a message queue (such as Kafka). The cleansing and conversion module connects to the downstream of the data access module and includes: an ETL engine for filtering abnormal data, correcting formats, and standardizing to a predetermined pattern; a rule engine (such as Drools) for generating data labels and diversion preprocessing instructions based on preset business rules; The intelligent traffic diversion module, connected downstream of the cleaning and conversion module, includes: a dual engine of rule engines and machine learning models, which integrate data labels and model prediction results (such as LSTM priority scores) to generate diversion decisions; a dynamic routing unit, which distributes data to real-time processing channels, batch processing channels, or cold / hot storage areas; and a backpressure control unit, which monitors downstream loads and provides feedback to adjust data flow rate. The storage and computing module includes: hot and cold separation storage units, storing hot data in an in-memory database (such as Redis) and cold data in a distributed file system (such as HDFS); stream processing engines (such as Flink) and batch processing engines (such as Spark), supporting real-time computing and offline model training; state snapshot and fault recovery units, ensuring exactly-once semantics of data processing; The service security module includes: a dynamic encryption gateway that uses AES-256 to encrypt data transmission; an RBAC access control unit that provides a multi-tenant permission-isolated data service interface through an API gateway; The device also includes: a dynamic resource scheduler that implements elastic scaling of containerized modules based on Kubernetes; a federated learning unit that is deployed on edge nodes and generates a global diversion decision model by aggregating local model parameters.

[0019] Furthermore, the protocol parsing engine of the data access module also includes: a dynamic protocol loading plug-in that supports hot-swapping of industrial protocols (OPC-UA) and Internet protocols (HTTP / 2); and a VXLAN tunnel encapsulation module to achieve secure transmission of data across networks.

[0020] Furthermore, the dynamic resource scheduler performs the following operations: triggering a photoelectric signal alarm based on the backlog of the Kafka queue and adjusting the parallelism of the Flink operator; and allocating computing resources using three-level priority tags (urgent / high / normal).

[0021] A big data integration and distribution method according to the device includes the following steps: S1. Multi-data collection access and protocol parsing: Debezium is used to capture database change logs, Filebeat is used to collect file logs, and these logs are parsed into a unified format; Debezium is used to capture database change logs, Filebeat is used to collect file logs, and these logs are parsed into a unified format; data is written to Kafka partition queues, and topics are divided by protocol type; S2: Data cleaning and standard conversion, writing data into Kafka partition queues, dividing topics by protocol type and calling the rule engine to add business tags to the data; S3: Diversion decision-making and execution: The rule engine matches label routing data; the federated learning model predicts data priority and dynamically adjusts routing weights; S4: Hierarchical storage and computing processing: real-time aggregation and statistics of hot data, with the results stored in the OLAP database; cold data is periodically trained on the LSTM model to update the diversion decision parameters; S5: Federated learning collaborative optimization: cold data is used to periodically train the LSTM model and update the diversion decision parameters. The central server aggregates and generates a global model and sends it to the diversion module. Furthermore, the S3 diversion decision and execution also includes: when the Kafka backlog exceeds the threshold, triggering a red light signal and reducing the data intake rate; using idempotence design to avoid repeated consumption and resulting in erroneous results.

[0022] Example 1: Overall process of data integration and diversion device This embodiment describes the workflow of the device of the present invention in a typical big data scenario, involving the entire process from data collection and processing to data distribution, storage, and service.

[0023] Multi-source data access: Multi-source adapter modules located at each data source initiate data collection tasks. For example, the Debezium connector deployed at the database source monitors updates to the business database and sends the changed data as events to a Kafka topic. At the log file source, Filebeat reads newly added log lines and pushes them to the corresponding Kafka topic. At the IoT device source, the edge gateway subscribes to industrial sensor data via MQTT or polls PLC registers via Modbus. The collected data is standardized by the protocol parsing engine and sent to Kafka. The Kafka cluster establishes separate topics for caching data from different sources.

[0024] Real-time cleaning and conversion: The Flink job of the cleaning and conversion layer continuously consumes data streams from the above-mentioned Kafka topic. For each piece of data, the format is first standardized and verified and cleaned (such as removing abnormal characters and filling in missing field default values); then the data is mapped into a unified internal object. For example, a database update event is converted into a standard JSON format, and the fields are uniformly named. Next, the Drools rule engine makes business rule judgments on the data based on the preset rule set: for example, it detects whether the temperature sensor data exceeds the threshold range. If so, the "alarm" label is added to the object. The judgment results of the rule engine and the added labels will be attached to the data object for subsequent use. After this series of ETL processing, the data is output to the downstream diversion decision module.

[0025] Intelligent Triage Decision-Making: In the intelligent triage layer, the dual-engine modules of rules and models analyze and make decisions on data from the cleansing and transformation layer. They first apply the triage rules configured in the rules engine to make preliminary judgments. For example, based on the "alarm" tag in the data, urgent alarm data can be immediately marked as requiring a fast alarm channel. Simultaneously, for complex scenarios, pre-trained machine learning models are used for evaluation. For example, for user behavior log data, the model can score the impact of the log on business metrics. The results of the rules and models work together: the triage module assigns a destination identifier to each data entry (such as "real-time stream processing" or "batch storage"). Next, the triage execution unit routes the data to different channels or storage based on these identifiers. Specifically, high-priority data is sent directly to the real-time computing channel (for example, to another Kafka topic for subscription by the real-time alarm processing service); general business data is sent to the standard processing channel (for example, to the next Kafka topic for batch processing); and certain data objects requiring long-term preservation are directly written to cold storage in the data lake via the storage interface. If the model's online learning mode is enabled, some data and subsequent results will be fed back to the federated learning module for model updates. The entire diversion process occurs in milliseconds, ensuring that data flowing through the system is quickly classified and processed.

[0026] Data storage and computing: Data routed to different channels enters the storage and computing layer. For data from real-time channels, such as alarm events, complex event processing (CEP) or aggregation calculations may be immediately performed by the real-time computing engine (Flink task), and the results are then pushed to the in-memory database for query by the front-end system. This hot data is also synchronously written to hot storage (such as Redis or HBase) for short-term historical queries. For data from batch channels, data from Kafka is periodically extracted in batches and written by Spark jobs to distributed file storage (such as Parquet files on HDFS) as cold data archives. Spark offline tasks can be run daily or weekly to analyze cold data, generate reports, or train new machine learning models. For example, a week's worth of logs can be aggregated to calculate user retention rates, or the models used in the intelligent distribution layer can be trained and updated. The trained models are distributed to the model engine of the intelligent distribution layer through the model publishing module, enabling continuous iterative optimization of the models. The storage computing layer is responsible for the migration of hot and cold data: when the data in the hot storage exceeds a certain age (such as 3 months) and is no longer frequently accessed, the system automatically marks it as cold data and transfers it to cold storage (such as cloud object storage), retaining only index or summary information in the hot storage, thereby reducing storage costs.

[0027] Service Calls and Security Controls: External applications or users access processed data and analysis results through a unified API interface provided by the service and security layer. For example, operations personnel can call an API from the front-end dashboard to obtain a real-time alert list (the request is routed by the API gateway to the alert service, which retrieves the latest alert data from Redis and returns it to the front-end). Another example is a data scientist calling the analysis report API to obtain a weekly report generated by batch processing (the request is routed by the API gateway to the report service, which retrieves the results from HDFS cold storage). In these interactions, the API gateway performs identity authentication and RBAC permission checks on each request to ensure that only authorized users can access specific data. Data is automatically encrypted and decrypted by the gateway (using AES-256 keys) during transmission to ensure secure communication. All request and response processes are logged for review. Notably, when new services are added or policies are updated, the dynamic encryption gateway allows security policies and routing rules to be updated without downtime, ensuring continuous service availability. Through this layer, the functionality of the entire big data integration and distribution system is exposed to the public as a service, in a secure and controllable manner.

[0028] Federated Learning Collaborative Optimization (Optional Process): During the above process, the federated learning modules of the present invention can operate in parallel to enhance the intelligent decision-making capabilities of the model engine. Assuming that the present device is deployed on multiple geographically distributed nodes, each node processes local business data and cannot directly share data for privacy reasons. The federated learning module in each node then uses local data to perform model training (for example, training a predictive model to assess the business impact of logs). After local training is complete, only the model parameter gradients are uploaded to a central server for aggregation. Upon receiving model updates from each node, the central server performs a weighted average to generate a global model, which is then distributed to the model engines of each node. This allows each node to collaboratively train a more accurate model without exchanging raw data. Once the global model is updated, the intelligent traffic diversion layer uses the new model for decision-making, thereby continuously improving the traffic diversion effect. For example, federated learning allows a node to benefit from the data contributions of other nodes, even if it lacks sample data for a certain category, thereby improving the model's accuracy in identifying rare events. The entire federated learning process is executed asynchronously in the background, without affecting the real-time performance of foreground data processing.

[0029] This embodiment demonstrates how the device of the present invention operates in a complex big data environment. Through the above process, it can be seen that the present invention realizes the full link of data from collection and access, cleaning and standardization, intelligent judgment and diversion, classification and storage processing to final security services. Each step adopts corresponding optimization technology and innovative design to ensure that the system runs efficiently, reliably and intelligently. In actual applications, users can cut or expand part of the process as needed, for example, only use part of the hierarchical functions, or adjust the rules and models to adapt to specific scenarios. The architecture of the present invention has good versatility and flexibility, and can be applied to various fields such as industrial Internet, smart cities, financial risk control, large enterprise data platforms, etc., to solve the pain points of multi-source data fusion and intelligent distribution.

[0030] Those skilled in the art will appreciate that the various embodiments disclosed above may be modified and altered without departing from the essence of the invention. Therefore, the scope of protection of the present invention shall be defined by the appended claims.

[0031] It should be noted that not all steps and units in the above processes are required, and certain steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.

[0032] The specific embodiments set forth above describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance, or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0033] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. It will be apparent to those of ordinary skill in the art that various modifications to the present disclosure are possible, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein. The general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A big data integration and distribution unit with a layered architecture, characterized by: Includes the following modules: The data access module includes a multi-source adapter unit and a protocol parsing engine to access structured, semi-structured, and unstructured data sources; parses industrial IoT protocols (including MQTT and Modbus) and network protocols (including VXLAN and GTP) and converts them into a unified internal format; and buffers output data streams through a message queue (such as Kafka). The cleansing and conversion module connects to the downstream of the data access module and includes: an ETL engine for filtering abnormal data, correcting formats, and standardizing to a predetermined pattern; a rule engine (such as Drools) for generating data labels and diversion preprocessing instructions based on preset business rules; The intelligent traffic diversion module, connected downstream of the cleaning and conversion module, includes: a dual engine of rule engines and machine learning models, which integrate data labels and model prediction results (such as LSTM priority scores) to generate diversion decisions; a dynamic routing unit, which distributes data to real-time processing channels, batch processing channels, or cold / hot storage areas; and a backpressure control unit, which monitors downstream loads and provides feedback to adjust data flow rate. The storage and computing module includes: hot and cold separation storage units, storing hot data in an in-memory database (such as Redis) and cold data in a distributed file system (such as HDFS); stream processing engines (such as Flink) and batch processing engines (such as Spark), supporting real-time computing and offline model training; state snapshot and fault recovery units, ensuring exactly-once semantics of data processing; The service security module includes: a dynamic encryption gateway that uses AES-256 to encrypt data transmission; an RBAC access control unit that provides a multi-tenant permission-isolated data service interface through an API gateway; The device also includes: a dynamic resource scheduler that implements elastic scaling of containerized modules based on Kubernetes; a federated learning unit that is deployed on edge nodes and generates a global diversion decision model by aggregating local model parameters.

2. The hierarchical big data integration and distribution unit according to claim 1, characterized in that: The protocol parsing engine of the data access module also includes: a dynamic protocol loading plug-in that supports hot-swapping of industrial protocols (OPC-UA) and Internet protocols (HTTP / 2); and a VXLAN tunnel encapsulation module to achieve secure transmission of data across networks.

3. The hierarchical big data integration and distribution unit component and method of use according to claim 1, characterized in that: The dynamic resource scheduler performs the following operations: triggering photoelectric signal alarms based on the backlog in the Kafka queue and adjusting the parallelism of Flink operators; and allocating computing resources using three-level priority tags (urgent / high / normal).

4. A method for integrating and distributing big data according to the device of any one of claims 1 to 3, characterized in that: The following steps are involved: S1. Multi-data collection access and protocol parsing: Debezium is used to capture database change logs, Filebeat is used to collect file logs, and these logs are parsed into a unified format; Debezium is used to capture database change logs, Filebeat is used to collect file logs, and these logs are parsed into a unified format; data is written to Kafka partition queues, and topics are divided by protocol type; S2: Data cleaning and standard conversion, writing data into Kafka partition queues, dividing topics by protocol type and calling the rule engine to add business tags to the data; S3: Diversion decision-making and execution: The rule engine matches label routing data; the federated learning model predicts data priority and dynamically adjusts routing weights; S4: Hierarchical storage and computational processing: real-time aggregation and statistics of hot data, with the results stored in the OLAP database; cold data is periodically trained on the LSTM model to update the diversion decision parameters; S5: Federated learning collaborative optimization, periodically training the LSTM model with cold data, and updating the diversion decision parameters; the central server aggregates and generates a global model and sends it to the diversion module.

5. A big data integration and diversion method according to claim 4, characterized in that: The S3 diversion decision and execution also includes: when the Kafka backlog exceeds the threshold, triggering a red light signal and reducing the data intake rate; using idempotence design to avoid repeated consumption and resulting in erroneous results.

Images