NiFi-based large-data-volume high-reliability data acquisition system and method

Abstract

The invention relates to a large-data-volume and high-reliability data acquisition system and method based on NiFi, and belongs to the technical field of big data processing and data governance. According to the system, a data assembly line with NiFi as a core is constructed, high-grade characteristics of data traceability, replay, flow control, extended processor architecture and the like of the data assembly line are integrated, and efficient collection and cleaning of multi-source heterogeneous data are achieved; rabbitMQ / Kafka is adopted as asynchronous message middleware for system decoupling, regular storage of data is completed in combination with a database temporary table and a timed batch processing mechanism, and the problem of message blocking in a high-concurrency scene is effectively avoided. Dynamic current-limiting control and an elastic acquisition agent cluster are introduced into the system, stable access and processing of mass data are supported, the reliability, traceability and maintainability of a data link are guaranteed, and the technical problems that a traditional acquisition system is prone to data loss, difficult to track abnormities, high in processing delay and the like under a high load are solved.

Description

Technical Field

[0001] This invention belongs to the field of big data processing and governance technology, and in particular to a high-reliability data acquisition system and method for large data volumes based on NiFi. Background Technology

[0002] In high-frequency data scenarios such as the Industrial Internet of Things (IIoT), logistics loading and unloading monitoring, and financial transaction records, data acquisition systems serve as the "first line of defense" in information flow, undertaking the core functions of acquiring, preprocessing, and initially distributing raw data. However, with the expansion of business scale and the increase in data complexity, existing systems have generally exposed the following technical bottlenecks, which seriously restrict their reliability, flexibility, and scalability.

[0003] I. Untraceable Data: Missing Processing Tracks Lead to Operational and Maintenance Challenges Most current systems adopt a "one-way pipeline" architecture, lacking fine-grained logging in the intermediate stages from data acquisition to storage (such as format conversion, field mapping, and preliminary validation). When data anomalies occur (such as missing fields or sudden numerical changes) or business process failures (such as failed downstream service calls), operations personnel can only obtain the final output or general error messages, unable to trace back to which specific stage the data was modified, which module processed it, and what the processing parameters were. For example, in a logistics loading and unloading monitoring system, vehicle load data collected by sensors was automatically corrected to a "reasonable range value" during the cleaning stage, but the original out-of-limit data and the correction logic were not recorded, making it difficult to determine whether the subsequent tracing was due to sensor failure or system misjudgment. This "black box" processing mode greatly increases the cost of fault location, especially in long-link, multi-node distributed scenarios, where problem-solving efficiency may drop by more than 70%.

[0004] II. Rigid Processing Capacity: Stability Risks Under Burst Traffic: Traditional systems are typically designed based on fixed resource pools (such as preset thread counts and queue lengths), making their processing capacity strongly tied to hardware configuration and lacking elastic scaling mechanisms. When business peaks suddenly arrive (such as order surges during e-commerce promotions or sensor data bursts during peak port loading and unloading operations), the system often experiences a surge in processing latency due to message queue backlogs; if the preset threshold is exceeded, it directly triggers service crashes (such as memory overflows or thread deadlocks). For example, a port video surveillance data acquisition system needed to process real-time streams from hundreds of high-definition cameras simultaneously during typhoon weather. The fixed allocation of 10 processing threads was saturated within 30 minutes, and subsequent data was discarded due to queue overload, creating blind spots in monitoring during critical periods. This "rigid bottleneck" makes the system difficult to adapt to non-steady-state business demands, significantly reducing availability.

[0005] 3. High system coupling: rigid modules hinder functional expansion. Most systems deeply bind functional modules such as data acquisition (e.g., sensor access, network packet capture), preprocessing (e.g., format standardization, deduplication), and business logic processing (e.g., rule validation, classification labeling), achieving collaboration through tightly coupled code structures or single-process calls. While this design simplifies initial development, it sacrifices module independence—when a new data source needs to be added (e.g., adding a new IoT device protocol) or the processing logic needs to be adjusted (e.g., modifying data cleaning rules), the entire system must be redeployed or the code must be modified. For example, an industrial data acquisition platform initially only supported Modbus protocol devices. Later, when OPC UA devices were added, because the acquisition and processing modules shared the same communication framework, the development team had to refactor the core code, which took up to two months and severely impacted business iteration efficiency.

[0006] IV. Fixed cleaning logic: difficult to adapt to dynamic business rules. The data cleansing logic of existing systems (such as outlier filtering, deduplication, and field completion) is usually hard-coded into the program or defined by limited rule templates through static configuration files (such as JSON / XML). When business needs change (such as regulatory requirements for adding mandatory data fields or analytical models requiring finer data granularity) or data source characteristics change (such as sensor accuracy upgrades causing the original "reasonable range" to become invalid), the system cannot be flexibly adjusted. For example, a financial institution's transaction data collection system was originally set to "automatically fill in 0 if the amount field is empty," but new compliance requirements require retaining the original empty value to distinguish between "no transaction" and "zero transaction." Because the cleansing logic cannot be dynamically modified, it can only be temporarily corrected through manual intervention, leading to data consistency risks.

[0007] V. Large Data Volume Blocking: Write Database Bottleneck Dragging Down Overall Performance To ensure data persistence, most systems adopt a "real-time database write" strategy, where collected data is initially processed and then directly inserted into relational databases (such as MySQL and Oracle) in batches. However, in high-concurrency write scenarios, the database's row-level and table-level locking mechanisms can lead to severe resource contention—when multiple write threads operate on the same table simultaneously, later requests must wait for preceding transactions to release locks, resulting in thread blocking, soaring write latency, or even deadlocks. For example, a video surveillance system in a logistics park generates over 5,000 vehicle trajectory data points per second during peak hours. The average response time for database write operations increased from the normal 50ms to over 2 seconds, not only slowing down the real-time display functionality of the front end but also preventing new data collection tasks from being submitted due to prolonged connection pool occupation, creating a vicious cycle of "the more congested, the slower it gets." The aforementioned bottlenecks collectively make it difficult for existing data acquisition systems to meet the needs of complex business scenarios in terms of reliability, scalability, and adaptability. There is an urgent need to break through these limitations through architectural restructuring (such as introducing message middleware for decoupling and adopting microservice design) and technology upgrades (such as using distributed storage to replace centralized databases and using rule engines to achieve dynamic cleaning), and evolve towards a more "highly elastic, observable, and easily scalable" intelligent system. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and propose a high-reliability data acquisition system and method based on NiFi for large data volumes. This system can achieve efficient, reliable, and traceable acquisition and processing of multi-source heterogeneous data, effectively addressing the challenges of system stability and scalability in high-concurrency, large-data-volume scenarios.

[0009] The technical problem solved by this invention is achieved through the following technical solution: A NiFi-based high-reliability data acquisition system for large data volumes includes a data acquisition access module, a data cleaning and enhancement module, a data traceability and replay module, a message buffering and decoupling module, a batch normalization processing module, a dynamic rate limiting control module, and an elastic acquisition agent module. The data acquisition and access module receives data, which then sequentially enters the data cleaning and enhancement module, the traceability and replay module, the message buffering and decoupling module, and the batch normalization processing module. These steps ensure data optimization and system stability, respectively. The dynamic rate limiting control module monitors the status of each link in the system in real time and adjusts the traffic of the data acquisition and access module accordingly. The elastic acquisition proxy module works in conjunction with the data acquisition and access module, supporting horizontal scaling. The data acquisition and access module receives data from multiple heterogeneous data sources. The data cleaning and enhancement module performs format conversion, field mapping, null value filling, and data anonymization on the data. The data traceability and replay module records the entire lifecycle event trajectory during data processing. The message buffering and decoupling module pushes the processed data to a designated RabbitMQ queue. When the message is consumed by downstream consumers and written to a temporary table in the database, it enters the batch normalization processing module. The dynamic rate limiting control module collects the NiFi queue backlog, RabbitMQ queue length, and database response latency in real time, and adjusts the backpressure threshold on the NiFi connection through a feedback algorithm. The elastic acquisition proxy module has its task load uniformly scheduled by the NiFi cluster coordinator, supporting horizontal scaling to cope with traffic peaks.

[0010] Furthermore, the data traceability and replay module includes full-link event tracking, a visual traceability analysis interface, and an abnormal data replay mechanism. The link event tracking is used to record every operation event that the data undergoes in NiFi. The specific implementation method is as follows: enable NiFi's Provenance Repository traceability repository, configure it to record all types of events, including INGRESS data entry, FORK fork, ROUTE routing, MODIFY modification, CONTENT_MODIFIED content change, and EGRESS output, with a storage period of no less than 30 days, and use a Write-Ahead Log mechanism to ensure the persistence of traceability logs. The visual source tracing analysis interface is used to provide a timeline view to display the data flow path and processing latency. The specific implementation method is as follows: through NiFi's built-in UI or by integrating a custom Grafana plugin, a timeline view is provided to display the complete flow path of a single FlowFile from source to destination, and the processing latency, operation type and operator information of each node are overlaid; it supports searching by UUID, time range and source system; The abnormal data replay mechanism is used to re-inject failed or erroneous data from any processing node for reprocessing. The specific implementation method is as follows: when a data processing failure is detected, the system automatically extracts its corresponding ProvenanceEvent ID and calls the NiFi REST API / provenance-events / {id} / replay interface to re-inject the FlowFile at the specified Processor node; at the same time, the original context attributes are preserved to ensure semantic consistency in the replay process.

[0011] A data acquisition method for a high-reliability data acquisition system based on NiFi for large data volumes includes the following steps: Step 1: The data acquisition and access module receives data from multiple heterogeneous data sources and injects it into the NiFi data stream pipeline; Step 2: The data output from the NiFi data stream pipeline is transmitted to the data cleaning and enhancement module, where format conversion, field mapping, null value filling, and desensitization are completed in the NiFi processor chain. Step 3: During the data cleaning and enhancement module processing, the data traceability and replay module automatically records the entire lifecycle event trajectory of each data packet FlowFile; Step 4: The cleaned data is pushed to the specified queue in RabbitMQ by the message buffer and decoupling module through the PutAMQP or PublishMQTT processor; Step 5: After the RabbitMQ message is consumed by the downstream consumer, it is written to the database temporary table and then enters the batch smoothing processing module. The batch smoothing processing module executes a timed ETL task to write the smoothed data to the target business table. Step 6: The dynamic flow limiting control module performs overall control by collecting real-time data on NiFi queue backlog, RabbitMQ queue length, and database response latency. It then adjusts the back pressure threshold on the NiFi connection using a feedback algorithm to control the overall system flow and prevent congestion. Step 7: The elastic acquisition agent module is deployed as a distributed cluster, with the NiFi cluster coordinator uniformly scheduling the task load. It supports horizontal scaling to cope with traffic peaks and ensures the stable operation of the system in high-concurrency scenarios.

[0012] Furthermore, step 3 includes the following steps: Step 3.1: Implement data validation and completion for specific business rules based on Java / Python extensions; Step 3.2: Integrate the ExecuteScript processor to support the dynamic execution of cleaning logic using Groovy and Jython scripts; Step 3.3: Attach context metadata to the data packet using the UpdateAttribute processor.

[0013] Furthermore, step 4 includes the following steps: Step 4.1: Establish a Topic Exchange distribution mechanism based on data type, priority, region, and other dimensions; Step 4.2: Ensure messages are not lost due to service interruption; Step 4.3: Capture the message of processing failure for subsequent troubleshooting and retrying.

[0014] Furthermore, step 5 includes the following steps: Step 5.1: The RabbitMQ consumer writes the original message to a temporary table in the database to avoid lock contention caused by directly operating the main table; Step 5.2: Execute ETL scripts periodically using Quartz or Spring Scheduler to clean, deduplicate, and aggregate the data in the temporary table before writing it into the formal table; Step 5.3: Partition the temporary table by time or business dimension to improve the efficiency of batch processing.

[0015] Furthermore, step 6 includes the following steps: Step 6.1: Collect metrics such as NiFi queue backlog, RabbitMQ queue length, and database write latency; Step 6.2: Dynamically adjust the flow threshold of the NiFi connection based on the PID control principle; Step 6.3: Set up high-priority channels for critical business data to ensure low-latency transmission.

[0016] The advantages and positive effects of this invention are: 1. Achieve full lifecycle traceability and replayability of data, improving system observability and fault location efficiency; 2. By employing dynamic rate limiting and elastic proxy mechanisms, the system is ensured to operate stably under high-concurrency scenarios; 3. By combining message middleware with batch processing, the database write bottleneck can be effectively alleviated; 4. Supports flexible access and dynamic cleaning of multi-source heterogeneous data to adapt to changes in complex business needs. Attached Figure Description

[0017] Figure 1 This is a schematic diagram showing the module composition and connection relationship of the system of the present invention; Figure 2 This is a schematic diagram of the NIFI configuration of the present invention; Figure 3 This is a schematic diagram of the data acquisition process of this invention; Figure 4This is a schematic diagram of the data acquisition and decoupling process of the present invention; Figure 5 This is a schematic diagram illustrating the application of the CDC architecture of the present invention in other applications. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to the accompanying drawings.

[0019] Figure 1 The overall framework of the project includes the NIFI processing optimization section and the business processing section.

[0020] Figure 2 The main focus is on showing screenshots of the overall grouping, data acquisition, and cleaning processes obtained from NIFI. Figure 1 The left half of the diagram shows how all configurations can be achieved through a visual configuration interface.

[0021] A NiFi-based high-reliability data acquisition system for large datasets includes a data acquisition and access module, a data cleaning and enhancement module, a data traceability and replay module, a message buffering and decoupling module, a batch normalization processing module (the first few modules correspond to the ReceiveAndClean group), a dynamic rate limiting control module, and an elastic acquisition proxy module (corresponding to the DataLimit group). Some overall processing and algorithms are implemented within... Figure 1 In FlowMonitor, this is achieved through a custom component. The data acquisition and access module receives data, which then sequentially enters the data cleaning and enhancement module, the traceability and replay module, the message buffering and decoupling module, and the batch normalization processing module. These steps ensure data optimization and system stability, respectively. The dynamic rate limiting control module monitors the status of each link in the system in real time and adjusts the traffic of the data acquisition and access module accordingly. The elastic acquisition proxy module works in conjunction with the data acquisition and access module, supporting horizontal scaling. The data acquisition and access module receives data from multiple heterogeneous data sources. The data cleaning and enhancement module performs format conversion, field mapping, null value filling, and data anonymization on the data. The data traceability and replay module records the entire lifecycle event trajectory during data processing. The message buffering and decoupling module pushes the processed data to a designated RabbitMQ queue. When the message is consumed by downstream consumers and written to a temporary table in the database, it enters the batch normalization processing module. The dynamic rate limiting control module collects the NiFi queue backlog, RabbitMQ queue length, and database response latency in real time, and adjusts the backpressure threshold on the NiFi connection through a feedback algorithm. The elastic acquisition proxy module has its task load uniformly scheduled by the NiFi cluster coordinator, supporting horizontal scaling to cope with traffic peaks.

[0022] Furthermore, the data traceability and replay module includes full-link event tracking, a visual traceability analysis interface, and an abnormal data replay mechanism. The link event tracking is used to record every operation event that the data undergoes in NiFi. The specific implementation method is as follows: enable NiFi's Provenance Repository traceability repository, configure it to record all types of events, including INGRESS data entry, FORK fork, ROUTE routing, MODIFY modification, CONTENT_MODIFIED content change, and EGRESS output, with a storage period of no less than 30 days, and use a Write-Ahead Log mechanism to ensure the persistence of traceability logs. The visual source tracing analysis interface is used to provide a timeline view to display the data flow path and processing latency. The specific implementation method is as follows: through NiFi's built-in UI or by integrating a custom Grafana plugin, a timeline view is provided to display the complete flow path of a single FlowFile from source to destination, and the processing latency, operation type and operator information of each node are overlaid; it supports searching by UUID, time range and source system; The abnormal data replay mechanism is used to re-inject failed or erroneous data from any processing node for reprocessing. The specific implementation method is as follows: when a data processing failure is detected, the system automatically extracts its corresponding ProvenanceEvent ID and calls the NiFi REST API / provenance-events / {id} / replay interface to re-inject the FlowFile at the specified Processor node; at the same time, the original context attributes are preserved to ensure semantic consistency in the replay process.

[0023] A data acquisition method for a high-reliability data acquisition system based on NiFi for large data volumes includes the following steps: Step 1: The data acquisition and access module receives data from multiple heterogeneous data sources and injects it into the NiFi data stream pipeline; Step 2: The data output from the NiFi data stream pipeline is transmitted to the data cleaning and enhancement module, where format conversion, field mapping, null value filling, and desensitization are completed in the NiFi processor chain. Step 3: During the data cleaning and enhancement module processing, the data traceability and replay module automatically records the entire lifecycle event trajectory of each data packet FlowFile; Step 4: The cleaned data is pushed to the specified queue in RabbitMQ by the message buffer and decoupling module through the PutAMQP or PublishMQTT processor; Step 5: After the RabbitMQ message is consumed by the downstream consumer, it is written to the database temporary table and then enters the batch smoothing processing module. The batch smoothing processing module executes a timed ETL task to write the smoothed data to the target business table. Step 6: The dynamic flow limiting control module performs overall control by collecting real-time data on NiFi queue backlog, RabbitMQ queue length, and database response latency. It then adjusts the back pressure threshold on the NiFi connection using a feedback algorithm to control the overall system flow and prevent congestion. Step 7: The elastic acquisition agent module is deployed as a distributed cluster, with the NiFi cluster coordinator uniformly scheduling the task load. It supports horizontal scaling to cope with traffic peaks and ensures the stable operation of the system in high-concurrency scenarios.

[0024] Furthermore, step 3 includes the following steps: Step 3.1: Implement data validation and completion for specific business rules based on Java / Python extensions; Step 3.2: Integrate the ExecuteScript processor to support the dynamic execution of cleaning logic using Groovy and Jython scripts; Step 3.3: Attach context metadata to the data packet using the UpdateAttribute processor.

[0025] Furthermore, step 4 includes the following steps: Step 4.1: Establish a Topic Exchange distribution mechanism based on data type, priority, region, and other dimensions; Step 4.2: Ensure messages are not lost due to service interruption; Step 4.3: Capture the message of processing failure for subsequent troubleshooting and retrying.

[0026] Furthermore, step 5 includes the following steps: Step 5.1: The RabbitMQ consumer writes the original message to a temporary table in the database to avoid lock contention caused by directly operating the main table; Step 5.2: Execute ETL scripts periodically using Quartz or Spring Scheduler to clean, deduplicate, and aggregate the data in the temporary table before writing it into the formal table; Step 5.3: Partition the temporary table by time or business dimension to improve the efficiency of batch processing.

[0027] Furthermore, step 6 includes the following steps: Step 6.1: Collect metrics such as NiFi queue backlog, RabbitMQ queue length, and database write latency; Step 6.2: Dynamically adjust the flow threshold of the NiFi connection based on the PID control principle; Step 6.3: Set up high-priority channels for critical business data to ensure low-latency transmission.

[0028] Figure 3 and Figure 4 The corresponding business logic processing involves some implementation details, mainly reflecting the decoupling of the business logic. The processing of various data will not affect each other. The entire system is in a low-coupling state, with a single responsibility, strong robustness, and strong scalability.

[0029] Figure 5 This showcases additional application use cases of the CDC and channel technologies used in the platform. The data bus component is the technical framework adopted within the project, primarily supporting real-time broadcasting and synchronization of data changes. Although it is only used in this project for broadcasting abnormal and alarm data and some high real-time data notifications, it can actually be used more broadly for interaction between multiple systems, as illustrated in the order system shown in the figure.

[0030] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. A high-reliability data acquisition system for large data volumes based on NiFi, characterized in that: It includes a data acquisition and access module, a data cleaning and enhancement module, a data traceability and replay module, a message buffering and decoupling module, a batch normalization and processing module, a dynamic rate limiting control module, and an elastic acquisition agent module. The data acquisition and access module receives data, which then sequentially enters the data cleaning and enhancement module, the traceability and replay module, the message buffering and decoupling module, and the batch normalization processing module. These steps ensure data optimization and system stability, respectively. The dynamic rate limiting control module monitors the status of each link in the system in real time and adjusts the traffic of the data acquisition and access module accordingly. The elastic acquisition proxy module works in conjunction with the data acquisition and access module, supporting horizontal scaling. The data acquisition and access module receives data from multiple heterogeneous data sources. The data cleaning and enhancement module performs format conversion, field mapping, null value filling, and data anonymization on the data. The data traceability and replay module records the entire lifecycle event trajectory during data processing. The message buffering and decoupling module pushes the processed data to a designated RabbitMQ queue. When the message is consumed by downstream consumers and written to a temporary table in the database, it enters the batch normalization processing module. The dynamic rate limiting control module collects the NiFi queue backlog, RabbitMQ queue length, and database response latency in real time, and adjusts the backpressure threshold on the NiFi connection through a feedback algorithm. The elastic acquisition proxy module has its task load uniformly scheduled by the NiFi cluster coordinator, supporting horizontal scaling to cope with traffic peaks.

2. The NiFi-based high-reliability data acquisition system for large data volumes according to claim 1, characterized in that: The data traceability and replay module includes full-link event tracking, a visual traceability analysis interface, and an abnormal data replay mechanism. The link event tracking is used to record every operation event that the data undergoes in NiFi. The specific implementation method is as follows: enable NiFi's Provenance Repository traceability repository, configure it to record all types of events, including INGRESS data entry, FORK fork, ROUTE routing, MODIFY modification, CONTENT_MODIFIED content change, and EGRESS output. The storage period is not less than 30 days, and the Write-Ahead Log mechanism is used to ensure the persistence of traceability logs. The visual source tracing analysis interface is used to provide a timeline view to display the data flow path and processing latency. The specific implementation method is as follows: through NiFi's built-in UI or by integrating a custom Grafana plugin, a timeline view is provided to display the complete flow path of a single FlowFile from source to destination, and the processing latency, operation type and operator information of each node are overlaid; it supports searching by UUID, time range and source system; The abnormal data replay mechanism is used to re-inject failed or erroneous data from any processing node for reprocessing. The specific implementation method is as follows: when a data processing failure is detected, the system automatically extracts its corresponding Provenance EventID and calls the NiFi REST API / provenance-events / {id} / replay interface to re-inject the FlowFile at the specified Processor node; at the same time, the original context attributes are preserved to ensure semantic consistency in the replay process.

3. A data acquisition method for a high-reliability data acquisition system based on NiFi with large data volumes as described in any one of claims 1 to 2, characterized in that: Includes the following steps: Step 1: The data acquisition and access module receives data from multiple heterogeneous data sources and injects it into the NiFi data stream pipeline; Step 2: The data output from the NiFi data stream pipeline is transmitted to the data cleaning and enhancement module, where format conversion, field mapping, null value filling, and desensitization are completed in the NiFi processor chain. Step 3: During the data cleaning and enhancement module processing, the data traceability and replay module automatically records the entire lifecycle event trajectory of each data packet FlowFile; Step 4: The cleaned data is pushed to the specified queue in RabbitMQ by the message buffer and decoupling module through the PutAMQP or PublishMQTT processor; Step 5: After the RabbitMQ message is consumed by the downstream consumer, it is written to the database temporary table and then enters the batch smoothing processing module. The batch smoothing processing module executes a timed ETL task to write the smoothed data to the target business table. Step 6: The dynamic flow limiting control module performs overall control by collecting real-time data on NiFi queue backlog, RabbitMQ queue length, and database response latency. It then adjusts the back pressure threshold on the NiFi connection using a feedback algorithm to control the overall system flow and prevent congestion. Step 7: The elastic acquisition agent module is deployed as a distributed cluster, with the NiFi cluster coordinator uniformly scheduling the task load. It supports horizontal scaling to cope with traffic peaks and ensures the stable operation of the system in high-concurrency scenarios.

4. The data acquisition method of a high-reliability data acquisition system based on NiFi for large data volumes according to claim 3, characterized in that: Step 3 includes the following steps: Step 3.1: Implement data validation and completion for specific business rules based on Java / Python extensions; Step 3.2: Integrate the ExecuteScript processor to support the dynamic execution of cleaning logic using Groovy and Jython scripts; Step 3.3: Attach context metadata to the data packet using the UpdateAttribute processor.

5. The data acquisition method of a high-reliability data acquisition system based on NiFi for large data volumes according to claim 3, characterized in that: Step 4 includes the following steps: Step 4.1: Establish a Topic Exchange distribution mechanism based on data type, priority, region, and other dimensions; Step 4.2: Ensure messages are not lost due to service interruption; Step 4.3: Capture the message of processing failure for subsequent troubleshooting and retrying.

6. The data acquisition method of a high-reliability data acquisition system based on NiFi for large data volumes according to claim 3, characterized in that: Step 5 includes the following steps: Step 5.1: The RabbitMQ consumer writes the original message to a temporary table in the database to avoid lock contention caused by directly operating the main table; Step 5.2: Execute ETL scripts periodically using Quartz or Spring Scheduler to clean, deduplicate, and aggregate the data in the temporary table before writing it into the formal table; Step 5.3: Partition the temporary table by time or business dimension to improve the efficiency of batch processing.

7. The data acquisition method of a high-reliability data acquisition system based on NiFi for large data volumes according to claim 3, characterized in that: Step 6 includes the following steps: Step 6.1: Collect metrics such as NiFi queue backlog, RabbitMQ queue length, and database write latency; Step 6.2: Dynamically adjust the flow threshold of the NiFi connection based on the PID control principle; Step 6.3: Set up high-priority channels for critical business data to ensure low-latency transmission.

Images