High-availability passenger flow data processing system
By using a highly available data processing system, the problems of data loss and errors in the existing passenger flow statistics system under network fluctuations and high concurrency were solved, achieving data integrity and accuracy and ensuring the reliability of business decisions.
Patent Information
- Application Number
- CN202511515775.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-22
- Publication Date
- 2026-03-20
AI Technical Summary
Existing passenger flow statistics systems often suffer from data loss, delays, and errors when faced with network fluctuations, equipment failures, and high-concurrency data reporting. Furthermore, they lack the ability to handle complex scenarios, resulting in insufficient data accuracy and availability, which impacts business decisions.
A highly available data processing system is adopted, including a data acquisition and local caching module, a cloud data access and buffering module, a highly available data cleaning process module, and a data storage and application module. Through edge caching and a multi-stage cloud cleaning process, data integrity and quality are ensured.
It effectively prevents data loss, improves data accuracy and availability, supports stable operation in scenarios with large data volumes, outputs high-quality standardized data, and provides reliable basis for business decision-making.
Smart Images
Figure CN121705342A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, specifically to a highly available passenger flow data processing system. Background Technology
[0002] Customer traffic in offline retail stores is a key indicator for evaluating store operations, optimizing merchandise displays, improving the effectiveness of marketing activities, and rationally allocating human resources. Accurate, real-time, and complete customer traffic data is crucial for businesses to formulate operating strategies, especially during promotional events when customer traffic surges and the amount of data explodes. In current technologies, some offline store customer traffic statistics systems mainly rely on sensors deployed at store entrances to collect data. The data is aggregated and uploaded to a cloud server through the store's local network. The cloud server typically performs some basic data aggregation and storage. Some solutions may include batch data cleaning scripts that run after the data is entered into the database or at a specific time window to perform some simple verification and transformation on the data. However, existing passenger flow statistics systems often suffer from data loss, delays, and errors when faced with network fluctuations, equipment failures, and high-concurrency data reporting. Even if data is successfully uploaded to the cloud, the raw data may contain "dirty data" caused by sensor errors, network transmission errors, and duplicate reporting. Without effective processing, this data will seriously affect the accuracy of the analysis results, leading to data gaps or distortions, failing to truly reflect the actual passenger flow situation, and ultimately impacting business decisions. Furthermore, current technologies have the following problems: If the store's network connection is interrupted or unstable, the edge acquisition device will be unable to upload data to the cloud in real time. If the local caching capacity is insufficient or there is no caching mechanism, the customer flow data during the network interruption will be permanently lost. Existing cloud-based cleaning processes are often relatively simple and lack the ability to handle complex scenarios (such as data disorder, diverse formats, and hidden errors). The cleaning process itself may become a single point of failure. Once the cleaning service is abnormal, it will lead to data backlog or "dirty data" flowing directly into downstream applications. Without robust and highly available cleaning mechanisms, even with a large volume of data, its quality cannot be effectively guaranteed. Using data containing errors, duplicates, and omissions directly for analysis can lead to misleading conclusions. In the data cleaning process, if a monolithic or inflexible architecture is used, it is easy to create processing bottlenecks when faced with the massive amount of data to be cleaned from multiple stores during promotional activities, resulting in data delays and affecting the timeliness of the data. If errors occur during the cleaning process, there may be a lack of effective error isolation, retry and manual intervention mechanisms, making it difficult to trace the source of the problem data and the state before and after cleaning. Summary of the Invention
[0003] This invention provides a highly available passenger flow data processing system that can effectively solve the existing problems mentioned in the background art.
[0004] To achieve the above objectives, the present invention provides the following technical solution: a highly available customer flow data processing system that incorporates REID calculation logic into the overall scheduling framework, and the function call entry point determines which store data to run based on the store's capabilities. Furthermore, all technical frameworks used are deployed in a highly available manner, and a large task is divided into several smaller task areas for execution. The execution status of each step is recorded in the tracking table to improve scalability and maintainability. It includes a data acquisition and local caching module, a cloud data access and buffering module, a high-availability data cleaning process module, and a data storage and application module.
[0005] According to the above technical solution, the data acquisition and local caching module is responsible for capturing customer flow information in real time at the store and performing preliminary data buffering and fault tolerance processing. Specifically, it collects raw customer flow data information through sensor devices deployed in the store and sends the raw data to the local edge node. This module can enable the local caching mechanism when the network is unstable to prevent data loss. The sensor devices include cameras and IoT devices, used to capture raw passenger flow data, which specifically includes personnel trajectory, personnel location, timestamp, and personnel characteristics. The edge node refers to a local computing node, which includes an edge server. It is responsible for receiving sensor data and temporarily storing it. When the network is interrupted, the node caches the data and automatically retransmits it after the network is restored.
[0006] According to the above technical solution, in the specific implementation process, the data acquisition and local caching module sends data to the edge node in real time through sensor devices. The edge node adopts a high availability design and supports data buffering and retransmission logic to ensure that the data will not be permanently lost when there are network fluctuations or equipment failures.
[0007] According to the above technical solution, the cloud data access and buffering module, as the data access gateway of the cloud platform, is responsible for receiving data packets from various edge nodes through the cloud entry point, and performing preliminary security verification and buffering processing to ensure that the data smoothly enters the cleaning process. After preliminary source authentication and security verification, the original data packets are quickly sent to the original data message queue. The data access gateway, as a front-end service unit of the cloud platform, is responsible for receiving data packets uploaded by edge nodes and performing source authentication, security verification, and preliminary format verification. The raw data message queue is a high-throughput, high-availability message queue used to buffer data. It has the characteristics of high throughput and high availability, plays the role of peak shaving and valley filling, and writes data to HDFS storage. The raw data message queue can also decouple data, avoid direct coupling between the access layer and the cleaning service, and support asynchronous processing. HDFS storage, as a distributed file system, is used to persistently store raw data packets containing multiple passenger flow records, ensuring data traceability and redundant backup.
[0008] According to the above technical solution, in the specific implementation process, after the gateway receives the data, it quickly pushes it into the original data message queue. The original data message queue dynamically adjusts the data writing rate according to the load. The data is then written to the HDFS storage in batches, supporting high-concurrency scenarios and preventing data backlog from causing system crashes.
[0009] According to the above technical solution, the high-availability data cleaning process module is responsible for performing multi-stage cleaning and standardization processing on the original passenger flow data to remove dirty data and ensure data quality. Dirty data specifically includes duplicate data, data with incorrect format, and abnormal data. The high-availability data cleaning process module adopts a microservice architecture, supports elastic scaling and task scheduling, and includes a task scheduling and distribution unit and a multi-stage cleaning processing unit. The task scheduling and distribution unit is responsible for pulling data from Hive and HDFS, and distributing cleaning tasks according to the current cluster load and data characteristics. Specifically, it distributes data to specific cleaning work nodes, supports parallelism control, and prevents resource overload. The multi-stage cleaning process unit includes a first processing stage, a second processing stage, and a third processing stage.
[0010] According to the above technical solution, the first processing stage is the format verification and standardization processing stage, which performs structure verification on the data and verifies the data schema, including unifying the time format to UTC and standardizing the device ID, to ensure that the data conforms to the predefined schema.
[0011] According to the above technical solution, the second processing stage is the data cleaning processing stage, which involves obtaining the stores with configured capabilities from the configuration table and starting the cleaning process according to the set number of stores to be cleaned, so as to effectively prevent the cleaning task from failing due to too many stores. The corresponding code logic is executed through K8s operators. When there are too many requests, subsequent tasks are blocked to prevent the cleaning from failing due to too many cleaning tasks. Airflow sets an upper limit on task parallelism, allowing a maximum of a dozen or so tasks to be executed at a time. This also effectively prevents memory overflow issues. Since it is started using Kubernetes, when faced with increased business volume, simply adding more machines and adding the corresponding taint information can achieve horizontal scaling and solve the problem of large data volumes. In the second processing stage, when performing data cleaning, K8s is used to execute specific cleaning logic, including deduplication, outlier handling, and specific cleaning logic. Deduplication is based on similarity algorithms that take into account the characteristics of customer flow data. These algorithms include cosine similarity calculation to remove duplicate records. Outlier handling identifies and corrects illogical data by applying business rules. Specific cleaning logic is tailored to the configured store capabilities and dynamically adjusts cleaning parameters to prevent task failure.
[0012] According to the above technical solution, the third processing stage is the cleaning failure processing stage. When the cleaning task fails, it will automatically retry. All tasks will have 3 retry opportunities. If the result is cleaning failure or the execution time is too long and the customer does not get the data for the day, a message will be sent to the customer's notification group to inform the customer that the data for the day is unavailable. The error message will not be sent to the customer to avoid making the wrong decision. At the same time, it will trigger the alarm mechanism, send a message to the Lark alarm group, and notify the dedicated oncall personnel to solve the problem; In addition, after the data cleaning is successful, the cleaning results need to be output, specifically writing all the result data and verification data into the corresponding tables to facilitate auditing and backtracking.
[0013] According to the above technical solution, the data storage and application module is responsible for storing the cleaned high-quality passenger flow data and providing data services for subsequent business applications. The final data obtained after cleaning will be persistently stored in HDFS to ensure data persistence, queryability and high availability access. HDFS is a distributed file system that stores large files in blocks across multiple machines, ensuring reliable and efficient data access. It uses a multi-replica mechanism to store data and archive cleaned data, ensuring high reliability and long-term storage of data and meeting data archiving needs. At the same time, to support low-latency, high-concurrency real-time analysis and querying, the cleaned data is re-uploaded to Pinot, supporting low-latency queries.
[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention, through edge-end caching and retransmission and a multi-stage, highly available deep cleaning process in the cloud, not only ensures no data loss but also significantly improves data accuracy, consistency, and availability, providing a solid foundation for business decisions. The cloud-based cleaning service adopts a microservice architecture, database decoupling, and elastic scaling design, which can easily handle large data volume scenarios in promotional activities, ensuring the stable operation of the cleaning service and the timeliness of data processing. Targeted cleaning stages can effectively handle various types of dirty data, including duplicate data, format errors, outliers, and missing values, outputting high-quality standardized data. The modularity and elastic scaling capabilities of the cleaning service make the system easy to expand to adapt to business growth. Clear processes and logs facilitate monitoring and maintenance. The high-quality customer flow data processed by the highly available cleaning service provides merchants with more reliable and accurate decision-making basis. In summary, this invention features extremely high data integrity and quality, improved overall system availability and robustness, powerful data cleansing capabilities, scalability and maintainability, and ensures the accuracy of data-driven decision-making. This allows for the construction of a system that not only guarantees data integrity at the edge but also ensures the final data is normal and usable in the cloud through highly available cleansing processes and services. Attached Figure Description
[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0016] In the attached diagram: Figure 1 This is an architecture diagram of the passenger flow system of this invention; Figure 2 This is a flowchart illustrating the passenger flow system of the present invention. Detailed Implementation
[0017] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0018] Example: Figure 1-2 As shown, the present invention provides a technical solution, a highly available customer flow data processing system, which puts the REID calculation logic into the overall scheduling framework, and the function call entry point determines which store data to run based on the store's capabilities. Moreover, all the technical frameworks used are deployed in a highly available manner, and a large task is split into several small task areas for execution. The execution status of each step is recorded in the tracking table, which improves scalability and maintainability. It includes a data acquisition and local caching module, a cloud data access and buffering module, a high-availability data cleaning process module, and a data storage and application module.
[0019] Based on the above technical solution, the data acquisition and local caching module is responsible for capturing customer flow information in real time at the store and performing preliminary data buffering and fault tolerance processing. Specifically, it collects raw customer flow data through sensor devices deployed in the store and sends the raw data to the local edge node. This module can enable the local caching mechanism to prevent data loss when the network is unstable. Sensor devices, including cameras and IoT devices, are used to capture raw passenger flow data, which specifically includes passenger trajectories, passenger locations, timestamps, and passenger characteristics. Edge nodes refer to local computing nodes, including edge servers, which are responsible for receiving sensor data and temporarily storing it. When the network is interrupted, the node caches the data and automatically retransmits it after the network is restored.
[0020] Based on the above technical solution, in the specific implementation process, the data acquisition and local caching module sends data to the edge node in real time through sensor devices. The edge node adopts a high availability design and supports data buffering and retransmission logic to ensure that data is not permanently lost in the event of network fluctuations or device failures.
[0021] Based on the above technical solution, the cloud data access and buffer module, as the data access gateway of the cloud platform, is responsible for receiving data packets from various edge nodes through the cloud entry point, and performing preliminary security verification and buffering processing to ensure that the data smoothly enters the cleaning process. After preliminary source authentication and security verification, the original data packets are quickly sent to the original data message queue. As a front-end service unit of the cloud platform, the data access gateway is responsible for receiving data packets uploaded by edge nodes and performing source authentication, security verification, and preliminary format verification. The raw data message queue is a high-throughput, high-availability message queue used to buffer data. It has the characteristics of high throughput and high availability, plays the role of peak shaving and valley filling, and writes data to HDFS storage. The raw data message queue can also decouple data, avoid direct coupling between the access layer and the cleaning service, and support asynchronous processing. HDFS storage, as a distributed file system, is used to persistently store raw data packets containing multiple passenger flow records, ensuring data traceability and redundant backup.
[0022] Based on the above technical solution, in the specific implementation process, after the gateway receives the data, it quickly pushes it into the raw data message queue. The raw data message queue dynamically adjusts the data writing rate according to the load. The data is then written to the HDFS storage in batches, supporting high-concurrency scenarios and preventing data backlog from causing system crashes.
[0023] Based on the above technical solution, the high-availability data cleaning process module is responsible for performing multi-stage cleaning and standardization processing on the original passenger flow data to remove dirty data and ensure data quality. Dirty data specifically includes duplicate data, data with incorrect format, and abnormal data. The high-availability data cleaning process module adopts a microservice architecture, supports elastic scaling and task scheduling, and includes a task scheduling and distribution unit and a multi-stage cleaning processing unit. The task scheduling and distribution unit is responsible for pulling data from Hive and HDFS and distributing cleaning tasks dynamically based on the current cluster load and data characteristics. Specifically, it distributes data to specific cleaning worker nodes, supports parallelism control, and prevents resource overload. To achieve intelligent scheduling and high-availability allocation of tasks across multiple nodes, this system designs a task allocation formula with an adaptive penalty factor to dynamically adjust task weights based on node performance and stability.
[0024] in, The task weight assigned to the i-th node; The weight representing the computing power of a node (determined by a combination of CPU, memory, and network bandwidth); Indicates the current load of the node; This indicates the recent task failure rate of the node; δ is a dynamic penalty factor used to prevent nodes that frequently make mistakes from being reassigned tasks.
[0025] This formula enables the system to automatically avoid unstable nodes, achieve dynamic load balancing and fault tolerance optimization in task allocation, and significantly improve the stability of task execution and the overall availability of the system.
[0026] The multi-stage cleaning and processing unit includes a first processing stage, a second processing stage, and a third processing stage.
[0027] Based on the above technical solution, the first processing stage is the format verification and standardization stage, which verifies the data structure and data schema, including unifying the time format to UTC and standardizing the device ID, to ensure that the data conforms to the predefined schema.
[0028] Based on the above technical solution, the second processing stage is the data cleaning stage, which involves obtaining the stores with configured capabilities from the configuration table and starting the cleaning process according to the set number of stores to be cleaned, so as to effectively prevent the cleaning task from failing due to too many stores. The corresponding code logic is executed through K8s operators. When there are too many requests, subsequent tasks are blocked to prevent the cleaning from failing due to too many cleaning tasks. Airflow sets an upper limit on task parallelism, allowing a maximum of a dozen or so tasks to be executed at a time. This also effectively prevents memory overflow issues. Since it is started using Kubernetes, when faced with increased business volume, simply adding more machines and adding the corresponding taint information can achieve horizontal scaling and solve the problem of large data volumes. In the second processing stage, when performing data cleaning, Kubernetes and K8s are used to execute specific cleaning logic, including deduplication, outlier handling, and specific cleaning logic. Deduplication is based on similarity algorithms that take into account the characteristics of customer flow data. These algorithms include cosine similarity calculation to remove duplicate records. Outlier handling identifies and repairs illogical data, specifically sudden changes in customer flow, which are handled using business rules, including inbound and outbound balance checks. Specific cleaning logic is tailored to the configured store capabilities, obtained from the configuration table, and dynamically adjusts cleaning parameters to prevent task failure. Cosine similarity calculation: Similarity = (vector A * vector B) / (length of vector A * length of vector B), where * represents the dot product operation, and the vector length refers to the Euclidean norm, L2 norm.
[0029] To further improve the cleaning effect and data quality, the system introduces a comprehensive quality evaluation formula during the cleaning process to quantitatively evaluate the cleaning results in real time. The calculation method is as follows:
[0030] in: This indicates the overall cleaning quality score; This is a similarity index for passenger flow data, used to measure the adequacy of duplicate removal. This represents abnormal volatility, used to characterize the proportion of outlier data. This represents the rule repair rate, which measures the percentage of data that has been successfully corrected by system rules. α, β, and γ are weighting coefficients, which are automatically adjusted by the system based on the data characteristics of different stores.
[0031] Using the above formula, the system can dynamically evaluate the cleaning quality and adaptively optimize the cleaning strategy at different stages, thereby ensuring the accuracy and consistency of the output data.
[0032] Based on the above technical solution, the third processing stage is the cleaning failure handling stage. When the cleaning task fails, it will automatically retry. All tasks will have 3 retry opportunities. If the result is cleaning failure or the execution time is too long and the customer does not get the data for the day, a message will be sent to the customer's notification group to inform the customer that the data for the day is unavailable. This will prevent the customer from being given an error message and making an incorrect decision. At the same time, it will trigger the alarm mechanism, send a message to the Lark alarm group, and notify the dedicated oncall personnel to solve the problem; In addition, after the data cleaning is successful, the cleaning results need to be output, specifically writing all the result data and verification data into the corresponding tables to facilitate auditing and backtracking.
[0033] Based on the above technical solution, the data storage and application module is responsible for storing the cleaned high-quality passenger flow data and providing data services for subsequent business applications. The final data obtained after cleaning will be persistently stored in HDFS to ensure data persistence, queryability and high availability access. HDFS is a distributed file system that stores large files in blocks across multiple machines, ensuring reliable and efficient data access. It uses a multi-replica mechanism to store data and archive cleaned data, ensuring high reliability and long-term storage of data and meeting data archiving needs. At the same time, to support low-latency, high-concurrency real-time analysis and querying, the cleaned data is re-uploaded to Pinot, supporting low-latency queries.
[0034] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A highly available passenger flow data processing system, characterized in that: The REID calculation logic is incorporated into the overall scheduling framework, and the function call entry point determines which store data to run based on the store's capabilities. Furthermore, all the technical frameworks used are deployed in a highly available manner, and a large task is broken down into several smaller task areas for execution. The execution status of each step is recorded in the event tracking table. It includes a data acquisition and local caching module, a cloud data access and buffering module, a high-availability data cleaning process module, and a data storage and application module.
2. The highly available passenger flow data processing system according to claim 1, characterized in that: The data acquisition and local caching module is responsible for capturing customer flow information in real time at the store and performing preliminary data buffering and fault tolerance processing. Specifically, it collects raw customer flow data through sensor devices deployed in the store and sends the raw data to the local edge node. The sensor devices include cameras and IoT devices, used to capture raw passenger flow data, which specifically includes personnel trajectory, personnel location, timestamp, and personnel characteristics. The edge node refers to a local computing node, which includes an edge server. It is responsible for receiving sensor data and temporarily storing it. When the network is interrupted, the node caches the data and automatically retransmits it after the network is restored.
3. The highly available passenger flow data processing system according to claim 2, characterized in that: In its implementation, the data acquisition and local caching module transmits data to edge nodes in real time via sensor devices. The edge nodes are designed for high availability and support data buffering and retransmission logic to ensure that data is not permanently lost in the event of network fluctuations or device failures.
4. The highly available passenger flow data processing system according to claim 1, characterized in that: The cloud data access and buffering module, as the data access gateway of the cloud platform, is responsible for receiving data packets from various edge nodes through the cloud entry point, and performing preliminary security verification and buffering processing to ensure that the data smoothly enters the cleaning process. After preliminary source authentication and security verification, the original data packets are quickly sent to the original data message queue. The data access gateway, as a front-end service unit of the cloud platform, is responsible for receiving data packets uploaded by edge nodes and performing source authentication, security verification, and preliminary format verification. The raw data message queue is a high-throughput, high-availability message queue used to buffer data. It has the characteristics of high throughput and high availability, plays the role of peak shaving and valley filling, and writes data to HDFS storage. The raw data message queue can also decouple data, avoid direct coupling between the access layer and the cleaning service, and support asynchronous processing. HDFS storage, as a distributed file system, is used to persistently store raw data packets containing multiple passenger flow records, ensuring data traceability and redundant backup.
5. A highly available passenger flow data processing system according to claim 4, characterized in that: In its implementation, the cloud data access and buffering module quickly pushes data into the raw data message queue after the gateway receives the data. The raw data message queue dynamically adjusts the data writing rate according to the load, and the data is then written to the HDFS storage in batches, supporting high-concurrency scenarios and preventing data backlog from causing system crashes.
6. A highly available passenger flow data processing system according to claim 1, characterized in that: The high-availability data cleaning process module is responsible for performing multi-stage cleaning and standardization on the original passenger flow data to remove dirty data and ensure data quality. Dirty data specifically includes duplicate data, data with incorrect format, and abnormal data. The high-availability data cleaning process module adopts a microservice architecture, supports elastic scaling and task scheduling, and includes a task scheduling and distribution unit and a multi-stage cleaning processing unit. The task scheduling and distribution unit is responsible for pulling data from Hive and HDFS, and distributing cleaning tasks according to the current cluster load and data characteristics. Specifically, it distributes data to specific cleaning work nodes, supports parallelism control, and prevents resource overload. The multi-stage cleaning process unit includes a first processing stage, a second processing stage, and a third processing stage.
7. A highly available passenger flow data processing system according to claim 6, characterized in that: The first processing stage is the format verification and standardization stage, which performs structure verification on the data and verifies the data schema, including unifying the time format to UTC and standardizing the device ID to ensure that the data conforms to the predefined schema.
8. A highly available passenger flow data processing system according to claim 6, characterized in that: The second processing stage is the data cleaning stage, which involves retrieving stores with configured capabilities from the configuration table and starting the cleaning process according to the set number of stores to be cleaned, in order to effectively prevent the cleaning task from failing due to too many stores. The corresponding code logic is executed through K8s operators. When there are too many requests, subsequent tasks are blocked to prevent the cleaning from failing due to too many cleaning tasks. Airflow sets an upper limit on task parallelism, allowing a maximum of a dozen or so tasks to be executed at a time. This also effectively prevents memory overflow issues. Since it is started using Kubernetes, when faced with increased business volume, simply adding more machines and adding the corresponding taint information can achieve horizontal scaling and solve the problem of large data volumes. In the second processing stage, when performing data cleaning, K8s is used to execute specific cleaning logic, including deduplication, outlier handling, and specific cleaning logic. Deduplication is based on similarity algorithms that take into account the characteristics of customer flow data. These algorithms include cosine similarity calculation to remove duplicate records. Outlier handling identifies and corrects illogical data by applying business rules. Specific cleaning logic is tailored to the configured store capabilities and dynamically adjusts cleaning parameters to prevent task failure.
9. A highly available passenger flow data processing system according to claim 6, characterized in that: The third processing stage is the cleaning failure handling stage. When the cleaning task fails, it will automatically retry. All tasks will have 3 retry opportunities. If the result is cleaning failure or the execution time is too long and the customer does not get the data for the day, a message will be sent to the customer's notification group to inform the customer that the data for the day is unavailable. This will prevent the customer from making wrong decisions by sending error messages. At the same time, it will trigger the alarm mechanism, send a message to the Lark alarm group, and notify the dedicated oncall personnel to solve the problem; In addition, after the data cleaning is successful, the cleaning results need to be output, specifically writing all the result data and verification data into the corresponding tables.
10. A highly available passenger flow data processing system according to claim 1, characterized in that: The data storage and application module is responsible for storing the cleaned, high-quality passenger flow data and providing data services for subsequent business applications. The final data obtained after cleaning will be persistently stored in HDFS to ensure data persistence, queryability, and high availability access. HDFS is a distributed file system that stores large files in blocks across multiple machines, ensuring reliable and efficient data access. It uses a multi-replica mechanism to store data and archive cleaned data, ensuring high reliability and long-term storage of data and meeting data archiving needs. At the same time, to support low-latency, high-concurrency real-time analysis and querying, the cleaned data is re-uploaded to Pinot, supporting low-latency queries.