Industrial data monitoring system, method and medium based on streaming computing engine

Abstract

The present invention provides an industrial data monitoring system based on a streaming computing engine, including a status monitoring module that uses Prometheus, which is natively supported by the streaming computing engine, to collect and store real-time indicators, periodically captures all sub-node indicators through a meta node for aggregation, provides a unified data source to Grafana for visualization, and configures alarms through a dashboard; a resource allocation module that analyzes real-time tasks, provides sufficient memory for the job, and processes task messages in real time. It combines relevant indicators obtained from the real-time task memory analysis and the rationality of the real-time task concurrency to derive a preset value for real-time task resources, adjusts real-time task resources, and achieves real-time task resource configuration; an abnormal status module that uses SideOutput to collect data on errors in each link of the streaming job, aggregates them into a unified error stream, and issues real-time alarms by discovering patterns that meet set characteristics in the streaming data. The present invention combines complex event processing with a streaming computing engine to enhance the real-time nature and applicability of alarms.

Description

Technical Field

[0001] The present invention relates to the field of data monitoring technology, and in particular to an industrial data monitoring system, method, and medium based on a streaming computing engine. Background Art

[0002] The Industrial Internet is a new industrial ecology, key infrastructure and new application model that is the result of the deep integration of new-generation information and communication technology and the industrial economy. It is based on the network, centered on the platform, with data as the element and security as the guarantee. It improves the efficiency of industrial production through the collection and mining of data, the application of technology and the intelligent transformation of processes.

[0003] With the advancement of informatization and intelligent transformation of industrial production, the amount of data generated during production is increasing exponentially. This complex data is distributed across sensor devices, IoT devices, e-commerce, enterprise communication tools, and other platforms. Its complex sources, unstructured nature, and uneven distribution pose challenges to data value mining and the processing and analysis of unstructured data.

[0004] The diversification of data sources places high demands on the connectivity and scalability of data integration systems. Distributed big data storage systems, while providing enhanced capabilities and relieving pressure on businesses, have led to an accelerated expansion of data volumes. This rapid growth in data volumes places higher demands on the throughput and real-time performance of data integration platforms. Of course, as a fundamental data-related system, data accuracy is a fundamental requirement.

[0005] Furthermore, the system needs to understand and integrate data scattered across different business teams, manage it effectively, and ensure secure data access. This entire process is relatively complex. While platformization can automate complex processes, the inherently high costs of data integration cannot be completely eliminated. Therefore, maximizing process reusability and controllability remains a constant challenge for data integration systems.

[0006] At the same time, streaming data now faces the problem of needing to query persistent databases, which puts pressure on the underlying database and remains unresolved.

[0007] After searching, patent document CN109637090A discloses a method for building a disaster monitoring and early warning platform based on the SOA architecture. This method addresses the current problem that geological disaster monitoring and early warning systems are mostly carried out through manual inspections and monitoring equipment threshold alarms. However, due to the diversity and uncertainty of the factors that induce geological disasters, and the complex geographical environment factors that are often intertwined, data management and evaluation become extremely difficult. This invention fully utilizes the low coupling characteristics of the SOA architecture to better integrate AI early warning services with the monitoring system. At the same time, it uses the advantages of machine learning algorithms to process complex data and accurately classify them. Based on the hazard evaluation of risk sources and the evaluation of monitored objects, it monitors the data of each monitored object in real time, predicts the level of a series of geological disasters of different intensities that may occur within a certain period of time in the danger zone, and proposes various risk reduction countermeasures based on the characteristics of different risk zones, providing auxiliary decision-making for geological disaster monitoring and early warning. However, this existing technology still cannot process complex events and cannot provide real-time alerts for patterns in streaming data that meet certain characteristics and trigger corresponding subsequent actions.

[0008] Patent document CN111475682A discloses an intelligent operation and maintenance platform based on a super-large-scale data system, which adopts an open and scalable data acquisition underlying architecture, connects to various types of data sources, and lowers the threshold for big data acquisition. It provides a full-perspective view of all types of data in the system, tracks the entire life cycle of each data node, and shares and displays the link relationship between each data node. For example, through system node log monitoring, link monitoring of multiple dimensions such as users, data, job tasks, APIs, and services can be implemented to sort out data lineage relationships, ensure the traceability of fine-grained data and operations, and complete the full life cycle management of data. Although this existing technology can perform highlighted display and comparison verification with original data to ensure the integrity and auditability of data, it still cannot solve the mining of data value and the processing and analysis of unstructured data.

[0009] Therefore, there is an urgent need to develop and design a system and method that can solve the problems encountered by streaming computing services when industrial intelligent transformation is implemented. Summary of the Invention

[0010] In response to the defects in the existing technology, the purpose of the present invention is to provide an industrial data monitoring system, method and medium based on a streaming computing engine, so that streaming data does not need to query a persistent database and will not put any pressure on the underlying database.

[0011] According to the present invention, an industrial data monitoring system based on a streaming computing engine is provided, comprising:

[0012] Status monitoring module: Prometheus, natively supported by the streaming computing engine, is used to collect and store real-time metrics. Meta nodes are used to periodically aggregate metrics from all child nodes. This unified data source is provided to Grafana for visualization and alarm configuration is performed through the dashboard.

[0013] Resource allocation module: This module analyzes real-time tasks and provides sufficient memory for them. It also processes task messages in real time. Based on the relevant indicators obtained from the real-time task memory analysis and the rationality of the real-time task concurrency, it determines the preset value of real-time task resources and adjusts the real-time task resources to achieve real-time task resource allocation.

[0014] Abnormal status module: Use SideOutput to collect error data from each link of the streaming job, aggregate it into a unified error stream, and trigger corresponding subsequent actions to issue real-time alerts by discovering patterns that meet the set characteristics in the streaming data.

[0015] Preferably, the status monitoring module is input into each data source, and the metadata service generates a set of timed heartbeat signals, which are consumed by the job. After the heartbeat signal flows into each job, it passes through each node along with the data flow, labels each node, and then skips the processing logic of the node to flow to the next node. When it flows through the end of the job component, it will be collected to the display page through JMX.

[0016] Preferably, the status monitoring module can determine the time it takes for a piece of data to be processed by the entire pipeline and the time it takes for each node to process the data through the time when the heartbeat signal is generated, the time when it flows into the job, and the time when it reaches each node, thereby determining the performance bottleneck of the job.

[0017] Preferably, the status monitoring module obtains the offset consumed in the streaming computing engine job through JMX to check the squeeze situation of the persistent message topic in the message queue. At the same time, the streaming computing collects the back pressure situation of the job node through the JMX provided by the engine to comprehensively analyze and infer whether data accumulation occurs in the job.

[0018] Preferably, the status monitoring module adjusts the default mechanism of the log4j log framework, splits the task log by day, and regularly cleans up expired logs to avoid abnormal tasks frequently filling up the disk and causing the computing node to become unavailable. At the same time, agents are deployed on all computing nodes to collect logs in real time, extract them into message queues, and then aggregate them into non-relational databases.

[0019] Preferably, the resource allocation module periodically scans all running streaming computing tasks according to the configured interval, combines the real-time task GC log, and calculates the recommended heap memory size of the streaming computing task according to the memory optimization rules, and compares it with the heap memory of the actually allocated streaming computing task. If the difference between the two is too large, it is considered that there is waste in the memory configuration of the streaming computing task, and an alarm is generated to prompt the task scheduling for optimization.

[0020] Preferably, the resource allocation module determines whether the input per unit time of the data source consumed by the real-time task matches the message processing capabilities of each operator / task of the real-time task;

[0021] Through the native K8s support provided by the streaming computing engine, jobs can actively request resources from the scheduler.

[0022] Preferably, the error flow record in the abnormal status module contains the system preset error code, original input data, error class and error information. The error flow information is classified and written into the distributed file storage unit. The user can know whether the data is normal by monitoring the storage directory.

[0023] According to the present invention, an industrial data monitoring method based on a streaming computing engine includes the following steps:

[0024] Status monitoring steps: Use Prometheus, natively supported by the streaming computing engine, to collect and store real-time metrics. Use the meta node to periodically capture and aggregate metrics from all child nodes. This unified data source is provided to Grafana for visualization and alarm configuration through the dashboard.

[0025] Resource allocation step: Analyze real-time tasks to provide sufficient memory for the job, and process task messages in real time. Combined with the relevant indicators obtained from the real-time task memory analysis and the rationality of the real-time task concurrency, the preset value of the real-time task resources is obtained, and the real-time task resources are adjusted to achieve the real-time task resource allocation;

[0026] Abnormal status step: Use SideOutput to collect error data from each link of the streaming job and aggregate it into a unified error stream. By discovering patterns that meet the set characteristics in the streaming data, corresponding subsequent actions are triggered and real-time alarms are issued.

[0027] According to a computer-readable storage medium storing a computer program provided by the present invention, the steps of the above method are implemented when the computer program is executed by a processor.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention can enhance the real-time performance and applicability of alarms by combining complex event processing with a streaming computing engine.

[0030] 2. The present invention uses the real-time alarm CEP (full name: Complex Event Processing) capability to trigger corresponding subsequent actions by discovering patterns that meet certain characteristics in streaming data. It supports both simple stateless pattern matching based on a single event (for example, filtering based on a certain field in the event) and complex stateful pattern matching across events based on association / aggregation / time windows (for example, calculating the moving average of a sliding time window).

[0031] 3. The present invention directly acts on streaming data without querying the persistent database and does not generate any pressure on the underlying database.

[0032] 4. The present invention adapts to the ever-changing rule conditions through the self-learning CEP engine, in order to solve the problems encountered by streaming computing services when industrial intelligent transformation is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0034] Figure 1 This is a flowchart of the steps of the industrial data monitoring method based on the streaming computing engine in the present invention;

[0035] Figure 2 This is a task scheduling flow chart of the industrial data monitoring system based on the streaming computing engine in the present invention;

[0036] Figure 3 This is a flow chart of the overall framework of the data processing system based on the streaming computing engine in the present invention;

[0037] Figure 4 This is a flow chart of the steps of the data processing method based on the streaming computing engine in the present invention. DETAILED DESCRIPTION

[0038] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, without departing from the scope of the present invention, a number of variations and improvements may be made by those skilled in the art. These all fall within the scope of protection of the present invention.

[0039] like Figure 2 As shown, the present invention provides an industrial data monitoring system based on a streaming computing engine, comprising:

[0040] Status monitoring module: Prometheus, which is natively supported by the streaming computing engine, is used to collect and store real-time indicators. The meta node periodically captures all sub-node indicators for aggregation, and provides a unified data source to Grafana for visualization and alarm configuration through the dashboard. Specifically:

[0041] The front-end interface designed for the streaming computing engine provides a wealth of runtime information to help users understand the current status of tasks. However, it lacks access to historical status monitoring, making it difficult for users to understand the historical status of tasks. This paper uses Prometheus, which is natively supported by the computing engine, to collect and store real-time metrics. The meta node periodically aggregates metrics from all child nodes, facilitating unified data sources for visualization in Grafana and dashboard-based alarm configuration.

[0042] During a streaming computing job, data constantly flows into the cleaning process, making it difficult to monitor the job's performance. Even with checkpoints enabled, it's impossible to determine whether or how much data has been lost. Therefore, this invention incorporates a heartbeat mechanism, injecting heartbeat information into each job to monitor its performance.

[0043] With each incoming data source, the metadata service generates a set of timed heartbeat signals, which are consumed by the job. After the heartbeat information flows into each job, it passes through each node along with the data flow. Each node is labeled with the current node, and then the processing logic of the node is skipped before flowing to the next node. When it passes the end of the job component, it is collected and displayed on the page via JMX.

[0044] This metric includes the time it takes for a heartbeat signal to be generated, when it flows into the job, and when it reaches each node. This metric can be used to determine the time it takes for a piece of data to be processed by the entire pipeline and the time it takes for each node to process the data, thereby identifying the performance bottleneck of the job.

[0045] Because heartbeat signals are sent periodically, each job should receive the same number of heartbeat messages. If the final number of metrics sent is inconsistent with the expected number, further investigation can be performed to determine whether data has been lost.

[0046] Throughput and latency are the most important indicators for measuring real-time task performance, and these two indicators are often used to adjust task concurrency and resource allocation. If the computing engine's native latency parameter is enabled to track task delays, it will significantly affect cluster and task performance. This invention uses message topic consumption accumulation as an indicator to measure task latency. The offset consumed in the engine job is obtained through JMX to check the squeeze of persistent message topics in the message queue. At the same time, the back pressure of the job node is collected through JMX provided by the engine to comprehensively analyze and infer whether the job has experienced data accumulation.

[0047] To accommodate large-scale data processing, streaming computing engines typically provide distributed computing capabilities. All tasks submitted to the platform are uniformly dispatched by a scheduler to any computing node. Consequently, task logs are distributed across different machines, making it difficult for users to locate them. This paper adjusts the default mechanism of the log4j logging framework, partitioning task logs by day and regularly cleaning out expired logs to prevent abnormal tasks from frequently filling up disks and rendering computing nodes unavailable. Furthermore, agents are deployed on all computing nodes to collect logs in real time, extracting them into message queues and then aggregating them into non-relational databases such as Elasticsearch. This makes it easy for users to quickly find the corresponding logs for a job when they need to locate and identify abnormal information through logs.

[0048] Resource allocation module: Analyzes real-time tasks to provide sufficient memory for the job, processes task messages in real time, and combines relevant indicators obtained from real-time task memory analysis and the rationality of real-time task concurrency to derive preset values ​​for real-time task resources, adjusts real-time task resources, and achieves real-time task resource allocation. Specifically:

[0049] The analysis of real-time task resources mainly includes two points:

[0050] On the one hand, we analyze real-time tasks from the perspective of runtime heap memory, starting with the memory occupied by the job, and providing sufficient memory to ensure stable job execution.

[0051] On the other hand, we start from the real-time task message processing capability to ensure that the data processing needs are met while using CPU resources as reasonably as possible.

[0052] Then, combined with the relevant indicators obtained from the real-time task memory analysis and the rationality of the real-time task concurrency, a real-time task resource preset value is obtained, and the real-time task resources are adjusted to ultimately achieve the goal of rationalizing the real-time task resource configuration, thereby better reducing the cost of machine use.

[0053] In terms of task memory management, the system will periodically scan all running streaming computing tasks according to the configured interval, combine the real-time task GC log, and calculate the recommended heap memory size for the streaming computing task based on memory optimization rules. It will then be compared with the heap memory actually allocated for the streaming computing task. If the difference between the two is too large, it can be considered that the memory configuration of the streaming computing task is wasted, and an alarm will be generated to prompt the task scheduler for optimization.

[0054] Analyzing the message processing capabilities of real-time job tasks primarily involves determining whether the input per unit time of the data source consumed by the real-time task matches the message processing capabilities of each operator / task in the real-time task. By monitoring data source traffic and combining it with the throughput of the operators in the job, we can determine whether the job is experiencing data skew or traffic backpressure. Using job optimization rules, we can adjust operator parallelism and optimize job resource allocation.

[0055] With the native Kubernetes support provided by the streaming computing engine, jobs can proactively request resources from the scheduler, enabling elastic scaling of job resources to cope with sudden data spikes. This architecture fully automates task resource optimization. By combining historical resource usage over different time periods, it automatically infers and adjusts resource allocation for real-time tasks, thereby improving overall real-time cluster resource utilization.

[0056] Abnormal status module: Use SideOutput to collect error data from each link of the streaming job, aggregate it into a unified error stream, and trigger corresponding subsequent actions and issue real-time alerts by discovering patterns that meet the set characteristics in the streaming data. Specifically:

[0057] Error logs contain the system's preset error code, the original input data, the error type, and the error message. Generally, error data is categorized and written to a distributed file storage system. Users can monitor the storage directory to determine whether the data is normal.

[0058] There are usually three situations when recovering data.

[0059] 1. Abnormal data formats, such as incomplete logs due to truncation or timestamps that do not conform to the agreed format. In this case, the system provides precise offline batch jobs to repair the data, skipping the abnormal data events or re-correcting the data and re-entering the operation process to backfill the data into the original data pipeline.

[0060] 2. Job pipeline anomalies. For example, if the actual data schema has changed but the flow table configuration has not been updated, a field may contain null values, but the overall job does not have an anomaly. This is indicated by indicator output alarms, prompting the user to perform data processing.

[0061] 3. Data link anomalies. For example, if a data source's master-slave cluster switches abnormally but the consumption configuration remains unchanged, this can cause downstream data applications to time out. Usually, the system's retry and exception recovery mechanisms ensure timely switching to a healthy data source, and alarms are generated if a job fails.

[0062] During the data replenishment job, the online flow table configuration is first updated to the latest version, switching to a healthy data source to prevent further abnormal data generation. However, some partitions in the storage may still be abnormal. Therefore, a separate data replenishment job is launched to specifically repair the abnormal data. The output data is written to a temporary directory, and the partition location is switched on the metastore to replace the original abnormal directory. This data replenishment process is transparent to users performing offline queries. Finally, at an appropriate time, the data in the abnormal partition is replaced and the location is restored.

[0063] like Figure 1 An industrial data monitoring method based on a stream computing engine provided by the present invention includes the following steps:

[0064] Status monitoring steps: Use Prometheus, natively supported by the streaming computing engine, to collect and store real-time metrics. Use the meta node to periodically capture and aggregate metrics from all child nodes. This unified data source is provided to Grafana for visualization and alarm configuration through the dashboard.

[0065] Resource allocation step: Analyze real-time tasks to provide sufficient memory for the job, and process task messages in real time. Combined with the relevant indicators obtained from the real-time task memory analysis and the rationality of the real-time task concurrency, the preset value of the real-time task resources is obtained, and the real-time task resources are adjusted to achieve the real-time task resource allocation;

[0066] Abnormal status step: Use SideOutput to collect error data from each link of the streaming job and aggregate it into a unified error stream. By discovering patterns that meet the set characteristics in the streaming data, corresponding subsequent actions are triggered and real-time alarms are issued.

[0067] According to a computer-readable storage medium storing a computer program provided by the present invention, the steps of the above method are implemented when the computer program is executed by a processor.

[0068] The present invention uses a streaming engine to perform query optimization:

[0069] In dimension table association scenarios, dimension tables frequently change, especially with new dimensions. However, if the association operation occurs before the dimension is added, association failures are common. To address these scenarios, the system has developed tailored exception handling strategies. If an exception occurs during association, the data is temporarily cached and retried later. The number of attempts can be controlled, and rules for delayed association can be customized.

[0070] An operator has also been added to SQL support to support delayed joins of dimension tables. When a join fails, the local cache does not cache empty dataset results. Instead, the data is temporarily stored in a state backend and retried based on a set timer and its retry count.

[0071] Analyzing the job structure topology reveals that computation operators and associated computation operators are linked together in streaming engine jobs. This is because there is no unique key splitting semantics. When the job parallelism is high, each subtask associated with a dimension table accesses all cache space, placing significant pressure on the cache.

[0072] However, observing the SQL implementation of management operations reveals that equijoins naturally possess hashing properties. When packaging operators, this configuration is directly exposed, allowing users to use the key associated with the dimension table as a hash condition to partition data. This ensures that the access space between the subtasks of each downstream operator is independent, significantly improving cache hit rates at job startup.

[0073] The present invention uses a streaming engine to perform index optimization:

[0074] To speed up data retrieval, databases often create indexes for data in advance. This index is then used to locate the starting point of the data before scanning, thereby accelerating data retrieval. Traditional databases commonly use row indexes, which create indexes based on one or several fields and store the index results in a tree structure. This type of index is accurate down to the row level and offers the highest indexing efficiency.

[0075] Some big data projects support row indexing, but the downside is that the large amount of indexed data can cause delays in writing and retrieving data. Platforms primarily process collected data, such as sensor data, which is characterized by high repetition rates and often yields very few analytical results. A tiny fraction of target behaviors can be hidden within the vast amount of data, often accounting for only one thousandth or less. Therefore, choosing a more cost-effective block indexing solution is sufficient for current application scenarios.

[0076] Most existing solutions store index data in the form of files on disk, and add some cache mechanisms to accelerate data access. However, the system stores index data directly in the database. This is mainly due to the following two considerations:

[0077] 1. Transaction. Generally speaking, column-stored files cannot be updated. However, when the system regularly optimizes file distribution, it merges the contents of multiple files. To ensure query consistency, the database needs to provide transaction capabilities.

[0078] 2. Performance: The database has strong read, write, and retrieval capabilities, and can even push predicates down to the database for completion. The database's high compression ratio can further save storage.

[0079] The present invention uses a streaming engine to perform exception optimization:

[0080] Because streaming jobs require caching a large amount of intermediate processes and applying for considerable computing resources, it takes about one or two minutes from job failure to job restart, which is unacceptable for some online business scenarios.

[0081] An analysis of the process by which job exceptions occur reveals that the main bottlenecks are anomaly detection and initialization costs. Anomaly detection is limited by the interface polling interval, and resource initialization is constrained by the container initialization step. The system has optimized these two aspects to quickly detect inactivity during job execution. Furthermore, resources are reserved to avoid resource allocation and initialization time when an outage occurs.

[0082] The system adds a majority connectivity detection service on top of the streaming computing engine. Multiple working nodes in the detection service cluster will periodically detect the connectivity of each machine in the cluster. Since it is a majority service, its credibility is guaranteed.

[0083] In addition, in terms of reserved resources, the system has expanded the resource application model of streaming engine jobs. Resource redundancy parameters can be set when submitting streaming jobs. When the redundancy parameters are activated, the amount of redundant resources will be automatically guaranteed to be higher than the amount of resource loss caused by a single point failure, and the aggregation of redundant resources will be avoided in resource allocation.

[0084] To address the issue of fully loaded cluster hardware resources, the system controls the data source consumption rate. A coordinator is introduced to periodically check resource consumption on the job aggregator and the progress of data source watermarks. Based on the global status, the system predicts the maximum position each data source is allowed to read next and distributes this information to all data sources. The data source determines its reading rate based on the maximum position and its current position. The data consumption rate is also limited based on load. Dynamically adjusting the consumption rate of all data sources ensures the stability of streaming jobs.

[0085] Prometheus is an open-source service monitoring system and time series database. Meta is an auxiliary tag in the HTML header section, located at the top of a document and containing no content. Grafana is a dashboard and graph editor. SideOutput is any number of additional side output result streams. JMX is a framework for embedding management capabilities into applications, devices, and systems. An agent is an autonomous software or hardware entity. Log4j is an open-source Apache project that allows you to control the delivery of log information to consoles, files, GUI components, and even socket servers, NT event loggers, and UNIX Syslog daemons. A transaction generally refers to something to be done or done. In computer terminology, it refers to a unit of program execution that accesses and potentially updates various data items in a database. Performance is an API for front-end performance monitoring. It can monitor page performance. A new API introduced by the W3C Performance Group can monitor white screen time, initial screen time, user-accessible time points, total page download time, DNS query time, TCP connection time, and more.

[0086] Based on the above invention, Figure 3 As shown, the present invention provides a data processing system based on a stream computing engine, comprising:

[0087] Data collection module: After acquiring the original data from the terminal, it will simultaneously compile the codec protocol of the sending and receiving ends, match the corresponding table information from the unbounded data stream to complete the collection of information events, and clean and filter the business field information. Specifically:

[0088] The collection and extraction of data from messages are relatively complex and involve many challenges, such as how to process sensor signal data, how to parse database binary log files, and how to filter business logs in the production process.

[0089] When processing sensor signal data, after collecting and obtaining the raw data from the terminal device, it is necessary to synchronously compile the encoding and decoding protocols of the sending and receiving ends, considering the reliability and efficiency of data transmission, and at the same time compress the periodically uploaded data to improve data transmission efficiency.

[0090] When parsing database binary log files, you first need to match the corresponding table information from the unbounded data stream to complete the information event collection. Log files are written in a mixed format, and records in row and statement formats must be parsed based on the record storage format. Log files are not updated until the streaming computing engine commits transactions and flushes them to disk. It is necessary to record the log consumption point to deal with abnormal job restarts.

[0091] When screening business logs in the production process, the logs contain many internal business fields, which need to be cleaned and screened. The function of extracting the business field information carried in the log message header is realized, thereby providing the ability to extract and analyze business logs.

[0092] The device diagnostic data involved in the above scenario is collected through MEMS sensors. Analog values ​​such as humidity, temperature, and pressure are converted into digital signals and transmitted to the diagnostic system. The data processing part needs to reconstruct the binary data in the sensor register into data that is useful for analysis. Specifically, it includes:

[0093] 1. Restore the measured value data by collecting calculation formulas.

[0094] 2. By recording the initial values ​​collected by the sensor, the zero bias error of the collected data is eliminated in the measured value.

[0095] 3. Select a reasonable cutoff frequency for the filter to reduce out-of-band noise and minimize errors caused by random noise signals.

[0096] After unbounded information streams are collected or pushed into the system, they are first sent to the consumption queue. The high throughput provided by the message queue smooths out data peaks and fills valleys, preventing the sudden increase in pressure on the processing system caused by sensor data uploads at the same collection interval. Multi-partition configuration is used for message queue topics to improve concurrency efficiency.

[0097] In the data management and acquisition stage, a very rich set of connector components are provided, including multiple storage supports such as HDFS and Kafka. At the same time, it supports multi-parallel consumption of data stored in message queues to ensure that the consumption speed of the data link does not become a short board at the entrance.

[0098] Existing streaming computing engines, including Flink used in the system, currently do not provide management functions for the entire data set. The present invention has made certain extensions through the database interface, and the data information of all connected data sources is transmitted to the downstream through the database.

[0099] Pre-filtering module: The collected and filtered information is filtered and deduplicated using the lightweight data provided by the Bloom filter, and the massive data provided by the Key-Value storage is simultaneously filtered and deduplicated. Specifically:

[0100] Upstream processes often easily cause data duplication in data collection, message delivery, format parsing, and other links. Data duplication can affect downstream applications such as data monitoring. In addition to traditional uses such as UV counting, the significance of deduplication lies in eliminating dirty data generated by unreliable data sources—that is, the impact of duplicated reported or delivered data—to make the results of streaming computing more accurate.

[0101] The present invention implements a lightweight data filtering provided by a Bloom filter, which will lead to high downstream statistics when the upstream data source can only guarantee at least once delivery semantics.

[0102] Bloom filters use built-in hash functions to deduplicate data. Their advantages are:

[0103] 1. The data itself does not need to be stored during calculation, and is only represented by bits. Therefore, the space occupied has a huge advantage over traditional methods, and the data can be kept confidential.

[0104] 2. The deduplication algorithm has high execution efficiency, and the time complexity of insertion and query is O(k).

[0105] 3. Hash functions are independent of each other and can be calculated in parallel at the hardware instruction level if necessary to increase processing efficiency.

[0106] However, Bloom filters cannot guarantee complete filtering accuracy and are not suitable for scenarios that require 100% accuracy.

[0107] For scenarios requiring accurate deduplication, this invention implements a method for deduplicating massive amounts of data through key-value storage. By leveraging the key-value storage features of the RocksDB state backend provided by the Flink engine, message records with the same key are merged. This approach is suitable for scenarios with high requirements for business data and low tolerance for errors. While providing accurate filtering capabilities, it is necessary to implement detailed state management for the filtering process, preventing unlimited state growth by setting timeouts and configuring incremental checkpoints.

[0108] Considering the problem of state bloat that can occur when data keys occupy a large amount of space, a hash algorithm is used to convert the key into an integer before storage, ensuring that it occupies a maximum of 8 bytes. However, since the hash algorithm cannot guarantee the absence of conflicts, its use depends on the business scenario. The deduplicated data stream is then filtered using a filter operator that implements custom business thresholds or custom filtering rules to filter out illegal data, reducing downstream computing pressure. The raw data is also unpacked and packaged into the required data format according to business needs. Finally, the data stream records entering the streaming computing system are timestamped to facilitate subsequent processing.

[0109] Data preprocessing module: The streaming computing engine provides unified stream and batch processing, avoiding the waste of resources caused by developing two sets of offline and real-time services, and abstracting the data flow by providing SQL support. Specifically:

[0110] During the data preprocessing and feature engineering phases, the streaming computing engine is dedicated to providing unified streaming and batch computing capabilities, avoiding the resource waste associated with developing both offline and real-time services. Furthermore, by providing SQL support to abstract data flows, the efficiency of job development is improved. Based on these characteristics, compared to traditional solutions, the present invention can unify the development logic of both real-time and offline links through a unified computing engine, thereby avoiding inconsistent computational semantics caused by inconsistent data processing links, eliminating the need for repeated cleaning and filtering of downstream data.

[0111] The filtered industrial data also needs to be effectively trimmed during the data preprocessing stage:

[0112] 1. Obtain the data packet sent by the collector from the message queue and parse it according to the transmission protocol.

[0113] 2. Split the data stream by collector and device identification, and then collect the data with the same identification into the same data stream.

[0114] 3. Combine data according to pre-set data processing groups and merge the data of association analysis into the same group.

[0115] 4. Aggregate the data according to the time window, and regenerate the aggregated data into an unbounded stream by sliding the window.

[0116] 5. Clean the data based on simple threshold logic, and record the data that does not meet the requirements or is obviously abnormal to the abnormal branch flow for system optimization and analysis.

[0117] 6. Record the percentage of valid data and the number of processing rule hits during data processing for system optimization and analysis.

[0118] Data lineage module: Through the SQL abstract processing provided by the streaming computing engine, a script is written in SQL to describe the flow of data, and then submitted to the computing platform for parsing and starting the job. Specifically:

[0119] The SQL abstraction provided by the streaming computing engine allows for rapid depiction of the data processing process within a task. Therefore, the primary method for submitting and constructing streaming computing tasks in the system is to write a script in SQL to describe the data flow, then submit it to the platform for parsing and initiating the job.

[0120] Typically, during development, one must first define data metadata using DDL (Database Schema Definition Language) and then write DML (Database Manipulation Language) to specify data processing. In many tasks, DDL statements are repeated but do not benefit from job development. Metadata information is also stored in the persistent storage metastore component and is also accessible from upstream and downstream catalogs (databases). Therefore, this solution defines real-time data streams as real-time tables and stores them in the metastore component. Users can directly select the required data source from the metastore and complete job development by simply writing DML statements. When submitting a task, the system supplements the converted DDL statements based on the upstream and downstream catalogs to form a complete StreamingSQL script for submission. This further simplifies job development and streamlines the development processes for batch and stream processing, making the development steps more consistent.

[0121] In addition, the solution's rule engine supports the continuous upgrading of configuration rules. The metadata hosted in the metastore can provide a certain degree of compatibility for inconsistent data schemas introduced by different data nodes. During the data processing process, pre-processed data is usually written to the message queue and then trained online. The training process is continuous, during which dynamic models are continuously generated and then pushed to the online inference module for inference. The characteristics of online machine learning are dynamic updates, continuous training, and continuous verification of models. At the same time, more complex strategies such as model monitoring, model deployment, and model rollback are required. At the computing node, the data structure may change through dimension table association and multi-stream convergence. By tracing the lineage relationship of the data schema, support can be provided for the changed data structure.

[0122] like Figure 4 As shown, the present invention also provides a data processing method based on a stream computing engine, comprising the following steps:

[0123] Data collection steps: After obtaining the original data from the terminal, the encoding and decoding protocols of the sending and receiving ends are compiled synchronously, and the corresponding table information is matched from the unbounded data stream to complete the collection of information events, while the business field information is cleaned and screened.

[0124] Pre-filtering step: The collected and filtered information is filtered and deduplicated using the lightweight data provided by the Bloom filter, and the massive data provided by the Key-Value storage is simultaneously filtered and deduplicated.

[0125] Data preprocessing step: The streaming computing engine provides unified stream and batch processing to avoid resource waste caused by developing two sets of offline and real-time services, and abstracts the data flow by providing SQL support.

[0126] Data lineage step: Through the SQL abstract processing provided by the streaming computing engine, a script is written in SQL language to describe the flow of data, and then submitted to the computing platform for parsing and starting the job.

[0127] The present invention further provides a computer-readable storage medium storing a computer program, which implements the steps of the above method when executed by a processor.

[0128] The main way to build tasks in this invention is SQL, and a StreamingSQL editor has been developed based on this, which provides support for system-defined SQL specifications and is compatible with StromSQL, SparkSQL, KSQL, HiveSQL, and FlinkSQL syntax. In addition to providing support for engine dialect SQL, SQL templates are also provided. This allows platform users to quickly develop SQL tasks for preset scenarios based on SQL templates. In addition, a web-style guide page is provided for business personnel to help them build streaming processing tasks by filling in forms without writing code.

[0129] In addition, the editor also provides a component library for custom functions, which contains a wealth of built-in functions, including time functions, collection functions, Json processing functions, and string functions. The rich built-in functions can facilitate user development and save users the time of self-development. The component library also contains instructions and use cases written by developers for platform users to search and review. The component library also provides a cloud platform for system users to upload and download custom functions and custom operators accumulated in the business, which facilitates data sharing when different systems are deployed, as well as subsequent development support.

[0130] The editor also implements syntax detection and intelligent prompts for StreamingSQL. Users can perform real-time syntax detection and receive prompts to a certain extent while writing statements. It also provides metadata completion for tables and fields involved in the statement, optimizing the development experience.

[0131] The editor also supports online debugging, a crucial capability for stream computing. This avoids the need to adjust tasks based on observations after they are launched, saving development costs. The system can accept text files as data sources to verify that output meets expectations, and can also sample message topics in message queues or data in database tables for business logic verification.

[0132] During the SQL parsing process, the most complicated part is widening the data table through dimension table association. In the invention, table association is performed through hot storage. After the data is imported from the data source, the system uses Async I / O technology to access the backend, and the system backend uses the Data Accessor interface to access the backend storage. The system backend storage supports obtaining data from distributed storage systems, structured databases, NoSQL databases, message queues and other data pipelines, and the backend will cache the data in the LRU Cache module. By broadcasting in the message queue when the dimension table is updated, the situation of untimely update of dimension data is reduced. In the system, storage support for a variety of big data tools has been developed for the data after the dimension table is associated, which greatly increases the compatibility of the system.

[0133] Because rule matching scenarios and business requirements are constantly changing, rules often need to be adjusted based on these changes. Business personnel can use the front-end feature management interface to add, delete, modify, and query rule factors in the rule base. By combining different rule factors, they construct different feature rules. They can then perform simple tests on the constructed rules using a specific set of test results. Once the test results are satisfactory, the rules are saved as a whole. This eliminates the need to directly modify the rule base for each operation, thereby decoupling business rules from rule factors.

[0134] In terms of measuring the overall effect, the rationality of the rules is determined by analyzing the hit rate of the rules and the alarm frequency of the rules.

[0135] It is necessary to determine whether the rules are invalid, such as a sudden decrease in the interception rate.

[0136] Determine whether the rule is redundant, for example, a rule has never intercepted any events.

[0137] Determine whether there are loopholes in the rules, such as when an operation is performed but the expected effect is not achieved.

[0138] In addition, the grayscale release mechanism of rules is also built based on the overall effectiveness evaluation of the rules. The rule release process is optimized by setting whitelists for different regions, importing real data test results, or limiting the flow of the official environment.

[0139] In terms of accumulating, analyzing and optimizing data, it is necessary to discover combination rules. By identifying a combination of certain behaviors, we can prevent the situation where each step in the operation is normal, but the cumulative results of the behaviors are abnormal. We also need to make group identification, such as using graph analysis technology to discover groups and label them, to prevent the situation where each part performs normally, but abnormal conditions occur in the entire area.

[0140] In the pattern matching of the invented rules, the Rate algorithm is used to improve matching efficiency and reduce the time redundancy caused by repeated calculations. When the number of rules and fact samples is large, each fact data needs to be matched with the Alpha node in the Rete network. Most rules contain the same conditional atoms, that is, there are conditional atoms that are simultaneously included in multiple rules. Matching each Alpha node in turn will result in a certain amount of time waste. Therefore, a pre-matching module aggregates multiple rules into a small number of rule groups. Through rule group screening, some normal data is filtered out in the pre-matching stage, reducing the number of fact and node matches. The implementation logic is to divide rules containing multiple identical conditional atoms into the same rule group, and the conditional atoms that appear the most times in the rule group are used as the characteristic conditions of the rule group. The full amount of data can be filtered out by filtering the rule group in the pre-matching module, and the rule judgment within the rule group is executed on the remaining samples.

[0141] The goal of rule learning is to generate a set of rules that covers as many examples as possible. The most direct approach is sequential coverage, or rule-by-rule induction. This is also the simplest training concept for rule learning. After each rule is generated in the training set, the examples covered by that rule are removed from the training set, and the remaining examples are used to train another set of rules. Of course, to avoid overfitting, algorithms generally incorporate pruning optimization strategies.

[0142] To train a highly available model, excellent features are essential. The important feature of excellent features is high discrimination, which requires close cooperation between algorithm developers and highly sensitive business personnel to analyze the configured rules and extract better rule factors. At the same time, algorithm developers should also be very familiar with the business process and create highly available features based on actual conditions. For actual business, in many cases, the addition of an accurate feature is more effective than model parameter tuning or even model optimization. At the same time, accurate features can also prevent overfitting. In the present invention, detailed business parameters are abstracted for combination screening through the analysis and processing of historical data and the process of industrial monitoring business.

[0143] A key requirement for model training is to maximize the number of samples. However, monitoring services inherently have this limitation. After configuring anomaly rules, very few cases are actually screened, resulting in a significant imbalance between positive and negative samples. Therefore, we employ a resampling method to increase the number of positive samples. Specifically, we resample the numerical features contained in the samples using the SMOTE method, and the labeled features using random selection, and then combine the two to create a new sample. During actual training, algorithm parameters such as the maximum number of iterations, step size, and maximum tree depth must be adjusted multiple times to select the parameters that maximize the recognition rate.

[0144] As the event stream processed by the system continues to grow, business rules will evolve along with the data set. By validating the effectiveness of rules against real-time business data and archived historical data, the rule engine will have a certain degree of evolutionary capability.

[0145] The workflow for combining a data processing system based on a streaming computing engine with a data monitoring system based on a streaming computing engine is as follows:

[0146] 1. According to the needs of the scenario, users upload custom functions developed for business logic in the form of jar packages through the upload module, and expand the definition of data processing operators and SQL functions required in the data processing process.

[0147] 2. Users use SQL statements to describe the data flow within a job, organize and orchestrate the context of job data processing, and adjust the parallelism of each processing node and the overall resource allocation of the job as needed. This also supports uploading JAR packages and implementing interface-based task orchestration for more precise control of the job process.

[0148] 3. Add judgment rules to the rule engine according to business needs, including threshold judgment, pattern judgment, combination logic judgment, etc. Complex association judgment rules also support user customization. After uploading the jar package that implements the system-specified interface, it can be configured in the system.

[0149] 4. Configure the system's rule learning module based on operational needs and enable the platform's system evolution function. This continuously trains the rule model as the system processes data. By learning user interaction logic and operational data, including traffic peaks and valleys, rule hit rates, and resource usage, the platform's rules are optimized and evolved. Detailed rules that better meet the business operation characteristics are manually verified and then released online.

[0150] 5. Configure the data source and data sink channel for the job, supporting reading and writing data from distributed storage systems, structured databases, NoSQL databases, message queues, and other data channels. It also supports the construction and parsing of multiple data formats such as JSON, Protobuffer, and Avro.

[0151] 6. Submit the job to the job system for orchestration. First, logical decomposition generates processing modules and verifies the syntax and custom package specifications of the job flow. A directed acyclic graph (DAG) constructed from these modules is displayed on the page, clearly illustrating the job's data flow and the order in which operators are orchestrated. Once verified, the job is submitted to the task scheduling module to request resources and initiate the job.

[0152] 7. After a job request obtains job resources from the resource cluster, it continuously retrieves data with corresponding tags from the data source. After data processing according to the configured rules and rule judgment by the rule engine, it is output to the real-time display module or data sinking. During the job, users can use the task monitoring module to obtain job details, such as job throughput, network I / O, resource usage, and module backpressure, allowing users to understand the operation status of the job.

[0153] 8. Based on user configuration, confirmation points are generated during the operation process, either by time period or batch, containing all status information for the current operation. Simultaneously, business archive points are generated at intervals between business processes, containing the calculation results of the current operation. These two mechanisms enable the accumulation of job metrics and analysis of job status. Furthermore, recovery capabilities are provided when an operation unit experiences an anomaly.

[0154] 9. When an abnormal situation occurs in the operation process, abnormal recovery will be carried out according to the user configuration, including attempting to restart the job, attempting to recover from the confirmation point, recording abnormal data, or rapid failure and then pushing abnormal handling configurations. The task status in the job monitoring module will also be updated in real time to help operation and maintenance personnel promptly know the task abnormality information

[0155] 10. When a job terminates, the resource monitoring module retrieves the current task status information and reclaims resources, releasing computing resources, clearing file caches, and retaining or clearing archive points and confirmation points based on user configuration. The stream computing job is terminated after storing the task status data in the time series database.

[0156] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0157] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and that those skilled in the art may make various changes or modifications within the scope of the claims, without affecting the essence of the present invention. Unless there is a conflict, the embodiments of this application and the features in the embodiments may be combined with each other in any manner.

Claims

1. An industrial data monitoring system based on a streaming computing engine, characterized in that: include: Status monitoring module: Prometheus, natively supported by the streaming computing engine, is used to collect and store real-time metrics. Meta nodes are used to periodically aggregate metrics from all child nodes. This unified data source is provided to Grafana for visualization and alarm configuration is performed through the dashboard. Resource allocation module: This module analyzes real-time tasks and provides sufficient memory for them. It also processes task messages in real time. Based on the relevant indicators obtained from the real-time task memory analysis and the rationality of the real-time task concurrency, it determines the preset value of real-time task resources and adjusts the real-time task resources to achieve real-time task resource allocation. Abnormal status module: Uses SideOutput to collect error data from each link of the streaming job, aggregates it into a unified error stream, and triggers corresponding subsequent actions and generates real-time alerts by discovering patterns that meet the set characteristics in the streaming data; The status monitoring module adjusts the default mechanism of the log4j logging framework to split task logs by day and regularly clean up expired logs to avoid abnormal tasks frequently filling up the disk and causing computing nodes to become unavailable. At the same time, agents are deployed on all computing nodes to collect logs in real time, extract them into message queues, and then aggregate them into a non-relational database. The error flow record in the abnormal status module contains the system preset error code, original input data, error class and error information. The error flow information is classified and written into the distributed file storage unit. The user can know whether the data is normal by monitoring the storage directory.

2. The industrial data monitoring system based on the stream computing engine according to claim 1 is characterized in that: The status monitoring module transmits data to each data source, and the metadata service generates a set of timed heartbeat signals, which are consumed by the job. After the heartbeat signal flows into each job, it passes through each node along with the data flow, labels each node, and then skips the processing logic of the node to flow to the next node. When it flows through the end of the job component, it will be collected and displayed on the page through JMX.

3. The industrial data monitoring system based on the stream computing engine according to claim 2, characterized in that: The status monitoring module can determine the time it takes for a piece of data to be processed by the entire pipeline and the time it takes for each node to process the data through the time when the heartbeat signal is generated, the time when it flows into the job, and the time when it reaches each node, thereby determining the performance bottleneck of the job.

4. The industrial data monitoring system based on the stream computing engine according to claim 1, characterized in that: The status monitoring module obtains the offset consumed in the streaming computing engine job through JMX to check the squeeze situation of the persistent message topic in the message queue. At the same time, the streaming computing collects the back pressure situation of the job node through the JMX provided by the engine to comprehensively analyze and infer whether data accumulation occurs in the job.

5. The industrial data monitoring system based on the stream computing engine according to claim 1, characterized in that: The resource allocation module periodically scans all running streaming computing tasks according to the configured interval, combines the real-time task GC log, and calculates the recommended heap memory size of the streaming computing task according to the memory optimization rules, and compares it with the heap memory of the actually allocated streaming computing task. If the difference between the two is too large, it is considered that there is a waste of memory configuration of the streaming computing task, and an alarm is generated to prompt the task scheduling for optimization.

6. The industrial data monitoring system based on the stream computing engine according to claim 1, characterized in that: The resource allocation module determines whether the input per unit time of the data source consumed by the real-time task matches the message processing capabilities of each operator / task of the real-time task; Through the native K8s support provided by the streaming computing engine, jobs can actively request resources from the scheduler.

7. An industrial data monitoring method based on a streaming computing engine, characterized in that: The steps include: Status monitoring steps: Use Prometheus, natively supported by the streaming computing engine, to collect and store real-time metrics. Use the meta node to periodically capture and aggregate metrics from all child nodes. This unified data source is provided to Grafana for visualization and alarm configuration through the dashboard. Resource allocation step: Analyze real-time tasks to provide sufficient memory for the job, and process task messages in real time. Combined with the relevant indicators obtained from the real-time task memory analysis and the rationality of the real-time task concurrency, the preset value of the real-time task resources is obtained, and the real-time task resources are adjusted to achieve the real-time task resource allocation; Abnormal status step: Use SideOutput to collect error data from each link of the streaming job and aggregate it into a unified error stream. By discovering patterns that meet the set characteristics in the streaming data, corresponding subsequent actions are triggered and real-time alerts are issued; In the status monitoring step, the default mechanism of the log4j log framework is adjusted to split the task logs by day and regularly clean up expired logs to avoid abnormal tasks frequently filling up the disk and causing computing nodes to become unavailable. At the same time, agents are deployed on all computing nodes to collect logs in real time, extract them into message queues, and then aggregate them into a non-relational database. The error flow record in the abnormal state step contains the system preset error code, original input data, error class and error information. The error flow information is classified and written into the distributed file storage unit. The user can know whether the data is normal by monitoring the storage directory.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 7 are implemented.

Images