Industrial data processing system and method based on stream computing engine, medium

Abstract

The application provides a data processing system based on a stream computing engine, comprising a data acquisition module: after acquiring original data from a terminal, a coding and decoding protocol of a sending end and a receiving end is compiled and sent synchronously, corresponding table information is matched from a non-bound data stream, and acquisition of information events is completed; a pre-filtering module: information screened by acquisition is provided with lightweight data filtering by a Bloom filter, and massive data filtering provided by K-V storage is synchronously utilized; a data preprocessing module: the stream computing engine provides unified stream processing of streams and batches, avoids resource waste caused by two sets of business development of offline and real-time, and provides SQL support to abstract data flow; a data blood relationship module: SQL abstract processing provided by the stream computing engine is adopted, a script is written by using SQL language to describe the flow of data, and then the script is submitted to a platform to parse and start a job. The application realizes mining of data value and processing and analysis of unstructured data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data processing system, method, and medium based on a streaming computing engine. Background Technology

[0002] The Industrial Internet is a new industrial ecosystem, key infrastructure, and new application model that deeply integrates the next generation of information and communication technologies with the industrial economy. It is based on networks, with platforms as the hub, data as the element, and security as the guarantee. It improves the efficiency of industrial production by collecting and mining data, applying technologies, and intelligently transforming processes.

[0003] With the advancement of informatization and intelligent transformation in industrial production, the amount of data generated during the production process is increasing exponentially. This complex data is distributed across sensor devices, IoT devices, e-commerce, enterprise communication tools, and more, and is characterized by its complex sources, unstructured nature, and uneven distribution, posing challenges to data value mining and the processing and analysis of unstructured data.

[0004] The diversification of data sources places higher demands on the connectivity and scalability of data integration systems. While distributed big data storage systems bring greater capabilities to businesses, they also relieve business pressure, thus accelerating the expansion of data volume. This rapid increase in data volume requires higher throughput and real-time performance from data integration platforms. Of course, as a fundamental data-related system, data accuracy is a basic requirement.

[0005] Furthermore, the system needs to understand and integrate data scattered across different business teams, manage it effectively, and ensure secure data access, making the entire process relatively complex. While platformization can automate complex processes, the inherent high costs of data integration cannot be completely eliminated through platformization. Therefore, maximizing the reusability and controllability of processes remains a continuous challenge for data integration systems.

[0006] Meanwhile, the problem of streaming data needing to query persistent databases, which puts pressure on the underlying database, remains unresolved.

[0007] A search revealed patent document CN111475682A, which discloses an intelligent operation and maintenance platform based on a large-scale data system. This platform employs an open and scalable data acquisition architecture, connecting to various data sources and lowering the barrier to entry for big data collection. It provides a comprehensive view of all types of data within the system, tracks the entire lifecycle of each data node, and shares the link relationships between them. For example, through system node log monitoring, it enables multi-dimensional link monitoring of users, data, tasks, APIs, and services to clarify data lineage, ensuring fine-grained data and operational traceability, and completing full lifecycle management of data. While this existing technology can highlight and compare raw data to ensure data integrity and auditability, it still cannot solve the problems of data value mining and unstructured data processing and analysis.

[0008] Patent document CN109637090A discloses a method for constructing a disaster monitoring and early warning platform based on SOA architecture. This method addresses the shortcomings of current geological disaster monitoring and early warning systems, which mostly rely on manual patrols and threshold alarms from monitoring equipment. However, the diversity and uncertainty of factors triggering geological disasters, coupled with complex geographical environmental factors, make data management and evaluation extremely difficult. This invention fully utilizes the low-coupling characteristics of SOA architecture to better integrate AI early warning services with the monitoring system. It also leverages the advantages of machine learning algorithms in processing complex data and accurate classification. Based on the hazard assessment of risk sources and the evaluation of monitored objects, it monitors data from various monitored objects in real time, predicts the levels of geological disasters of different intensities that may occur within a certain period in the danger zone, and proposes various risk reduction strategies based on the characteristics of different risk areas, providing auxiliary decision-making for geological disaster monitoring and early warning. However, this existing technology still cannot handle complex events and cannot provide real-time alerts for patterns in streaming data that match certain characteristics and trigger corresponding subsequent actions.

[0009] Therefore, there is an urgent need to develop and design a system and method that can solve the problems encountered by streaming computing services when implementing intelligent transformation in industry. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the purpose of this invention is to provide a data processing system, method, and medium based on a streaming computing engine, which enables the mining of data value and the processing and analysis of unstructured data, while improving the reusability and controllability of the process.

[0011] A data processing system based on a streaming computing engine, according to the present invention, includes:

[0012] Data acquisition module: After acquiring raw data from the terminal, it synchronously compiles the encoding and decoding protocols for the sending and receiving ends, matches the corresponding table information from the unbounded data stream to complete the acquisition of information events, and cleans and filters the business field information at the same time.

[0013] Pre-filtering module: The collected and filtered information is deduplicated using lightweight data filtering provided by Bloom filter, and simultaneously deduplicated using massive data filtering provided by Key-Value storage;

[0014] Data preprocessing module: The streaming computing engine provides the ability to unify streaming and batch processing, unifying the message processing path, avoiding the waste of resources caused by developing two sets of business processes for offline and real time, and abstracting the data flow by providing SQL support;

[0015] Data lineage module: Through the SQL abstraction processing provided by the streaming computing engine, SQL scripts are written to describe the flow of data, and then submitted to the computing platform for parsing and startup.

[0016] Preferably, when collecting information events in the data acquisition module, the log file is written in a mixed format. It is necessary to parse the records in row format and statement format separately according to the record storage format. The log file will only be updated after the streaming computing engine submits the transaction and refreshes it to the disk. It is necessary to record the log consumption point to deal with the abnormal restart of the job.

[0017] Preferably, after the data acquisition module collects the unbounded data stream, the unbounded data stream is sent to the consumption queue. The high throughput processing provided by the message queue completes the peak smoothing and valley filling of the data. The message topics of the message queue are set up with multiple partitions.

[0018] Preferably, the data acquisition module provides HDFS and Kafka storage support during the data management and acquisition phases, and also supports the consumption of data stored in message queues with multiple degrees of parallelism, ensuring that the consumption speed of the data link does not become a bottleneck at the entry point.

[0019] Preferably, the pre-filtering module uses the hash function built into the Bloom filter to deduplicate the data;

[0020] -Use Key-Value storage to merge message records that have the same key;

[0021] - A hash algorithm is used to convert the key into an integer for storage, so that it occupies a maximum of 8 bytes.

[0022] Preferably, the pre-filtering module further filters out invalid data from the deduplicated data stream by using a filter operator with custom business thresholds or custom filtering rules.

[0023] Preferably, during the development of the data lineage module, a database schema definition language is first defined to specify the data metadata, and then a data manipulation language is written to specify the data processing procedure. The real-time data stream is defined as a real-time table and stored in the metadata storage component. Users can directly select the required data source from it, and then only need to write statements in the data manipulation language to complete the job development.

[0024] Preferably, when a task is submitted, the data lineage module supplements the database schema definition language statements converted from upstream and downstream databases to form a complete Streaming SQL script for submission, thus connecting the development processes of batch processing and stream processing.

[0025] A data processing method based on a streaming computing engine, provided by the present invention, includes the following steps:

[0026] Data acquisition steps: After acquiring raw data from the terminal, the encoding and decoding protocols of the sending and receiving ends are compiled synchronously. The corresponding table information is matched from the unbounded data stream to complete the acquisition of information events. At the same time, the business field information is cleaned and filtered.

[0027] Pre-filtering steps: The collected and filtered information is deduplicated using lightweight data filtering provided by the Bloom filter, and simultaneously deduplicated using massive data filtering provided by Key-Value storage;

[0028] Data preprocessing steps: The streaming computing engine provides the capability of integrating streaming and batch processing to unify the message processing path, avoiding the waste of resources caused by developing two sets of business processes for offline and real-time, and abstracting the data flow by providing SQL support;

[0029] Data lineage steps: Through the SQL abstraction processing provided by the streaming computing engine, a script is written in SQL language to describe the flow of data, and then submitted to the computing platform for parsing and startup.

[0030] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, it implements the steps of the above-described method.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This invention achieves the mining of data value and the processing and analysis of unstructured data by compressing and transmitting raw data, defining and mining data indicators, preprocessing data used for aggregation calculation, and standardizing and governing data lineage.

[0033] 2. This invention acquires raw data and then simultaneously develops encoding and decoding protocols for the sending and receiving ends, taking into account the reliability and efficiency of data transmission. At the same time, it compresses the periodically uploaded data to improve data transmission efficiency.

[0034] 3. This invention cleans and filters the many internal business fields contained in the logs during the production process, thereby enabling the extraction of business field information carried in the log message header, and thus providing the ability to extract and analyze business logs.

[0035] 4. This invention uses the hash function built into the Bloom filter to remove duplicate data, so that the data itself does not need to be stored during the operation, but only represented by bits. Therefore, it has a huge advantage in space usage compared with the traditional method, and can keep the data confidential. The deduplication algorithm has high execution efficiency, with the time complexity of insertion and query being O(k). The hash functions are independent of each other and can be computed in parallel at the hardware instruction level if needed, increasing processing efficiency.

[0036] 5. In the case of changes in data structure caused by dimension table association and multi-stream convergence in computing nodes, this invention can provide support for the changed data structure by tracing the lineage of the data schema. Attached Figure Description

[0037] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 This is a flowchart illustrating the overall framework of the data processing system based on a streaming computing engine in this invention.

[0039] Figure 2 This is a flowchart of the data processing method based on a streaming computing engine in this invention.

[0040] Figure 3 This is a task scheduling flowchart for the industrial data monitoring system based on a streaming computing engine in this invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0042] like Figure 1 As shown, the present invention provides a data processing system based on a streaming computing engine, comprising:

[0043] Data Acquisition Module: After acquiring raw data from the terminal, it synchronously develops encoding and decoding protocols for both the sending and receiving ends. It then matches corresponding table information from the unbounded data stream to complete the acquisition of information events, while simultaneously cleaning and filtering business field information. Specifically:

[0044] Data collection and extraction from messages are quite complex, involving many challenges, such as how to process sensor signal data, how to parse database binary log files, and how to filter business logs during the production process.

[0045] When processing sensor signal data, after acquiring the raw data from the terminal device, it is necessary to simultaneously develop encoding and decoding protocols for the sending and receiving ends, taking into account the reliability and efficiency of data transmission, and compressing the periodically uploaded data to improve data transmission efficiency.

[0046] When parsing database binary log files, the first step is to match the corresponding table information from the unbounded data stream to collect information events. The log files are written in a mixed format, requiring the parsing of row and statement records based on their storage format. The log files are only updated after the streaming engine commits a transaction and flushes it to disk. Therefore, it's necessary to record log consumption points to handle abnormal job restarts.

[0047] When filtering business logs during the production process, the logs contain many internal business fields, which need to be cleaned and filtered. This enables the extraction of business field information carried in the log message header, thereby providing the ability to extract and analyze business logs.

[0048] The equipment diagnostic data involved in the above scenario is collected by MEMS sensors, which convert analog detection values ​​such as humidity, temperature, and pressure into digital signals and transmit them to the diagnostic system. The data processing section needs to reconstruct the binary data in the sensor registers into data useful for analysis, specifically including:

[0049] 1. The measured data is reconstructed by collecting and calculating the formula.

[0050] 2. By recording the initial values ​​collected by the sensor, the zero-bias error of the collected data is eliminated in the measured values.

[0051] 3. Select a reasonable cutoff frequency for the filter to reduce out-of-band noise and minimize errors caused by random noise signals.

[0052] After the unbounded information stream is collected or pushed to the system, it is first sent to the consumption queue. The high throughput provided by the message queue smooths out data spikes and troughs, preventing a sharp increase in system load due to sensor data uploads at the same collection interval. Multiple partitions are used for message topics in the message queue to improve concurrency efficiency.

[0053] During the data management and acquisition phase, a rich set of connector components is provided, including support for multiple storage methods such as HDFS and Kafka. It also supports the consumption of data stored in message queues with multiple degrees of parallelism, ensuring that the consumption speed of the data link does not become a bottleneck at the entry point.

[0054] Existing streaming computing engines, including Flink used in the system, do not currently provide management functions for the entire dataset. This invention extends this functionality through a database interface, allowing data from all connected data sources to be transmitted downstream via the database.

[0055] The pre-filtering module: This module uses a lightweight Bloom filter to deduplicate the collected information, and simultaneously utilizes a key-value store to deduplicate the massive amounts of data. Specifically:

[0056] Upstream processes often result in data duplication during data collection, message delivery, and format parsing. Data duplication can affect downstream applications such as data monitoring. In addition to traditional uses such as UV statistics, the significance of deduplication lies in eliminating dirty data generated by unreliable data sources—that is, the impact of duplicate data reports or duplicate data deliveries—so that the results generated by streaming computing are more accurate.

[0057] This invention implements a lightweight data filtering method provided by a Bloom filter, which can lead to an overestimation of downstream statistical data when the upstream data source can only guarantee at least once delivery semantics.

[0058] Bloom filters deduplicate data using a built-in hash function, and their advantages are:

[0059] 1. The operation does not require storing the data itself, but only uses bits to represent it. Therefore, it has a huge advantage in space usage compared to traditional methods, and can keep the data confidential.

[0060] 2. The deduplication algorithm has high execution efficiency, with the time complexity of insertion and query both being O(k).

[0061] 3. Hash functions are independent of each other and can be computed in parallel at the hardware instruction level if needed, increasing processing efficiency.

[0062] However, Bloom filters cannot guarantee complete filtration accuracy, and are not suitable for scenarios requiring 100% accuracy.

[0063] For scenarios requiring accurate deduplication, this invention implements a deduplication method for massive amounts of data using key-value storage features. It leverages the key-value storage features of a RocksDB state backend, such as those provided by the Flink engine, to merge message records with the same key. This is suitable for situations with high requirements for business data and zero tolerance for errors. While providing accurate filtering capabilities, it is necessary to implement fine-grained state management for the filtering jobs, preventing unlimited state growth by setting timeouts and configuring incremental checkpoints.

[0064] To address the issue of state bloat when data keys occupy a large amount of space, a hash algorithm is used to convert the keys into integers before storage, ensuring they occupy a maximum of 8 bytes. However, since hash algorithms cannot guarantee against collisions, their use needs to be determined based on the business scenario. The deduplicated data stream is then filtered for invalid data using custom business thresholds or custom filtering rules, reducing downstream computational pressure. Simultaneously, the original data is unpacked and packaged into the required data format according to business needs. Finally, a timestamp is added to the data stream records entering the streaming computing system for easy subsequent processing.

[0065] Data preprocessing module: The streaming computing engine provides unified streaming and batch processing capabilities, unifying the message processing path and avoiding resource waste caused by developing two separate sets of business logic for offline and real-time applications. Furthermore, it abstracts the data flow by providing SQL support. Specifically:

[0066] During the data preprocessing and feature engineering stages, the streaming computing engine aims to provide unified computing capabilities for both streaming and batch processing, avoiding resource waste caused by developing separate offline and real-time business processes. Furthermore, it improves job development efficiency by providing SQL support for abstracting data flow. Based on these characteristics, compared to traditional solutions, this invention can unify the development logic of real-time and offline processes through a unified computing engine, thereby avoiding inconsistencies in computational semantics caused by inconsistent data processing paths. This eliminates the need for repeated cleaning and filtering of downstream data.

[0067] After filtering, industrial data still needs to undergo effective trimming during the data preprocessing stage:

[0068] 1. Consume data packets sent by the collector from the message queue and parse them according to the transmission protocol.

[0069] 2. The data stream is split by the acquisition device and the device identifier, and then the data with the same identifier is included in the same data stream.

[0070] 3. Combine data according to the pre-defined data processing groups, and merge the data of the correlation analysis into the same group.

[0071] 4. Aggregate the data based on the time window, and regenerate the unbounded stream by sliding the window.

[0072] 5. Clean the data based on simple threshold logic, and record data that does not meet the requirements or is obviously abnormal to the abnormal branch stream for system optimization and analysis.

[0073] 6. Record the percentage of valid data and the number of processing rule hits during the data processing process for system optimization and analysis.

[0074] Data lineage module: Utilizing the SQL abstraction provided by the streaming computing engine, it uses SQL scripts to describe the data flow and then submits them to the computing platform for parsing and job startup. Specifically:

[0075] Leveraging the SQL abstraction capabilities provided by the streaming computing engine, the data processing flow in a task can be quickly described. Therefore, the primary method for submitting streaming computing tasks in the system is to write SQL scripts to describe the data flow, and then submit them to the platform for parsing and job startup.

[0076] Typically, development requires first defining DDL (Database Schema Definition Language) to specify data metadata, and then writing DML (Database Manipulation Language) to specify the data processing procedures. In most tasks, the DDL definition statements are repetitive and do not yield any benefits in job development. Metadata information is also stored in the persistent storage metastore component and can also be retrieved from upstream and downstream catalogs (databases). Therefore, this solution defines the real-time data stream as a real-time table stored in the metastore component. Users can directly select the required data source and then only need to write DML statements to complete job development. When the task is submitted, the system will supplement the transformed DDL statements based on the upstream and downstream catalogs to form a complete StreamingSQL script for submission. This further reduces the difficulty of job development and also further streamlines the development processes of batch processing and stream processing, making the development steps of both more similar.

[0077] Furthermore, the solution's rule engine supports continuous upgrades to configuration rules. Metadata hosted in the metastore provides a degree of compatibility with inconsistencies in data schemas introduced from different data nodes. During data processing, preprocessed data is typically written to a message queue and then trained online. This training process is continuous, generating dynamic models that are then pushed to the online inference module for inference. Online machine learning is characterized by dynamic model updates, continuous training, and constant validation. It also requires sophisticated model monitoring, deployment, and rollback strategies. At compute nodes, data structures may change due to dimension table joins and multi-stream convergence; tracing the data schema lineage provides support for handling these changes.

[0078] like Figure 2 As shown, the present invention also provides a data processing method based on a streaming computing engine, comprising the following steps:

[0079] Data acquisition steps: After acquiring raw data from the terminal, the encoding and decoding protocols of the sending and receiving ends are developed simultaneously. The corresponding table information is matched from the unbounded data stream to complete the acquisition of information events. At the same time, the business field information is cleaned and filtered.

[0080] Pre-filtering steps: The collected and filtered information is deduplicated using lightweight data filtering provided by the Bloom filter, and simultaneously deduplicated using massive data filtering provided by Key-Value storage.

[0081] Data preprocessing steps: The streaming computing engine provides the ability to unify streaming and batch processing, unifying the message processing path and avoiding the waste of resources caused by developing two sets of business processes for offline and real time. It also provides SQL support to abstract the data flow.

[0082] Data lineage steps: Through the SQL abstraction processing provided by the streaming computing engine, a script is written in SQL language to describe the flow of data, and then submitted to the computing platform for parsing and startup.

[0083] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0084] This invention primarily uses SQL to construct tasks, and based on this, a StreamingSQL editor was developed. This editor provides support for system-defined SQL specifications and is compatible with StormSQL, SparkSQL, KSQL, HiveSQL, and FlinkSQL syntax. In addition to providing support for engine dialect SQL, it also provides SQL templates. This allows platform users to quickly develop SQL tasks for preset scenarios based on these templates. Furthermore, a web-based guide page is provided for business users, helping them build streaming processing tasks without writing code by filling in forms.

[0085] In addition, the editor provides a component library for custom functions, containing a rich set of built-in functions, including time functions, collection functions, JSON processing functions, and string functions. This rich set of built-in functions facilitates user development, saving users time compared to developing their own. The component library also includes user manuals and use cases written by developers for platform users to search and consult. The component library also provides a cloud platform for system users to upload and download custom functions and operators accumulated in their business processes, facilitating data sharing across different system deployments and subsequent development support.

[0086] The editor also implements syntax checking and intelligent suggestions for StreamingSQL. Users can perform real-time syntax checks and receive some hints while writing statements. It can also provide metadata completion for tables and fields involved in the statements, thus optimizing the development experience.

[0087] The editor also supports online debugging, which is crucial for streaming computing. This avoids the need to adjust tasks after deployment based on observed results, saving development costs. The system can accept text files as a data source to verify if the output meets expectations, and can also sample message topics from message queues or data from database tables for business logic validation.

[0088] The most complex aspect of SQL parsing is widening data tables through dimension table joins. This invention utilizes hot storage for these joins. After data is imported from the data source, the system uses Async I / O technology to access the backend, which in turn uses a Data Accessor interface to access its storage. The backend storage supports data retrieval from distributed storage systems, structured databases, NoSQL databases, message queues, and other data pipelines, while simultaneously caching data in an LRU cache module. Broadcasting updates to dimension tables via message queues reduces the likelihood of delayed dimension data updates. The system incorporates storage support for various big data tools for the data after dimension table joins, significantly enhancing system compatibility.

[0089] Because rule matching conditions and business needs are constantly changing, rules often need to be frequently adjusted to adapt to these changes. Business personnel can use the feature management interface on the front end to add, delete, modify, and query rule factors in the rule base, constructing different feature rules by combining different rule factors. They can then specify a set of test results to perform simple tests on the constructed rules, and save the entire set only after the test results are satisfactory. This avoids directly modifying the rule base with each operation, thus decoupling business rules from rule factors.

[0090] In terms of measuring overall effectiveness, the rationality of a rule is determined by analyzing its hit rate and the frequency of alarms.

[0091] It is necessary to determine whether the rule has become invalid, such as a sudden drop in the interception rate.

[0092] Determine if a rule is redundant, for example, if a rule has never blocked any events.

[0093] Determine if there are loopholes in the rules, such as performing an operation but not achieving the expected result.

[0094] In addition, the rule gray-scale deployment mechanism is also based on the overall effect evaluation of the rules. The rule release process is optimized by setting whitelists for different regions, importing real data test results, or limiting the rate of deployment to the official environment.

[0095] In terms of accumulating, analyzing, and optimizing data, it is necessary to discover combination rules and identify combinations of certain behaviors to prevent situations where each step in the operation is normal and usable, but the cumulative results of the behaviors are abnormal. It is also necessary to perform group identification, such as by using graph analysis technology to discover groups and label them, to prevent situations where each part performs normally, but the entire area shows abnormalities.

[0096] In the pattern matching of the invented rules, the Rate algorithm is used to improve matching efficiency and reduce time redundancy caused by repeated calculations. When there are many rules and fact samples, each fact needs to be matched with the Alpha node in the Rete network. Most rules contain the same condition atoms, that is, there are condition atoms contained by multiple rules at the same time. Matching each Alpha node sequentially will waste some time. Therefore, a pre-matching module aggregates multiple rules into a small number of rule groups. By filtering the rule groups, some normal data is filtered out in the pre-matching stage, reducing the number of times facts and nodes are matched. The implementation logic is to divide rules containing multiple identical condition atoms into the same rule group, and the condition atom that appears most frequently in the rule group is used as the feature condition of the rule group. The full data is filtered out by the rule groups in the pre-matching module, and the rules within the rule group are applied to the remaining samples.

[0097] The goal of rule learning is to generate a rule set that covers as many examples as possible. The most direct way is sequential coverage, that is, summarizing rule by rule. This is also the simplest training idea for rule learning. After each rule is generated in the training set, the samples covered by that rule are removed from the training set, and then another set of rules is trained using the remaining samples. Of course, to avoid overfitting, most algorithms incorporate pruning optimization strategies.

[0098] To train a highly usable model, excellent features are essential. A key characteristic of excellent features is high discriminative power. This requires close collaboration between algorithm developers and highly perceptive business personnel to analyze the configured rules, extract superior rule factors, and, simultaneously, ensure the algorithm developers are thoroughly familiar with the business processes to create highly usable features tailored to the specific circumstances. In practical business applications, in many cases, adding an accurate feature is more effective than model parameter tuning or even model optimization. Furthermore, accurate features can prevent overfitting. This invention uses the analysis and processing of historical data and the streamlining of industrial monitoring processes to abstract detailed business parameters that can be combined and filtered.

[0099] A crucial requirement for model training is maximizing the number of samples. However, surveillance operations inherently suffer from this limitation; after configuring anomaly rules, very few cases are actually detected, leading to a severe imbalance between positive and negative samples. Therefore, a resampling method was developed to increase the number of positive samples. Specifically, the numerical features of the samples are resampled using the SMOTE method, while the labeled features are resampled using a random selection method. Finally, the two are combined into new samples. During actual training, parameters such as the maximum number of iterations, step size, and maximum tree depth need to be adjusted multiple times to select the parameters with the highest recognition rate.

[0100] As the event stream processed by the system grows, the business rules evolve along with the dataset. After validating the rules using real-time business data and archived historical data, the rule engine gains a certain degree of evolutionary capability.

[0101] Based on the above invention, an industrial data monitoring system based on a streaming computing engine can be incorporated into the Industrial Internet of Things (IIoT), comprising:

[0102] like Figure 3 As shown, the status monitoring module uses Prometheus, a streaming computing engine natively supported, for real-time metric collection and storage. It periodically scrapes and aggregates metrics from all child nodes via the meta node, providing a unified data source to Grafana for visualization and alert configuration. Specifically:

[0103] The front-end interface designed for streaming computing engines provides users with a wealth of runtime information to understand the current running status of tasks. However, it suffers from the inability to obtain historical status monitoring, preventing users from understanding the historical running status of tasks. This invention uses Prometheus, natively supported by the computing engine, for real-time metric collection and storage. Metrics from all child nodes are periodically captured and aggregated through the meta node, facilitating a unified data source for visualization in Grafana and alarm configuration via the dashboard.

[0104] During streaming computing jobs, the continuous influx of data into the cleaning process makes it difficult to monitor the job's progress. Even with checkpoints enabled, it's impossible to determine whether data has been lost or how much data has been lost. Therefore, this invention designs a heartbeat mechanism, injecting heartbeat information into each job to monitor its operation.

[0105] With each incoming data source, the metadata service generates a set of periodic heartbeat signals, which are consumed by the jobs. After the heartbeat information flows into each job, it passes through each node along with the data stream, being tagged with the current node's information before skipping that node's processing logic and moving on to the next node. At the end of the job component, the data is collected and displayed on the page via JMX.

[0106] This metric includes the time the heartbeat signal is generated, the time it flows into the job, and the time it reaches each node. By using this metric, we can determine the time it takes for a piece of data to be processed by the entire pipeline and the time each node takes to process the data, thus identifying the performance bottleneck of the job.

[0107] Since the heartbeat signal is sent periodically, the number of heartbeat messages received by each job should be consistent. If the final number of indicators sent is inconsistent with the expectation, it can be further determined whether there is data loss.

[0108] Throughput and latency are the most important metrics for measuring the performance of real-time tasks, and these two metrics are often used to adjust task concurrency and resource configuration. Enabling the native latency parameter tracking of the computing engine can significantly impact cluster and task performance. This invention uses message topic consumption backlog as a metric for measuring task latency. It uses JMX to obtain the consumption offset in the engine job to check the back pressure of persistent message topics in the message queue. Simultaneously, it uses the JMX provided by the engine to collect the back pressure of job nodes for comprehensive analysis and inference of whether the job is experiencing data backlog.

[0109] To accommodate large-scale data processing, streaming computing engines typically offer distributed computing capabilities. All tasks submitted to the platform are uniformly scheduled to any computing node by the scheduler. Therefore, task execution logs are distributed across different machines, making it difficult for users to locate specific logs. This invention addresses this by adjusting the default mechanism of the log4j logging framework, splitting task logs daily, and periodically cleaning up expired logs. This prevents abnormal tasks from frequently filling up disks and causing computing nodes to become unavailable. Simultaneously, agents are deployed on all computing nodes to collect logs in real time, extracting them into message queues and then aggregating them into non-relational databases such as Elasticsearch. This allows users to quickly find the corresponding logs for a job when they need to locate and determine anomalies through log analysis.

[0110] Resource allocation module: Analyzes real-time tasks and provides sufficient memory for the jobs. It also processes task messages in real-time. Based on relevant metrics from real-time task memory analysis and the rationality of real-time task concurrency, it derives preset values ​​for real-time task resources and adjusts them to achieve optimal real-time task resource configuration. Specifically:

[0111] The approach to real-time task resource analysis mainly includes two points:

[0112] On the one hand, we start by analyzing the memory occupied by the job and the runtime heap memory to provide sufficient memory to ensure stable job execution.

[0113] On the other hand, we can start with real-time task message processing capabilities to ensure that data processing needs are met while making the most rational use of CPU resources.

[0114] Then, by combining the relevant indicators obtained from the real-time task memory analysis and the rationality of the real-time task concurrency, a preset value for real-time task resources is obtained, and the real-time task resources are adjusted to ultimately achieve the goal of rationalizing the configuration of real-time task resources, thereby better reducing the cost of machine use.

[0115] Regarding the management of task memory, the system will periodically scan all running streaming tasks according to the configured intervals, combine real-time task GC logs, and calculate the recommended heap memory size for streaming tasks based on memory optimization rules. This size will be compared with the actual allocated heap memory for the streaming tasks. If the difference between the two is too large, it can be considered that the memory configuration of the streaming tasks is wasted, and an alarm will be generated to prompt the task scheduler for optimization.

[0116] In analyzing the message processing capabilities of real-time job tasks, the main approach is to determine whether the input per unit time of the data source consumed by the real-time task matches the message processing capabilities of each operator / task. By monitoring the traffic of the data source and combining it with the throughput of the operators in the job, it is determined whether the job will experience data skew or traffic backpressure. Based on the job optimization rules, adjustments to operator parallelism and optimization of job resource allocation are suggested.

[0117] Leveraging the native Kubernetes support provided by the streaming computing engine, jobs can proactively request resources from the scheduler, achieving elastic scaling of job resources to handle sudden data spikes. Based on this, the optimization of task resources is fully automated. It automatically predicts and adjusts the resource configuration of real-time tasks by combining historical resource usage data from different time periods, thereby improving the overall resource utilization of the real-time cluster.

[0118] Anomaly Status Module: Utilizes SideOutput to collect error data from each stage of the streaming job, aggregating it into a unified error stream. By identifying patterns matching predefined characteristics within the streaming data, it triggers corresponding subsequent actions for real-time alerts. Specifically:

[0119] Error logs contain system-preset error codes, raw input data, error types, and error messages. Generally, error data is categorized and written to a distributed file storage system, and users can monitor the storage directory to determine if the data is normal.

[0120] There are generally three scenarios when recovering data.

[0121] 1. Abnormal data format, such as logs being truncated and incomplete or timestamps not conforming to the agreed format. In this case, the system provides a way to repair the data through precise offline batch jobs, skipping the processing of abnormal data events or re-entering the job process after correcting the data, and backfilling the data into the original data pipeline.

[0122] 2. Anomalies in the job pipeline, such as changes to the actual data schema but failure to update the flow table configuration, may result in null values ​​for certain fields, even though the overall job does not show any anomalies. An alert will be displayed via metric output to remind the user to perform data correction.

[0123] 3. Data link anomalies, such as a failover due to an anomaly in the data source master-slave cluster, but without corresponding changes to the consumer configuration, can lead to timeouts in downstream data applications. Normally, the system's retry and anomaly recovery mechanisms can promptly switch to a healthy data source, and alerts will be issued if a job fails.

[0124] During the data replenishment operation, the online flow table configuration is first updated to the latest version, and a healthy data source is switched to ensure that no more abnormal data is generated. At this point, some partitions in the storage are still abnormal. Therefore, a separate data replenishment job is issued specifically to repair the abnormal data. The output data is written to a temporary directory, and the partition location on the metastore is switched to replace the original abnormal directory. Therefore, this data replenishment process is transparent to users performing offline queries. Finally, the data in the abnormal partitions is replaced and the location is restored at an appropriate time.

[0125] Query optimization using streaming engines:

[0126] In scenarios involving dimension table joins, the dimension tables frequently change, especially with the addition of new dimensions, while the join operation occurs before the dimension is added, often resulting in join failures. To address this, the system has developed exception handling strategies tailored to different needs. If an exception occurs during a join, the data is temporarily cached and the attempt is retried, with the number of attempts controllable and customizable delayed join rules.

[0127] Additionally, an operator supporting delayed joins to dimension tables has been added to SQL support. When a join fails, the local cache will not cache an empty dataset result; instead, the data will be temporarily stored in a state backend, and retrying will be performed later based on the set timer and its retry count.

[0128] Through job structure topology analysis, it's clear that computational operators and associated computational operators are connected together during streaming engine job execution. This is because it lacks unique key-based routing semantics. When job parallelism is high, each dimension table's associated subtask accesses the entire cache space, placing significant pressure on the cache.

[0129] However, observing the SQL implementation of management operations, equi-joins inherently possess hashing properties. The configuration is directly exposed when wrapping the operators, allowing users to use the key of the dimension table association as a hashing condition to partition the data. This ensures that the access space between subtasks of each downstream operator is independent, significantly improving the cache hit rate at the start of the job.

[0130] Optimize indexing using a streaming engine:

[0131] To accelerate data retrieval, databases often create indexes beforehand, using these indexes to locate the starting position of the data before scanning, thus speeding up the process. Traditional databases commonly use row-level indexes, created by indexing one or more fields. The index results are stored in a tree structure, offering row-level precision and the highest indexing efficiency.

[0132] Some big data projects also support row indexing, but its drawback is that the large amount of index data can cause latency in writing and retrieval. Platforms primarily process collected data, such as sensor data, which is characterized by very high repetition, while the analysis results are often very few. A very small number of target behaviors are hidden within the massive amounts of data, often accounting for only one-thousandth or even less. Therefore, choosing a more cost-effective block indexing solution is sufficient to support current application scenarios.

[0133] Existing solutions often store index data as files on disk, along with caching mechanisms to accelerate data access. However, this system stores the index data directly in the database. This is primarily due to the following two considerations:

[0134] 1. Transaction: Generally speaking, columnar storage files are often not updatable. However, when the system periodically optimizes the file distribution, it will perform multiple file content merging operations. In order to ensure query consistency, the database needs to provide transaction capabilities.

[0135] 2. Performance: The database has strong read, write, and retrieval capabilities, and can even push predicates down to the database to complete the task. The high compression ratio of the database can also further save storage space.

[0136] Optimize for anomalies using a streaming engine:

[0137] Because streaming jobs require caching a large number of intermediate processes and requesting significant computing resources, it can take about one to two minutes for a job to restart after an abnormal failure. This is unacceptable for some online business scenarios.

[0138] Analysis of the job anomaly generation process revealed that anomaly detection and initialization were the main bottlenecks. Anomaly detection was constrained by the interface polling interval, while resource initialization was constrained by the container initialization steps. The system optimized these two aspects to enable rapid detection of job failures during runtime. Furthermore, resources were reserved to save time spent requesting resources and initializing them when a crash occurs.

[0139] The system adds a majority-based connectivity detection service on top of the streaming computing engine. Multiple worker nodes in the detection service cluster periodically check the connectivity of each machine in the cluster. Since it is majority-based, its reliability is guaranteed.

[0140] In addition, regarding resource reservation, the system has expanded the resource request model for streaming engine jobs. When a streaming job is submitted, resource redundancy parameters can be set. When the redundancy parameters are activated, the system will automatically ensure that the amount of redundant resources is higher than the amount of resources lost due to a single point of failure, and will avoid the clustering of redundant resources in resource arrangement.

[0141] Meanwhile, to address situations where cluster hardware resources are fully loaded, the system controls the consumption rate of data sources. A coordinator is introduced to periodically check the resource consumption on the job setter and the progress of data source watermarking. Based on the overall status, it predicts the maximum allowable reading position for each data source and then distributes this prediction to all data sources. Data sources determine their reading speed based on the obtained maximum position and their current position, and limit the data consumption rate according to the load. This dynamic adjustment of the consumption rate of all data sources ensures the stability of the streaming job.

[0142] Prometheus is an open-source service monitoring system and time-series database; meta is an auxiliary tag in the HTML `<head>` section, located at the beginning of a document and containing no content; Grafana is a dashboard and graphical editor; SideOutput is a stream of any number of additional side output results; JMX is a framework for embedding management functions into applications, devices, and systems; agent is a software or hardware entity that can act autonomously; log4j is an Apache open-source project that allows control over the destination of log information, such as the console, files, GUI components, or even socket servers, NT event loggers, and UNIX Syslog daemons. A transaction generally refers to something that needs to be done or has been done. In computer terminology, it refers to a unit of program execution that accesses and may update various data items in a database. Performance is a front-end performance monitoring API. It can detect page performance; a new API introduced by the W3C Performance Group, it can detect white screen time, first screen time, user-operable time points, total page download time, DNS lookup time, TCP connection time, etc.

[0143] The workflow after combining a data processing system based on a streaming computing engine with a data monitoring system based on a streaming computing engine is as follows:

[0144] 1. Depending on the scenario requirements, users can upload custom functions developed for business logic via the upload module in the form of a JAR file, extending the definition of data processing operators and SQL functions needed in the data processing process.

[0145] 2. Users describe the data flow in the job using SQL statements, organize and arrange the sequential relationships of data processing, and then adjust the parallelism of each processing node and the overall resource allocation of the job as needed. It also supports uploading JAR packages to implement the interface and orchestrate the task process for more precise control over the job process.

[0146] 3. Add judgment rules to the rule engine according to business needs, including threshold judgment, pattern judgment, combination logic judgment, etc. For complex association judgment rules, users can also customize them. After uploading the JAR package that implements the interface specified by the system, it can be configured in the system.

[0147] 4. Configure the system's rule learning module according to the operational needs, enable the platform's system evolution function, and continuously train the rule model during system data processing. Optimize and evolve the platform's rules by learning user interaction logic and operational data, including data traffic peaks and troughs, rule hit rates, and resource consumption. The evolved, more refined rules that better suit the characteristics of business operations are then manually confirmed before being deployed.

[0148] 5. Configure the data source and data migration channel for the job, supporting reading and writing data from distributed storage systems, structured databases, NoSQL databases, message queues, and other data channels. It also supports the construction and parsing of various data formats such as JSON, protobuffer, and Avro.

[0149] 6. Submit to the job system for orchestration. First, the processing modules are generated through logical decomposition, while the syntax and custom package writing standards in the job flow are checked. A directed acyclic graph constructed by modules is displayed on the page, which can clearly show the data flow of the job and the arrangement order of operators. After checking for errors, it is submitted to the task scheduling module to request resources and start the job.

[0150] 7. After a job request obtains job resources from the resource cluster, it continuously retrieves data with corresponding tags from the data source. Following data processing according to configured rules, the data is then evaluated by the rule engine before being output to the real-time display module or further processed. During the job process, users can monitor job details through the task monitoring module, including job throughput, network I / O, resource usage, and module backpressure, allowing them to understand the job's running status.

[0151] 8. Based on user configuration, confirmation points will be generated during the job process, either in time periods or in job batches, containing all status information of the current job. Simultaneously, business archive points will be generated at intervals in the business process, containing the calculation results of the current job. Combined with these two mechanisms, job metrics can be accumulated and job status analyzed. Furthermore, the ability to recover from anomalies in job units is provided.

[0152] 9. When an anomaly occurs during the job workflow, recovery will be performed according to user configuration. This includes attempting job restart, attempting recovery from the confirmation point, recording abnormal data, or fast failure followed by push notification, among other anomaly handling configurations. The task status in the job monitoring module will also be updated in real time to help operations and maintenance personnel promptly obtain information about task anomalies.

[0153] 10. When the job terminates, the resource monitoring module will reclaim resources after obtaining the current task's status information, releasing computing resources, clearing file cache, and retaining or clearing save points and confirmation points according to user configuration. The streaming computing job will then terminate after storing the task's status data in the time-series database.

[0154] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0155] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A data processing system based on a streaming computing engine, characterized in that, include: Data acquisition module: After acquiring raw data from the terminal, it synchronously compiles the encoding and decoding protocols for the sending and receiving ends, matches the corresponding table information from the unbounded data stream to complete the acquisition of information events, and cleans and filters the business field information at the same time. Pre-filtering module: The collected and filtered information is deduplicated using lightweight data filtering provided by Bloom filter, and simultaneously deduplicated using massive data filtering provided by Key-Value storage; Data preprocessing module: The streaming computing engine provides the ability to unify the message processing path by stream and batch processing, avoiding the waste of resources caused by developing two sets of business processes for offline and real time, and abstracting the data flow by providing SQL support; Data lineage module: Through the SQL abstraction processing provided by the streaming computing engine, the SQL language is used to write scripts to describe the flow of data, and then submitted to the computing platform for parsing and starting the job; The pre-filtering module uses the hash function built into the Bloom filter to deduplicate the data; -Use Key-Value storage to merge message records that have the same key; - A hash algorithm is used to convert the key into an integer for storage, so that it occupies a maximum of 8 bytes; During development, the data lineage module first defines a database schema definition language to specify data metadata, then writes a data manipulation language to specify the data processing procedure, defines the real-time data stream as a real-time table and stores it in the metadata storage component. Users can directly select the required data source from it, and then only need to write statements in the data manipulation language to complete the job development. When a task is submitted, the data lineage module supplements the converted database schema definition language statements based on the upstream and downstream databases to form a complete Streaming SQL script for submission, thus connecting the development processes of batch processing and stream processing. The data preprocessing module is also used to: when performing dimension table association, if the association fails, temporarily store the data to be associated in the status backend, and retry the association according to the preset timer and number of retries.

2. The data processing system based on a streaming computing engine according to claim 1, characterized in that, When collecting information events in the data acquisition module, the log file is written in a mixed format. It is necessary to parse the records in row format and statement format separately according to the record storage format. The log file will only be updated after the streaming computing engine submits the transaction and refreshes it to the disk. It is necessary to record the log consumption point to deal with the abnormal restart of the job.

3. The data processing system based on a streaming computing engine according to claim 1, characterized in that, After the data acquisition module collects the unbounded data stream, the unbounded data stream is sent to the consumption queue. The high throughput processing provided by the message queue completes the peak smoothing and valley filling of the data. The message topics of the message queue are set up with multiple partitions.

4. The data processing system based on a streaming computing engine according to claim 1, characterized in that, The data acquisition module provides HDFS and Kafka storage support during the data management and acquisition phases, and also supports the consumption of data stored in message queues with multiple degrees of parallelism, ensuring that the consumption speed of the data link does not become a bottleneck at the entry point.

5. The data processing system based on a streaming computing engine according to claim 1, characterized in that, The pre-filtering module further filters out invalid data from the deduplicated data stream by using a filter operator with custom business thresholds or custom filtering rules.

6. A method for implementing the data processing system based on a streaming computing engine as described in claim 1, characterized in that, Includes the following steps: Data acquisition steps: After acquiring raw data from the terminal, the encoding and decoding protocols of the sending and receiving ends are compiled synchronously. The corresponding table information is matched from the unbounded data stream to complete the acquisition of information events. At the same time, the business field information is cleaned and filtered. Pre-filtering steps: The collected and filtered information is deduplicated using lightweight data filtering provided by the Bloom filter, and simultaneously deduplicated using massive data filtering provided by Key-Value storage; Data preprocessing steps: The streaming computing engine provides unified streaming and batch processing, avoiding the waste of resources caused by developing two sets of business processes for offline and real time, and abstracting the data flow by providing SQL support; Data lineage steps: Through the SQL abstraction processing provided by the streaming computing engine, a script is written in SQL language to describe the flow of data, and then submitted to the computing platform for parsing and startup.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method of claim 6.

Images