Data processing method of industrial database

Abstract

The invention provides a data processing method of an industrial database, and relates to the technical field of data processing. The data processing method of the industrial database comprises the following steps: S1, constructing a stream batch integrated processing engine, and carrying out joint calculation on real-time data stream and historical data; s2, dynamically allocating a storage structure according to an entity identifier by adopting an entity partition storage model; s3, executing real-time query based on an incremental graph traversal technology, and dynamically pruning in a traversal process; s4, entity behavior abnormity is detected in real time through the space-time fusion model; and S5, compressing the event stream data according to a causal rule. According to the technology, real-time and historical data are processed by utilizing a stream batch integrated engine, and full-dimension rapid decision analysis is realized. Incremental graph traversal, space-time fusion anomaly detection and the like respectively improve the query efficiency and accurately position faults. The expenditure is reduced through causal rule compression and the like, the end-to-end credibility is guaranteed through the quaternion metadata block and verification technology, the storage read-write efficiency is improved through thermal scheduling, and the problems of response, detection and storage are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a data processing method for an industrial database. Background Technology

[0002] Industrial databases are database systems specifically designed for the industrial sector. They are used to store, manage, and process various types of data generated during industrial production processes. This data includes equipment operating parameters, production process data, quality inspection information, and raw material inventory. They feature high reliability, real-time performance, and high concurrency processing capabilities, adapting to the stringent data requirements of industrial environments. They can operate stably in complex and harsh industrial settings, ensuring no data loss; and they can quickly respond to and process large amounts of real-time data. Data processing is necessary because data generated in industrial production often suffers from incompleteness, inaccuracy, and inconsistent formats. Directly using this raw data can affect the accuracy and reliability of decision-making. Data processing can clean up noise and errors in the data, improving data quality. Industrial data is massive, containing a wealth of valuable information, but this information is often hidden within the vast amounts of data. Data processing can extract valuable information, such as identifying potential problems in the production process, optimizing production processes, and predicting equipment failures. Different industrial systems and equipment may generate data with different formats and standards. Data processing can transform and integrate this data, enabling data sharing and interaction, breaking down information silos, and allowing different departments and systems to collaborate more effectively.

[0003] Traditional industrial data processing architectures generally suffer from serious design flaws. Real-time computing and historical analysis systems are isolated, forming computational silos. This prevents real-time data streams from being linked to historical behavior patterns in real time, resulting in a lack of complete data support for critical decisions. Graph traversal queries rely on static pre-computed paths, failing to dynamically adapt to changes in equipment status, generating a large amount of ineffective computational load and causing severe query response delays. Anomaly detection technologies are limited to single-dimensional analysis, either focusing only on temporal fluctuations while ignoring spatial correlations, or relying on topology while disregarding temporal evolution patterns, leading to a persistently high rate of missed early fault signals. Event data processing uses simple time window aggregation, failing to distinguish between causal relationships and random coincidences, burying key event patterns in noisy data. Storage systems rigidly allocate physical resources, storing high-frequency access data mixed with low-frequency, cold data, resulting in low read / write efficiency and high costs. Storing raw time-series values ​​in their entirety is extremely wasteful of storage space, and differential compression technology struggles to address sudden value jumps. Data integrity assurance relies on traditional backup mechanisms, resulting in lengthy fault recovery times that fail to meet the real-time requirements of industrial scenarios. This technological deficiency leads to insufficient value mining of massive amounts of device data, lagging and passive operation and maintenance, and uncontrolled growth in storage costs. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a data processing method for industrial databases, which solves the problems of insufficient data value mining and high storage costs in existing technologies.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data processing method for an industrial database, comprising the following steps: S1. Build a unified stream and batch processing engine to perform joint calculations on real-time data streams and historical data; S2. Adopt a physical partitioned storage model and dynamically allocate storage structure according to entity identifier; S3. Perform real-time queries based on incremental graph traversal technology and dynamically prune branches during the traversal process; S4. Real-time detection of entity behavior anomalies using a spatiotemporal fusion model; S5. Compress event stream data based on causal rules.

[0006] Preferably, the implementation method of the dynamic pruning is as follows: Calculate the current path during traversal. Confidence level:

[0007] in: This represents the path confidence level (range [0,1]), used to evaluate path effectiveness; For the first Real-time weight of the edge (dynamically calculated based on historical interaction frequency). This is the number of edges contained in the current path; It is the status score of the target node (generated by preset scoring rules); When the confidence level is lower than the dynamic threshold The traversal of this path will terminate at that time.

[0008] Preferably, the dynamic threshold is calculated as follows: A sliding window statistical mechanism is used:

[0009] in: It is the historical path confidence mean (based on the most recent) (Data calculation within the time window) It is the standard deviation of the confidence level of the historical path; Sensitivity adjustment factor (preset range 0.5 to 3.0); The time window Dynamically configure based on business scenarios.

[0010] Preferably, the spatiotemporal fusion anomaly detection includes: (a) For each entity Establish a state prediction model:

[0011] (b) Calculate joint loss by fusing spatial topology information:

[0012] in: It is a physical entity exist The actual observed value at time; It is a predicted value (output by the Kalman filter model); For graph structure feature vectors (dimension ≥ 16); Representing entities The set of direct neighbors; This is the spatial error weighting coefficient (preset value > 0).

[0013] Preferably, the feature vector is generated using a graph attention mechanism: An improved attention weight calculation method is used:

[0014] in: It is a physical entity For entities Attention weights; This represents a vector concatenation operation; It is a trainable parameter vector (with the same dimension as the concatenated vector); The activation function has a fixed negative slope parameter of 0.01.

[0015] Preferably, the causal rule compression includes: a) Calculation of the strength of causal relationship of the event:

[0016] in: It is an event lead to The conditional probability; The time decay factor (range 0.01-0.1). It is an event arrive The delay time (in milliseconds); b) Only when the intensity value exceeds the threshold Compression is triggered when the value is within the preset range of 0.3-0.8.

[0017] Preferably, the specific method for calculating the conditional probability is as follows: Use an event counting and statistics mechanism:

[0018] in: Indicates an event The number of occurrences during the observation period; yes After it happens within the time window (Preset 200ms-5s) The number of times it occurs; Smoothing factor (default value) ), used to avoid division by zero errors.

[0019] Preferably, the historical data storage employs differential encoding: Perform data compression based on time series:

[0020] in: It is a moment The differential encoded value; It is the original value at the current moment; It is the value from the previous moment.

[0021] Preferably, a metadata block structure corresponding to the compressed data block is generated, and the metadata block structure is as follows: Quad-tuple packaging:

[0022] in: It is a unique identifier for the entity (a string of 16 bytes in length); and Indicates the time range of the data block (Unix timestamp); It is the SHA-256 hash value (32 bytes) of the data block.

[0023] Preferably, the hash value is calculated as follows:

[0024] in, It is a differentially encoded data sequence, arranged in timestamp order.

[0025] This invention provides a data processing method for industrial databases. It has the following beneficial effects: This invention provides a data processing method for industrial databases. This technology deeply integrates real-time and historical data processing capabilities through a stream-batch integrated engine, overcoming the architectural limitations of traditional industrial databases. It enables rapid real-time graph traversal queries while simultaneously providing high-precision equipment anomaly warnings. Key technological advantages include a dynamic pruning mechanism that intelligently filters invalid computation paths, reducing query latency and improving real-time response. Spatiotemporal fusion anomaly detection technology simultaneously captures equipment state anomalies and spatial topology correlation risks, significantly improving fault prediction accuracy. A causal rule compression engine constructs a high-value information extraction network by quantifying event correlation strength, improving the compression efficiency of the original event stream while ensuring the integrity of key data. The storage layer employs an optimization strategy combining entity partitioning and differential coding, significantly reducing storage resource consumption while ensuring rapid data retrieval. A metadata block hash verification mechanism ensures end-to-end data integrity, and an intelligent thermal scheduling strategy optimizes the layered storage efficiency of hot and cold data. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] like Figure 1 As shown, this embodiment of the invention provides a data processing method for an industrial database, including the following steps: S1. Build a unified stream and batch processing engine to perform joint calculations on real-time data streams and historical data; S2. Adopt a physical partitioned storage model and dynamically allocate storage structure according to entity identifier; S3. Perform real-time queries based on incremental graph traversal technology and dynamically prune branches during the traversal process; S4. Real-time detection of entity behavior anomalies using a spatiotemporal fusion model; S5. Compress event stream data based on causal rules.

[0029] The implementation method of dynamic pruning is as follows: Calculate the current path during traversal. Confidence level:

[0030] in: This represents the path confidence level (range [0,1]), used to evaluate path effectiveness; For the first Real-time weight of the edge (dynamically calculated based on historical interaction frequency). This is the number of edges contained in the current path; It is the status score of the target node (generated by preset scoring rules); When the confidence level is lower than the dynamic threshold The traversal of this path will terminate at that time.

[0031] The dynamic threshold is calculated as follows: A sliding window statistical mechanism is used:

[0032] in: It is the historical path confidence mean (based on the most recent) (Data calculation within the time window) It is the standard deviation of the confidence level of the historical path; Sensitivity adjustment factor (preset range 0.5 to 3.0); Time window Dynamically configure based on business scenarios.

[0033] Spatiotemporal fusion anomaly detection includes: (a) For each entity Establish a state prediction model:

[0034] (b) Calculate joint loss by fusing spatial topology information:

[0035] in: It is a physical entity exist The actual observed value at time; It is a predicted value (output by the Kalman filter model); For graph structure feature vectors (dimension ≥ 16); Representing entities The set of direct neighbors; This is the spatial error weighting coefficient (preset value > 0).

[0036] Feature vectors are generated using a graph attention mechanism: An improved attention weight calculation method is used:

[0037] in: It is a physical entity For entities Attention weights; This represents a vector concatenation operation; It is a trainable parameter vector (with the same dimension as the concatenated vector); The activation function has a fixed negative slope parameter of 0.01.

[0038] Causal rule compression includes: a) Calculation of the strength of causal relationship of the event:

[0039] in: It is an event lead to The conditional probability; The time decay factor (range 0.01-0.1). It is an event arrive The delay time (in milliseconds); b) Only when the intensity value exceeds the threshold Compression is triggered when the value is within the preset range of 0.3-0.8.

[0040] The specific method for calculating conditional probability is as follows: Use an event counting and statistics mechanism:

[0041] in: Indicates an event The number of occurrences during the observation period; yes After it happens within the time window (Preset 200ms-5s) The number of times it occurs; Smoothing factor (default value) ), used to avoid division by zero errors.

[0042] Historical data storage uses differential encoding: Perform data compression based on time series:

[0043] in: It is a moment The differential encoded value; It is the original value at the current moment; It is the value from the previous moment.

[0044] Generate a metadata block structure corresponding to the compressed data block. The metadata block structure is as follows: Quad-tuple packaging:

[0045] in: It is a unique identifier for the entity (a string of 16 bytes in length); and Indicates the time range of the data block (Unix timestamp); It is the SHA-256 hash value (32 bytes) of the data block.

[0046] The hash value is calculated as follows:

[0047] in, It is a differentially encoded data sequence, arranged in timestamp order.

[0048] The specific implementation process is as follows: System initialization and data reception Deploy a unified stream and batch processing engine, configuring real-time data access ports and historical database connection channels. Start the entity partition storage controller, dividing physical storage areas based on the device's unique identifier (EntityID), creating an independent data partition for each device, and setting up a capacity monitor. When real-time data streams (such as sensor readings and device logs) continuously input, the engine automatically distributes the processing: time-sensitive data is directed to the real-time computing thread, while batch processing tasks are imported into the distributed computing cluster. Simultaneously, an entity relationship graph loading mechanism is established, loading a pre-defined device topology (such as factory equipment connection relationships) from the metadata database and mapping it to an in-memory graph structure for incremental computing. The storage controller dynamically monitors partition load, automatically splitting sub-partitions when the data volume of a specific device exceeds a threshold, ensuring continuous optimization of the storage structure.

[0049] Real-time query and abnormal status linkage mechanism After a user submits an entity association query request, the incremental graph traversal module loads the adjacency relationships of the target entity and searches along each edge path of the graph. Dynamic pruning is implemented during the traversal: the confidence score of the current path is calculated in real time (path weight product × target node state score), and the latest historical mean and standard deviation of confidence scores are simultaneously calculated using a sliding window to calculate a dynamic threshold (mean - standard deviation × adjustment factor). When the confidence score falls below the threshold, the branch is immediately terminated, retaining only high-confidence paths to complete the query response. The parallel anomaly detection engine continuously collects entity time-series data, predicts the state value of the next time step using a state-space model, calculates the spatial consistency error (difference in features between adjacent entities) using entity feature vectors generated by a graph neural network, and fuses the time-series prediction bias (squared difference between measured and predicted values) to generate a joint anomaly score. Entities exceeding the threshold trigger real-time alarms.

[0050] Causal compression and differential storage execution The event stream analyzer scans the input event sequence and, based on event counter statistical rules, captures the frequency of event Y within a fixed time window (200ms-5s) after a specific event X occurs. This frequency is divided by the total frequency of event X, and a smoothing factor is added to calculate the conditional probability value. The causal strength (conditional probability × time decay exponent) is calculated by combining the event delay time (accurate to milliseconds) and a preset decay factor. When the strength exceeds a preset threshold (0.3-0.8), the compression rule engine is activated, transforming the original event stream into a causal rule triple (X, Y, strength) for storage. Simultaneously, the historical data compression module performs differential encoding on the original time-series value sequence: using the first valid value as a reference point, subsequent values ​​only store the difference from the previous value, forming a differential sequence chain.

[0051] Data verification and intelligent storage scheduling Metadata structures are generated for each compressed differential data block: an embedded unique entity identifier (16-byte string), precise recording of the start and end timestamp of the data block (Unix timestamp format), and an integrity hash calculation of the differential sequence using the SHA-256 algorithm. The final result is encapsulated as a four-tuple metadata set (entity identifier / start time / end time / hash value). The storage scheduler dynamically migrates data based on query heatmap results: frequently accessed data blocks are automatically scheduled to high-speed SSD storage, while infrequently accessed data is downgraded to mechanical hard disk storage. A hash verification mechanism is triggered after each data read, comparing the stored value with the calculated value. In case of data loss, a historical backup and recovery process is initiated, forming a closed-loop storage quality assurance system.

[0052] This technology constructs an integrated stream and batch processing engine to overcome data limitations, enabling rapid fusion of real-time data streams and historical databases to support multi-dimensional analysis and decision-making. Incremental graph traversal introduces a dynamic pruning mechanism, intelligently filtering inefficient branches based on real-time path confidence calculation and sliding window threshold adjustment, thus improving the efficiency of complex relational queries. The spatiotemporal fusion anomaly detection model overcomes single-dimensional limitations, simultaneously capturing device state prediction deviations and spatial topological neighbor feature consistency errors, achieving rapid and accurate early fault location. The causal rule compression engine quantifies causal strength through time window event statistics and decay factors, intelligently extracting high-value event patterns and improving original data compression efficiency while retaining key information. Entity partitioned storage and differential coding are collaboratively optimized, managing data independently by device and converting time-series values ​​into incremental sequences, significantly reducing storage requirements while enabling rapid historical value reconstruction. The quadruple metadata block design, combined with blockchain-level hash verification technology, constructs an end-to-end data trust guarantee system. The intelligent thermal sensing scheduler dynamically migrates hot and cold data to hierarchical storage media, achieving high read and write efficiency. The end-to-end technology closed loop solves the three major industry shortcomings of response latency, anomaly detection failure, and storage bloat, truly realizing the optimization of predictive maintenance and real-time decision-making capabilities.

[0053] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data processing method for an industrial database, characterized in that, Includes the following steps: S1. Build a unified stream and batch processing engine to perform joint calculations on real-time data streams and historical data; S2. Adopt a physical partitioned storage model and dynamically allocate storage structure according to entity identifier; S3. Perform real-time queries based on incremental graph traversal technology and dynamically prune branches during the traversal process; S4. Real-time detection of entity behavior anomalies using a spatiotemporal fusion model; S5. Compress event stream data based on causal rules.

2. The data processing method for an industrial database according to claim 1, characterized in that: The implementation method of the dynamic pruning is as follows: Calculate the current path during traversal. Confidence level: ; When the confidence level is lower than the dynamic threshold The traversal of this path will terminate at that time.

3. The data processing method for an industrial database according to claim 2, characterized in that: The dynamic threshold is calculated as follows: A sliding window statistical mechanism is used: ; The time window Dynamically configure based on business scenarios.

4. The data processing method for an industrial database according to claim 1, characterized in that: The spatiotemporal fusion anomaly detection includes: (a) For each entity Establish a state prediction model: ; (b) Calculate joint loss by fusing spatial topology information: in: The feature vectors of the graph structure (dimension ≥ 16).

5. The data processing method for an industrial database according to claim 4, characterized in that: The feature vector is generated through a graph attention mechanism: An improved attention weight calculation method is used: .

6. The data processing method for an industrial database according to claim 1, characterized in that: The causal rule compression includes: a) Calculation of the strength of causal relationship of the event: ; in: It is an event lead to The conditional probability; b) Only when the intensity value exceeds the threshold Compression is triggered when the value is within the preset range of 0.3-0.

8.

7. The data processing method for an industrial database according to claim 6, characterized in that: The specific method for calculating the conditional probability is as follows: Use an event counting and statistics mechanism: .

8. The data processing method for an industrial database according to claim 1, characterized in that: The historical data storage uses differential encoding: Perform data compression based on time series: .

9. The data processing method for an industrial database according to claim 8, characterized in that: Generate a metadata block structure corresponding to the compressed data block, wherein the metadata block structure is as follows: Quad-tuple packaging: ; in: It is the SHA-256 hash value of the data block.

10. The data processing method for an industrial database according to claim 9, characterized in that: The hash value is calculated as follows: .

Images