Data processing method and device based on streaming processing engine, computer equipment, medium and product

Abstract

The invention relates to a data processing method and device based on a streaming processing engine, computer equipment, a medium and a product. The method comprises the following steps: collecting power equipment data through network equipment, packaging the power equipment data into a standard data packet, transmitting the standard data packet to a message queue, distributing the power equipment data in the standard data packet to a stream processing engine through the message queue, performing multi-level verification on the distributed power equipment data through a stream processing engine to obtain abnormal data in the power equipment data, correcting the abnormal data through the stream processing engine to obtain corrected data, storing qualified data into a time sequence database, and writing target abnormal data into a cache database. By adopting the method, the data processing efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment, medium and product based on a streaming engine. Background Technology

[0002] Data processing is a core component of modern information systems. With the continuous development of data-intensive applications, traditional batch processing methods are struggling to meet real-time demands, particularly in high-frequency data acquisition and processing scenarios, where their shortcomings are becoming increasingly apparent. Traditional data acquisition relies on fixed patterns, making it difficult to adapt to diverse data interfaces and lacking the ability to flexibly adjust the acquisition frequency. This results in low data acquisition efficiency and an inability to meet the needs of high-frequency data scenarios. Summary of the Invention

[0003] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, medium, and product based on a streaming processing engine that can improve data processing efficiency in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a data processing method based on a streaming engine, including:

[0005] Data from power equipment is collected through network devices and encapsulated into standard data packets; the network devices include distributed sensor networks and application programming interfaces (APIs).

[0006] Standard data packets are transmitted to a message queue, and the power equipment data in the standard data packets is distributed to the stream processing engine through the message queue;

[0007] The distributed power equipment data is subjected to multi-level verification through a stream processing engine to identify abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification and correlation verification.

[0008] The abnormal data is corrected by the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target abnormal data.

[0009] Qualified data is stored in a time-series database, while target abnormal data is written to a cache database. The cache database uses containers to isolate the computational resources of the target abnormal data.

[0010] In one embodiment, the step of collecting power equipment data via a network device includes:

[0011] Using a preset duration as a window, and following a preset sliding step size, the power equipment data is sampled via a sliding window through network devices; the sliding step size is determined according to data acquisition requirements; the power equipment data includes equipment indication data and equipment demand data.

[0012] In one embodiment, the step of performing multi-level verification on the distributed power equipment data using a stream processing engine includes:

[0013] The streaming engine verifies the timestamp compliance and field integrity of the distributed power equipment data.

[0014] The business logic rules are executed through the stream processing engine to perform logical verification on the power equipment data. The business logic rules include the equipment value increment verification rules and the equipment demand threshold determination rules.

[0015] A dynamic baseline model is built based on historical equipment data using a stream processing engine. The power equipment data is then correlated and verified based on the comparison results between the power equipment data and the data in the dynamic baseline model.

[0016] In one embodiment, the step of correcting anomalous data using a stream processing engine to obtain corrected data includes:

[0017] Determine the anomaly type of the abnormal data;

[0018] For anomalous data with sensor drift error as the anomaly type, the Kalman filter algorithm is used to correct the anomalous data;

[0019] For abnormal data where the main acquisition channel fails, the corresponding power equipment data is reacquired through the backup data channel.

[0020] In one embodiment, the method further includes:

[0021] The standard data packet is divided into first device data and second device data; the first device data consists of power equipment data that meets the verification conditions; the second device data consists of power equipment data that does not meet the verification conditions; the verification conditions include verification complexity conditions and verification data volume conditions.

[0022] The first device data is sent to the edge node of the stream processing engine via a message queue, and the second device data is sent to the central node of the stream processing engine via a message queue.

[0023] The distributed power equipment data is verified at multiple levels through both edge nodes and central nodes.

[0024] In one embodiment, the method further includes:

[0025] Generate a first checksum based on the standard data packet, and transmit the first checksum and the standard data packet to the message queue.

[0026] Obtain the second checksum generated by the message queue based on the transmitted standard data packets;

[0027] The integrity of the power equipment data in the standard data packet is verified based on the first check code and the second check code.

[0028] Secondly, this application also provides a data processing apparatus based on a streaming engine, comprising:

[0029] The data acquisition module is used to collect power equipment data through network devices and encapsulate the power equipment data into standard data packets; the network devices include a distributed sensor network and an application programming interface;

[0030] The data distribution module is used to transmit standard data packets to the message queue, and then distribute the power equipment data in the standard data packets to the stream processing engine through the message queue;

[0031] The multi-level verification module is used to perform multi-level verification on the distributed power equipment data through the stream processing engine to obtain abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification and correlation verification.

[0032] The anomaly correction module is used to correct anomalous data through the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target anomalous data.

[0033] The data storage module is used to store qualified data in the time-series database and write target abnormal data to the cache database; the cache database uses containers to isolate the computing resources of the target abnormal data.

[0034] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of the first aspects.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method steps of any one of the first aspects.

[0036] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method steps of any one of the first aspects.

[0037] The aforementioned data processing method, apparatus, computer equipment, medium, and product based on a streaming engine collects power equipment data through network devices, encapsulates the power equipment data into standard data packets, transmits the standard data packets to a message queue, distributes the power equipment data in the standard data packets to the streaming engine through the message queue, performs multi-level verification on the distributed power equipment data through the streaming engine to obtain abnormal data in the power equipment data, corrects the abnormal data through the streaming engine to obtain corrected data, stores qualified data in a time-series database, and writes target abnormal data to a cache database. This can improve data verification efficiency and data accuracy, achieve long-term complete storage of high-frequency time-series data, and ensure fast querying of abnormal data, thereby improving resource utilization. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is an application environment diagram of a data processing method based on a streaming engine in one embodiment;

[0040] Figure 2 This is a flowchart illustrating a data processing method based on a streaming engine in one embodiment;

[0041] Figure 3 This is a flowchart illustrating a data processing method based on a streaming engine in another embodiment;

[0042] Figure 4 This is a structural block diagram of a data processing device based on a streaming processing engine in one embodiment;

[0043] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] The data processing method based on a streaming engine provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 is used to collect power equipment data through network devices, encapsulate the power equipment data into standard data packets, transmit the standard data packets to a message queue, distribute the power equipment data in the standard data packets to a stream processing engine through the message queue, perform multi-level verification on the distributed power equipment data through the stream processing engine, obtain abnormal data in the power equipment data, correct the abnormal data through the stream processing engine, obtain corrected data, store qualified data in a time-series database, and write target abnormal data to a cache database. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Headset devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0046] In one exemplary embodiment, such as Figure 2 As shown, a data processing method based on a streaming processing engine is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 210. Wherein:

[0047] S202: Collect power equipment data through network devices and encapsulate the power equipment data into standard data packets; network devices include distributed sensor networks and application programming interfaces.

[0048] Optionally, the distributed sensor network consists of multiple sensor nodes deployed on different power equipment (such as transformers and switchgear), capable of collecting physical quantities of the equipment (such as current, voltage, and temperature) in real time, and supporting parallel access of multi-source data. The application programming interface (API) provides an interface for data interaction between different systems, supporting protocols such as HTTP / REST, MQTT, and Modbus, and is used to access existing data interfaces in the power system to achieve data interoperability. Real-time physical quantity data of power equipment is collected through the distributed sensor network, employing a sliding window mechanism to achieve high-frequency sampling at the 1-second level. The sampling frequency can be dynamically adjusted according to changes in power load (such as peak hours), simultaneously acquiring key power indicators such as readings and demand. The sliding window mechanism is a data sampling strategy that continuously collects power equipment data within a fixed time window (such as 1 second), achieving high-frequency, continuous data sampling, and the window size can be dynamically adjusted to change the sampling frequency. All collected data is encapsulated into standard data packets according to specified protocol standards to ensure data format consistency in subsequent transmission and processing, avoiding processing delays caused by format incompatibility.

[0049] S204: Transmit standard data packets to the message queue, and distribute the power equipment data in the standard data packets to the stream processing engine through the message queue.

[0050] Optionally, Kafka, a high-throughput distributed message middleware, enables data temporary storage, buffering, and distribution, supports high-concurrency data access, avoids data congestion, and ensures data transmission stability. Flink, a real-time stream processing framework, features low latency and high throughput, enabling real-time computation and processing of continuously generated data streams, supporting millisecond-level response times. It is the core carrier for real-time verification and correction in this solution. Kafka's data organization method allocates different types of power equipment data (such as transformer data and switchgear data) to different topic partitions, achieving data classification storage and targeted distribution, ensuring that the stream processing engine can accurately acquire target data.

[0051] Optionally, standard data packets are first transmitted to a Kafka message queue. Kafka's high concurrency capabilities are leveraged to handle the massive amounts of data generated by high-frequency data collection. A temporary buffer prevents overload caused by directly transmitting data to the stream processing engine. Kafka uses a Topic partitioning mechanism to distribute different types of power equipment data to the Flink stream processing engine, ensuring that Flink can quickly acquire the corresponding data type. This provides low-latency data input for subsequent real-time verification and correction, achieving seamless integration of data collection and processing.

[0052] S206: Perform multi-level verification on the distributed power equipment data through the stream processing engine to obtain abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification and correlation verification.

[0053] Optionally, format verification refers to verifying the basic legality of power equipment data, including timestamp compliance (whether the data time is synchronized with the system time) and field completeness (whether key fields such as indicated values ​​and demand are missing). This is a fundamental step in data verification. Based on a real-time verification engine, relying on the Flink stream processing engine, millisecond-level multi-level verification is implemented, including format verification, logical verification, and correlation verification. Format verification verifies whether the data timestamp is compliant (e.g., avoiding invalid historical data) and whether the fields are complete. If an anomaly is found, a re-collection mechanism is triggered to re-collect the equipment data, ensuring the basic legality of the data. Logical verification loads power business rules (e.g., power indicators should increase over time, and demand should not exceed the equipment's rated capacity) through a rules engine, performs logical judgments on the data, and verifies whether the data packet has been tampered with using an XOR check algorithm, filtering out logical anomalies and tampered data. Correlation verification calls a dynamic baseline model to compare real-time data with the baseline (e.g., whether the load exceeds the normal fluctuation range), detecting data mutations. Finally, data that does not meet any of the verification criteria is marked as abnormal data and enters the subsequent correction stage.

[0054] S208: Correct the abnormal data through the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target abnormal data.

[0055] Optionally, based on the dynamic correction module, abnormal data correction is performed in the Flink stream processing engine to ensure the continuity of verification and correction. For sensor drift errors (such as current sensor accuracy deviation), a Kalman filter algorithm is used to dynamically correct abnormal data, keeping the error within a low range and ensuring the data meets the accuracy requirements of power metering. When the main acquisition channel fails (e.g., due to sensor damage), the system automatically switches to the backup channel to acquire data and marks the data with a replacement identifier. This data, after verification, becomes the corrected and qualified data, ensuring the continuity of data acquisition. Simultaneously, through online machine learning, the verification threshold and correction model are dynamically optimized based on real-time power data. Abnormal data that cannot be corrected using the above methods is marked as uncorrected target abnormal data and enters a special storage stage.

[0056] S210: Store qualified data in the time-series database and write target abnormal data to the cache database; the cache database uses containers to isolate the computational resources of the target abnormal data.

[0057] Optionally, the corrected, qualified data is written to a time-series database. Leveraging its high throughput, high compression ratio, and efficient time-dimensional query capabilities, this ensures the long-term integrity of high-frequency power data storage and meets subsequent power metering and data analysis needs. Uncorrected target anomaly data is written to a cache database. Utilizing the cache database's millisecond-level query capabilities, maintenance personnel can easily access anomaly data instantly and quickly locate equipment faults. The cache database is encapsulated using Docker container technology, isolating its computing resources from high-priority tasks such as Flink real-time verification and time-series database storage, preventing the storage and querying of target anomaly data from consuming core task resources. The time-series database (InfluxDB) is specifically designed for high-frequency time-series data (such as power equipment data collected every second). It supports efficient time-dimensional queries (such as querying the load change of a device over a specific hour) and has a high compression ratio, making it suitable for long-term storage of large amounts of qualified data. The cache database (Redis) is a high-performance key-value pair cache database with millisecond-level query response speed, suitable for storing data that needs to be accessed immediately (such as target anomaly data), avoiding the consumption of time-series database resources and improving query efficiency. Docker containers are a lightweight virtualization technology that encapsulates applications and their dependencies into independent containers, isolating computing resources and preventing resource contention between different tasks (such as abnormal data storage and real-time verification), thus ensuring the stable operation of high-priority tasks. By using Docker containers, the computing resources (CPU, memory) of the cache database (which stores the target abnormal data) are isolated from other core tasks (such as real-time verification and qualified data storage), ensuring that the processing of the target abnormal data does not affect the core functions of the system.

[0058] In the aforementioned data processing method based on a streaming engine, power equipment data is collected through network devices, encapsulated into standard data packets, and transmitted to a message queue. The message queue then distributes the power equipment data from the standard data packets to the streaming engine. The streaming engine performs multi-level verification on the distributed power equipment data to identify abnormal data. The streaming engine then corrects the abnormal data to obtain corrected data. Qualified data is stored in a time-series database, while target abnormal data is written to a cache database. This approach improves data verification efficiency and accuracy, ensuring long-term complete storage of high-frequency time-series data while guaranteeing rapid retrieval of abnormal data, thereby improving resource utilization.

[0059] In an exemplary embodiment, the step of collecting power equipment data through a network device includes: sampling the power equipment data through a sliding window with a preset duration as a window and a preset sliding step size; the sliding step size is determined according to the data collection requirements; the power equipment data includes equipment indication data and equipment demand data.

[0060] Optionally, network devices include distributed sensor networks (deployed at power equipment sites to collect physical quantity data such as current, voltage, and temperature) and application programming interfaces (APIs), which form the hardware and software interface foundation for power equipment data acquisition. The fixed time period set in the sliding window sampling, where the preset duration is typically 1 second (adjustable according to power scenario requirements, such as shortening to 0.5 seconds during peak load), is used to define the data time range for a single sampling. The sliding step size refers to the interval at which the window moves along the time axis; it is a core parameter for frequency adjustment and can be determined according to data acquisition requirements.

[0061] Optionally, the preset duration and sliding step size can be determined based on the power scenario requirements (such as daily monitoring and peak load warning). Through the adapter pattern, the distributed sensor network and the API interface respond synchronously to the sliding window sampling commands. The distributed sensors collect device readings (such as real-time current readings), while the API interface retrieves device demand data (such as historical maximum load values) from the power system. This ensures that both types of core data are collected within the same window period, avoiding analysis errors caused by data time differences. The raw data acquired by the sliding window sampling is encapsulated into standard data packets according to preset standards, laying the foundation for subsequent transmission to the Kafka message queue and entry into the Flink stream processing engine. This achieves seamless integration of sampling, encapsulation, and processing, avoiding data interruption.

[0062] In this embodiment, by using a preset time period as a window and a preset sliding step size, the network device performs sliding window sampling on the power equipment data, which enables high-frequency acquisition and dynamic adaptation, ensuring applicability to power scenarios, without the need for additional data conversion steps, thereby improving overall processing efficiency.

[0063] In an exemplary embodiment, the step of performing multi-level verification on distributed power equipment data through a stream processing engine includes: verifying the timestamp compliance and field integrity of the distributed power equipment data through the stream processing engine; executing business logic rules through the stream processing engine to perform logical verification on the power equipment data; the business logic rules include equipment value increment verification rules and equipment demand threshold determination rules; and constructing a dynamic baseline model based on historical equipment data through the stream processing engine, and performing correlation verification on the power equipment data based on the comparison results between the power equipment data and the data in the dynamic baseline model.

[0064] Optionally, the stream processing engine, specifically the Flink stream processing engine, is the core carrier for achieving millisecond-level data processing and verification. It features low latency and high throughput, enabling real-time computation and rule execution on continuously distributed power equipment data streams, supporting the efficient implementation of multi-level verification. Multi-level verification of distributed power equipment data includes format verification, logical verification, and correlation verification. Format verification includes timestamp compliance verification and field integrity verification. Timestamp compliance verification verifies the legality of the power equipment data timestamps, including correct timestamp format and time synchronization, and is one of the core dimensions of format verification. Field integrity verification verifies whether the power equipment data contains key business fields, checking for missing values, demand, equipment number, collection time, and other critical fields to prevent subsequent processing failures due to missing fields.

[0065] Optionally, logical verification refers to verifying power equipment data by loading and setting business logic rules through a rule engine. These business logic rules include rules for verifying the incrementality of equipment indications and rules for determining equipment demand thresholds. Correlation verification refers to using a model built on historical data. By analyzing historical operating data of power equipment, a baseline range for normal equipment operation is generated. By comparing real-time power equipment data with the normal data range in the baseline model, data mutations and anomalies are detected, overcoming the limitations of format verification and logical verification, which only focus on single data points or static rules.

[0066] In this embodiment, the distributed power equipment data is verified at multiple levels through a stream processing engine, which can improve the comprehensiveness of the verification, reduce the omission and misjudgment of abnormal data, and thus improve the reliability of the power equipment data.

[0067] In an exemplary embodiment, the step of correcting abnormal data using a stream processing engine to obtain corrected data includes: determining the abnormal type of the abnormal data; correcting the abnormal data with a Kalman filter algorithm for abnormal data whose abnormal type is sensor drift error; and re-acquiring the corresponding power equipment data through a backup data channel for abnormal data whose abnormal type is the failure of the main acquisition channel.

[0068] Optionally, anomalous data refers to data that has been determined to be non-compliant through multi-level verification. The core anomalous data types include sensor drift error (accuracy deviations caused by long-term sensor use or environmental influences, such as persistently high / low readings from current sensors) and main acquisition channel failure (failure of the primary acquisition path, such as distributed sensor failure or main API interface interruption, resulting in the inability to acquire valid data). For anomalous data with sensor drift error, Flink loads Kalman filter algorithm parameters, combines the inherent noise characteristics of power equipment sensors, and presets state prediction equations and observation update equations. Using the anomalous data caused by sensor drift as input, the Kalman filter calculates and corrects the error, outputting corrected, qualified data. The Kalman filter algorithm, through a prediction-update loop, dynamically corrects drift error by combining the noise characteristics of the sensor-acquired data, reducing data deviation and making the corrected data closer to the true value, with an error correction accuracy of up to 0.05%.

[0069] Furthermore, for abnormal data where the primary acquisition channel fails, Flink sends a channel switching command to the multi-source data acquisition module through a preset interface, stopping the use of the primary acquisition channel (such as faulty sensors or interrupted APIs) and starting a backup data channel. The backup channel (such as a backup sensor or backup API) re-acquires the indicated and demand data of the corresponding power equipment at the same acquisition frequency as the primary channel (e.g., 1 second), ensuring that the data time dimension is consistent with the acquisition cycle before the primary channel failure and avoiding data gaps. Valid re-acquired data is marked with a backup channel replacement identifier, and information such as the switching time and channel type is recorded. This data is then sent to Flink for rapid format verification; data that passes verification is considered corrected and qualified. If the backup channel also fails to acquire valid data, the abnormal data is determined to be uncorrected target abnormal data and enters the subsequent cache storage stage. The primary acquisition channel is the priority path for daily acquisition of power equipment data, while the backup data channel is a redundant path when the primary channel fails, ensuring uninterrupted data acquisition, and the replaced data is marked with a replacement identifier.

[0070] In this embodiment, by determining the anomaly type of abnormal data, for abnormal data of sensor drift error type, the abnormal data is corrected by Kalman filtering algorithm; for abnormal data of main acquisition channel failure type, the corresponding power equipment data is re-acquired by backup data channel. This can accurately correct sensor error, improve data accuracy, accurately adapt to high frequency scenarios, reduce data invalidity, and thus further improve overall processing efficiency.

[0071] In an exemplary embodiment, the method further includes: dividing a standard data packet into first device data and second device data; the first device data is power equipment data that meets the verification conditions; the second device data is power equipment data that does not meet the verification conditions; the verification conditions include verification complexity conditions and verification data volume conditions; sending the first device data to the edge node of the stream processing engine through a message queue, and sending the second device data to the central node of the stream processing engine through a message queue; and performing multi-level verification on the distributed power equipment data through the edge node and the central node respectively.

[0072] Optionally, the first device data and the second device data are two categories of data divided according to verification conditions. The first device data consists of power equipment data that meets the verification conditions (i.e., low complexity and small data volume), suitable for rapid processing at edge nodes. The second device data consists of power equipment data that does not meet the verification conditions (i.e., high complexity and large data volume), requiring in-depth processing at the central node. This division aims to achieve hierarchical verification and resource optimization. The verification conditions include verification complexity conditions and verification data volume conditions. Verification complexity conditions refer to the difficulty of the verification logic; for example, format verification (timestamp / field integrity) alone is considered low complexity, while correlation verification is considered high complexity. Verification data volume conditions refer to the scale of a single batch of data; for example, 1-second sampling data from a single device is considered small data volume, while hourly aggregated data from multiple devices is considered large data volume.

[0073] Optionally, the stream processing engine includes edge nodes and central nodes. Edge nodes are deployed at power equipment sites or near the data acquisition end. While their computing resources are relatively limited, they offer short data transmission distances and low latency, and are responsible for processing data from the first device. Central nodes are deployed in the cloud or data centers, possessing ample computing resources (CPU / memory) and historical data storage capabilities. They are responsible for processing data from the second device. Together, they form a distributed verification architecture. The message queue uses Topic partitioning or tagging mechanisms to route data from the first device to the corresponding Kafka partition on the edge node, and data from the second device to the partition on the central node, ensuring accurate and low-latency data distribution.

[0074] In this embodiment, by dividing the standard data packet into first device data and second device data, the first device data is sent to the edge node of the stream processing engine through a message queue, and the second device data is sent to the central node of the stream processing engine through a message queue. Multi-level verification of the distributed power equipment data is performed through the edge node and the central node, which can improve the verification efficiency, avoid congestion caused by node load, accurately optimize resource allocation, and reduce operating costs.

[0075] In an exemplary embodiment, the method further includes: generating a first checksum based on a standard data packet; transmitting the first checksum and the standard data packet to a message queue; obtaining a second checksum generated by the message queue based on the transmitted standard data packet; and performing integrity verification on the power equipment data in the standard data packet based on the first checksum and the second checksum.

[0076] Optionally, the first checksum is an encoding generated by the data acquisition layer using a specific verification algorithm after the standard data packet is generated but before it is transmitted to the message queue. It characterizes the integrity of the original standard data packet and serves as the benchmark for subsequent comparison and verification. The second checksum is an encoding recalculated by the message queue after receiving the standard data packet, based on the same verification algorithm as the first checksum. It is used to compare with the first checksum to determine whether the data is complete during transmission. By comparing the consistency of the first and second checksums, a verification method is used to determine whether the standard data packet has been lost, tampered with, or damaged during transmission from the data acquisition layer to the message queue. If they are completely consistent, it is determined that the standard data packet has not been tampered with or lost during transmission, and the integrity verification passes. Kafka then distributes the standard data packet to the Flink stream processing engine according to the normal process. If they are inconsistent, it is determined that the data packet has an integrity problem (such as byte loss or malicious tampering during transmission). Kafka sends a re-acquisition command to the data acquisition layer, requesting the acquisition module to regenerate and transmit the standard data packet for the device until the integrity verification passes.

[0077] In this embodiment, by generating a first checksum based on a standard data packet, transmitting the first checksum and the standard data packet to a message queue, obtaining a second checksum generated by the message queue based on the transmitted standard data packet, and performing integrity verification on the power equipment data in the standard data packet based on the first checksum and the second checksum, the reliability of the power equipment data can be guaranteed and the data transmission efficiency can be improved.

[0078] In one exemplary embodiment, such as Figure 3 As shown, a data processing method based on a streaming processing engine is provided, which includes the following steps:

[0079] (1) Sliding window acquisition: The power equipment data is sampled by a network device with a preset time period as a window and a preset sliding step size. The sliding step size is determined according to the data acquisition requirements. The power equipment data includes equipment indication data and equipment demand data.

[0080] (2) Standard data encapsulation: Power equipment data is encapsulated into standard data packets; network devices include distributed sensor networks and application programming interfaces. The standard data packets are transmitted to a message queue, and the power equipment data in the standard data packets is distributed to the stream processing engine through the message queue.

[0081] (3) Transmission integrity verification: Generate a first check code based on the standard data packet, and transmit the first check code and the standard data packet to the message queue; obtain the second check code generated by the message queue based on the transmitted standard data packet; perform integrity verification on the power equipment data in the standard data packet based on the first check code and the second check code.

[0082] (4) Node-based distribution: The standard data packet is divided into first device data and second device data; the first device data is power equipment data that meets the verification conditions; the second device data is power equipment data that does not meet the verification conditions; the verification conditions include verification complexity conditions and verification data volume conditions; the first device data is distributed to the edge nodes of the stream processing engine through the message queue, and the second device data is distributed to the central node of the stream processing engine through the message queue; the distributed power equipment data is verified at multiple levels through the edge nodes and the central node respectively.

[0083] (5) Multi-level verification: The time stamp compliance and field integrity of the distributed power equipment data are verified by the stream processing engine; the business logic rules are executed by the stream processing engine to perform logical verification on the power equipment data; the business logic rules include the equipment value increment verification rules and the equipment demand threshold judgment rules; the dynamic baseline model is built by the stream processing engine based on historical equipment data, and the power equipment data is correlated and verified according to the comparison results between the power equipment data and the data in the dynamic baseline model.

[0084] (6) Abnormal Data Correction: Obtain abnormal data from power equipment data; multi-level verification includes format verification, logical verification, and correlation verification. Determine the abnormal type of the abnormal data; for abnormal data of sensor drift error type, correct the abnormal data using the Kalman filter algorithm; for abnormal data of main acquisition channel failure type, re-acquire the corresponding power equipment data through the backup data channel. Among them, the corrected data includes corrected qualified data and uncorrected target abnormal data.

[0085] (7) Distributed storage: Qualified data is stored in the time-series database, and target abnormal data is written to the cache database; the cache database uses containers to isolate the computing resources of the target abnormal data.

[0086] In this embodiment, power equipment data is collected through network devices, encapsulated into standard data packets, and transmitted to a message queue. The message queue then distributes the power equipment data from the standard data packets to a stream processing engine. The stream processing engine performs multi-level verification on the distributed power equipment data to identify abnormal data. The stream processing engine then corrects the abnormal data to obtain corrected data. Qualified data is stored in a time-series database, while target abnormal data is written to a cache database. This process improves data verification efficiency and accuracy, ensuring long-term complete storage of high-frequency time-series data while guaranteeing rapid retrieval of abnormal data, thereby improving resource utilization.

[0087] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0088] Based on the same inventive concept, this application also provides a streaming engine-based data processing apparatus for implementing the above-described streaming engine-based data processing method. The solution provided by this apparatus is similar to the implementation described in the above-described method. Therefore, the specific limitations in one or more streaming engine-based data processing apparatus embodiments provided below can be found in the limitations of the streaming engine-based data processing method described above, and will not be repeated here.

[0089] In one exemplary embodiment, such as Figure 4 As shown, a data processing device based on a streaming engine is provided, comprising: a data acquisition module 10, a data distribution module 20, a multi-level verification module 30, an anomaly correction module 40, and a data storage module 50, wherein:

[0090] The data acquisition module 10 is used to acquire power equipment data through network devices and encapsulate the power equipment data into standard data packets; the network devices include a distributed sensor network and an application programming interface.

[0091] The data distribution module 20 is used to transmit standard data packets to a message queue, and then distribute the power equipment data in the standard data packets to the stream processing engine through the message queue.

[0092] The multi-level verification module 30 is used to perform multi-level verification on the distributed power equipment data through the stream processing engine to obtain abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification and correlation verification.

[0093] The anomaly correction module 40 is used to correct the abnormal data through the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target abnormal data.

[0094] The data storage module 50 is used to store qualified data in the time-series database and write target abnormal data to the cache database; the cache database uses containers to isolate the computing resources of the target abnormal data.

[0095] In an exemplary embodiment, the data acquisition module 10 is further configured to perform sliding window sampling of power equipment data through a network device, with a preset duration as a window and a preset sliding step size; the sliding step size is determined according to the data acquisition requirements; the power equipment data includes equipment indication data and equipment demand data.

[0096] In an exemplary embodiment, the multi-level verification module 30 is further configured to verify the timestamp compliance and field integrity of the distributed power equipment data through a stream processing engine; to perform logical verification on the power equipment data by executing business logic rules through the stream processing engine; the business logic rules include equipment value increment verification rules and equipment demand threshold determination rules; and to construct a dynamic baseline model based on historical equipment data through the stream processing engine, and to perform correlation verification on the power equipment data based on the comparison results between the power equipment data and the data in the dynamic baseline model.

[0097] In an exemplary embodiment, the anomaly correction module 40 is further configured to determine the anomaly type of the abnormal data; for abnormal data whose anomaly type is sensor drift error, the abnormal data is corrected by a Kalman filter algorithm; for abnormal data whose anomaly type is failure of the main acquisition channel, the corresponding power equipment data is reacquired through a backup data channel.

[0098] In an exemplary embodiment, the data distribution module 20 is further configured to divide the standard data packet into first device data and second device data; the first device data is power equipment data that meets the verification conditions; the second device data is power equipment data that does not meet the verification conditions; the verification conditions include verification complexity conditions and verification data volume conditions; the first device data is sent to the edge node of the stream processing engine through a message queue, and the second device data is sent to the central node of the stream processing engine through a message queue; the distributed power equipment data is verified at multiple levels through the edge node and the central node respectively.

[0099] In an exemplary embodiment, the data distribution module 20 is further configured to generate a first checksum based on a standard data packet, transmit the first checksum and the standard data packet to a message queue; obtain a second checksum generated by the message queue based on the transmitted standard data packet; and perform integrity verification on the power equipment data in the standard data packet based on the first checksum and the second checksum.

[0100] The modules in the aforementioned data processing device based on a streaming engine can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0101] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data processing method based on a streaming engine. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0102] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0103] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: collecting power equipment data through a network device and encapsulating the power equipment data into standard data packets; the network device includes a distributed sensor network and an application programming interface; transmitting the standard data packets to a message queue, and distributing the power equipment data in the standard data packets to a stream processing engine through the message queue; performing multi-level verification on the distributed power equipment data through the stream processing engine to obtain abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification, and correlation verification; correcting the abnormal data through the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target abnormal data; storing the qualified data in a time-series database and writing the target abnormal data to a cache database; the cache database uses containers to isolate the computational resources of the target abnormal data.

[0104] In one embodiment, the process of a processor executing a computer program involving the collection of power equipment data via a network device includes: sampling the power equipment data via a sliding window using a preset duration as a window and a preset sliding step size; the sliding step size is determined according to the data collection requirements; the power equipment data includes equipment indication data and equipment demand data.

[0105] In one embodiment, when the processor executes a computer program, it involves performing multi-level verification of the distributed power equipment data through a stream processing engine. This includes: verifying the timestamp compliance and field integrity of the distributed power equipment data through the stream processing engine; executing business logic rules through the stream processing engine to perform logical verification on the power equipment data; the business logic rules include equipment value increment verification rules and equipment demand threshold determination rules; and constructing a dynamic baseline model based on historical equipment data through the stream processing engine, and performing correlation verification on the power equipment data based on the comparison results between the power equipment data and the data in the dynamic baseline model.

[0106] In one embodiment, when the processor executes a computer program, it involves correcting abnormal data through a stream processing engine to obtain corrected data, including: determining the abnormal type of the abnormal data; correcting the abnormal data using a Kalman filter algorithm for abnormal data whose abnormal type is sensor drift error; and re-acquiring the corresponding power equipment data through a backup data channel for abnormal data whose abnormal type is the failure of the main acquisition channel.

[0107] In one embodiment, when the processor executes the computer program, it further performs the following steps: dividing a standard data packet into first device data and second device data; the first device data is power device data that meets the verification conditions; the second device data is power device data that does not meet the verification conditions; the verification conditions include verification complexity conditions and verification data volume conditions; sending the first device data to the edge node of the stream processing engine through a message queue, and sending the second device data to the central node of the stream processing engine through a message queue; performing multi-level verification on the distributed power device data through the edge node and the central node respectively.

[0108] In one embodiment, when the processor executes the computer program, it further performs the following steps: generating a first checksum based on a standard data packet, transmitting the first checksum and the standard data packet to a message queue; obtaining a second checksum generated by the message queue based on the transmitted standard data packet; and performing integrity verification on the power equipment data in the standard data packet based on the first checksum and the second checksum.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: collecting power equipment data through a network device and encapsulating the power equipment data into standard data packets; the network device includes a distributed sensor network and an application programming interface; transmitting the standard data packets to a message queue, and distributing the power equipment data in the standard data packets to a stream processing engine through the message queue; performing multi-level verification on the distributed power equipment data through the stream processing engine to obtain abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification, and correlation verification; correcting the abnormal data through the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target abnormal data; storing the qualified data in a time-series database and writing the target abnormal data to a cache database; the cache database uses containers to isolate the computational resources of the target abnormal data.

[0110] In one embodiment, the computer program, when executed by a processor, involves collecting power equipment data via a network device, including: sampling power equipment data via a sliding window with a preset duration as a window and a preset sliding step size; the sliding step size is determined according to data collection requirements; the power equipment data includes equipment indication data and equipment demand data.

[0111] In one embodiment, when the computer program is executed by the processor, it involves multi-level verification of the distributed power equipment data through a stream processing engine, including: verifying the timestamp compliance and field integrity of the distributed power equipment data through the stream processing engine; executing business logic rules through the stream processing engine to perform logical verification on the power equipment data; the business logic rules include equipment value increment verification rules and equipment demand threshold determination rules; and constructing a dynamic baseline model based on historical equipment data through the stream processing engine, and performing correlation verification on the power equipment data based on the comparison results between the power equipment data and the data in the dynamic baseline model.

[0112] In one embodiment, when the computer program is executed by the processor, it involves correcting abnormal data through a stream processing engine to obtain corrected data, including: determining the abnormal type of the abnormal data; correcting the abnormal data using a Kalman filter algorithm for abnormal data whose abnormal type is sensor drift error; and re-acquiring the corresponding power equipment data through a backup data channel for abnormal data whose abnormal type is the failure of the main acquisition channel.

[0113] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: dividing a standard data packet into first device data and second device data; the first device data is power device data that meets the verification conditions; the second device data is power device data that does not meet the verification conditions; the verification conditions include verification complexity conditions and verification data volume conditions; sending the first device data to the edge nodes of the stream processing engine through a message queue, and sending the second device data to the central node of the stream processing engine through a message queue; performing multi-level verification on the distributed power device data through the edge nodes and the central node respectively.

[0114] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating a first checksum based on a standard data packet, transmitting the first checksum and the standard data packet to a message queue; obtaining a second checksum generated by the message queue based on the transmitted standard data packet; and performing integrity verification on the power equipment data in the standard data packet based on the first checksum and the second checksum.

[0115] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: collecting power equipment data through a network device and encapsulating the power equipment data into standard data packets; the network device includes a distributed sensor network and an application programming interface; transmitting the standard data packets to a message queue, and distributing the power equipment data in the standard data packets to a stream processing engine through the message queue; performing multi-level verification on the distributed power equipment data through the stream processing engine to obtain abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification, and correlation verification; correcting the abnormal data through the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target abnormal data; storing the qualified data in a time-series database and writing the target abnormal data to a cache database; the cache database uses containers to isolate the computational resources of the target abnormal data.

[0116] In one embodiment, the computer program, when executed by a processor, involves collecting power equipment data via a network device, including: sampling power equipment data via a sliding window with a preset duration as a window and a preset sliding step size; the sliding step size is determined according to data collection requirements; the power equipment data includes equipment indication data and equipment demand data.

[0117] In one embodiment, when the computer program is executed by the processor, it involves multi-level verification of the distributed power equipment data through a stream processing engine, including: verifying the timestamp compliance and field integrity of the distributed power equipment data through the stream processing engine; executing business logic rules through the stream processing engine to perform logical verification on the power equipment data; the business logic rules include equipment value increment verification rules and equipment demand threshold determination rules; and constructing a dynamic baseline model based on historical equipment data through the stream processing engine, and performing correlation verification on the power equipment data based on the comparison results between the power equipment data and the data in the dynamic baseline model.

[0118] In one embodiment, when the computer program is executed by the processor, it involves correcting abnormal data through a stream processing engine to obtain corrected data, including: determining the abnormal type of the abnormal data; correcting the abnormal data using a Kalman filter algorithm for abnormal data whose abnormal type is sensor drift error; and re-acquiring the corresponding power equipment data through a backup data channel for abnormal data whose abnormal type is the failure of the main acquisition channel.

[0119] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: dividing a standard data packet into first device data and second device data; the first device data is power device data that meets the verification conditions; the second device data is power device data that does not meet the verification conditions; the verification conditions include verification complexity conditions and verification data volume conditions; sending the first device data to the edge nodes of the stream processing engine through a message queue, and sending the second device data to the central node of the stream processing engine through a message queue; performing multi-level verification on the distributed power device data through the edge nodes and the central node respectively.

[0120] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating a first checksum based on a standard data packet, transmitting the first checksum and the standard data packet to a message queue; obtaining a second checksum generated by the message queue based on the transmitted standard data packet; and performing integrity verification on the power equipment data in the standard data packet based on the first checksum and the second checksum.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0123] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method based on a streaming engine, characterized in that, The method includes: Data from power equipment is collected via network devices and encapsulated into standard data packets; the network devices include a distributed sensor network and an application programming interface. The standard data packet is transmitted to a message queue, and the power equipment data in the standard data packet is distributed to the stream processing engine through the message queue; The streaming engine performs multi-level verification on the distributed power equipment data to identify abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification, and correlation verification. The abnormal data is corrected by the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target abnormal data. The qualified data is stored in a time-series database, and the target abnormal data is written to a cache database; the cache database uses containers to isolate the computational resources of the target abnormal data.

2. The method according to claim 1, characterized in that, The collection of power equipment data through network devices includes: Using a preset duration as a window, and following a preset sliding step size, the power equipment data is sampled via a network device through a sliding window; the sliding step size is determined according to the data acquisition requirements; the power equipment data includes equipment indication data and equipment demand data.

3. The method according to claim 1, characterized in that, The multi-level verification of the distributed power equipment data through the stream processing engine includes: The streaming engine verifies the timestamp compliance and field integrity of the distributed power equipment data. The stream processing engine executes business logic rules to perform logical verification on the power equipment data; the business logic rules include equipment value increment verification rules and equipment demand threshold determination rules. The streaming engine constructs a dynamic baseline model based on historical device data, and performs correlation verification on the power device data based on the comparison results between the power device data and the data in the dynamic baseline model.

4. The method according to claim 1, characterized in that, The step of correcting the abnormal data through the stream processing engine to obtain corrected data includes: Determine the anomaly type of the abnormal data; For the abnormal data whose anomaly type is sensor drift error, the abnormal data is corrected using a Kalman filter algorithm; For abnormal data of the aforementioned anomaly type, where the main acquisition channel fails, the corresponding power equipment data is reacquired through the backup data channel.

5. The method according to claim 1, characterized in that, The method further includes: The standard data packet is divided into first device data and second device data; the first device data consists of power equipment data that meets the verification conditions; the second device data consists of power equipment data that does not meet the verification conditions; the verification conditions include verification complexity conditions and verification data volume conditions. The first device data is sent to the edge node of the stream processing engine through the message queue, and the second device data is sent to the central node of the stream processing engine through the message queue. The distributed power equipment data is verified at multiple levels through the edge nodes and the central node, respectively.

6. The method according to claim 1, characterized in that, The method further includes: A first checksum is generated based on the standard data packet, and the first checksum and the standard data packet are transmitted to the message queue. Obtain the second checksum generated by the message queue based on the transmitted standard data packet; The integrity of the power equipment data in the standard data packet is verified based on the first check code and the second check code.

7. A data processing device based on a streaming engine, characterized in that, The device includes: The data acquisition module is used to collect power equipment data through network devices and encapsulate the power equipment data into standard data packets; the network devices include a distributed sensor network and an application programming interface. The data distribution module is used to transmit the standard data packet to the message queue, and distribute the power equipment data in the standard data packet to the stream processing engine through the message queue; A multi-level verification module is used to perform multi-level verification on the distributed power equipment data through the stream processing engine to obtain abnormal data in the power equipment data; the multi-level verification includes format verification, logical verification and correlation verification; An anomaly correction module is used to correct the abnormal data through the stream processing engine to obtain corrected data; the corrected data includes corrected qualified data and uncorrected target abnormal data. The data storage module is used to store the qualified data in a time-series database and write the target abnormal data to a cache database; the cache database uses containers to isolate the computing resources of the target abnormal data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images