Multi-source heterogeneous data integration method and system for smart energy management

Abstract

The invention belongs to the technical field of data management, and discloses a smart energy management-oriented multi-source heterogeneous data integration method and system. The method comprises the following steps: collecting multi-source heterogeneous data through a gateway, and carrying out adaptive protocol adaptation and standardization processing to obtain a standardized data set with a label; deploying a dynamic sensing message queue, splitting a data processing step into independent processing units, and processing the standardized data set to obtain a to-be-processed data stream; obtaining and carrying out dynamic routing and load balancing scheduling on a to-be-processed data flow based on multi-dimensional sensing data to obtain an optimized processing path; obtaining fusion data processed by the optimization processing path, performing unified storage, and performing fault isolation and monitoring feedback through a unified gateway; according to the invention, stable, efficient and continuous operation of the smart energy data processing pipeline under high-concurrency impact is ensured.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and more specifically, to a method and system for integrating multi-source heterogeneous data for smart energy management. Background Technology

[0002] In the typical architecture of current smart energy management platforms, when a critical downstream system on which the platform depends, such as the enterprise resource planning system responsible for settlement, the customer relationship management system that manages customer interactions, or the specific energy control system that directly controls the charging and discharging of energy storage, becomes slow to respond or completely unavailable due to internal failures, planned maintenance, or sudden overload, the platform aims to provide robust standard fault-tolerance mechanisms, such as exponential backoff retry strategies and thread pool waiting. Under the impact of dynamically influxed high-concurrency data streams, its behavior will be reversed.

[0003] Tens of thousands of real-time data requests are not immediately rejected, but instead accumulate in the thread pool of upstream services and are repeatedly retried. This first leads to the rapid exhaustion of application service threads that handle these requests, subsequently causing the application service itself to crash. This failure can retrace upstream like dominoes: message queues quickly back up due to consumer stagnation, and the data access layer eventually becomes congested because it cannot obtain the release of core processing power. The entire system starts from a single vulnerable point at the very bottom and experiences a chain reaction of failures from the inside out and from the bottom up, resulting in a complete paralysis of the data processing pipeline and forming an upstream service avalanche. More seriously, in smart energy scenarios, such avalanches not only mean IT service interruptions, but can also directly lead to physical risks such as real-time energy scheduling misalignment and delayed fault isolation. Traditional circuit breaker or degradation strategies are often slow to react and have limited effectiveness in the face of such systemic, chain-reaction failures.

[0004] Under the continuous impact of high-concurrency data streams, existing technologies rely on standard fault-tolerance mechanisms such as passive retries and circuit breakers, which not only fail to effectively isolate faults but also rapidly exhaust upstream resources due to the continuous accumulation and retries of requests, triggering a cascading failure effect between services and causing the entire data processing pipeline to be paralyzed.

[0005] In view of this, the present invention proposes a method and system for integrating multi-source heterogeneous data for smart energy management to solve the above problems. Summary of the Invention

[0006] To overcome the aforementioned shortcomings of existing technologies and achieve the above objectives, this invention provides the following technical solution: a multi-source heterogeneous data integration method for smart energy management, comprising:

[0007] Multi-source heterogeneous data is collected through a gateway, and adaptive protocol adaptation and standardization processing are performed to obtain a labeled standardized dataset.

[0008] Deploy a dynamically aware message queue and receive labeled standardized datasets. Break down the data processing steps into independent processing units, process the standardized datasets, and obtain the data stream to be processed.

[0009] By acquiring and dynamically routing and load balancing the data stream to be processed based on multi-dimensional perception data, an optimized processing path can be obtained.

[0010] The system acquires and integrates the data after processing through the optimized processing path, stores it uniformly, and uses a unified gateway for fault isolation and monitoring feedback.

[0011] Furthermore, methods for obtaining the data stream to be processed include:

[0012] Deploy a dynamically aware message queue and receive labeled standardized datasets, then partition and store the standardized datasets according to the labels;

[0013] Break down the data processing steps into independent processing units;

[0014] The processing unit acts as a consumer of the dynamic sensing message queue. The consumer starts a status feedback thread and sends status data to the dynamic sensing message queue every preset number of seconds.

[0015] The dynamically aware message queue has a built-in status analysis module that calculates the processing capability score for each consumer based on the status data fed back by the consumer, and dynamically adjusts the data delivery rate based on the processing capability score to obtain the initial data stream.

[0016] The process execution engine collects the status data of each processing unit in real time and feeds the status data back to the dynamic perception message queue to obtain the data stream to be processed.

[0017] Furthermore, the processing unit includes a data cleaning unit, a format conversion unit, a unit conversion unit, a data aggregation unit, and a data verification unit.

[0018] Furthermore, the status data includes CPU utilization, memory utilization, network I / O load, current task queue length, data processing latency, and service availability status; wherein, the service availability status includes normal, degraded, and unavailable.

[0019] Furthermore, the method for calculating each consumer's processing ability score based on consumer feedback status data includes:

[0020] Calculate the ratio of CPU utilization to the value 100 to obtain the first ratio, and calculate the difference between the value 1 and the first ratio to obtain the CPU idle time.

[0021] Calculate the ratio of memory usage to the value 100 to obtain the second ratio, and calculate the difference between the value 1 and the second ratio to obtain the memory freeness.

[0022] Calculate the ratio of the preset delay threshold to the data processing delay to obtain the third ratio. Then, obtain the smaller value between the value 1 and the third ratio to obtain the delay satisfaction.

[0023] The processing power score for each consumer is obtained by weighting CPU idle time, memory idle time, and latency satisfaction.

[0024] Furthermore, methods for fault isolation and monitoring feedback through a unified gateway include:

[0025] The fused data is classified according to data type and object model label. The fused data includes cleaned real-time data, aggregated statistical data, verified business data, and alarm event data.

[0026] Construct a unified energy data model, define the core entities, entity attributes, and entity relationships of energy data, and establish a unified semantic index for all stored data;

[0027] Deploy a distributed data service gateway as the sole entry point for all external data services, providing a unified data access interface for upper-layer applications;

[0028] The distributed data service gateway monitors the health status of the backend's dependent internal services in real time, with monitoring metrics including interface response time, error rate, and number of concurrent requests.

[0029] A preset circuit breaker threshold is set. When the monitoring index of an internal service exceeds the circuit breaker threshold for S consecutive periods, the gateway will automatically trigger the circuit breaker and suspend calls to the corresponding internal service. The circuit breaker duration is dynamically adjusted according to the exponential backoff strategy.

[0030] During the circuit breaker period, the distributed data service gateway executes a predefined degradation strategy.

[0031] Furthermore, methods for obtaining labeled, standardized datasets include:

[0032] A gateway is deployed at the edge layer. The gateway receives heterogeneous data from multiple sources and writes it to a data buffer pool. When the amount of buffered data in the data buffer pool reaches a preset high water level threshold, the gateway sends a buffer overflow warning. When it falls below a preset low water level threshold, the gateway initiates a batch data upload mechanism.

[0033] The data buffer pool performs preliminary verification on multi-source heterogeneous data; generates an initial verification flag based on the verification results; multi-source heterogeneous data that passes verification is marked as pending parsing, and multi-source heterogeneous data that fails verification is logged and a retry request is triggered on the device.

[0034] The protocol is identified and the format is converted for multi-source heterogeneous data in the data buffer pool that is to be parsed; and the converted data is labeled with object model based on a unified data model to form a labeled standardized dataset.

[0035] Furthermore, methods for obtaining optimized processing paths include:

[0036] Collect multi-dimensional perception data, including cluster resource indicators, data flow business characteristics, and downstream service health status; cluster resource indicators include CPU utilization and memory utilization; data flow business characteristics include data priority and latency sensitivity thresholds; downstream service health status includes processing latency and heartbeat status.

[0037] Set routing rules based on the data priority and latency sensitivity threshold in the data tags;

[0038] When the preset expansion or reduction conditions are met, the corresponding expansion or reduction is triggered.

[0039] Furthermore, methods for dynamically adjusting the data delivery rate based on processing capacity scores include:

[0040] When the processing capability score is not lower than the first threshold, data is delivered at the first rate; when it is not lower than the second threshold but lower than the first threshold, data is delivered at the second rate; when it is not lower than the third threshold but lower than the second threshold, data is delivered at the third rate; when the processing capability score is lower than the third threshold or the service availability status is unavailable, data delivery to the corresponding consumer is suspended, the data stream of the corresponding topic is temporarily persisted to the backup storage partition of the dynamic awareness message queue, and marked as pending delivery status; when the consumer does not provide status data for N consecutive times, data delivery to it is suspended, the data is temporarily stored in the persistent storage partition of the dynamic awareness message queue, and re-delivered after the consumer recovers.

[0041] A multi-source heterogeneous data integration system for smart energy management, implementing the aforementioned multi-source heterogeneous data integration method for smart energy management, includes:

[0042] Data access module: Collects multi-source heterogeneous data through the gateway, performs adaptive protocol adaptation and standardization processing, and obtains a labeled standardized dataset;

[0043] Message receiving module: Deploys a dynamically aware message queue and receives labeled standardized datasets, breaks down data processing steps into independent processing units, processes the standardized datasets, and obtains the data stream to be processed;

[0044] Perception and scheduling module: acquires and dynamically routes and load balances the data stream to be processed based on multi-dimensional perception data to obtain an optimized processing path;

[0045] Storage monitoring module: acquires the fused data after optimization processing path, stores it uniformly, and performs fault isolation and monitoring feedback through a unified gateway.

[0046] The technical effects and advantages of this invention regarding the multi-source heterogeneous data integration method and system for smart energy management are as follows:

[0047] This invention transforms the traditional passive fault-tolerance mechanism by constructing a multi-layered, proactive, and resilient data processing pipeline. At the data access source, edge buffering and verification are used for peak shaving and invalid data filtering. At the core processing layer, dynamic-aware message queues and atomic units enable intelligent traffic control and precise fault isolation based on real-time processing capabilities. Finally, a unified gateway at the service exit implements centralized circuit breaking and structured degradation, forming a collaborative defense system that spans the entire data lifecycle. This invention significantly advances protective actions, shifting from relying on retries and circuit breaking after errors occur to continuous perception and predictive intervention of resource status and traffic pressure. This effectively breaks the avalanche chain of request accumulation, resource exhaustion, and subsequent failures, ultimately ensuring that the smart energy management system can maintain the stable and reliable operation of the data processing pipeline even when facing high-concurrency, multi-source, heterogeneous data streams. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the multi-source heterogeneous data integration method for smart energy management according to the present invention;

[0049] Figure 2 This is a schematic diagram of the method for obtaining a data stream to be processed according to the present invention;

[0050] Figure 3 This is a schematic diagram of the method for obtaining an optimized processing path according to the present invention;

[0051] Figure 4 This is a schematic diagram of the multi-source heterogeneous data integration system for smart energy management according to the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Example 1:

[0054] Please see Figure 1 As shown, this embodiment provides a method for integrating multi-source heterogeneous data for smart energy management, including:

[0055] Multi-source heterogeneous data is collected through a gateway, and adaptive protocol adaptation and standardization processing are performed to obtain a labeled standardized dataset.

[0056] Methods for obtaining labeled, standardized datasets include:

[0057] Deploying gateways at the edge layer allows them to receive heterogeneous data from multiple sources and write it to a data buffer pool. When the amount of buffered data in the data buffer pool reaches a preset high-water mark threshold, the gateway sends a buffer overflow warning. When the amount of data falls below a preset low-water mark threshold, the gateway initiates a batch data upload mechanism. By using local buffering at the edge and dynamic data flow control, peak shaving and valley filling are proactively implemented to prevent high-concurrency data flows from directly impacting the core system. This reduces the risk of request backlog at the source and solves the problem of upstream resources being continuously occupied under passive fault tolerance mechanisms.

[0058] The data buffer pool performs preliminary verification on multi-source heterogeneous data, including verification of data format validity, numerical range rationality, and timestamp continuity, and generates an initial verification flag based on the verification results. Multi-source heterogeneous data that passes verification is marked as pending parsing, while multi-source heterogeneous data that fails verification is logged and triggers device-side retry requests with a preset retry frequency and maximum number of attempts. Invalid data such as those with illegal formats and abnormal values ​​are filtered in advance to prevent invalid requests from entering subsequent processing and consuming resources. Targeted retries are triggered on the device side after verification failure, which can reduce the impact of retry storms on the gateway and reduce the risk of fault propagation.

[0059] The system performs protocol identification and format conversion on multi-source heterogeneous data awaiting parsing in the data buffer pool. Based on a unified data model, it tags the converted data with object model labels, forming a standardized dataset with labels. Specifically, the gateway's built-in protocol parsing engine automatically matches the corresponding protocol parsing rules based on the data header identifier, uniformly converting the parsed multi-source heterogeneous data into JSON or Avro intermediate formats to ensure data structure consistency. The unified data model tags the converted data with object model labels, such as device identifier labels representing device ID, affiliated site, and device type; data attribute labels representing data types as real-time sampled values ​​or statistical values; and time labels representing data acquisition timestamps. The protocol parsing engine automatically matches protocol parsing rules, achieving adaptive adaptation and unified format conversion of heterogeneous data, reducing parsing overhead in subsequent processing flows, avoiding processing bottlenecks caused by inconsistent data structures, and lowering the probability of resource exhaustion leading to avalanches. Standardized labels achieve data semantic uniformity, providing a foundation for subsequent label-based dynamic scheduling and fault isolation, supporting proactive fault tolerance mechanisms, and preventing processing pipeline blockages caused by data chaos.

[0060] Deploy a dynamically aware message queue and receive labeled standardized datasets. Break down the data processing steps into independent processing units, process the standardized datasets, and obtain the data stream to be processed.

[0061] Reference Figure 2 Methods for obtaining the data stream to be processed include:

[0062] Deploy a dynamically aware message queue and receive standardized datasets with labels. Partition the standardized datasets according to the labels and store them in partitions. Physical isolation of data of different business types is achieved through label partitioning, which avoids the spread of a single business data stream failure to the entire link. At the same time, it lays the foundation for subsequent accurate scheduling, reduces data processing conflicts under high concurrency, and reduces the risk of resource exhaustion.

[0063] The data processing steps are broken down into independent processing units, each performing only a single function. Each processing unit is defined as a consumer, including a data cleaning unit, a format conversion unit, a unit conversion unit, a data aggregation unit, and a data verification unit. Specifically, the data cleaning unit fills in missing values ​​and corrects outliers, the format conversion unit adapts intermediate formats to storage formats, the unit conversion unit and the data aggregation unit aggregate data by time window or device type, and the data verification unit performs secondary deep verification, such as cross-device data consistency verification. By decoupling complex processing flows into atomic units, precise fault isolation is achieved, and faulty units can be quickly bypassed or replaced, thus solving the problem of pipeline paralysis caused by single-point failures in traditional integrated processes.

[0064] Each processing unit initiates a status feedback thread, sending status data to the dynamic sensing message queue every preset number of seconds, such as 1 second. The status data includes CPU utilization, memory utilization, network I / O load, current task queue length, data processing latency, and service availability status. Among them, service availability status includes normal, degraded, and unavailable. By actively reporting core status data, the message queue can perceive downstream processing capabilities in real time, replacing the traditional passive fault tolerance circuit breaker, providing data support for dynamically adjusting the delivery rate, and avoiding blind request backlog from the source.

[0065] The dynamically aware message queue has a built-in status analysis module that calculates a processing capability score for each consumer based on status data reported by the consumers. Specifically, it calculates the processing capability score for each consumer based on CPU utilization, memory utilization, data processing latency, and a preset latency threshold, including:

[0066] Calculate the ratio of CPU utilization to the value 100 to obtain the first ratio, and calculate the difference between the value 1 and the first ratio to obtain the CPU idle time.

[0067] Calculate the ratio of memory usage to the value 100 to obtain the second ratio, and calculate the difference between the value 1 and the second ratio to obtain the memory freeness.

[0068] Calculate the ratio of the preset delay threshold to the data processing delay to obtain the third ratio. Then, obtain the smaller value between the value 1 and the third ratio to obtain the delay satisfaction.

[0069] The processing capability score for each consumer is obtained by weighting CPU idle time, memory idle time, and latency satisfaction. The CPU weight, memory weight, and latency weight can be configured according to the actual business scenario: for compute-intensive tasks, the CPU weight can be appropriately increased (e.g., 0.4-0.5); for memory-intensive tasks, the memory weight can be appropriately increased (e.g., 0.4-0.5); for latency-sensitive tasks, the latency weight can be appropriately increased (e.g., 0.4-0.6). The typical default configuration is: CPU weight 0.3, memory weight 0.3, and latency weight 0.4.

[0070] The data delivery rate is dynamically adjusted based on the processing capability score to obtain the initial data stream. When the processing capability score is not lower than the first threshold, data is delivered at the first rate. When it is not lower than the second threshold but lower than the first threshold, data is delivered at the second rate. When it is not lower than the third threshold but lower than the second threshold, data is delivered at the third rate. The first threshold is typically set based on the system's optimal performance range, generally between 0.8 and 0.85, indicating excellent processing capability. The second threshold is typically set based on the system's normal load range, generally between 0.6 and 0.7, indicating good processing capability. The third threshold is typically set based on the system's degraded operation threshold, generally between 0.5 and 0.6, indicating basically acceptable processing capability. The first rate is set by the system... The maximum theoretical processing capacity is determined based on hardware configuration and business needs; the second rate is 50%-70% of the first rate, serving as a degraded processing rate; the third rate is 20%-40% of the first delivery rate, serving as a low-load processing rate; when the processing capacity score is below the third threshold or the service availability status is unavailable, data delivery to the corresponding consumer is suspended, the data stream of the corresponding topic is temporarily persisted to the backup storage partition of the dynamically aware message queue, and marked as pending delivery recovery; based on the processing capacity score, fine-grained control of the data stream is achieved, with speed reduction during high load and suspension of delivery and persistence during failure, proactively smoothing peaks to prevent downstream processing units from being overwhelmed by high-concurrency data streams, and resolving the avalanche cause of continuous resource exhaustion under passive retries or circuit breakers.

[0071] When a consumer fails to provide status data N times consecutively, data delivery to it is suspended, and the data is temporarily stored in the persistent storage partition of the dynamic awareness message queue. Delivery resumes once the consumer recovers, preventing upstream data backlog caused by downstream failures. N is set based on system reliability requirements, typically 3-5 times. Unresponsive, faulty consumers are quickly identified and isolated. Persistent storage ensures no data loss and prevents queue resources from being exhausted due to continuous data accumulation caused by consumer failures, thus blocking the path of fault propagation upstream. When the faulty consumer recovers and its processing capability score rises above the acceptable threshold, the dynamic awareness message queue will resume delivering persistent data at a controllable rate, such as 50% of the base rate, gradually returning to the normal rate once its status stabilizes.

[0072] The process execution engine collects the execution status of each processing unit in real time and feeds the status data back to the status analysis module of the dynamically aware message queue. This serves as supplementary information for adjusting the data delivery rate of the dynamically aware message queue, thus obtaining the data stream to be processed. The process execution engine itself can suggest adjustments to the task distribution strategy or request scaling up or down from the resource management platform based on the overall system load, rather than directly interfering with the real-time delivery rate of the dynamically aware message queue. This ensures clear responsibility for traffic control and avoids conflicts. By incorporating the execution status of processing units into the queue control criteria, a closed-loop feedback between the queue and processing units is formed, allowing for more precise delivery rate adjustments and avoiding resource waste or processing bottlenecks caused by single-dimensional status judgments.

[0073] Meanwhile, the process execution engine dynamically adjusts the process execution concurrency based on the load status feedback of downstream computing nodes, achieving dynamic matching between data flow and processing capacity. By adapting the process processing capacity to the load of downstream computing nodes in real time, dynamic matching between data flow processing concurrency and computing resources is achieved, preventing process execution from becoming a bottleneck in the data processing pipeline and reducing the risk of request backlog and resource exhaustion caused by processing capacity mismatch.

[0074] By acquiring and dynamically routing and load balancing the data stream to be processed based on multi-dimensional perception data, an optimized processing path can be obtained.

[0075] Reference Figure 3 Methods for obtaining optimized processing paths include:

[0076] Collect multi-dimensional perception data, including cluster resource indicators, data flow business characteristics, and downstream service health status. Cluster resource indicators include CPU utilization and memory utilization. Data flow business characteristics include data priority and latency sensitivity threshold. Downstream service health status includes processing latency and heartbeat status. Among them, data priority is obtained by encoding the data priority of different businesses, and the data priority includes high priority, medium priority, and low priority. The latency sensitivity threshold is set according to the business type and actual situation. For example, the latency sensitivity threshold for real-time control is generally no higher than 50ms, the latency sensitivity threshold for monitoring and alarm is generally no higher than 100ms, and the latency sensitivity threshold for statistical analysis is generally no higher than 5000ms. When a processing unit loses its heartbeat or does not respond for 3 consecutive times, it is marked as a fault.

[0077] Set routing rules based on the data priority and latency sensitivity threshold in the data tags;

[0078] When CPU utilization is below the CPU health threshold, memory utilization is below the memory health threshold, and the current data processing latency does not exceed the latency tolerance multiple of the latency sensitivity threshold, the node is marked as a healthy node. The CPU health threshold is set according to the efficiency inflection point of the operating system scheduler, usually set to 70%-80% to avoid CPU overload affecting processing stability. The memory health threshold is set according to the efficiency of memory paging and garbage collection mechanisms, usually set to 75%-85% to leave a safety margin for memory usage. The latency tolerance multiple is set according to the acceptable latency fluctuation range of the business and the historical latency distribution, usually set to 1.2-1.5 to allow for a certain amount of processing latency fluctuation.

[0079] Dynamic routing decisions are made based on the following rules:

[0080] Priority rules: High-priority data is routed to the healthy node with the lowest processing latency; medium-priority data is routed to the healthy node with the lowest overall load; low-priority data is distributed among all healthy nodes using a weighted round-robin method.

[0081] Delay guarantee rule: For delay-sensitive data, i.e., the data delay sensitivity threshold is not higher than 100ms, it must be routed to a node whose current processing delay is lower than 70% of the delay sensitivity threshold; if there is no node that meets the condition, select the node whose delay is closest to the delay sensitivity threshold and trigger a delay alarm.

[0082] Load balancing rules: The overall load rate of a node is obtained by weighting CPU utilization and memory utilization. The CPU load weight and memory load weight can be configured according to the task characteristics. For CPU-intensive tasks, the CPU weight can be set to 0.6-0.7; for memory-intensive tasks, the memory weight can be set to 0.6-0.7. The typical default configuration is CPU weight 0.6 and memory weight 0.4. The minimum overall load rate priority strategy is adopted to assign new tasks to the healthy node with the lowest overall load rate. When the difference in the overall load rate of multiple nodes is less than 5%, round-robin allocation is used to avoid hotspots.

[0083] Fault avoidance rules: Automatically skip nodes marked as faulty; skip nodes that have failed at least once in the last 3 minutes.

[0084] When the overall system latency satisfaction rate is lower than the latency satisfaction rate expansion threshold for a continuous A collection period, and the average CPU utilization rate of the cluster is higher than the CPU expansion threshold or the average memory utilization rate is higher than the memory expansion threshold, expansion is triggered. Specifically, one processing unit is added each time, and the evaluation is repeated after an interval of at least 2 minutes.

[0085] When the overall system latency satisfaction rate is higher than the latency satisfaction rate reduction threshold for a continuous B collection cycle, and the average CPU utilization rate of the cluster is lower than the CPU reduction threshold and the average memory utilization rate is lower than the memory reduction threshold, a reduction is triggered. Specifically, one processing unit is reduced each time to ensure that at least three processing units are retained.

[0086] The latency satisfaction threshold for expansion is set based on service quality, typically between 85% and 92%. A latency satisfaction rate below this value indicates insufficient system performance. The latency satisfaction threshold for reduction is set based on maximizing resource utilization efficiency, i.e., maximizing resource utilization while meeting service quality requirements, typically between 95% and 98%. A latency satisfaction rate above this value indicates potential resource overcapacity. The CPU expansion threshold is set based on long-term average utilization and the actual expansion coefficient, typically between 75% and 85%, indicating CPU resource strain. The memory expansion threshold is set based on long-term average utilization and the actual expansion coefficient, typically between 80% and 90%, indicating memory resource strain. The CPU reduction threshold is set based on long-term average utilization and the actual reduction coefficient, typically between 25% and 35%, indicating sufficient CPU resources. The memory reduction threshold is set based on long-term average utilization and the actual reduction coefficient, typically between 35% and 45%, indicating sufficient memory resources. A and B are set based on response performance, with A typically set to 3-5 cycles and B typically set to 8-12 cycles to avoid frequent expansion and reduction.

[0087] The system acquires and integrates the data after processing through the optimized processing path, stores it uniformly, and uses a unified gateway for fault isolation and monitoring feedback.

[0088] Methods for fault isolation and monitoring feedback through a unified gateway include:

[0089] The fused data is classified according to data type and object model label. The fused data includes cleaned real-time data, aggregated statistical data, verified business data, and alarm event data, etc.

[0090] A unified energy data model is constructed, defining core entities, entity attributes, and entity relationships for energy data, and establishing a unified semantic index for all stored data. Core entities include equipment, measurement points, business processes, and users; entity attributes include equipment models, measurement point units, and business priorities; and entity relationships include the attribution relationship between equipment and measurement points, and the association relationship between business processes and data. The unified energy data model eliminates semantic ambiguity between different data sources, enabling one-time modeling and multi-source reuse, supporting cross-data source queries and analysis in upper-layer applications. Data is categorized and stored according to data type to adapt to different storage needs. The unified energy data model eliminates heterogeneous data ambiguity through semantic indexing, enabling one-time modeling and multi-source reuse, reducing processing overhead and resource waste in subsequent cross-data source queries, avoiding processing pipeline blockage caused by data chaos, and reducing the risk of data avalanche.

[0091] Deploy a distributed data service gateway as the sole entry point for all external data services, providing a unified data access interface for upper-layer applications; converge all external data services to a unified gateway to achieve centralized management and control of interface access, avoid blind spots in fault monitoring and chaotic management caused by scattered interfaces, provide a foundation for subsequent unified circuit breaking, degradation, and rate limiting, and block the path of fault propagation from the data service exit end.

[0092] The distributed data service gateway monitors the health status of the backend's dependent internal services in real time. The monitoring indicators include interface response time, error rate, and number of concurrent requests. By monitoring the core health indicators of backend services in real time, it proactively detects service anomalies, replacing the traditional passive waiting mode for fault triggers. This provides accurate and timely triggering basis for circuit breaking and degradation, and avoids resource exhaustion caused by the continuous escalation of faults.

[0093] A preset circuit breaker threshold is set. When the monitoring metric of an internal service exceeds the threshold for S consecutive periods, the gateway automatically triggers the circuit breaker, suspending calls to the corresponding internal service. The circuit breaker duration is dynamically adjusted according to an exponential backoff strategy. The circuit breaker threshold is set according to the actual situation. S is set according to system stability requirements, usually 2-4 periods. A precise triggering mechanism based on consecutive periods exceeding the threshold is adopted to avoid false circuit breakers caused by single metric fluctuations. The circuit breaker duration is dynamically adjusted according to an exponential backoff strategy to prevent gateway resource jitter caused by frequent circuit breakers or recovery. Calls to faulty services are proactively suspended, and faults are isolated from the service dependency layer to prevent upstream requests from continuously accumulating and exhausting gateway resources.

[0094] During the circuit breaker period, the distributed data service gateway executes a predefined degradation strategy. After the circuit breaker is broken, it does not directly reject upstream requests, but returns cached data, default values, or simplified data to avoid upstream applications initiating high-frequency retries due to service unavailability. This reduces the consumption of gateway connection resources by retry storms and prevents the chain reaction of failure, retries, resource exhaustion, and cascading failures.

[0095] Example 2:

[0096] Please see Figure 4 As shown, this embodiment provides a multi-source heterogeneous data integration system for smart energy management, including:

[0097] Data access module: Collects multi-source heterogeneous data through the gateway, performs adaptive protocol adaptation and standardization processing, and obtains a labeled standardized dataset;

[0098] Message receiving module: Deploys a dynamically aware message queue and receives labeled standardized datasets, breaks down data processing steps into independent processing units, processes the standardized datasets, and obtains the data stream to be processed;

[0099] Perception and scheduling module: acquires and dynamically routes and load balances the data stream to be processed based on multi-dimensional perception data to obtain an optimized processing path;

[0100] Storage monitoring module: acquires the fused data after optimization processing path, stores it uniformly, and performs fault isolation and monitoring feedback through a unified gateway.

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0102] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for integrating multi-source heterogeneous data for smart energy management, characterized in that, include: Multi-source heterogeneous data is collected through a gateway, and adaptive protocol adaptation and standardization processing are performed to obtain a labeled standardized dataset. Deploy a dynamically aware message queue and receive labeled standardized datasets. Break down the data processing steps into independent processing units, process the standardized datasets, and obtain the data stream to be processed. By acquiring and dynamically routing and load balancing the data stream to be processed based on multi-dimensional perception data, an optimized processing path can be obtained. The system acquires and integrates the data after processing through the optimized processing path, stores it uniformly, and uses a unified gateway for fault isolation and monitoring feedback.

2. The multi-source heterogeneous data integration method for smart energy management according to claim 1, characterized in that, Methods for obtaining the data stream to be processed include: Deploy a dynamically aware message queue and receive labeled standardized datasets, then partition and store the standardized datasets according to the labels; Break down the data processing steps into independent processing units; The processing unit acts as a consumer of the dynamic sensing message queue. The consumer starts a status feedback thread and sends status data to the dynamic sensing message queue every preset number of seconds. The dynamically aware message queue has a built-in status analysis module that calculates the processing capability score for each consumer based on the status data fed back by the consumer, and dynamically adjusts the data delivery rate based on the processing capability score to obtain the initial data stream. The process execution engine collects the status data of each processing unit in real time and feeds the status data back to the dynamic perception message queue to obtain the data stream to be processed.

3. The multi-source heterogeneous data integration method for smart energy management according to claim 2, characterized in that, The processing unit includes a data cleaning unit, a format conversion unit, a unit conversion unit, a data aggregation unit, and a data verification unit.

4. The multi-source heterogeneous data integration method for smart energy management according to claim 2, characterized in that, The status data includes CPU utilization, memory utilization, network I / O load, current task queue length, data processing latency, and service availability status; wherein, the service availability status includes normal, degraded, and unavailable.

5. The multi-source heterogeneous data integration method for smart energy management according to claim 2, characterized in that, Methods for calculating each consumer's processing ability score based on consumer feedback status data include: Calculate the ratio of CPU utilization to the value 100 to obtain the first ratio, and calculate the difference between the value 1 and the first ratio to obtain the CPU idle time. Calculate the ratio of memory usage to the value 100 to obtain the second ratio, and calculate the difference between the value 1 and the second ratio to obtain the memory freeness. Calculate the ratio of the preset delay threshold to the data processing delay to obtain the third ratio. Then, obtain the smaller value between the value 1 and the third ratio to obtain the delay satisfaction. The processing power score for each consumer is obtained by weighting CPU idle time, memory idle time, and latency satisfaction.

6. The multi-source heterogeneous data integration method for smart energy management according to claim 1, characterized in that, Methods for fault isolation and monitoring feedback through a unified gateway include: The fused data is classified according to data type and object model label. The fused data includes cleaned real-time data, aggregated statistical data, verified business data, and alarm event data. Construct a unified energy data model, define the core entities, entity attributes, and entity relationships of energy data, and establish a unified semantic index for all stored data; Deploy a distributed data service gateway as the sole entry point for all external data services; The distributed data service gateway monitors the health status of the backend's dependent internal services in real time, with monitoring metrics including interface response time, error rate, and number of concurrent requests. A preset circuit breaker threshold is set. When the monitoring index of an internal service exceeds the circuit breaker threshold for S consecutive periods, the gateway will automatically trigger the circuit breaker and suspend calls to the corresponding internal service. The circuit breaker duration is dynamically adjusted according to the exponential backoff strategy. During the circuit breaker period, the distributed data service gateway executes a predefined degradation strategy.

7. The multi-source heterogeneous data integration method for smart energy management according to claim 1, characterized in that, Methods for obtaining labeled, standardized datasets include: A gateway is deployed at the edge layer. The gateway receives heterogeneous data from multiple sources and writes it to a data buffer pool. When the amount of buffered data in the data buffer pool reaches a preset high water level threshold, the gateway sends a buffer overflow warning. When it falls below a preset low water level threshold, the gateway initiates a batch data upload mechanism. The data buffer pool performs preliminary verification on multi-source heterogeneous data; generates an initial verification flag based on the verification results; multi-source heterogeneous data that passes verification is marked as pending parsing, and multi-source heterogeneous data that fails verification is logged and a retry request is triggered on the device. The protocol is identified and the format is converted for multi-source heterogeneous data in the data buffer pool that is to be parsed; and the converted data is labeled with object model based on a unified data model to form a labeled standardized dataset.

8. The multi-source heterogeneous data integration method for smart energy management according to claim 1, characterized in that, Methods for obtaining optimized processing paths include: Collect multi-dimensional perception data, including cluster resource indicators, data flow business characteristics, and downstream service health status; cluster resource indicators include CPU utilization and memory utilization; data flow business characteristics include data priority and latency sensitivity thresholds; downstream service health status includes processing latency and heartbeat status. Set routing rules based on the data priority and latency sensitivity threshold in the data tags; When the preset expansion or reduction conditions are met, the corresponding expansion or reduction is triggered.

9. The multi-source heterogeneous data integration method for smart energy management according to claim 2, characterized in that, Methods for dynamically adjusting data delivery rates based on processing capacity scores include: When the processing capability score is not lower than the first threshold, data is delivered at the first rate; when it is not lower than the second threshold but lower than the first threshold, data is delivered at the second rate; when it is not lower than the third threshold but lower than the second threshold, data is delivered at the third rate; when the processing capability score is lower than the third threshold or the service availability status is unavailable, data delivery to the corresponding consumer is suspended, the data stream of the corresponding topic is temporarily persisted to the backup storage partition of the dynamic awareness message queue, and marked as pending delivery status; when the consumer does not provide status data for N consecutive times, data delivery to it is suspended, the data is temporarily stored in the persistent storage partition of the dynamic awareness message queue, and re-delivered after the consumer recovers.

10. A multi-source heterogeneous data integration system for smart energy management, implementing the multi-source heterogeneous data integration method for smart energy management as described in any one of claims 1-9, characterized in that, include: Data access module: Collects multi-source heterogeneous data through the gateway, performs adaptive protocol adaptation and standardization processing, and obtains a labeled standardized dataset; Message receiving module: Deploys a dynamically aware message queue and receives labeled standardized datasets, breaks down data processing steps into independent processing units, processes the standardized datasets, and obtains the data stream to be processed; Perception and scheduling module: acquires and dynamically routes and load balances the data stream to be processed based on multi-dimensional perception data to obtain an optimized processing path; Storage monitoring module: acquires the fused data after optimization processing path, stores it uniformly, and performs fault isolation and monitoring feedback through a unified gateway.

Images