Collaborative data processing method and system based on source network load storage integration

Abstract

The invention discloses a collaborative data processing method and system based on source-network-load-storage integration. The collaborative data processing method and system are used for data acquisition, processing and optimal scheduling of multi-source heterogeneous equipment. The method comprises the following steps: registering a power generation side device, an energy storage side device, a power distribution network device and a load side device, obtaining self-description information, automatically identifying a communication protocol and loading a corresponding drive; a standardized data stream is generated through protocol conversion and semantic mapping, edge nodes execute real-time feature extraction and preliminary calculation, and a cloud platform performs global state estimation, prediction analysis and optimization scheduling; establishing a time delay monitoring and task priority mechanism to realize edge and cloud dynamic collaboration; performing quality evaluation and anomaly repair on the transmission data to generate a high-quality data set; a probability distribution model is established based on data, an uncertainty weight is calculated, feature fusion and weighted dimension reduction are performed through a collaborative data processing engine, a weighted confidence matrix is formed and input into optimal scheduling, system robustness is enhanced, and the method is suitable for a power system with a high new energy proportion and large load fluctuation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of power systems and energy management, and in particular to a collaborative data processing method and system based on source-grid-load-storage integration. BACKGROUND

[0002] With the increasing proportion of new energy power generation and the rapid development of multi-energy complementary systems, the power system is transforming from the traditional single generation-transmission-distribution-load mode to the intelligent collaborative mode of source-grid-load-storage integration. The source-grid-load-storage integrated system can realize efficient energy utilization and safe and reliable system operation by coordinating multiple types of devices and resources on the generation side, energy storage side, distribution network, and load side. With the rapid increase in new energy installed capacity, the volatility and uncertainty of renewable energy sources such as wind power and photovoltaic power pose higher requirements for the stable operation of the power system. At the same time, the rapid development of energy storage systems makes source-grid-load-storage integration possible, providing a technical foundation for achieving multiple objectives such as economy, reliability, renewable energy consumption, and energy storage life extension.

[0003] The existing technology has the following problems: In the current system, different communication protocols and data interfaces are used for various devices, resulting in complex device access, non-uniform data formats, and difficulty in directly sharing information between system layers, which limits the real-time performance and accuracy of collaborative scheduling. Traditional scheduling systems usually rely on centralized computing and lack support for dynamic task allocation between edge nodes and cloud platforms, which cannot quickly adjust the task allocation ratio according to real-time load and network delay, resulting in scheduling response delay and control accuracy decline. Due to the presence of outliers, missing values, and drift in device collected data, and the volatility of new energy generation, existing systems have difficulty in effectively assessing data quality and quantifying uncertainty, affecting the accuracy of scheduling decisions and system robustness. The priority management for different task types (such as frequency regulation, energy balance, and prediction analysis) is not perfect, and the communication link delay is not dynamically monitored and predicted, making it impossible to guarantee real-time execution of critical tasks and posing a risk to system operation. SUMMARY

[0004] The purpose of the present application is to provide a collaborative data processing method and system based on source-grid-load-storage integration to achieve unified access, standardized data processing, edge-cloud collaborative scheduling, data quality enhancement, and uncertainty quantification for various devices and resources of source, grid, load, and storage, thereby improving the real-time performance, robustness, and overall scheduling efficiency of the system.

[0005] The purpose of the present application can be achieved through the following technical solutions:

[0006] The collaborative data processing method based on source-grid-load-storage integration includes the following steps:

[0007] S1, register the power generation side device, energy storage side device, power distribution network device and load side device, and obtain the self-description information of each device; automatically identify the communication protocol type of the device according to the self-description information, load the driving module corresponding to the communication protocol type; unify the data format and data label of different protocols through the protocol conversion unit and the semantic mapping unit, generate standardized data stream, and input the standardized data stream to the edge node for subsequent processing;

[0008] S2, the edge node performs preliminary calculation and rapid response processing on the standardized data stream to generate edge processing results; the edge processing results are uploaded to the cloud platform, and the cloud platform performs global state estimation, prediction analysis and optimization scheduling calculation; the cloud platform issues a cooperative scheduling instruction to the edge node according to the calculation result, and the edge node executes local control according to the received instruction; the data consistency between the edge node and the cloud platform is maintained through the time delay perception communication channel, and the task execution weight of the edge node and the cloud platform is dynamically allocated according to the real-time load by using the task scheduling module; when the communication is interrupted, the edge node calls the preset strategy library for autonomous control, and synchronizes the autonomous running data to the cloud platform after the communication is restored;

[0009] S3, based on the cooperative calculation process of the edge node and the cloud platform, set the task priority for the task type including frequency adjustment task, energy balance task and prediction analysis task; use the time delay monitoring model to monitor the communication link delay between the edge node and the cloud platform in real time, and dynamically adjust the edge and cloud task allocation ratio of the task scheduling module according to the monitoring result; the standardized data stream and the scheduling instruction are transmitted through the message queue buffer module and the lightweight data compression algorithm with low delay;

[0010] S4, quality evaluation is performed on the sampling data and scheduling feedback data transmitted through the time delay guarantee and task priority scheduling steps; a data confidence model is established to detect, interpolate repair and drift compensation operation on abnormal data; based on historical statistical characteristics and model residual error, the confidence interval of each data point is calculated, and a high-quality source network load storage cooperative data set with confidence label is output, which is fed back to the cloud platform for subsequent prediction analysis and scheduling optimization;

[0011] S5, based on the high-quality source network load storage cooperative data set, a probability distribution model is established for source data, network data, load data and storage data respectively, and the uncertainty weight is calculated; the uncertainty weight and the confidence index are used to construct a weighted confidence matrix; the high-quality data set is fused, calculated, and dimensionally analyzed by a cooperative data processing engine, and the weighted confidence matrix is provided to the optimization scheduling module of the cloud platform to form a robust enhanced scheduling decision.

[0012] Furthermore, data sample values ​​from different devices The data is parsed according to device type, communication protocol, and data interface format to generate a data vector. ;

[0013] Assign a uniform semantic label to each element of the data vector. The mapping function is defined as follows:

[0014]

[0015] in, For equipment type, For communication protocol type, For predefined semantic mapping functions;

[0016] Data vectors based on unified labels Perform format conversion to generate standardized vectors. ;

[0017] in, This is a format conversion function, including data type normalization, unit unification, and missing value filling;

[0018] Standardized vectors for all devices The data is merged according to time series and device identifier to generate a standardized data stream. Where N is the total number of connected devices and t is the time index;

[0019] The standardized data stream is provided to the edge and cloud collaborative hierarchical processing steps for subsequent real-time computing and optimized scheduling.

[0020] Furthermore, the standardized vector Mapping to a unified semantic space Generate semantic feature vectors :

[0021]

[0022] in, It is a semantic feature mapping function;

[0023] Semantic feature vectors from different devices are aggregated according to time series and device type to generate a global semantic matrix. Where N is the total number of connected devices and t is the time index, so that various types of data are kept aligned at the source, network, load and storage levels;

[0024] The edge node on the global semantic matrix Perform local feature extraction and preliminary calculations to generate edge processing results. ;

[0025] wherein, is an edge computing function, including key control signal extraction, real-time anomaly detection and preliminary prediction;

[0026] The edge processing result is uploaded to the cloud platform, and the cloud platform performs global state estimation, optimization scheduling calculation and prediction analysis to generate a global decision matrix ;

[0027] wherein, is a cloud global computing function, including global optimization scheduling and multi-device coordination decision;

[0028] According to the global decision matrix , the scheduling instruction is issued to the edge node, and the edge node and the cloud platform are cooperatively controlled and autonomously scheduled.

[0029] Further, the edge node detects the semantic feature vector and the key control signal in the edge processing result based on an event-driven mechanism, and generates a local task queue when the trigger condition is met.

[0030] wherein, is an event trigger function, is a trigger threshold;

[0031] The cloud platform dynamically calculates the task allocation ratio through the global task scheduler according to the edge processing result uploaded by the edge node, the global semantic matrix and the real-time network delay :

[0032]

[0033] wherein, is a task weight calculation function, N is the total number of access devices, is a weight coefficient;

[0034] When the communication link delay exceeds the set threshold , the cloud platform automatically migrates part of the computing task to the edge node for execution:

[0035]

[0036] wherein, The edge node executes the migrated task to ensure the real-time performance of the control signal as an indication function;

[0037] After the edge node completes the migration task, the task execution result is fused with the original edge processing result and uploaded to the cloud platform to update the global decision matrix

[0038]

[0039] Further, the edge node and the cloud platform perform real-time evaluation on the communication link delay through the time delay monitoring model to generate a time delay prediction value :

[0040]

[0041] wherein, is a time delay prediction function, is a historical time delay observation window length;

[0042] Each task is assigned a priority according to the type of task and a weight coefficient :

[0043] ;

[0044] wherein, γ is a task type weight coefficient, is a task time delay threshold, the is from the link time delay prediction step;

[0045] The edge node and the cloud platform use a lightweight message queue to cache task data and the decision matrix and apply a data compression function for fast transmission:

[0046]

[0047] wherein, is a transmission data packet;

[0048] The global task scheduler dynamically adjusts the task allocation ratio according to the predicted time delay and the task weight to reduce the control risk caused by time delay exceeding the threshold.

[0049] Further, the sampled data and scheduling feedback data transmitted through the time delay guarantee and task priority scheduling step Quality assessment is performed to calculate the initial confidence of each data point

[0050]

[0051] wherein, is the transmission data point, and are the historical statistical mean and standard deviation, respectively, is a constant to prevent zero value;

[0052] Interpolation repair and drift compensation are performed on the abnormal data points to generate repaired data

[0053]

[0054] wherein, is a polynomial interpolation function, is an abnormal threshold value;

[0055] The confidence interval of each data point is calculated based on the model residual of the repaired data

[0056]

[0057] wherein, is the residual standard deviation, is the standard normal quantile corresponding to the confidence level;

[0058] The repaired data is combined with the confidence interval label to generate a high-quality data set with confidence label

[0059]

[0060] wherein, M is the total number of data points.

[0061] The high-quality data set is fed back to the cloud platform for prediction analysis and optimization scheduling by the cloud global calculation function, forming a closed-loop control and optimization decision.

[0062] Further, based on the high-quality data set , a probability distribution model is established for source data, network data, load data and storage data respectively to generate a data probability vector

[0063]

[0064] wherein, is a probability distribution estimation function;​​​​​

[0065] According to the probability distribution vector Calculate the uncertainty weight of each data point

[0066]

[0067] Wherein, Indicates the maximum value of the data point probability distribution, reflecting the confidence level;

[0068] The collaborative data processing engine performs multi-layer feature fusion and principal component dimensionality reduction analysis on the data to generate a fusion feature matrix

[0069]

[0070] Wherein, Indicates the weighting operation according to the uncertainty weight N is the total number of devices, Is the principal component analysis function;

[0071] The fusion feature matrix Is combined with the uncertainty weight matrix To form a weighted confidence matrix

[0072]

[0073] The weighted confidence matrix Is input to the cloud global calculation function to adjust the global decision matrix .

[0074] A collaborative data processing system based on source, network, load and storage integration, the system uses the above method, the system comprises:

[0075] A device access module is used to register power generation side devices, energy storage side devices, power distribution network devices and load side devices, and obtain self-description information of each device. The device access module is also used to automatically identify the communication protocol type of the device according to the self-description information, and load the driving module corresponding to the communication protocol type;

[0076] A protocol conversion and semantic mapping unit is used to map the data of the accessed device through a unified semantic model to realize unified modeling and format standardization of various types of source, network, load and storage data at the semantic level, and generate a standardized data stream;

[0077] An edge-cloud collaborative processing module comprises:

[0078] ​​​Edge nodes are used to receive the standardized data stream, execute local task triggering, local feature extraction and preliminary calculation based on an event-driven mechanism, and generate edge processing results.

[0079] The cloud platform is used to receive the edge processing results and global semantic information, perform global state estimation, optimized scheduling calculation and predictive analysis, and generate global decision results; the edge nodes and the cloud platform dynamically adjust the task allocation ratio through a latency monitoring and prediction mechanism, and migrate some tasks to the edge nodes for execution when the latency exceeds the threshold.

[0080] The data quality enhancement module is used to assess the quality of the sampled data and scheduling feedback data transmitted through the edge-cloud collaborative processing steps, establish a data confidence model, perform detection, interpolation repair and drift compensation operations on abnormal data, and output a high-quality source-network-load-storage collaborative dataset with confidence labels.

[0081] The uncertainty quantification and collaborative computing module is used to establish a probability distribution model of source, network, load and storage data based on the high-quality dataset, calculate uncertainty weights, perform multi-layer feature fusion and principal component dimensionality reduction analysis, generate a weighted confidence matrix, and input the weighted confidence matrix into the optimization scheduling algorithm to improve system robustness.

[0082] Compared with the prior art, the present invention has the following technical effects:

[0083] By registering and collecting self-describing information from devices on the generation side, energy storage side, distribution network, and load side, and combining protocol conversion units and semantic mapping units, adaptive access and data standardization of heterogeneous devices with multiple protocols are achieved, solving the coordination problems caused by inconsistent device protocols, data silos, and interface differences in the existing system.

[0084] Edge nodes and the cloud platform combine local rapid response and global optimized scheduling through real-time task allocation, event-driven mechanisms, and time-latency-aware communication channels, thereby improving the real-time performance of critical control tasks and the overall system scheduling efficiency.

[0085] Anomaly detection, interpolation repair, drift compensation, and confidence labeling are performed on sampled data and scheduling feedback data to form a high-quality collaborative dataset. This provides reliable data support for cloud-based optimized scheduling and reduces the impact of data errors on scheduling accuracy. Uncertainty weights are calculated based on probability distribution models of source, network, load, and storage data. Multi-layer feature fusion and dimensionality reduction analysis are performed through a collaborative data processing engine, incorporating uncertainty information into a weighted confidence matrix. This provides a reference for optimized scheduling and improves the system's robustness and reliability under conditions of renewable energy fluctuations and load changes. Attached Figure Description

[0086] Figure 1This is a flowchart of the method of the present invention. Detailed Implementation

[0087] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0088] like Figure 1 The collaborative data processing method based on source-grid-load-storage integration shown includes the following steps:

[0089] S1. Register the generation-side equipment, energy storage-side equipment, distribution network equipment, and load-side equipment, and obtain the self-description information of each equipment; automatically identify the communication protocol type of the equipment based on the self-description information, and load the driver module corresponding to the communication protocol type; unify the data format and data tags of different protocols through the protocol conversion unit and semantic mapping unit, generate a standardized data stream, and input the standardized data stream to the edge node for subsequent processing; realize unified access of generation-side, energy storage-side, distribution network, and load-side equipment, automatically identify the equipment communication protocol, and solve the difficulty of accessing multi-protocol equipment; through protocol conversion and semantic mapping, standardize the data of different protocols, and provide a unified data foundation for subsequent edge and cloud processing.

[0090] S2. Edge nodes perform preliminary calculations and rapid response processing on the standardized data stream to generate edge processing results. These results are then uploaded to the cloud platform, where global state estimation, predictive analysis, and optimized scheduling calculations are performed. The cloud platform issues collaborative scheduling instructions to the edge nodes based on the calculation results, and the edge nodes execute local control according to the received instructions. Data consistency between the edge nodes and the cloud platform is maintained through a latency-aware communication channel, and the task scheduling module dynamically allocates task execution weights between the edge nodes and the cloud platform based on real-time load. When communication is interrupted, the edge nodes invoke a preset strategy library for autonomous control, and synchronize the autonomous operation data to the cloud platform after communication is restored. By performing local feature extraction and real-time calculations on the standardized data stream at the edge nodes, while the cloud platform performs global state estimation, predictive analysis, and optimized scheduling, the collaboration between rapid edge response and global optimization decision-making is achieved, improving system operating efficiency and scheduling accuracy, and ensuring data consistency and task continuity.

[0091] S3. Based on the collaborative computing process between the edge node and the cloud platform, task priorities are set for task types including frequency adjustment tasks, energy balance tasks, and predictive analysis tasks; a latency monitoring model is used to monitor the communication link latency between the edge node and the cloud platform in real time, and the edge and cloud task allocation ratio of the task scheduling module is dynamically adjusted according to the monitoring results; the standardized data stream and the scheduling instructions are transmitted with low latency through a message queue buffer module and a lightweight data compression algorithm; by setting priorities for task types, monitoring communication link latency, and dynamically adjusting the edge and cloud task allocation ratio, while using message queues and data compression technology, low-latency transmission of key data and scheduling instructions is achieved, thereby ensuring timely response to key tasks and improving the scheduling reliability of the system.

[0092] S4. The quality of the sampled data and scheduling feedback data transmitted through the aforementioned latency guarantee and task priority scheduling steps is assessed; a data confidence model is established, and abnormal data is detected, interpolated, repaired, and compensated for drift; the confidence interval for each data point is calculated based on historical statistical characteristics and model residuals, and a high-quality source-network-load-storage collaborative dataset with confidence labels is output. This high-quality dataset is fed back to the cloud platform for subsequent predictive analysis and scheduling optimization. By assessing the quality of transmitted data and scheduling feedback data, detecting anomalies, repairing them, and compensating for drift, a confidence-labeled dataset is established, providing a high-quality data foundation for subsequent predictive analysis and optimized scheduling, reducing the impact of data errors on scheduling decisions, and improving system reliability.

[0093] S5. Based on the high-quality source-grid-load-storage collaborative dataset, establish probability distribution models for source data, grid data, load data, and storage data respectively, and calculate uncertainty weights. Combine the uncertainty weights with the confidence index to construct a weighted confidence matrix. Use a collaborative data processing engine to perform fusion calculations, feature extraction, and dimensionality reduction analysis on the high-quality dataset, and provide the weighted confidence matrix to the cloud platform's optimization scheduling module to form robust scheduling decisions. By establishing probability distribution models for source, grid, load, and storage data to calculate uncertainty weights, and combining them with confidence scores to construct a weighted matrix, the collaborative data processing engine achieves multi-layer feature fusion and dimensionality reduction analysis, providing quantified uncertainty information for optimized scheduling and improving the robustness of scheduling decisions and system stability.

[0094] Specifically, data sample values ​​from different devices The data is parsed according to device type, communication protocol, and data interface format to generate a data vector. It enables unified parsing of raw data from multi-source heterogeneous devices, converting data from different devices into a processable vector form, thus providing a foundation for subsequent unified semantic mapping and standardized processing.

[0095] Assign a uniform semantic label to each element of the data vector. The mapping function is defined as follows:

[0096]

[0097] in, For equipment type, For communication protocol type, It provides a predefined semantic mapping function to map device data of different types and protocols to a unified semantic space, achieving consistency and comparability of various types of data such as source, network, load, and storage at the semantic level, and providing a unified identifier for standardized processing and subsequent collaborative computing.

[0098] Data vectors based on unified labels Perform format conversion to generate standardized vectors. ;

[0099] in, It provides format conversion functions, including data type normalization, unit unification, and missing value filling; ensuring consistency in data type, unit, and integrity across different devices, forming a unified standard data format, and providing reliable input for real-time computing and scheduling optimization at the edge and cloud.

[0100] Standardized vectors for all devices The data is merged according to time series and device identifier to generate a standardized data stream. Where N is the total number of access devices and t is the time index; standardized data from all access devices are integrated to form a unified data stream, providing continuous and complete data input for collaborative processing between edge nodes and the cloud platform.

[0101] The standardized data stream is provided to the edge and cloud collaborative hierarchical processing steps for subsequent real-time computation and optimized scheduling; the unified and standardized data is sent into the hierarchical processing system to realize real-time utilization and optimized scheduling of multi-source data, providing basic data support for the collaborative operation of the integrated source-grid-load-storage system.

[0102] Specifically, the standardized vector Mapping to a unified semantic space Generate semantic feature vectors :

[0103]

[0104] in, It is a semantic feature mapping function; it maps standardized data from different devices to a unified semantic space, achieving consistency and comparability of source, network, load, and storage data at the semantic level, and providing a unified feature foundation for subsequent global computation and collaborative scheduling.

[0105] Semantic feature vectors from different devices are aggregated according to time series and device type to generate a global semantic matrix. Where N is the total number of connected devices and t is the time index, the data of various types are kept aligned at the source, network, load and storage levels; the characteristics of various devices are aligned to form a unified matrix, realizing the synchronization and comparability of data at the source, network, load and storage levels, and providing a complete global view for collaborative computing at the edge and cloud.

[0106] The edge node on the global semantic matrix Perform local feature extraction and preliminary calculations to generate edge processing results. ;

[0107] in, It provides edge computing functions, including key control signal extraction, real-time anomaly detection, and preliminary prediction; it rapidly processes key features at edge nodes, enabling real-time response to local control signals and preliminary anomaly analysis, providing preliminary information for global scheduling in the cloud.

[0108] The edge processing result Uploaded to the cloud platform, the cloud platform performs global state estimation, optimized scheduling calculations, and predictive analysis to generate a global decision matrix. ;

[0109] in, It provides global computing functions in the cloud, including global optimization scheduling and multi-device coordination decision-making; it performs global optimization scheduling and predictive analysis on the entire system, realizes coordinated control of all levels of source, grid, load and storage, and improves system operating efficiency and scheduling accuracy.

[0110] According to the global decision matrix The system issues scheduling instructions to the edge nodes to perform collaborative control and autonomous scheduling between the edge nodes and the cloud platform; it enables real-time collaborative operation between the edge nodes and the cloud platform, ensures timely execution of critical tasks, and achieves autonomous scheduling when communication or computing load changes, thereby improving the robustness and flexibility of the system.

[0111] Specifically, the edge nodes, based on an event-driven mechanism, process the semantic feature vector. and the edge processing results The system detects key control signals and generates a local task queue when trigger conditions are met. ;

[0112] in, This is the event triggering function. To trigger thresholds; to achieve rapid response to critical events, the computation and control tasks of edge nodes are triggered in real time, ensuring the immediacy and agility of local control.

[0113] The cloud platform uses a global task scheduler to process edge processing results uploaded by the edge nodes. and global semantic matrix and real-time network latency Dynamically calculate task allocation ratio :

[0114]

[0115] in, Here is the function for calculating task weights, where N is the total number of connected devices. This is a weighting coefficient; based on real-time network conditions and task priorities, the task ratio between edge nodes and the cloud platform is dynamically allocated to improve resource utilization efficiency and ensure the response speed of critical tasks.

[0116] When the communication link is delayed Exceeding the set threshold At that time, the cloud platform will handle part of the computing tasks. Automatic migration to the edge node for execution:

[0117]

[0118] in, As an indicator function, the edge node executes the migrated task to ensure the real-time performance of the control signal; when the network latency exceeds the threshold, critical tasks are delegating to the edge node for execution to ensure the system's rapid response and real-time control capabilities to emergencies.

[0119] After the edge node completes the migration task, it will send the task execution result. Compared with the original edge processing result The data is merged and uploaded to the cloud platform to update the global decision matrix.

[0120]

[0121] This enables the fusion of computational results from the edge and the cloud, ensuring that the global decision matrix contains the latest local execution information, thereby improving the accuracy of global scheduling and the robustness of the system.

[0122] Specifically, the edge node and the cloud platform use the latency monitoring model to monitor the communication link latency. Perform real-time assessment and generate latency prediction values. :

[0123]

[0124] in, For delay prediction function, It determines the length of the historical latency observation window; it assesses and predicts network latency in real time, providing a basis for task scheduling and ensuring that critical tasks can still be executed on time under network fluctuations.

[0125] Assign priorities to each task based on its type. With weighting coefficients :

[0126] Adjust task weights based on predicted delays to prioritize critical tasks when delays may exceed thresholds, thereby improving scheduling reliability.

[0127] Where γ is the task type weight coefficient, The task latency threshold, the From the link delay prediction step;

[0128] The edge nodes and the cloud platform use lightweight message queues to cache task data. and the decision matrix and apply data compression functions. For fast transmission:

[0129]

[0130] in, To transmit data packets; improve the transmission efficiency of critical data and scheduling instructions, and ensure the real-time collaboration between the edge and the cloud.

[0131] The global task scheduler determines the predicted delay based on the global task scheduler. With task weight Dynamically adjust the task allocation ratio This reduces control risks caused by latency exceeding thresholds. By dynamically allocating task execution weights, edge and cloud collaborative optimization is achieved, improving the system's adaptability to network fluctuations and latency changes, and ensuring the continuity and stability of critical control tasks.

[0132] Specifically, the sampled data transmitted through the aforementioned latency guarantee and task priority scheduling steps... and scheduling feedback data Perform a quality assessment and calculate the initial confidence level for each data point. :

[0133]

[0134] in, To transmit data points, and These are the historical statistical mean and standard deviation, respectively. To prevent zero-value constants, the reliability of each data point is quantified to provide a basis for subsequent anomaly detection and data repair.

[0135] Interpolation repair and drift compensation are performed on outlier data points to generate repaired data. :

[0136]

[0137] in, It is a polynomial interpolation function. It sets an abnormal threshold; it repairs abnormal or missing data, ensures the continuity and accuracy of input data, and provides high-quality data for collaborative computing.

[0138] Calculate the confidence interval for each data point based on the repaired data and model residuals. :

[0139]

[0140] in, The standard deviation of the residuals. It provides the standard normal quantile corresponding to the confidence level; and provides a quantified uncertainty range for each data point to support subsequent weighted collaborative scheduling and risk assessment.

[0141] Data repair Combined with confidence interval labels, this generates a high-quality dataset with confidence level labels. :

[0142]

[0143] Where M is the total number of data points.

[0144] The high-quality dataset The data is fed back to the cloud platform for predictive analysis and optimized scheduling by the cloud-based global computing function, forming a closed-loop control and optimization decision-making process. This provides a reliable, high-confidence data foundation, making cloud-based global scheduling and predictive analysis more robust, and enabling optimized control and closed-loop collaborative operation of the source-grid-load-storage system.

[0145] Specifically, based on the aforementioned high-quality dataset Probability distribution models are established for source data, network data, load data, and storage data respectively, generating data probability vectors. :

[0146]

[0147] in, It serves as a probability distribution estimation function; it quantifies the uncertainty of each data point, providing probabilistic information for collaborative scheduling and risk control.

[0148] According to the probability distribution vector Calculate the uncertainty weight for each data point :

[0149]

[0150] in, It represents the maximum value of the probability distribution of data points, reflecting the confidence level; it transforms the uncertainty of the data into a value that can be used for weighting, providing a quantitative basis for subsequent feature fusion and principal component analysis.

[0151] The collaborative data processing engine performs multi-layer feature fusion and principal component dimensionality reduction analysis on the data to generate a fused feature matrix. :

[0152]

[0153] in, Indicates weighting by uncertainty Weighted operation, where N is the total number of devices. Principal component analysis (PCA) functions are used to extract core information from multi-level features, reduce redundant data, improve scheduling decision efficiency, and incorporate uncertainty information into feature representation.

[0154] The fused feature matrix With uncertainty weight matrix Combined, a weighted confidence matrix is ​​formed. We construct a weighted matrix that comprehensively considers data uncertainty and feature fusion to provide input for global computing in the cloud and enhance robustness during scheduling optimization.

[0155]

[0156] The weighted confidence matrix The data is input into the cloud-based global computation function to adjust the global decision matrix. By comprehensively considering data confidence and uncertainty in global scheduling, collaborative optimization of the source-grid-load-storage system can be achieved, thereby improving the stability and reliability of system operation.

[0157] A collaborative data processing system based on integrated source-grid-load-storage, the system employing the aforementioned method, the system comprising:

[0158] The device access module is used to register generation-side devices, energy storage-side devices, distribution network devices, and load-side devices, and to obtain self-description information for each device; the device access module is also used to automatically identify the communication protocol type of the device based on the self-description information, and to load the driver module corresponding to the communication protocol type.

[0159] The protocol conversion and semantic mapping unit is used to map the data of the access device through a unified semantic model to feature tags, realize unified modeling and format standardization of various types of data such as source, network, load and storage at the semantic level, and generate standardized data streams;

[0160] The edge-cloud collaborative processing module includes:

[0161] Edge nodes are used to receive the standardized data stream, execute local task triggering, local feature extraction and preliminary calculation based on an event-driven mechanism, and generate edge processing results.

[0162] The cloud platform is used to receive the edge processing results and global semantic information, perform global state estimation, optimized scheduling calculation and predictive analysis, and generate global decision results; the edge nodes and the cloud platform dynamically adjust the task allocation ratio through a latency monitoring and prediction mechanism, and migrate some tasks to the edge nodes for execution when the latency exceeds the threshold.

[0163] The data quality enhancement module is used to assess the quality of the sampled data and scheduling feedback data transmitted through the edge-cloud collaborative processing steps, establish a data confidence model, perform detection, interpolation repair and drift compensation operations on abnormal data, and output a high-quality source-network-load-storage collaborative dataset with confidence labels.

[0164] The uncertainty quantification and collaborative computing module is used to establish a probability distribution model of source, network, load and storage data based on the high-quality dataset, calculate uncertainty weights, perform multi-layer feature fusion and principal component dimensionality reduction analysis, generate a weighted confidence matrix, and input the weighted confidence matrix into the optimization scheduling algorithm to improve system robustness.

[0165] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for collaborative data processing based on source-network-payload integration, characterized in that, The method comprises the following steps: S1, register the power generation side equipment, energy storage side equipment, power distribution network equipment and load side equipment, and obtain the self-description information of each equipment; automatically identify the communication protocol type of the equipment according to the self-description information, load the driving module corresponding to the communication protocol type; unify the data format and data label of different protocols through the protocol conversion unit and the semantic mapping unit, generate standardized data stream, and input the standardized data stream to the edge node for subsequent processing; S2, the edge node performs preliminary calculation and rapid response processing on the standardized data stream to generate edge processing results; upload the edge processing results to the cloud platform, and perform global state estimation, prediction analysis and optimization scheduling calculation by the cloud platform; the cloud platform issues a cooperative scheduling instruction to the edge node according to the calculation result, and the edge node executes local control according to the received instruction; the data consistency between the edge node and the cloud platform is maintained through the time delay perception communication channel, and the task execution weight of the edge node and the cloud platform is dynamically allocated according to the real-time load by using the task scheduling module; when the communication is interrupted, the edge node calls the preset strategy library for autonomous control, and synchronizes the autonomous running data to the cloud platform after the communication is restored; S3, based on the cooperative calculation process of the edge node and the cloud platform, set the task priority for the task type including frequency adjustment task, energy balance task and prediction analysis task; use the time delay monitoring model to monitor the communication link delay between the edge node and the cloud platform in real time, and dynamically adjust the edge and cloud task allocation ratio of the task scheduling module according to the monitoring result; the standardized data stream and the scheduling instruction are transmitted through the message queue buffer module and the lightweight data compression algorithm with low delay; S4, quality evaluation is performed on the sampling data and scheduling feedback data transmitted through the time delay guarantee and task priority scheduling steps; a data confidence model is established to detect, interpolate and repair and drift compensation operation is performed on abnormal data; based on historical statistical characteristics and model residual error, the confidence interval of each data point is calculated, and a high-quality source network load storage cooperative data set with confidence label is output, which is fed back to the cloud platform for subsequent prediction analysis and scheduling optimization; S5, based on the high-quality source network load storage cooperative data set, probability distribution models are established for source data, network data, load data and storage data respectively, and uncertainty weight is calculated; the uncertainty weight and the confidence index are used to construct a weighted confidence matrix; the high-quality data set is fused, calculated, feature extracted and dimensionality reduced by the cooperative data processing engine, and the weighted confidence matrix is provided to the optimization scheduling module of the cloud platform to form a robust enhanced scheduling decision.

2. The method of claim 1, wherein, Sampling values of data from different devices Resolving by device type, communication protocol and data interface format, generating data vectors ; Assigning uniform semantic labels to elements of each data vector where the mapping function is defined as: ； wherein, is a device type, is a communication protocol type, is a predefined semantic mapping function; Converting data vectors according to uniform tags Performing format conversion to generate standardized vectors ; wherein, is a format conversion function, including data type normalization, unit unification, and missing value filling; Standardized vector of all devices Merge by time series and device identification, generate standardized data stream Where N is the total number of access devices, t is the time index The standardized data stream is provided to the edge and cloud cooperative hierarchical processing step for subsequent real-time calculation and optimization scheduling.

3. The method of claim 2, wherein, mapping the standardized vectors to a unified semantic space , generating semantic feature vectors : ； wherein, is a semantic feature mapping function; The semantic feature vectors from different devices are aggregated according to time sequences and device types to generate a global semantic matrix Wherein, N is the total number of access devices, t is the time index, so that various data are kept aligned at the source, network, load and reservoir levels. at the edge node, the global semantic matrix is divided into a plurality of sub-matrices performing local feature extraction and preliminary calculation to generate edge processing results ; wherein, are edge computing functions, including key control signal extraction, real-time anomaly detection, and preliminary prediction; The edge processing result is uploaded to a cloud platform, global state estimation, optimization scheduling calculation and prediction analysis are performed by the cloud platform, and a global decision matrix is generated The edge processing result is uploaded to a cloud platform, global state estimation, optimization scheduling calculation and prediction analysis are performed by the cloud platform, and a global decision matrix is generated ; wherein, is a cloud global computing function, including global optimization scheduling and multi-device coordination decision; According to the global decision matrix , a scheduling instruction is issued to the edge node, and the edge node and the cloud platform are cooperatively controlled and autonomously scheduled.

4. The method of claim 3, wherein, The edge nodes are based on an event-driven mechanism for the semantic feature vectors. and the edge processing results The system detects key control signals and generates a local task queue when trigger conditions are met. ; wherein, is an event trigger function, is a trigger threshold; The cloud platform calculates a global task scheduler according to the edge processing results uploaded by the edge nodes and a global semantic matrix and real-time network delay dynamically calculates a task allocation ratio : ； wherein, is a task weight calculation function, N is the total number of access devices, is a weight coefficient; When the communication link delay exceeds a set threshold the cloud platform automatically migrates part of the computing task to the edge node for execution: ； wherein, is an indicator function, the edge node performing the migrated task to guarantee real-time of the control signal; The edge node completes the migration task, and uploads the task execution result to the cloud platform and the original edge processing result fusion, and uploads to the cloud platform to update the global decision matrix : 。 5. The method of claim 4, wherein, The edge node performs real-time evaluation of the communication link delay through the latency monitoring model and generates a latency prediction value : ； wherein, is a latency prediction function, is a history latency observation window length; Assigning a priority to each task according to the task type with the weight coefficient : ； wherein γ is a task type weight coefficient, is a task delay threshold, the from the link delay prediction step; The edge nodes and the cloud platform use a lightweight message queue to cache task data and the decision matrix and apply a data compression function for fast transmission: ； wherein, to transmit the data packet; The global task scheduler adjusts the task allocation proportion according to the predicted time delay with task weight dynamically adjusts the task allocation proportion , reduces the control risk caused by time delay exceeding threshold.

6. The method of claim 5, wherein, sampled data transmitted via the latency guaranteed and task priority scheduling steps and scheduling feedback data perform quality assessment, calculate initial confidence of each data point : wherein, is the transmitted data point, and are the historical statistical mean and standard deviation, respectively, is a guard zero constant; interpolation repair and drift compensation are performed on the abnormal data points to generate repaired data : ； wherein is a polynomial interpolation function, is an anomaly threshold; calculating a confidence interval for each data point based on the repaired data and the model residual : ； wherein, is the residual standard deviation, is the standard normal quantile corresponding to the confidence level; repair data combining the confidence interval tags, generating a high-quality dataset with confidence tags : ； Wherein, M is the total number of data points; The high-quality data set are fed back to the cloud platform for prediction analysis and optimal scheduling by the cloud global computing function, forming a closed-loop control and optimal decision.

7. The method of claim 6, wherein, based on the high-quality data set , respectively, to generate data probability vectors : ； wherein, is a probability distribution estimation function; According to the probability distribution vector Computing an uncertainty weight for each data point : ； wherein, represents the maximum value of the data point probability distribution, reflecting the confidence level; The synergistic data processing engine performs multi-layer feature fusion and principal component dimension reduction analysis on the data to generate a fused feature matrix : ； wherein, denotes the uncertainty weight the weighting operation, N is the total number of devices, is a principal component analysis function; combining the fusion feature matrix with an uncertainty weight matrix to form a weighted confidence matrix : ； The weighted confidence matrix is inputted to the cloud global computing function for adjusting the global decision matrix .

8. A collaborative data processing system based on source, network, and payload integration, comprising: The system uses the method of any one of claims 1-7, and the system comprises: A device access module is configured to register power generation side devices, energy storage side devices, power distribution network devices, and load side devices, and obtain self-description information of each device. The device access module is further configured to automatically identify the communication protocol type of the device according to the self-description information, and load a driving module corresponding to the communication protocol type. A protocol conversion and semantic mapping unit is configured to map data of the access device through a unified semantic model to realize unified modeling and format standardization of source, network, load, and storage data at a semantic level, and generate a standardized data stream. An edge-cloud collaborative processing module includes: An edge node is configured to receive the standardized data stream, perform local task triggering, local feature extraction, and preliminary calculation based on an event-driven mechanism, and generate edge processing results. A cloud platform is configured to receive the edge processing results and global semantic information, perform global state estimation, optimization scheduling calculation, and prediction analysis, and generate global decision results. The edge node and the cloud platform dynamically adjust the task allocation ratio through a time delay monitoring and prediction mechanism, and migrate part of the tasks to the edge node for execution when the time delay exceeds a threshold. A data quality enhancement module is configured to perform quality evaluation on sampling data and scheduling feedback data transmitted through the edge-cloud collaborative processing step, establish a data confidence model, perform detection, interpolation repair, and drift compensation operations on abnormal data, and output a high-quality source, network, load, and storage collaborative data set with a confidence label. An uncertainty quantification and collaborative calculation module is configured to establish a probability distribution model of source, network, load, and storage data based on the high-quality data set, calculate an uncertainty weight, perform multi-layer feature fusion and principal component dimensionality reduction analysis, generate a weighted confidence matrix, and input the weighted confidence matrix into an optimization scheduling algorithm to improve system robustness.

9. A system comprising a memory, a processor, and a program stored in the memory, wherein, The processor executes the program to implement the method of any one of claims 1-7.

10. An electronic device, comprising: It includes: A processor; A memory for storing processor-executable instructions; Wherein the processor is configured to call the instructions stored in the memory to execute the method of any one of claims 1 to 7.

Images