A distributed heterogeneous data processing method and system applied to a smart grid

Abstract

The application discloses a kind of distributed heterogeneous data processing method and system applied to smart grid, it is related to data processing technical field.In method, the heterogeneous data of distributed node is obtained and is converted into general data stream, and the multi-level semantic anchor point of distributed node is generated based on topological information multi-level clustering.Extract the time feature and space feature of general data stream, and obtain spatiotemporal joint feature by fusion.Then, forward consistency distillation is carried out, and the spatiotemporal joint feature corresponding to the semantic anchor point is mapped to the shared semantic space to obtain the mapping feature vector, and the multi-level feature similarity is calculated using the multi-head attention mechanism.Then, reverse consistency distillation is carried out, and the reverse consistency check parameter is obtained by using the reversible check network to perform reverse consistency check on the mapping feature vector.Finally, according to the feature similarity and the reverse check parameter, the heterogeneous data is processed by selecting the synchronization strategy.The realization of data processing spans from bit level to semantic level, and multi-granularity consistency check is realized.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a distributed heterogeneous data processing method and system for smart grids. Background Technology

[0002] In the field of distributed heterogeneous data processing, due to issues such as heterogeneous node hardware, dynamic changes in network topology, and uneven data distribution, existing technologies only guarantee the consistency of the bitstream of the original data and cannot identify differences in the semantic understanding of the data among different nodes. Traditional data processing methods struggle to achieve efficient and consistent data synchronization. Moreover, existing solutions typically employ centralized processing or simple distributed consensus protocols (such as Paxos or Raft), but these methods have significant drawbacks when dealing with large-scale heterogeneous nodes: on the one hand, centralized architectures are prone to becoming performance bottlenecks and have a high risk of single-point failures; on the other hand, simple consensus protocols cannot effectively handle semantic differences between heterogeneous nodes, leading to a decrease in data accuracy. In recent years, deep learning-based distributed data processing methods have gradually emerged, but most of these methods focus on single-modal feature extraction, neglecting semantic-level processing and lacking the ability to achieve multi-level semantic consistency. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, embodiments of this application provide a distributed heterogeneous data processing method and system applied to smart grids, which can realize the leap from bit-level to semantic level data processing and achieve multi-granularity consistency verification.

[0004] In a first aspect, embodiments of this application provide a distributed heterogeneous data processing method applied to smart grids, including:

[0005] Acquire heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a general data stream; wherein the distributed nodes are distributed sensor nodes;

[0006] Based on the topological information of the multiple distributed nodes, multi-level clustering is performed on the multiple distributed nodes to generate multi-level semantic anchors.

[0007] Spatiotemporal features are extracted from the general data stream to obtain temporal features and spatial features, and the temporal features and spatial features are jointly encoded to obtain spatiotemporal joint features;

[0008] A forward consistency distillation is performed to map the spatiotemporal joint features corresponding to the semantic anchors to the shared semantic space to obtain a mapped feature vector, and a multi-head attention mechanism is used to calculate multi-level feature similarity based on the mapped feature vector;

[0009] Perform reverse consistency distillation by using a preset reversibility verification network to perform reverse consistency verification on the mapped feature vector to obtain reverse verification parameters.

[0010] Based on the feature similarity and the reverse verification parameters, a corresponding synchronization strategy is selected to process the heterogeneous data.

[0011] In some embodiments of this application, the topology information includes geographic location information, communication latency parameters, and computing power indicators; the semantic anchors include physical layer anchors, regional layer anchors, and global layer anchors; the step of performing multi-level clustering of multiple distributed nodes based on the topology information of multiple distributed nodes to generate multi-level semantic anchors includes:

[0012] Based on the geographic location information, the communication delay parameters, and the computing power indicators, the distributed nodes are clustered using a clustering algorithm to generate the physical layer anchor points;

[0013] Based on the geographic location information, the communication delay parameters, and the computing power indicators, the physical layer anchor points are clustered using a hierarchical clustering algorithm to generate the regional layer anchor points;

[0014] The network topology of the regional layer anchor points is aggregated using a graph aggregation algorithm to generate the global layer anchor points.

[0015] In some embodiments of this application, the step of extracting temporal and spatial features from the general data stream to obtain temporal and spatial features, and jointly encoding the temporal and spatial features to obtain spatiotemporal joint features, includes:

[0016] The general data stream is modeled using a gated loop unit, and a time window is set to obtain the temporal characteristics of data changes in the time series.

[0017] Spatial features are extracted from the general data stream using a deformable convolutional neural network, and spatial features of data changes are obtained based on learnable offsets.

[0018] The spatiotemporal joint features are obtained by cross-modal fusion of the temporal and spatial features based on a multi-head self-attention mechanism.

[0019] In some embodiments of this application, the step of mapping the spatiotemporal joint features corresponding to the semantic anchor to a shared semantic space to obtain a mapped feature vector, and calculating multi-level feature similarity based on the mapped feature vector using a multi-head attention mechanism, includes:

[0020] The spatiotemporal joint features of the physical layer anchor, the regional layer anchor, and the global layer anchor are respectively mapped to the shared semantic space to obtain the corresponding mapping feature vectors of a preset dimension;

[0021] A first similarity is obtained by using a multi-head attention mechanism to calculate the cosine similarity between the mapping feature vectors of the physical layer anchor points and the mapping feature vectors of the regional layer anchor points.

[0022] A second similarity is obtained by calculating the cosine similarity between the mapped feature vectors of the regional layer anchor points and the mapped feature vectors of the global layer anchor points using a multi-head attention mechanism.

[0023] The first similarity and the second similarity are fused to obtain the feature similarity.

[0024] In some embodiments of this application, the reversibility verification network includes a generator and a discriminator; the step of using a preset reversibility verification network to perform reverse consistency verification on the mapped feature vector to obtain reverse verification parameters includes:

[0025] The generator is used to perform an encoding-decoding transformation mapping on the mapped feature vector to obtain the reconstructed features;

[0026] The difference index between the reconstructed features and the spatiotemporal joint features is calculated to obtain the reconstruction error;

[0027] The discriminator is used to determine the authenticity of the reconstructed features, and discrimination parameters are obtained.

[0028] The reverse verification parameters are obtained based on the reconstruction error and the discrimination parameters.

[0029] In some embodiments of this application, the method further includes:

[0030] A predetermined number of historical synchronization strategy records are stored using a consistent memory buffer to obtain multiple decision storage items; wherein, the decision storage item is a quadruple, including a state vector, action space, immediate reward, and next state;

[0031] The decision storage item is optimized using a Q-learning decision maker. A synchronous policy in the action space is selected and executed according to the state vector, and the immediate reward is calculated accordingly. The next state is then obtained and updated to the consistent memory buffer.

[0032] Based on the optimized decision storage items, the hyperparameters of the Q-learning decision engine are dynamically adjusted to obtain a dynamic arbitration engine.

[0033] In some embodiments of this application, selecting a corresponding synchronization strategy to process the heterogeneous data based on the feature similarity and the reverse verification parameter includes:

[0034] A state vector is constructed based on the feature similarity and the reverse verification parameters;

[0035] The dynamic arbitration engine selects and executes a synchronization strategy in the action space based on the state vector; wherein the synchronization strategy includes a forced synchronization strategy, an asynchronous compensation strategy, and a partial rollback strategy.

[0036] When the forced synchronization strategy is selected, all distributed nodes across the network are forced to synchronize and update all heterogeneous data.

[0037] When selecting the asynchronous compensation strategy, an incremental update mechanism is used to update the heterogeneous data that has changed in the distributed nodes;

[0038] When the local rollback strategy is selected, abnormal distributed nodes are identified and rolled back to the previous stable state, triggering the anomaly diagnosis mechanism to repair the abnormal heterogeneous data.

[0039] Secondly, embodiments of this application also provide a distributed heterogeneous data processing system for smart grids, comprising:

[0040] An acquisition module is used to acquire heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a general data stream; wherein, the distributed nodes are distributed sensor nodes;

[0041] The clustering module is used to perform multi-level clustering of the distributed nodes based on the topological information of the distributed nodes, and generate multi-level semantic anchors.

[0042] The feature extraction module is used to extract spatiotemporal features from the general data stream to obtain temporal features and spatial features, and to jointly encode the temporal features and spatial features to obtain spatiotemporal joint features;

[0043] The forward distillation module is used to perform forward consistency distillation, which maps the spatiotemporal joint features corresponding to the semantic anchors to the shared semantic space to obtain the mapped feature vector, and uses a multi-head attention mechanism to calculate multi-level feature similarity based on the mapped feature vector;

[0044] The reverse distillation module is used to perform reverse consistency distillation. It uses a preset reversibility verification network to perform reverse consistency verification on the mapped feature vector to obtain reverse verification parameters.

[0045] The processing module is used to select a corresponding synchronization strategy to process the heterogeneous data based on the feature similarity and the reverse verification parameters.

[0046] Thirdly, embodiments of this application also provide an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the distributed heterogeneous data processing method for smart grids as described in the first aspect of this application.

[0047] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a program that is executed by a processor to implement the distributed heterogeneous data processing method for smart grids as described in the first aspect of this application.

[0048] The embodiments of this application include at least the following beneficial effects:

[0049] This application provides a method and system for processing distributed heterogeneous data in a smart grid. The method involves acquiring heterogeneous data from multiple distributed nodes and converting it into a general data stream. Then, based on the topology information of the distributed nodes, multi-level clustering is performed to generate semantic anchors corresponding to each level. Spatiotemporal features are extracted from the general data stream to obtain temporal and spatial features, which are then jointly encoded to obtain spatiotemporal joint features. Forward consistency distillation is then performed, including mapping the spatiotemporal joint features corresponding to the semantic anchors to a shared semantic space to obtain mapped feature vectors, and using a multi-head attention mechanism to calculate multi-level feature similarities based on the mapped feature vectors. Reverse consistency distillation is then performed, including using a pre-defined reversibility verification network to perform reverse consistency verification on the mapped feature vectors to obtain reverse verification parameters. Finally, based on the feature similarity and reverse verification parameters, a corresponding synchronization strategy is selected to process the heterogeneous data. This constructs a multi-level semantic anchor system that not only captures the direct physical layer correlations of heterogeneous data but also delves into their deep semantic connections, maintaining semantic consistency across nodes and levels in a heterogeneous environment. The multi-granularity consistency verification mechanism primarily achieves deep data fusion across time and space dimensions through forward consistency distillation, while reverse consistency distillation employs a reversible verification network for bidirectional verification, ensuring comprehensive consistency verification from micro-bit changes to macro-semantic expressions. This bidirectional, multi-granularity processing paradigm not only significantly improves the accuracy and reliability of heterogeneous data processing but also enables a leap in data processing from the bit-level to the semantic level, achieving multi-granularity consistency verification.

[0050] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0051] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0052] Figure 1 This is a flowchart illustrating a distributed heterogeneous data processing method for smart grids provided in one embodiment of this application;

[0053] Figure 2 yes Figure 1 A flowchart illustrating step S102;

[0054] Figure 3 yes Figure 1 A flowchart illustrating step S103;

[0055] Figure 4 yes Figure 1 A flowchart illustrating step S104;

[0056] Figure 5 yes Figure 1 A flowchart illustrating step S105;

[0057] Figure 6 This is a flowchart illustrating a distributed heterogeneous data processing method for smart grids provided in another embodiment of this application;

[0058] Figure 7 yes Figure 1 A flowchart illustrating step S106;

[0059] Figure 8 This is a schematic diagram of a distributed heterogeneous data processing system module for smart grids provided in one embodiment of this application;

[0060] Figure 9 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application.

[0061] Reference numerals: Acquisition module 100, Clustering module 200, Feature extraction module 300, Forward distillation module 400, Reverse distillation module 500, Processing module 600, Electronic device 1000, Processor 1001, Memory 1002. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0064] In the description of this application, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0065] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0066] In the description of this application, unless otherwise expressly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0067] In the intersection of distributed computing and machine learning, distributed heterogeneous data processing remains a core technological challenge, especially in scenarios with high consistency requirements such as financial distributed ledgers, industrial IoT, and cross-cloud database synchronization. Traditional distributed consensus algorithms (such as Paxos and Raft) primarily focus on the bit-level consistency of the original data, ensuring consistency only at the bitstream level while neglecting semantic-level understanding. This limitation leads to different semantic interpretations on different nodes in heterogeneous environments, even if the data bits are consistent, differences in computing power, data representation, or model parameters can result in different interpretations, severely impacting system reliability and decision consistency. Furthermore, while existing distributed machine learning methods alleviate data inconsistency to some extent, gradient inconsistencies still exist between parameter server architecture layers, causing a decrease in global model convergence speed or even training failure. However, with the explosive growth of data scale and the increasing heterogeneity of systems, traditional methods struggle to simultaneously meet the multiple requirements of high consistency, high fault tolerance, and high scalability.

[0068] Meanwhile, traditional data synchronization technologies often rely on centralized processing architectures or simple distributed consistency protocols, which face significant performance bottlenecks in heterogeneous node environments. Centralized systems are prone to becoming single points of failure, and their processing efficiency drops sharply as the number of nodes increases. While classic protocols such as Paxos and Raft offer high fault tolerance, they cannot properly handle semantic differences between different nodes, leading to the long-term neglect of semantic inconsistencies in data. In recent years, although deep learning-based distributed data processing technologies have made progress, they still have significant limitations—existing methods often focus only on single-modal feature extraction (such as processing only visual or textual data), lacking cross-modal semantic association modeling capabilities, and are still lacking in hierarchical semantic consistency processing. This technological status quo makes it difficult for traditional methods to achieve efficient and robust multi-granularity (bit-level, feature-level, semantic-level) consistency guarantees in dynamically changing heterogeneous network environments, especially under conditions of high network latency and limited bandwidth, further deteriorating the overall system performance. Therefore, there is an urgent need for a new solution that can transcend bit-level synchronization and achieve end-to-end semantic consistency modeling to improve the reliability and intelligent adaptability of complex distributed systems.

[0069] Based on this, this application provides a distributed heterogeneous data processing method and system for smart grids, constructing a multi-level semantic anchor system. This system not only captures the direct correlations of heterogeneous data at the physical layer but also delves into their deep semantic connections, maintaining semantic consistency across nodes and levels in a heterogeneous environment. The multi-granularity consistency verification mechanism primarily achieves deep data fusion in the temporal and spatial dimensions through forward consistency distillation, while reverse consistency distillation employs a reversible verification network for bidirectional verification, ensuring comprehensive consistency verification from micro-bit changes to macro-semantic expressions. This bidirectional, multi-granularity processing paradigm significantly improves the accuracy and reliability of heterogeneous data processing, while simultaneously enabling data processing to leap from the bit-level to the semantic level, achieving multi-granularity consistency verification.

[0070] This application provides a distributed heterogeneous data processing method and system for smart grids, which will be described in detail through the following embodiments. First, the distributed heterogeneous data processing method for smart grids in this application is described.

[0071] The distributed heterogeneous data processing method for smart grids provided in this application relates to the field of data processing technology, and more particularly to the field of distributed heterogeneous data processing technology. This method can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a local application, i.e., a program that needs to be installed in the operating system to run, such as a client supporting distributed heterogeneous data processing for smart grids, i.e., a program that only needs to be downloaded to a browser environment to run. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. This distributed heterogeneous data processing method for smart grids can be executed by the terminal or the server, or by the terminal and the server working together.

[0072] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). A database index compression system server can be installed on the server, allowing interaction with the terminal. For example, corresponding software can be installed on the server, such as applications implementing distributed heterogeneous data processing methods for smart grids, but is not limited to these forms. The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or network communication methods; this embodiment does not impose limitations.

[0073] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. The invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0074] The following describes a distributed heterogeneous data processing method for smart grids according to embodiments of the present invention.

[0075] Reference Figure 1 As shown, this application provides a distributed heterogeneous data processing method for smart grids, which may include, but is not limited to, the following steps S101 to S106.

[0076] Step S101: Obtain heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a general data stream.

[0077] In some embodiments, heterogeneous data from multiple distributed nodes is acquired. For example, in smart grid applications, heterogeneous data may originate from different sensors, including current sensors, voltage sensors, and PMU synchronization phasor measurement devices. This heterogeneous data is then converted into a common data stream. Specifically, corresponding protocol conversion modules can be deployed for different communication protocols (such as Modbus, OPC UA, REST API, etc.) to ensure unified data access. For instance, in smart grid scenarios, an OPC UA gateway can convert the IEC 61850 data format of PMU devices and voltage / current transformers into an internal common format.

[0078] It is understandable that a structured schema (such as Protocol Buffers, Avro, etc.) is used to define the logical structure of the data, ensuring that the data is parsed, cleaned, and formatted into a unified data structure before transmission. For example, measurement values ​​from different devices can be uniformly mapped to standardized data tuples containing timestamps, device IDs, measurement type, and numerical values. This allows the preprocessed data to be encapsulated into a common data stream format (such as gRPC data streams), although this embodiment does not impose any limitations on this.

[0079] Step S102: Based on the topological information of multiple distributed nodes, perform multi-level clustering on multiple distributed nodes to generate multi-level semantic anchors.

[0080] In some embodiments, clustering analysis is performed based on the topological information of distributed nodes (e.g., geographical location, communication latency, computing power, network connectivity, etc.) to divide nodes with similar attributes or communication relationships into different groups, thereby generating multi-level semantic anchors. Specifically, in smart grid applications, substation nodes with similar geographical locations are grouped together to reflect their physical distribution relationship in the power grid; in industrial IoT scenarios, devices that communicate frequently may be clustered together to reflect their correlation in data interaction. Through clustering, a more semantically consistent data association structure can be constructed, providing a hierarchical reference framework for subsequent feature extraction and consistency verification, and enhancing the collaborative processing capabilities of distributed nodes.

[0081] Understandably, multi-level semantic anchors are a hierarchical data organization structure used to establish semantic relationships between nodes in a distributed heterogeneous system. Clustering algorithms divide nodes with similar characteristics or topological relationships into groups of different levels, with each group corresponding to a semantic anchor as an abstract representation of that group of nodes. For example, in a smart grid scenario, devices within the same substation may form the first layer of semantic anchors; multiple substations in the same power supply area constitute the second layer; and key hub nodes across the entire network form the third layer. This hierarchical design preserves local details while supporting global collaborative optimization.

[0082] Step S103: Extract time and space features from the general data stream, and jointly encode the time and space features to obtain joint time and space features.

[0083] In some embodiments, spatiotemporal feature extraction is performed on general data streams to obtain temporal and spatial features. Specifically, features characterizing the dynamic properties of data are extracted based on the patterns of data change over time. For example, in power systems, fluctuations in parameters such as voltage and current over time can be captured, identifying abnormal operating conditions such as voltage dips or frequency changes lasting for several cycles, thus obtaining the corresponding temporal features. Based on the spatial distribution and correlation of data, features reflecting the spatial characteristics between nodes are extracted. For example, in substation voltage monitoring scenarios, deformable convolutional neural networks can automatically focus on specific areas of voltage waveform distortion, accurately locate the dip in phase B voltage, thereby identifying the specific phase and location of the anomaly, and obtaining the corresponding spatial features.

[0084] In some embodiments, the extracted temporal and spatial features are jointly encoded to form a unified spatiotemporal joint feature representation. The spatiotemporal joint features contain both dynamic change information in the time dimension and distribution characteristics in the spatial dimension, providing a comprehensive and accurate feature foundation for subsequent consistency distillation. Taking substation voltage anomaly detection as an example, it can simultaneously capture the spatial distortion features of the voltage waveform (such as phase B voltage dips) and temporal persistence features (such as voltage drops lasting for 5 cycles), thereby achieving accurate identification and location of abnormal operating conditions.

[0085] Step S104: Perform forward consistency distillation, map the spatiotemporal joint features corresponding to the semantic anchors to the shared semantic space to obtain the mapped feature vector, and use the multi-head attention mechanism to calculate the multi-level feature similarity based on the mapped feature vector.

[0086] In some embodiments, spatiotemporal joint features corresponding to semantic anchors at different levels are mapped to a unified shared semantic space to obtain standardized mapped feature vectors. This not only preserves the key information of the original features but also eliminates the differences in dimensions and distributions between different nodes through standardization, ensuring that features from different nodes and at different levels can be compared and aligned within the same semantic framework.

[0087] In some embodiments, a multi-head attention mechanism is used to analyze the mapped feature vectors. Specifically, to deeply capture multi-dimensional semantic consistency, this application employs a multi-head attention mechanism with four parallel attention heads, each designed to focus on semantic associations in different dimensions (e.g., spatial consistency, temporal consistency, etc.). By calculating the similarity between features at different levels, the degree of semantic consistency of nodes at each level in the shared semantic space can be quantified.

[0088] Step S105: Perform reverse consistency distillation by using a preset reversibility verification network to perform reverse consistency verification on the mapped feature vector and obtain the reverse verification parameters.

[0089] In some embodiments, a pre-defined reversibility verification network is used to perform reverse verification on the mapped feature vector obtained by forward distillation, ensuring the accuracy and integrity of the feature mapping. The reversibility verification network, based on feature reconstruction and similarity evaluation mechanisms, analyzes whether the mapped features retain the key semantic information of the original data.

[0090] Therefore, through a two-way verification mechanism, the reliability of the forward distillation results is checked, while the reversibility of the reverse mapping is ensured, thereby improving the robustness of the entire consistent distillation process. The final output reverse verification parameters provide an important reference for the selection of subsequent synchronization strategies, ensuring that a high level of data consistency can be maintained even in heterogeneous environments.

[0091] Step S106: Select the corresponding synchronization strategy to process heterogeneous data based on feature similarity and reverse verification parameters.

[0092] In some embodiments, the optimal synchronization strategy for processing heterogeneous data is intelligently selected based on the multi-level feature similarity obtained from forward consistency distillation and the reverse verification parameters. For example, if the similarity is high and the verification parameters are normal, lightweight synchronization is performed; if consistency decreases but is recoverable, bidirectional verification synchronization is triggered; if severe distortion or anomalies occur, global forced synchronization is initiated. This mechanism dynamically adapts to changes in network state, ensuring that heterogeneous data maintains high consistency while collaborating efficiently.

[0093] This leads to the construction of a multi-level semantic anchor system, which not only captures the direct physical-level correlations of heterogeneous data but also delves into their deep semantic connections, maintaining semantic consistency across nodes and levels in heterogeneous environments. The multi-granularity consistency verification mechanism primarily achieves deep data fusion across time and space dimensions through forward consistency distillation, while reverse consistency distillation employs a reversible verification network for bidirectional verification, ensuring comprehensive consistency verification from micro-level bit changes to macro-level semantic expressions. This bidirectional, multi-granularity processing paradigm significantly improves the accuracy and reliability of heterogeneous data processing, while simultaneously enabling a leap from bit-level to semantic-level data processing, achieving multi-granularity consistency verification.

[0094] Reference Figure 2 As shown, in some embodiments of this application, the topology information includes geographical location information, communication latency parameters, and computing power indicators; the semantic anchors include physical layer anchors, regional layer anchors, and global layer anchors. Step S102 above: performing multi-level clustering of multiple distributed nodes based on the topology information of multiple distributed nodes to generate multi-level semantic anchors may include, but is not limited to, the following steps S201 to S203.

[0095] Step S201: Based on geographical location information, communication delay parameters, and computing power indicators, clustering algorithms are used to cluster distributed nodes to generate physical layer anchor points.

[0096] In some embodiments, the geographical location information is obtained by using GPS to obtain the physical coordinates (such as latitude and longitude) of each distributed node, which can ensure the spatial proximity of distributed nodes within the same anchor point; the communication delay parameter is based on the measured or estimated network delay matrix, which quantifies the network transmission efficiency between nodes and ensures that the communication delay of nodes within the anchor point meets the requirements; the computing power index includes hardware performance parameters such as CPU / GPU computing power and memory capacity, so that the computing load between anchor points is as balanced as possible.

[0097] Specifically, based on geographic location information, communication latency parameters, and computing power indicators, an improved k-means++ algorithm is used to cluster distributed nodes. The number of clusters *k* is dynamically optimized using silhouette coefficients to generate physical layer anchor points, ensuring that topology partitioning simultaneously satisfies communication latency constraints and computational load balancing. Mahalanobis distance is used to comprehensively evaluate multi-dimensional features, and k-means++ selects initial cluster centers probabilistically, avoiding the pitfalls of traditional k-means in getting stuck in local optima. Each physical layer anchor point represents a group of geographically proximate, low-latency, and similarly computed nodes. By dynamically adjusting the number of clusters, the physical layer anchor points achieve an optimal distribution in terms of spatial proximity, communication performance, and computing resources, providing a reliable foundation for subsequent hierarchical aggregation.

[0098] In some embodiments, when a new distributed node joins, a global recalculation is unnecessary. Using the k-means clustering centroid shift technique, only the centroids of the affected clusters need to be recalculated locally, reducing the time complexity from O(n) to O(log n). Specifically, after determining the cluster to which the new node belongs based on its distance from existing anchor points, only the centroid coordinates of that cluster are adjusted. A centroid offset threshold is set to prevent instability caused by frequent adjustments and ensure the continuity of the anchor point spatial distribution. This improves the dynamic response speed to new nodes, making it particularly suitable for highly dynamic industrial IoT environments (such as scenarios with frequent device start-stop). It significantly reduces computational overhead while maintaining clustering accuracy, providing efficient support for distributed applications with high real-time requirements.

[0099] Step S202: Based on geographical location information, communication delay parameters, and computing power indicators, a hierarchical clustering algorithm is used to cluster the physical layer anchor points to generate regional layer anchor points.

[0100] In some embodiments, a hierarchical clustering method is used to progressively merge geographically proximate physical layer anchors with low communication latency and matching computing capabilities into higher-level regional layer anchors. For example, in a smart grid scenario, physical layer anchors of the same substation and its surrounding adjacent sites can be clustered into the same regional layer anchor to support collaborative computing and consistency management within the region. The resulting regional layer anchors cover a certain geographical area, ensure efficient communication between nodes, and avoid excessive concentration of computing resources, providing a balanced and efficient intermediate hierarchical structure for subsequent global-level processing.

[0101] Step S203: Use graph aggregation algorithm to perform network topology aggregation on regional layer anchor points to generate global layer anchor points.

[0102] In some embodiments, graph aggregation algorithms are used to aggregate network topology of regional layer anchors to generate global layer anchors, enabling cross-regional collaborative management. Specifically, each regional layer anchor is treated as a graph node, and edge weights are established based on communication latency and data interaction frequency to form a network topology graph. Graph aggregation algorithms are applied to identify highly connected regions, merging closely related regional layer anchors into global layer anchors. For example, in a cross-regional power grid, multiple closely interconnected regional anchors are aggregated into provincial or national global anchors. The generated global layer anchors cover all key nodes in the network, ensuring cross-regional consistency and coordination capabilities while reducing the computational complexity of global management.

[0103] This approach utilizes a multi-level topological clustering architecture to construct a semantic anchor system covering the physical, regional, and global layers, achieving cross-level collaborative optimization from bit-level data synchronization to semantic-level alignment. Physical layer anchors are precisely clustered based on geographical location, communication latency, and computing power, ensuring the physical proximity and basic communication efficiency of heterogeneous nodes. Regional layer anchors further optimize the collaborative efficiency of nodes within a region through hierarchical aggregation, effectively reducing cross-node communication overhead. Global layer anchors utilize graph aggregation algorithms to construct cross-regional semantic associations, eliminating semantic gaps between different regions. This scheme overcomes the limitations of traditional consistency protocols that only handle bit-level consistency, achieving semantic-level semantic mapping and verification in a distributed system for the first time, accurately capturing deep data relationships across time, space, and logical dimensions. Through multi-level collaborative design, it not only improves the accuracy and efficiency of data fusion between distributed nodes but also significantly enhances the system's adaptability and fault tolerance in dynamic environments, providing an innovative solution for distributed collaboration in complex scenarios such as the Industrial Internet of Things and smart grids.

[0104] Reference Figure 3 As shown, in some embodiments of this application, the above step S103: extracting time and space features from the general data stream and jointly encoding the time and space features to obtain joint time and space features may include, but is not limited to, the following steps S301 to S303.

[0105] Step S301: Use a gated loop unit to perform time series modeling on the general data stream and set a time window to obtain the time characteristics of data changes in the time series.

[0106] In some embodiments, a gated recurrent unit (GRU) is used to perform time series modeling on the general data stream to extract time features reflecting dynamic changes. Specifically, the time window length can be set to 10 power frequency cycles (corresponding to 200ms@50Hz in a power system scenario) to ensure that the complete voltage / current fluctuation cycle is captured.

[0107] Furthermore, the hidden layer dimension of the Gated Recurrent Unit (GRU) is set to 64 dimensions, and the gating weights are initialized using Xavier to ensure stable gradient propagation and avoid the gradient vanishing / exploding problem in the early stages of training. The data sequence within the time window (such as voltage amplitude and current phase) is input into the GRU, the hidden state is updated step by step, and finally a vector representing the time dynamic features is output to obtain the corresponding time features.

[0108] Step S302: Spatial features are extracted from the general data stream using a deformable convolutional neural network, and spatial features of data changes are obtained based on learnable offsets.

[0109] In some embodiments, a Deformable Convolutional Network (Deformable CNN) is employed to extract spatial features from a general data stream, accurately capturing the spatial distribution characteristics and anomalous regions of the data. Specifically, a standard 3×3 convolutional kernel is used as the basic structure, with each kernel incorporating a learnable offset. This allows the kernel to dynamically adjust its sampling position, enhancing its ability to perceive spatial deformations. The offset, after constraint, supports local deformations of ±3 pixels, ensuring computational stability while being sufficient to cover anomalous regions such as voltage waveform distortion.

[0110] For example, in response to abnormal operating conditions such as voltage drops and waveform distortion in power systems, deformable convolutional networks can automatically focus on abnormal regions by adaptively adjusting the sampling positions. For instance, when a dip in the voltage waveform of a certain phase is detected, the network dynamically adjusts the sampling points of the convolutional kernels to enhance the feature extraction capability of that abnormal region and obtain the corresponding spatial features.

[0111] Step S303: Perform cross-modal fusion of temporal and spatial features based on the multi-head self-attention mechanism to obtain spatiotemporal joint features.

[0112] In some embodiments, a multi-head self-attention (MHSA) mechanism is employed to perform cross-modal fusion of temporal and spatial features, resulting in spatiotemporal joint features. Specifically, temporal features (such as voltage fluctuation trends) and spatial features (such as the location of abnormal waveform regions) are concatenated to form a multimodal input. The MHSA mechanism automatically learns the interaction relationship between temporal and spatial features, with different attention heads focusing on different semantic dimensions (such as temporal dynamics and spatial correlation). Then, the outputs of each attention head are weighted and summed to obtain the final spatiotemporal joint features, which simultaneously include temporal dynamic changes and spatial distribution characteristics.

[0113] Therefore, by employing deformable CNNs for spatial feature extraction and GRUs for temporal modeling, and utilizing a multi-head self-attention mechanism to achieve spatiotemporal feature fusion, the efficiency of distributed systems in processing heterogeneous data and their semantic consistency analysis capabilities are significantly improved. GRUs accurately capture temporal dynamic characteristics such as voltage fluctuations and frequency changes, while deformable CNNs use adaptive offsets to focus on spatially anomalous regions such as voltage drops and waveform distortions, achieving accurate extraction of spatiotemporal features. Furthermore, the multi-head self-attention mechanism enables cross-modal fusion of temporal and spatial features, significantly enhancing the distributed system's spatiotemporal collaborative processing capabilities for heterogeneous data.

[0114] Reference Figure 4 As shown, in some embodiments of this application, the above step S104: mapping the spatiotemporal joint features corresponding to the semantic anchor to the shared semantic space to obtain the mapped feature vector, and using the multi-head attention mechanism to calculate the multi-level feature similarity based on the mapped feature vector, may include, but is not limited to, the following steps S401 to S404.

[0115] Step S401: Map the spatiotemporal joint features of the physical layer anchor, the regional layer anchor, and the global layer anchor to the shared semantic space to obtain the corresponding mapping feature vectors of preset dimensions.

[0116] In some embodiments, a fully connected layer (FC) is used to reduce or increase the dimensionality of the spatiotemporal joint features of semantic anchors at each level, uniformly mapping them to a 256-dimensional shared semantic space to obtain the corresponding 256-dimensional mapped feature vectors. The mapping process preserves the key semantic information of the original features while eliminating dimensional differences between different levels, ensuring consistency in subsequent calculations.

[0117] Step S402: Use the multi-head attention mechanism to calculate the cosine similarity between the mapped feature vectors of the physical layer anchor points and the mapped feature vectors of the regional layer anchor points to obtain the first similarity.

[0118] In some embodiments, a cosine similarity-based quantization method performs deep semantic association analysis on physical layer anchors and regional layer anchors through a multi-head attention mechanism. First, the 256-dimensional mapping feature vectors of the physical layer and regional layer anchors are input into a multi-head attention module. Multiple attention heads are used in parallel to capture semantic associations in different dimensions, such as the degree of matching between temporal dynamic trends and spatial topological distribution features. The attention mechanism adaptively adjusts the weights of each feature, highlighting key similarity information and suppressing irrelevant noise. Then, based on the weighted feature representations, the cosine similarity between the two is calculated to obtain the first similarity, quantifying their proximity in the shared semantic space. It is understood that a higher similarity value indicates stronger consistency between the physical layer anchors and regional layer anchors in spatiotemporal features, enabling accurate identification of the semantic association strength between anchors at different levels and ensuring the synergy and consistency of distributed node data.

[0119] Step S403: Calculate the cosine similarity between the mapped feature vectors of the regional layer anchor points and the mapped feature vectors of the global layer anchor points using a multi-head attention mechanism to obtain the second similarity.

[0120] Similarly, a multi-head attention mechanism is used to perform deep semantic association analysis on the mapped feature vectors of regional layer anchors and global layer anchors, and cosine similarity is used to quantify the degree of matching between the two in the shared semantic space. Compared with traditional distance metrics, cosine similarity focuses more on vector direction rather than absolute difference, and can more accurately capture the logical association characteristics across layers in distributed systems. This provides an important basis for subsequent global consistency verification, ensuring that efficient data collaboration capabilities can still be maintained in complex network environments.

[0121] Step S404: The first similarity and the second similarity are fused to obtain the feature similarity.

[0122] In some embodiments, the two similarity metrics are integrated using a weighted summation method, and the weights can be dynamically adjusted according to the specific application scenario. For example, in scenarios with strong temporal dependencies, the weight of the first similarity can be increased to strengthen the temporal continuity constraints between the physical layer and the regional layer; while in wide-area collaborative scenarios, the weight of the second similarity can be enhanced to optimize cross-regional global consistency.

[0123] During the fusion process, similarity values ​​are not simply superimposed; an attention mechanism is also used to dynamically evaluate the importance of associations at different levels, ensuring that key levels have a more prominent impact on the final similarity. Through this hierarchical fusion strategy, feature similarity preserves the close relationships between local anchor points while also considering global semantic consistency, providing a comprehensive and reliable basis for subsequent consistency verification. This fusion method effectively balances local details with a global perspective, enabling accurate semantic alignment and efficient data collaboration in complex distributed environments.

[0124] Reference Figure 5 As shown, in some embodiments of this application, the above step S105: the reversibility verification network includes a generator and a discriminator; the mapping feature vector is reversed and verified using a preset reversibility verification network to obtain reverse verification parameters, and may also include, but is not limited to, the following steps S501 to S504.

[0125] Step S501: Use the generator to perform an encoding-decoding transformation mapping on the mapped feature vector to obtain the reconstructed features.

[0126] In some embodiments, the generator of the reversibility verification network uses a U-Net architecture to perform an encoding-decoding transformation on the mapped feature vector to obtain reconstructed features. Specifically, in the encoding stage, the feature dimension is gradually compressed through four layers of downsampling (max pooling) to extract multi-scale semantic information while retaining key features; in the decoding stage, the original feature dimension is gradually restored through four layers of upsampling (transposed convolution), and each decoding layer is fused with the skip connection of the corresponding encoding layer to enhance the ability to recover details.

[0127] Step S502: Calculate the difference index between the reconstructed features and the spatiotemporal joint features to obtain the reconstruction error.

[0128] In some embodiments, the mean squared error (MSE) of the reconstructed features and the spatiotemporal joint features is calculated to obtain the reconstruction error. That is, the average of the squared differences across each feature dimension is taken to obtain a scalar error value. This error value intuitively reflects the accuracy of the generator's reconstructed features—the smaller the error, the closer the reconstructed features are to the original features, the more complete the information preserved in the mapping process, and the stronger the reversibility; conversely, it indicates that there may be information loss or distortion in the features during the mapping process, requiring optimization of the network structure or parameters. Furthermore, cosine similarity can be combined to further analyze the directional consistency of feature vectors, ensuring not only numerical similarity but also semantic direction matching; this embodiment does not impose any limitations on this.

[0129] Step S503: Use a discriminator to determine the authenticity of the reconstructed features and obtain the discrimination parameters.

[0130] In some embodiments, a discriminator evaluates the realism of the reconstructed features, determining whether they conform to the distribution characteristics of spatiotemporal joint features. Specifically, a discriminator employing the PatchGAN architecture analyzes the feature map in a locally perceptive manner, dividing the input reconstructed features into multiple 5×5 regions (patch) and calculating the realism score for each patch. Each patch independently passes through multiple convolutional networks to extract features, ultimately outputting a 5×5 matrix, where each element represents the realism probability of the corresponding region. This design allows the discriminator to capture both global consistency and local detail features, avoiding the neglect of subtle differences due to overall discrimination. The final discrimination parameters are obtained by calculating the average or weighted score of the entire matrix, used to measure the similarity between the reconstructed features and the ground truth features.

[0131] Understandably, the discriminator's output can serve as a basis for backpropagation in generator optimization, and also provide auxiliary evaluation metrics for feature mapping quality, ensuring the accuracy and robustness of the back-verification process.

[0132] In some embodiments, the discriminator is trained using a loss function of Wasserstein distance plus a gradient penalty term, optimizing the invertibility of feature transformations through adversarial training. Specifically, the discriminator learns to distinguish between reconstructed features F2 and spatiotemporal joint features F1, while the generator optimizes the reconstruction process to make F2 as close to F1 as possible, satisfying the constraint G(F(x))≈x. The Wasserstein distance ensures that the discriminator provides a smoother gradient signal, while the gradient penalty term enhances training stability and prevents mode collapse. Finally, the realism matrix output by the discriminator is globally pooled to obtain scalar discriminant parameters, which are used to measure the reconstruction quality and guide the continuous optimization of the generator, thereby ensuring the accuracy and robustness of the reverse verification process.

[0133] Step S504: Obtain the reverse verification parameters based on the reconstruction error and the discrimination parameters.

[0134] In some embodiments, a weighted fusion or normalized fusion method is used to combine the reconstruction error with the discriminant parameters. For example, the reconstruction error can be normalized and then added to the discriminant parameters with fixed weights, or the contribution ratio of the two can be dynamically adjusted through a gating mechanism. The final reverse verification parameter contains both the absolute difference information of the reconstruction error and the relative distribution consistency assessment provided by the discriminator, thereby more accurately measuring the reversibility quality of the feature mapping.

[0135] Reference Figure 6 As shown, in some embodiments of this application, the distributed heterogeneous data processing method applied to smart grids may also include, but is not limited to, the following steps S601 to S603.

[0136] Step S601: Use a consistent memory buffer to store a preset number of historical synchronization strategy records to obtain multiple decision storage items.

[0137] In some embodiments, a consistency memory buffer stores historical synchronization policy records, providing experience replay support for bidirectional consistent distillation. This buffer employs a circular buffer structure with a fixed capacity of synchronization policy records for the most recent 100 decision cycles, ensuring efficient use of historical data within limited storage space. Each decision storage item is in the form of a quadruple, including a state vector, action space, immediate reward, and next state.

[0138] Furthermore, to improve sample utilization efficiency, the buffer employs a prioritized experience replay mechanism, using TD error (Temporal Difference Error) as the priority metric. Historical records that contribute more to policy optimization are replayed first. For example, when a synchronization strategy leads to a significant improvement in consistency, its stored items are given a higher sampling weight, accelerating the reproduction and learning of key experiences. This allows for dynamic adaptation to changes in the network environment, guiding the selection of the current synchronization strategy based on historical experience, and significantly improving the decision-making efficiency and robustness of bidirectional consistency distillation.

[0139] Step S602: Optimize the decision storage item using Q-learning decision maker, select synchronous policy in action space to execute according to state vector, calculate the corresponding immediate reward, obtain the next state and update it to the consistency memory buffer.

[0140] In some embodiments, the state vector is composed of multi-dimensional features, including backvalidation parameters, network latency, computational load, and layer similarity: feature similarity matrices for the physical layer, regional layer, and global layer (dimension is the number of layers × feature similarity). Synchronization strategies in the action space include: forced synchronization strategy, asynchronous compensation strategy, and local rollback strategy. The immediate reward is calculated based on the reward function R.

[0141] R = α⋅Conformity accuracy − β⋅Delay − γ⋅Energy consumption

[0142] Among them, α=0.6, β=0.3, and γ=0.1 are adjustable hyperparameters that can be adjusted according to the needs of the scenario.

[0143] Specifically, the Q-learning algorithm optimizes the historical policies in the consistency memory buffer, dynamically selecting the optimal synchronization policy for execution. After executing an action, the system enters the next state and stores the quadruple (state vector, action space, immediate reward, and next state) in the consistency memory buffer for subsequent training and optimization. Through continuous iteration, the Q-learning decision-maker gradually learns policy patterns that adapt to the dynamic network environment, achieving efficient and adaptive synchronization decision-making.

[0144] Step S603: Based on the optimized decision storage items, dynamically adjust the hyperparameters of the Q-learning decision engine to obtain the dynamic arbitration engine.

[0145] In some embodiments, based on the optimized decision storage item, an adaptive dynamic arbitration engine is constructed by adjusting the hyperparameters of the Q-learning decision maker online, thereby improving the optimization effect of the synchronization strategy. Specifically, a dynamic learning rate adjustment mechanism is adopted to adaptively adjust the learning rate of the Q-value according to the policy update frequency and magnitude recorded in the decision storage item. Simultaneously, the reward function weights are dynamically calibrated by analyzing historical reward feedback data to adjust the weight coefficients of consistency accuracy, network latency, and computational energy consumption in real time. Under high load or mission-critical scenarios, the weights of latency and energy consumption are increased accordingly to prioritize response speed and energy efficiency.

[0146] Furthermore, the arbitration engine fully leverages the priority replay function in the consistency memory buffer to reinforce high-value decision samples (such as records with large TD errors), enabling key experiences to influence Q-value updates more quickly and thus improving the overall policy learning efficiency. Through this mechanism, the dynamic arbitration engine can intelligently select the optimal synchronization strategy based on real-time monitored state vectors during operation, including actions such as forced synchronization, asynchronous compensation, or partial rollback. This not only effectively coordinates the trade-offs between consistency, latency, and energy consumption but also maintains efficient and stable decision-making capabilities in complex and ever-changing distributed environments.

[0147] Reference Figure 7 As shown, in some embodiments of this application, the above step S106: selecting the corresponding synchronization strategy to process heterogeneous data based on feature similarity and reverse verification parameters may include, but is not limited to, the following steps S701 to S705.

[0148] Step S701: Construct a state vector based on feature similarity and reverse verification parameters.

[0149] In some embodiments, a multi-dimensional state vector is constructed based on feature similarity, reverse verification parameters, and the real-time operating status of the system, providing a comprehensive decision-making basis for the dynamic selection of synchronization strategies. Specifically, feature similarity and reverse verification parameters serve as the core semantic features of the state vector. Simultaneously, two real-time system metrics are introduced: network latency and computational load. Network latency is obtained by periodically probing the communication delay between nodes, reflecting the current network transmission efficiency; computational load is dynamically calculated based on the CPU / GPU utilization of each node, characterizing resource constraints.

[0150] Thus, the state vector not only captures the semantic consistency features at the data level, but also incorporates the environmental constraints of the system runtime, enabling the selection of subsequent synchronization strategies to comprehensively consider the dynamic balance between data quality and system resources, thereby improving the robustness and efficiency of distributed heterogeneous data processing.

[0151] Step S702: The dynamic arbitration engine selects and executes the synchronization strategy in the action space based on the state vector.

[0152] In some embodiments, the optimal synchronization strategy in the action space is intelligently selected and executed based on the state vector by a dynamic arbitration engine. Specifically, there are: a forced synchronization strategy, which strictly aligns the heterogeneous data of all distributed nodes to ensure global consistency and is suitable for high-precision scenarios (such as financial transactions and medical data synchronization); an asynchronous compensation strategy, which allows some distributed nodes to delay synchronization, prioritizing system throughput and real-time performance and is suitable for scenarios that tolerate short-term inconsistencies, such as large-scale IoT monitoring; and a partial rollback strategy, which performs data rollback for abnormal nodes to isolate the impact of faults and ensure the overall stability of the system.

[0153] Understandably, during the decision-making process, the engine dynamically calculates the expected benefits of each strategy (such as consistency improvement, latency costs, and energy consumption) based on the quantified indicators of the state vector, and selects the strategy with the best overall benefit for execution. For example, when network latency is high, an asynchronous compensation strategy is prioritized to reduce communication overhead; if a significant decrease in data consistency is detected, a forced synchronization strategy is triggered to ensure data reliability. This mechanism achieves intelligent and adaptive synchronization strategies, significantly improving the collaborative efficiency and fault tolerance of distributed systems in complex environments.

[0154] Step S703: When the forced synchronization strategy is selected, all distributed nodes in the network are forced to synchronize and update all heterogeneous data.

[0155] In some embodiments, when the dynamic arbitration engine determines to execute a forced synchronization strategy, it immediately initiates a full data synchronization operation across the entire network to ensure strict consistency of data states across all distributed nodes. Specifically, the coordinating node broadcasts a forced synchronization command to the entire network, carrying the latest full heterogeneous data snapshot and related metadata (such as version number and timestamp). Upon receiving the command, each distributed node first verifies the integrity and consistency of the received data based on spatiotemporal joint characteristics and reversibility verification results. If the verification passes, data updates are executed. This strong consistency guarantee mechanism effectively solves the data inconsistency problem in a distributed environment, making it particularly suitable for scenarios with extremely high data accuracy requirements, such as financial transactions and industrial control.

[0156] In step S704, when selecting an asynchronous compensation strategy, an incremental update mechanism is used to update the heterogeneous data that has changed on the distributed nodes.

[0157] In some embodiments, when the dynamic arbitration engine determines to adopt an asynchronous compensation strategy, it initiates an incremental update mechanism to synchronize data across distributed nodes, minimizing overhead while ensuring data consistency. Unlike forced synchronization, the asynchronous compensation strategy only performs incremental updates on heterogeneous data that has changed. Specifically, it first identifies data portions that have changed since the last synchronization, including added, modified, or deleted records, and encapsulates these changes into incremental data packets. Subsequently, the coordinating node asynchronously distributes these incremental data packets to each target node, and each node independently applies these changes locally, while maintaining a version number or timestamp mechanism to track the data state. To ensure eventual consistency, a verification process can be triggered periodically or under specific conditions to check whether the data state of each node is consistent with the global state. This approach is suitable for scenarios with high real-time requirements but where short-term inconsistencies are acceptable, such as large-scale IoT monitoring and real-time data analysis, effectively improving system throughput and response speed while ensuring data reliability.

[0158] In step S705, when selecting a local rollback strategy, abnormal distributed nodes are identified and rolled back to the previous stable state, triggering an anomaly diagnosis mechanism to repair the abnormal heterogeneous data.

[0159] In some embodiments, when the dynamic arbitration engine determines to execute a partial rollback strategy, it identifies the abnormal distributed nodes and rolls their data state back to the previous stable version. Simultaneously, an anomaly diagnosis mechanism is triggered to repair the abnormal data. Specifically, through feature similarity analysis and reverse verification parameter evaluation, abnormal nodes with data consistency deviations exceeding a threshold are located and marked as rollback targets. Subsequently, the coordinating node sends rollback instructions to these target nodes, and each node restores its local data to the most recently verified state based on recorded historical version information, ensuring that abnormal data does not affect the overall system consistency. At the same time, the system initiates anomaly diagnosis, using techniques such as log analysis, data verification, and pattern recognition to locate the root cause of the anomaly, such as network transmission errors, compute node failures, or external interference. For automatically repairable issues, the system generates a correction plan and resynchronizes the affected data; for complex anomalies, a diagnostic report is generated and maintenance personnel are notified to intervene. This strategy ensures system stability while achieving precise isolation and repair of abnormal data. It is suitable for handling sudden failures or data pollution scenarios, effectively preventing the spread of anomalies and ensuring the continuous and reliable operation of the distributed system in complex environments.

[0160] In one embodiment of this application, the distributed heterogeneous data processing method applied to smart grids can correspond to a three-level architecture. Specifically, the first level is a dynamic distributed abstraction module (data perception layer), the second level is a bidirectional consistency distillation tower (feature verification layer), and the third level is a dynamic arbitration engine (decision execution layer). The following description uses substation voltage anomaly detection as a complete embodiment:

[0161] First, a 110kV substation acquired voltage signals via a PMU (Phasor Measurement Unit) at a sampling rate of 256 points / cycle (corresponding to a resolution of 5.12ms at a 50Hz power frequency), forming a high-density time-series data stream. Then, spatiotemporal feature extraction was performed. The GRU model captured a voltage drop lasting for 5 cycles (25.6ms) and predicted its recovery trend (expected to recover to 95% of the rated value by the 8th cycle). A deformable CNN detected a local dip in the B-phase voltage waveform (trough depth reaching 15% of the rated value), locating the anomaly between cycles 3 and 7, and identifying that this anomaly only affected phase B and not a three-phase global fault. Consistency arbitration was then performed. The physical layer anchor reported a similarity of 0.82 for the B-phase voltage anomaly (exceeding the preset threshold of 0.8), indicating that the anomaly has cross-node consistency. A reversibility verification network verified that the feature mapping was lossless (reconstruction error <3%), confirming the reliable data quality. Based on the current network latency (50ms) and computing load (70% CPU utilization), the arbitration engine selects an asynchronous compensation strategy, synchronizing only the voltage waveform data of the abnormal phase (phase B) to the associated node, instead of the full three-phase data.

[0162] Compared to traditional BFT consensus algorithms, which require all network nodes to achieve strong consensus on complete voltage data (three-phase × 256 points / cycle), resulting in a communication volume of 1.2MB / s and a decision latency of 120ms, this application's embodiment uses asynchronous compensation to synchronize only the abnormal phase data (single-phase × 256 points / cycle), reducing the communication volume to 450KB / s and optimizing the decision latency to 45ms, while maintaining the accuracy of anomaly identification (98.7% vs. 97.3% of traditional solutions). This example verifies the system's efficiency and accuracy in power system anomaly detection, significantly reducing communication overhead and improving real-time response capabilities.

[0163] Therefore, by employing a deep bidirectional verification mechanism and multi-level feature alignment technology, this method significantly improves the data consistency and processing efficiency of distributed systems in complex heterogeneous environments, demonstrating significant technical advantages and application value. Firstly, regarding data alignment capabilities, this method achieves accurate representation of heterogeneous data in terms of temporal dynamics and spatial distribution through multi-level semantic anchor construction and spatiotemporal feature extraction. This enables efficient alignment of features from different nodes and at different levels within a shared semantic space, making it particularly suitable for addressing sensor data heterogeneity issues in scenarios such as the Industrial Internet of Things (IIoT). Secondly, in terms of consistency verification, a mechanism combining forward and reverse consistency distillation is adopted. The forward approach quantifies the similarity of multi-level features through a multi-head attention mechanism, while the reverse approach verifies the reliability of feature mapping through a reversibility verification network. This ensures that data consistency remains accurately maintained even in complex network environments, avoiding calibration failures caused by semantic gaps in traditional methods. Furthermore, through a dynamic decision framework, a synchronization strategy is adaptively selected based on real-time feature similarity and reverse verification parameters, achieving a balance between high performance and low overhead, significantly improving system robustness and fault tolerance. Finally, the technology’s multi-granularity consistency verification capability (bit-level and semantic-level) breaks through the limitations of traditional consistency algorithms, enabling the system to operate stably in complex scenarios such as dynamic topology and network latency, and providing innovative solutions for high-reliability scenarios such as smart grids and cross-cloud database synchronization.

[0164] This application also provides a distributed heterogeneous data processing system for smart grids, which can implement the above-described distributed heterogeneous data processing method for smart grids, referencing... Figure 8 As shown in some embodiments of this application, a distributed heterogeneous data processing system applied to a smart grid includes:

[0165] The acquisition module 100 is used to acquire heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a general data stream; wherein, the distributed nodes are distributed sensor nodes.

[0166] Clustering module 200 is used to perform multi-level clustering of multiple distributed nodes based on the topological information of multiple distributed nodes, and generate multi-level semantic anchors.

[0167] The feature extraction module 300 is used to extract spatiotemporal features from a general data stream to obtain temporal and spatial features, and to jointly encode the temporal and spatial features to obtain spatiotemporal joint features.

[0168] The forward distillation module 400 is used to perform forward consistency distillation, which maps the spatiotemporal joint features corresponding to the semantic anchors to the shared semantic space to obtain the mapped feature vector, and uses a multi-head attention mechanism to calculate multi-level feature similarity based on the mapped feature vector;

[0169] The reverse distillation module 500 is used to perform reverse consistency distillation. It uses a preset reversibility verification network to perform reverse consistency verification on the mapped feature vector and obtains the reverse verification parameters.

[0170] The processing module 600 is used to select the corresponding synchronization strategy to process heterogeneous data based on feature similarity and reverse verification parameters.

[0171] The specific implementation of the distributed heterogeneous data processing system for smart grids in this embodiment is basically the same as the specific implementation of the distributed heterogeneous data processing method for smart grids described above, and will not be repeated here.

[0172] Figure 9 An electronic device 1000 provided in an embodiment of this application is shown. The electronic device 1000 includes: a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the computer program is executed, it is used to perform the above-described distributed heterogeneous data processing method applied to smart grids.

[0173] The processor 1001 and the memory 1002 can be connected via a bus or other means.

[0174] The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the distributed heterogeneous data processing method for smart grids described in the embodiments of this application. The processor 1001 implements the aforementioned distributed heterogeneous data processing method for smart grids by running the non-transitory software programs and instructions stored in the memory 1002.

[0175] The memory 1002 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function. The data storage area may store the distributed heterogeneous data processing method for smart grids described above. Furthermore, the memory 1002 may include high-speed random access memory (RAM) and non-transitory memory, such as at least one storage device, flash memory, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include remotely located memories 1002 relative to the processor 1001, which can be connected to the electronic device 1000 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0176] The non-transient software program and instructions required to implement the above-described distributed heterogeneous data processing method for smart grids are stored in memory 1002. When executed by one or more processors 1001, the above-described distributed heterogeneous data processing method for smart grids is executed, for example, executing... Figure 1 Method steps S101 to S106, Figure 2 Method steps S201 to S203, Figure 3 Method steps S301 to S303, Figure 4 Method steps S401 to S404 Figure 5 Method steps S501 to S504 Figure 6 Method steps S601 to S603, Figure 7 The method steps S701 to S705.

[0177] This application also provides a storage medium, which is a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the aforementioned distributed heterogeneous data processing method applied to a smart grid. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, enterprise intranets, local area networks, mobile communication networks, and combinations thereof.

[0178] The distributed heterogeneous data processing method and system for smart grids provided in this application constructs a multi-level semantic anchor system. This system not only captures the direct correlations of heterogeneous data at the physical layer but also delves into their deep semantic connections, maintaining semantic consistency across nodes and levels in a heterogeneous environment. The multi-granularity consistency verification mechanism primarily achieves deep data fusion in the temporal and spatial dimensions through forward consistency distillation, while reverse consistency distillation employs a reversible verification network for bidirectional verification, ensuring comprehensive consistency verification from micro-bit changes to macro-semantic expressions. This bidirectional, multi-granularity processing paradigm significantly improves the accuracy and reliability of heterogeneous data processing and enables a leap from the bit-level to the semantic level, achieving multi-granularity consistency verification.

[0179] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, storage device storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0181] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects. The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application.

Claims

1. A distributed heterogeneous data processing method applied to smart grids, characterized in that, include: Acquire heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a general data stream; wherein the distributed nodes are distributed sensor nodes; Based on the topological information of the multiple distributed nodes, multi-level clustering is performed on the multiple distributed nodes to generate multi-level semantic anchors. Spatiotemporal features are extracted from the general data stream to obtain temporal features and spatial features, and the temporal features and spatial features are jointly encoded to obtain spatiotemporal joint features; A forward consistency distillation is performed to map the spatiotemporal joint features corresponding to the semantic anchors to the shared semantic space to obtain a mapped feature vector, and a multi-head attention mechanism is used to calculate multi-level feature similarity based on the mapped feature vector; Perform reverse consistency distillation by using a preset reversibility verification network to perform reverse consistency verification on the mapped feature vector to obtain reverse verification parameters. Based on the feature similarity and the reverse verification parameters, a corresponding synchronization strategy is selected to process the heterogeneous data.

2. The distributed heterogeneous data processing method for smart grids according to claim 1, characterized in that, The topology information includes geographic location information, communication latency parameters, and computing power indicators; the semantic anchors include physical layer anchors, regional layer anchors, and global layer anchors. The step of performing multi-level clustering of the multiple distributed nodes based on their topological information to generate multi-level semantic anchors includes: Based on the geographic location information, the communication delay parameters, and the computing power indicators, the distributed nodes are clustered using a clustering algorithm to generate the physical layer anchor points; Based on the geographic location information, the communication delay parameters, and the computing power indicators, the physical layer anchor points are clustered using a hierarchical clustering algorithm to generate the regional layer anchor points; The network topology of the regional layer anchor points is aggregated using a graph aggregation algorithm to generate the global layer anchor points.

3. The distributed heterogeneous data processing method for smart grids according to claim 1, characterized in that, The step of extracting temporal and spatial features from the general data stream to obtain temporal and spatial features, and then jointly encoding the temporal and spatial features to obtain spatiotemporal joint features, includes: The general data stream is modeled using a gated loop unit, and a time window is set to obtain the temporal characteristics of data changes in the time series. Spatial features are extracted from the general data stream using a deformable convolutional neural network, and spatial features of data changes are obtained based on learnable offsets. The spatiotemporal joint features are obtained by cross-modal fusion of the temporal and spatial features based on a multi-head self-attention mechanism.

4. The distributed heterogeneous data processing method for smart grids according to claim 2, characterized in that, The step of mapping the spatiotemporal joint features corresponding to the semantic anchors to a shared semantic space to obtain a mapped feature vector, and then using a multi-head attention mechanism to calculate multi-level feature similarity based on the mapped feature vector, includes: The spatiotemporal joint features of the physical layer anchor, the regional layer anchor, and the global layer anchor are respectively mapped to the shared semantic space to obtain the corresponding mapping feature vectors of a preset dimension; A first similarity is obtained by using a multi-head attention mechanism to calculate the cosine similarity between the mapping feature vectors of the physical layer anchor points and the mapping feature vectors of the regional layer anchor points. A second similarity is obtained by calculating the cosine similarity between the mapped feature vectors of the regional layer anchor points and the mapped feature vectors of the global layer anchor points using a multi-head attention mechanism. The first similarity and the second similarity are fused to obtain the feature similarity.

5. The distributed heterogeneous data processing method for smart grids according to claim 1, characterized in that, The reversibility verification network includes a generator and a discriminator; the reverse consistency verification of the mapped feature vector using the preset reversibility verification network to obtain reverse verification parameters includes: The generator is used to perform an encoding-decoding transformation mapping on the mapped feature vector to obtain the reconstructed features; The difference index between the reconstructed features and the spatiotemporal joint features is calculated to obtain the reconstruction error; The discriminator is used to determine the authenticity of the reconstructed features, and discrimination parameters are obtained. The reverse verification parameters are obtained based on the reconstruction error and the discrimination parameters.

6. The distributed heterogeneous data processing method for smart grids according to any one of claims 1 to 5, characterized in that, The method further includes: A predetermined number of historical synchronization strategy records are stored using a consistent memory buffer to obtain multiple decision storage items; wherein, the decision storage item is a quadruple, including a state vector, action space, immediate reward, and next state; The decision storage item is optimized using a Q-learning decision maker. A synchronous policy in the action space is selected and executed according to the state vector. The immediate reward is calculated accordingly, the next state is obtained, and the consistency memory buffer is updated. Based on the optimized decision storage items, the hyperparameters of the Q-learning decision engine are dynamically adjusted to obtain a dynamic arbitration engine.

7. The distributed heterogeneous data processing method for smart grids according to claim 6, characterized in that, The step of selecting a corresponding synchronization strategy to process the heterogeneous data based on the feature similarity and the reverse verification parameters includes: A state vector is constructed based on the feature similarity and the reverse verification parameters; The dynamic arbitration engine selects and executes a synchronization strategy in the action space based on the state vector; wherein the synchronization strategy includes a forced synchronization strategy, an asynchronous compensation strategy, and a partial rollback strategy. When the forced synchronization strategy is selected, all distributed nodes across the network are forced to synchronize and update all heterogeneous data. When selecting the asynchronous compensation strategy, an incremental update mechanism is used to update the heterogeneous data that has changed in the distributed nodes; When the local rollback strategy is selected, abnormal distributed nodes are identified and rolled back to the previous stable state, triggering the anomaly diagnosis mechanism to repair the abnormal heterogeneous data.

8. A distributed heterogeneous data processing system for smart grids, characterized in that, include: An acquisition module is used to acquire heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a general data stream; wherein, the distributed nodes are distributed sensor nodes; The clustering module is used to perform multi-level clustering of the distributed nodes based on the topological information of the distributed nodes, and generate multi-level semantic anchors. The feature extraction module is used to extract spatiotemporal features from the general data stream to obtain temporal features and spatial features, and to jointly encode the temporal features and spatial features to obtain spatiotemporal joint features; The forward distillation module is used to perform forward consistency distillation, which maps the spatiotemporal joint features corresponding to the semantic anchors to the shared semantic space to obtain the mapped feature vector, and uses a multi-head attention mechanism to calculate multi-level feature similarity based on the mapped feature vector; The reverse distillation module is used to perform reverse consistency distillation. It uses a preset reversibility verification network to perform reverse consistency verification on the mapped feature vector to obtain reverse verification parameters. The processing module is used to select a corresponding synchronization strategy to process the heterogeneous data based on the feature similarity and the reverse verification parameters.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the distributed heterogeneous data processing method for smart grids as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, which is executed by a processor to implement the distributed heterogeneous data processing method for smart grids as described in any one of claims 1 to 7.

Images