Distributed heterogeneous data processing method and system based on bidirectional consistency distillation

Abstract

The invention discloses a distributed heterogeneous data processing method and system based on bidirectional consistency distillation, and relates to the technical field of data processing. According to the method, heterogeneous data of distributed nodes are obtained and converted into general data streams, and multi-level clustering is carried out on the distributed nodes based on topological information to generate multi-level semantic anchor points. And extracting time features and space features of the general data stream, and fusing to obtain space-time joint features. Then forward consistency distillation is carried out, space-time joint features corresponding to the semantic anchor points are mapped to a shared semantic space to obtain a mapping feature vector, and a multi-head attention mechanism is utilized to calculate multi-level feature similarity; and performing reverse consistency distillation, and performing reverse consistency verification on the mapping feature vector by using a reversibility verification network to obtain a reverse verification parameter. And finally, selecting a synchronization strategy to process the heterogeneous data according to the feature similarity and the reverse verification parameter. And the crossing of data processing from a bit level to a semantic level and multi-granularity consistency verification are realized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a distributed heterogeneous data processing method and system based on bidirectional consistency distillation. Background Art

[0002] In the field of distributed heterogeneous data processing, due to the heterogeneity of node hardware, dynamic changes in network topology, and uneven data distribution, existing technologies only ensure the consistency of the bit stream of the original data and cannot identify the differences in the semantic understanding of the data by different nodes. Traditional data processing methods have difficulty in achieving efficient and consistent data synchronization. In addition, existing technology solutions usually adopt centralized processing or simple distributed consistency protocols (such as Paxos or Raft), but these methods have significant defects when facing large-scale heterogeneous nodes: on the one hand, the centralized architecture can easily become a performance bottleneck and has a high risk of single point failure; on the other hand, simple consistency protocols cannot effectively handle the semantic differences between heterogeneous nodes, resulting in reduced data accuracy. In recent years, distributed data processing methods based on deep learning have gradually emerged, but most of these methods focus on single-modal feature extraction, ignore the processing of semantic levels, and lack the ability to maintain multi-level semantic consistency. Summary of the Invention

[0003] This application aims to solve at least one of the technical problems existing in the prior art. To this end, the embodiments of this application provide a distributed heterogeneous data processing method and system based on bidirectional consistency distillation, which can achieve a leap from bit-level to semantic-level data processing and realize multi-granularity consistency verification.

[0004] In a first aspect, an embodiment of the present application provides a distributed heterogeneous data processing method based on bidirectional consistency distillation, comprising:

[0005] Obtaining heterogeneous data from multiple distributed nodes and converting the heterogeneous data into a common data stream;

[0006] Performing multi-level clustering on the plurality of distributed nodes based on topological information of the plurality of distributed nodes to generate multi-level semantic anchor points;

[0007] Extracting spatiotemporal features from the general data stream to obtain a temporal feature and a spatial feature, and jointly encoding the temporal feature and the spatial feature to obtain a spatiotemporal joint feature;

[0008] Perform forward consistency distillation to map the spatiotemporal joint features corresponding to the semantic anchor points to a shared semantic space to obtain a mapping feature vector, and use a multi-head attention mechanism to calculate multi-level feature similarity based on the mapping feature vector;

[0009] Perform reverse consistency distillation, using a preset reversibility verification network to perform reverse consistency verification on the mapping feature vector to obtain reverse verification parameters;

[0010] According to the feature similarity and the reverse verification parameter, a corresponding synchronization strategy is selected to process the heterogeneous data.

[0011] In some embodiments of the present application, the topology information includes geographic location information, communication delay parameters, and computing power indicators; the semantic anchors include physical layer anchors, regional layer anchors, and global layer anchors; and the multi-level clustering of the multiple distributed nodes based on the topology information of the multiple distributed nodes to generate multi-level semantic anchors includes:

[0012] Clustering the distributed nodes using a clustering algorithm according to the geographic location information, the communication delay parameter, and the computing capability indicator to generate the physical layer anchor point;

[0013] Clustering the physical layer anchor points using a hierarchical clustering algorithm according to the geographic location information, the communication delay parameter, and the computing capability indicator to generate the regional layer anchor points;

[0014] A graph aggregation algorithm is used to perform network topology aggregation on the regional layer anchor points to generate the global layer anchor points.

[0015] In some embodiments of the present application, extracting spatiotemporal features from the general data stream to obtain temporal features and spatial features, and jointly encoding the temporal features and the spatial features to obtain spatiotemporal joint features, includes:

[0016] Using a gated recurrent unit to perform time series modeling on the general data stream, and setting a time window to obtain the temporal characteristics of data changes in the time series;

[0017] Performing spatial feature extraction on the general data stream using a deformable convolutional neural network, and obtaining spatial features of data changes based on a learnable offset;

[0018] The temporal features and the spatial features are cross-modally fused according to a multi-head self-attention mechanism to obtain the spatiotemporal joint features.

[0019] In some embodiments of the present application, mapping the spatiotemporal joint features corresponding to the semantic anchor points to a shared semantic space to obtain a mapping feature vector, and calculating multi-level feature similarity based on the mapping feature vector using a multi-head attention mechanism, includes:

[0020] Mapping the spatiotemporal joint features of the physical layer anchor point, the regional layer anchor point, and the global layer anchor point to the shared semantic space respectively to obtain corresponding mapping feature vectors of preset dimensions;

[0021] Performing cosine similarity calculation on the mapping feature vector of the physical layer anchor point and the mapping feature vector of the regional layer anchor point using a multi-head attention mechanism to obtain a first similarity;

[0022] Using a multi-head attention mechanism, a cosine similarity calculation is performed on the mapping feature vector of the regional layer anchor point and the mapping feature vector of the global layer anchor point to obtain a second similarity;

[0023] The first similarity and the second similarity are fused to obtain the feature similarity.

[0024] In some embodiments of the present application, the reversibility check network includes a generator and a discriminator; the reverse consistency check of the mapping feature vector using the preset reversibility check network to obtain the reverse check parameter includes:

[0025] Using the generator to perform encoding-decoding transformation mapping on the mapped feature vector to obtain a reconstructed feature;

[0026] Calculating a difference index between the reconstructed feature and the spatiotemporal joint feature to obtain a reconstruction error;

[0027] Using the discriminator to perform authenticity discrimination on the reconstructed features to obtain discrimination parameters;

[0028] The reverse check parameter is obtained according to the reconstruction error and the discrimination parameter.

[0029] In some embodiments of the present application, the method further comprises:

[0030] Using a consistent memory buffer to store a preset number of historical synchronization strategy records to obtain a plurality of decision storage items; wherein the decision storage items are four-tuples including a state vector, an action space, an immediate reward, and a next state;

[0031] Utilizing a Q-learning decider to optimize the decision storage item, selecting a synchronization strategy in the action space to execute according to the state vector, and correspondingly calculating the immediate reward, obtaining a next state, and updating the next state to the consistency memory buffer;

[0032] According to the optimized decision storage items, the hyperparameters of the Q-learning decision maker are dynamically adjusted to obtain a dynamic arbitration engine.

[0033] In some embodiments of the present application, selecting a corresponding synchronization strategy to process the heterogeneous data according to the feature similarity and the reverse verification parameter includes:

[0034] Constructing a state vector according to the feature similarity and the reverse verification parameter;

[0035] Utilizing the dynamic arbitration engine to select a synchronization strategy in the action space according to the state vector for execution; wherein the synchronization strategy includes a forced synchronization strategy, an asynchronous compensation strategy, and a local rollback strategy;

[0036] When the forced synchronization strategy is selected, all distributed nodes in the entire network are forced to synchronously update all heterogeneous data;

[0037] When the asynchronous compensation strategy is selected, an incremental update mechanism is used to update the changed heterogeneous data on the distributed nodes;

[0038] When the local rollback strategy is selected, abnormal distributed nodes are identified and rolled back to the last stable state, triggering the abnormal diagnosis mechanism to repair the abnormal heterogeneous data.

[0039] In a second aspect, the embodiments of the present application further provide a distributed heterogeneous data processing system based on bidirectional consistency distillation, including:

[0040] An acquisition module, configured to acquire heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a common data stream;

[0041] A clustering module, configured to perform multi-level clustering on the plurality of distributed nodes based on topological information of the plurality of distributed nodes to generate multi-level semantic anchor points;

[0042] a feature extraction module, configured to extract spatiotemporal features from the general data stream to obtain temporal features and spatial features, and jointly encode the temporal features and the spatial features to obtain spatiotemporal joint features;

[0043] A forward distillation module is used to perform forward consistency distillation, map the spatiotemporal joint features corresponding to the semantic anchor points to a shared semantic space to obtain a mapping feature vector, and calculate multi-level feature similarity based on the mapping feature vector using a multi-head attention mechanism;

[0044] A reverse distillation module is used to perform reverse consistency distillation, using a preset reversibility verification network to perform reverse consistency verification on the mapping feature vector to obtain reverse verification parameters;

[0045] A processing module is used to select a corresponding synchronization strategy to process the heterogeneous data according to the feature similarity and the reverse verification parameter.

[0046] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the distributed heterogeneous data processing method based on bidirectional consistency distillation as described in the embodiment of the first aspect of the present application.

[0047] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program, and the program is executed by a processor to implement the distributed heterogeneous data processing method based on bidirectional consistency distillation as described in the embodiment of the first aspect of the present application.

[0048] The embodiments of the present application include at least the following beneficial effects:

[0049] Embodiments of the present application provide a distributed heterogeneous data processing method and system based on bidirectional consistency distillation. The method obtains heterogeneous data from multiple distributed nodes and converts it into a common data stream. Based on their topological information, the distributed nodes are then clustered at multiple levels to generate semantic anchors corresponding to the multiple levels. Spatiotemporal feature extraction is then performed on the common data stream to obtain temporal and spatial features, which are then jointly encoded to obtain spatiotemporal joint features. Forward consistency distillation is then performed, including mapping the spatiotemporal joint features corresponding to the semantic anchors to a shared semantic space to obtain a mapping feature vector. Multi-head attention is then used to calculate multi-level feature similarity based on the mapping feature vector. Reverse consistency distillation is then performed, including performing a reverse consistency check on the mapping feature vector using a pre-set reversibility check network to obtain reverse check parameters. Finally, based on the feature similarity and reverse check parameters, a corresponding synchronization strategy is selected to process the heterogeneous data. This constructs a multi-level semantic anchor system that not only captures direct connections between heterogeneous data at the physical layer but also deeply explores deeper connections at the semantic level, maintaining semantic consistency across nodes and levels in heterogeneous environments. The multi-granularity consistency verification mechanism primarily achieves deep data integration across time and space through forward consistency distillation, while reverse consistency distillation employs a reversible verification network for bidirectional verification, ensuring comprehensive consistency verification from micro-bit changes to macro-semantic expression. This bidirectional, multi-granularity processing paradigm not only significantly improves the accuracy and reliability of heterogeneous data processing, but also enables a leap in data processing from the bit level to the semantic level, achieving multi-granularity consistency verification.

[0050] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0052] Figure 1 This is a flow chart of a distributed heterogeneous data processing method based on bidirectional consistency distillation provided by an embodiment of the present application;

[0053] Figure 2 yes Figure 1 Flow diagram of step S102;

[0054] Figure 3 yes Figure 1 Flow diagram of step S103;

[0055] Figure 4 yes Figure 1 Flow chart of step S104;

[0056] Figure 5 yes Figure 1 Flow chart of step S105;

[0057] Figure 6 This is a flow chart of a distributed heterogeneous data processing method based on bidirectional consistency distillation provided by another embodiment of the present application;

[0058] Figure 7 yes Figure 1 Flow chart of step S106;

[0059] Figure 8 This is a schematic diagram of a distributed heterogeneous data processing system module based on bidirectional consistency distillation provided by an embodiment of the present application;

[0060] Figure 9 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application.

[0061] Reference numerals: acquisition module 100 , clustering module 200 , feature extraction module 300 , forward distillation module 400 , reverse distillation module 500 , processing module 600 , electronic device 1000 , processor 1001 , memory 1002 . DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0064] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0065] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0066] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0067] At the intersection of distributed computing and machine learning, distributed heterogeneous data processing remains a core technical challenge, particularly critical in high-consistency scenarios such as financial distributed ledgers, the Industrial Internet of Things, and cross-cloud database synchronization. Traditional distributed consensus algorithms (such as Paxos and Raft) primarily focus on bit-level consistency of raw data, guaranteeing consistency only at the bitstream level while ignoring the semantic level of the data. This limitation results in heterogeneous node environments where, even if data is bit-identical, different nodes may produce different semantic interpretations due to differences in computing power, data representation, or model parameters, severely impacting system reliability and decision consistency. Furthermore, while existing distributed machine learning methods have alleviated data inconsistency to some extent, gradient inconsistencies persist across parameter server architecture layers, leading to slow global model convergence and even training failure. However, with the explosive growth of data volumes and increasing system heterogeneity, traditional methods struggle to simultaneously meet the multiple requirements of high consistency, high fault tolerance, and high scalability.

[0068] At the same time, traditional data synchronization technologies often rely on centralized processing architectures or simple distributed consensus protocols, which face significant performance bottlenecks in heterogeneous node environments. Centralized systems are prone to becoming bottleneck nodes with single points of failure, and processing efficiency decreases sharply with increasing node scale. While classic protocols such as Paxos and Raft offer high fault tolerance, they cannot properly handle semantic differences between nodes, leading to the long-standing neglect of inconsistencies at the semantic level. While recent progress has been made in distributed data processing technologies based on deep learning, significant limitations remain. Existing methods often focus solely on extracting features from a single modality (e.g., processing only a single type of data, such as visual or text), lack cross-modal semantic association modeling capabilities, and are still lacking in hierarchical semantic consistency processing. This current state of the art makes it difficult for traditional methods to achieve efficient and robust consistency guarantees at multiple granularities (bit-level, feature-level, and semantic-level) in dynamically changing heterogeneous network environments. This further degrades overall system performance, especially in situations with high network latency and limited bandwidth. Therefore, a new solution that transcends bit-level synchronization and achieves full-link semantic consistency modeling is urgently needed to improve the reliability and intelligent adaptability of complex distributed systems.

[0069] Based on this, the embodiment of the present application provides a distributed heterogeneous data processing method and system based on bidirectional consistency distillation, and constructs a multi-level semantic anchor system, which can not only capture the direct correlation of heterogeneous data at the physical layer, but also deeply explore the deep connection at the semantic level, and can maintain semantic consistency across nodes and levels in a heterogeneous environment. The multi-granularity consistency verification mechanism mainly realizes the deep fusion of data in time and space dimensions through forward consistency distillation, while the reverse consistency distillation uses a reversible verification network for bidirectional verification, ensuring all-round consistency verification from micro-bit changes to macro-semantic expressions. This bidirectional, multi-granularity processing paradigm not only significantly improves the accuracy and reliability of heterogeneous data processing, but also realizes the leap of data processing from the bit level to the semantic level, and realizes multi-granularity consistency verification. .

[0070] The embodiments of the present application provide a distributed heterogeneous data processing method and system based on bidirectional consistency distillation, which are specifically illustrated by the following embodiments. First, the distributed heterogeneous data processing method based on bidirectional consistency distillation in the embodiments of the present application is described.

[0071] The distributed heterogeneous data processing method based on bidirectional consistency distillation provided in the embodiment of the present application relates to the field of data processing technology, and in particular to the field of distributed heterogeneous data processing technology. The distributed heterogeneous data processing method based on bidirectional consistency distillation provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, the computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system before it can run, such as a client that supports distributed heterogeneous data processing based on bidirectional consistency distillation, that is, a program that can be run only by downloading it into a browser environment. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server through a network. The distributed heterogeneous data processing method based on bidirectional consistency distillation can be executed by the terminal or the server, or by the terminal and the server in collaboration.

[0072] In some embodiments, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, or smartwatch. The server can be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. It can also be a service node in a blockchain system, where each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The server can be installed with a database index compression system server, which can interact with the terminal through the server. For example, the server can be installed with corresponding software, which can be an application that implements a distributed heterogeneous data processing method based on bidirectional consistency distillation, etc., but is not limited to the above forms. The terminal and server can be connected via Bluetooth, Universal Serial Bus (USB), or network communication connection methods, which are not limited to this embodiment.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] The following describes a distributed heterogeneous data processing method based on bidirectional consistency distillation in an embodiment of the present invention.

[0075] Reference Figure 1 As shown, an embodiment of the present application provides a distributed heterogeneous data processing method based on bidirectional consistency distillation, which may include but is not limited to the following steps S101 to S106.

[0076] Step S101: Obtain heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a common data stream.

[0077] In some embodiments, heterogeneous data from multiple distributed nodes is acquired. For example, in smart grid applications, heterogeneous data may originate from different sensors, including current sensors, voltage sensors, and PMU synchronized phasor measurement devices. This heterogeneous data is converted into a common data stream. Specifically, corresponding protocol conversion modules can be deployed for different communication protocols (such as Modbus, OPC UA, and REST API) to ensure unified data access. For example, in a smart grid scenario, the IEC 61850 data format of PMU devices and voltage / current transformers can be converted to an internal common format through an OPC UA gateway.

[0078] It is understood that the use of a structured schema (such as Protocol Buffers, Avro, etc.) to define the logical structure of the data ensures that the data is parsed, cleaned, and formatted into a unified data structure before transmission. For example, the measurement values of different devices are uniformly mapped into standardized data tuples containing timestamps, device IDs, measurement type, and values. This encapsulates the preprocessed data into a common data stream format (such as a gRPC data stream), which is not limited in this embodiment.

[0079] Step S102 : performing multi-level clustering on the multiple distributed nodes based on the topological information of the multiple distributed nodes to generate multi-level semantic anchor points.

[0080] In some embodiments, cluster analysis is performed based on the topological information of distributed nodes (for example, including geographical location, communication delay, computing power, network connection, etc.), and nodes with similar attributes or communication relationships are divided into different groups, thereby generating multi-level semantic anchors. Specifically, in smart grid applications, substation nodes with similar geographical locations will be divided into the same group to reflect their physical distribution relationship in the power grid; in the industrial Internet of Things scenario, devices that communicate frequently may be clustered together to reflect their relevance in data interaction. Through clustering, a more semantically consistent data association structure can be constructed, providing a hierarchical reference framework for subsequent feature extraction and consistency verification, and enhancing the collaborative processing capabilities of distributed nodes.

[0081] It can be understood that multi-level semantic anchors are a hierarchical data organization structure used to establish semantic associations between nodes in distributed heterogeneous systems. Nodes with similar characteristics or topological relationships are divided into groups of different levels through clustering algorithms. Each group corresponds to a semantic anchor, which serves as an abstract representation of the group of nodes. For example, in a smart grid scenario, various devices in the same substation may form the first layer of semantic anchors; multiple substations in the same power supply area constitute the second layer of semantic anchors; and key hub nodes across the entire network form the third layer of semantic anchors. This hierarchical design retains local details while supporting global collaborative optimization.

[0082] Step S103 , performing spatiotemporal feature extraction on the general data stream to obtain temporal features and spatial features, and jointly encoding the temporal features and spatial features to obtain spatiotemporal joint features.

[0083] In some embodiments, spatiotemporal feature extraction is performed on a general data stream to obtain time features and spatial features. Specifically, based on the pattern of data changes over time, features that can characterize its dynamic characteristics are extracted. For example, in an electric power system, the fluctuation of parameters such as voltage and current over time can be captured, abnormal operating conditions such as voltage drops or frequency changes that last for several cycles can be identified, and corresponding time features can be obtained. Based on the distribution and correlation of data in the spatial dimension, features that can reflect the spatial characteristics between nodes are extracted. For example, in a substation voltage monitoring scenario, a deformable convolutional neural network can automatically focus on a specific area where the voltage waveform is distorted, accurately locate the depression position of the B-phase voltage, thereby identifying the specific phase and position where the anomaly occurs, and obtaining the corresponding spatial features.

[0084] In some embodiments, the extracted time features and spatial features are jointly encoded to form a unified joint time-space feature representation. The joint time-space feature contains both dynamic change information in the time dimension and distribution characteristics in the spatial dimension, providing a comprehensive and accurate feature basis for subsequent consistency distillation. Taking substation voltage anomaly detection as an example, it can simultaneously capture the spatial distortion characteristics of the voltage waveform (such as the B-phase voltage sag) and the temporal persistence characteristics (such as a voltage drop lasting 5 cycles), thereby achieving accurate identification and positioning of abnormal working conditions.

[0085] In step S104, forward consistency distillation is performed to map the spatiotemporal joint features corresponding to the semantic anchor points to the shared semantic space to obtain a mapping feature vector, and a multi-head attention mechanism is used to calculate the multi-level feature similarity based on the mapping feature vector.

[0086] In some embodiments, the spatiotemporal joint features corresponding to semantic anchors at different levels are mapped into a unified shared semantic space to obtain a standardized mapped feature vector. This not only preserves the key information of the original features, but also eliminates the dimensionality and distribution differences between different nodes through standardization, ensuring that features from different nodes and different levels can be compared and aligned within the same semantic framework.

[0087] In some embodiments, a multi-head attention mechanism is used to analyze the mapped feature vectors. Specifically, to deeply capture multi-dimensional semantic consistency, the present embodiment adopts a multi-head attention mechanism with four parallel attention heads, each of which is designed to focus on semantic associations of different dimensions (such as spatial consistency, temporal consistency, etc.). By calculating the similarity between features at different levels, the degree of semantic consistency of nodes at each level in the shared semantic space can be quantified.

[0088] Step S105 , performing reverse consistency distillation, using a preset reversibility verification network to perform reverse consistency verification on the mapping feature vector to obtain reverse verification parameters.

[0089] In some embodiments, the mapped feature vectors obtained by forward distillation are reverse-verified using a pre-set reversibility verification network to ensure the accuracy and information integrity of the feature mapping. Based on feature reconstruction and similarity assessment mechanisms, the reversibility verification network analyzes whether the mapped features retain the key semantic information of the original data.

[0090] This bidirectional verification mechanism not only verifies the reliability of the forward distillation results but also ensures the reversibility of the reverse mapping, thereby improving the robustness of the entire consistency distillation process. The resulting reverse verification parameters provide an important reference for selecting subsequent synchronization strategies, ensuring a high level of data consistency in heterogeneous environments.

[0091] Step S106: Select a corresponding synchronization strategy to process heterogeneous data based on feature similarity and reverse verification parameters.

[0092] In some embodiments, the optimal synchronization strategy for handling heterogeneous data is intelligently selected based on the multi-level feature similarity and reverse validation parameters derived from forward consistency distillation. For example, if the similarity is high and the validation parameters are normal, lightweight synchronization is performed; if the consistency degrades but is recoverable, bidirectional validation synchronization is triggered; if severe distortion or anomalies occur, global forced synchronization is initiated. This mechanism dynamically adapts to changes in network status, ensuring efficient collaboration and high consistency of heterogeneous data.

[0093] This has led to the construction of a multi-level semantic anchor system that can not only capture the direct correlation of heterogeneous data at the physical layer, but also deeply explore its deep connections at the semantic level, and maintain semantic consistency across nodes and levels in heterogeneous environments. The multi-granularity consistency verification mechanism mainly uses forward consistency distillation to achieve deep fusion of data in the time and space dimensions, while reverse consistency distillation uses a reversible verification network for bidirectional verification, ensuring all-round consistency verification from micro-bit changes to macro-semantic expressions. This bidirectional, multi-granularity processing paradigm not only significantly improves the accuracy and reliability of heterogeneous data processing, but also realizes the leap from bit level to semantic level in data processing, achieving multi-granularity consistency verification.

[0094] Reference Figure 2 As shown, in some embodiments of the present application, topology information includes geographic location information, communication delay parameters, and computing power indicators; semantic anchors include physical layer anchors, regional layer anchors, and global layer anchors. Step S102 above: performing multi-level clustering of the multiple distributed nodes based on their topology information to generate multi-level semantic anchors may include, but is not limited to, steps S201 to S203.

[0095] Step S201: clustering distributed nodes using a clustering algorithm based on geographic location information, communication delay parameters, and computing capability indicators to generate physical layer anchor points.

[0096] In some embodiments, geographic location information is obtained through GPS to obtain the physical coordinates (such as longitude and latitude) of each distributed node, which can ensure the spatial proximity of distributed nodes within the same anchor point; the communication delay parameter is based on the measured or estimated network delay matrix, which quantifies the network transmission efficiency between nodes and ensures that the communication delay of nodes within the anchor point meets the requirements; computing power indicators include hardware performance parameters such as CPU / GPU computing power and memory capacity, so that the computing load between anchor points is as balanced as possible.

[0097] Specifically, an improved k-means++ algorithm is used to cluster distributed nodes based on geographic location information, communication delay parameters, and computing power indicators. The silhouette coefficient is then used to dynamically optimize the number of clusters k to generate physical layer anchor points, ensuring that the topological partitioning satisfies both communication delay constraints and computational load balancing. Mahalanobis distance is used to comprehensively evaluate multidimensional features, and k-means++ probabilistically selects initial cluster centers to prevent traditional k-means from falling into local optimality. Each physical layer anchor point represents a group of distributed nodes with close geographic locations, low communication delays, and similar computing power. By dynamically adjusting the number of clusters, the physical layer anchor points achieve optimal distribution in terms of spatial proximity, communication performance, and computing resources, providing a reliable foundation for subsequent hierarchical aggregation.

[0098] In some embodiments, when a new distributed node is added, there is no need for global recalculation. Through the k-means clustering center drift technology, only the center of mass of the affected cluster needs to be locally recalculated, and the time complexity is reduced from O(n) to O(log n). Specifically, based on the distance between the new node and the existing anchor point, after determining the cluster to which it belongs, only the center of mass coordinates of the cluster are adjusted, and a center of mass offset threshold is set to prevent instability caused by frequent adjustments and ensure the continuity of the spatial distribution of the anchor points. The dynamic response speed to new nodes can be improved, which is especially suitable for highly dynamic industrial Internet of Things environments (such as scenarios where equipment frequently starts and stops). While ensuring clustering accuracy, it greatly reduces computing overhead and provides efficient support for distributed applications with high real-time requirements.

[0099] Step S202 : clustering the physical layer anchor points using a hierarchical clustering algorithm based on the geographic location information, the communication delay parameter, and the computing capability index to generate regional layer anchor points.

[0100] In some embodiments, a hierarchical clustering approach is used to gradually merge geographically proximal physical layer anchors with low communication latency and matching computing power into higher-level regional layer anchors. For example, in a smart grid scenario, the physical layer anchors of the same substation and its surrounding adjacent sites can be clustered into the same regional layer anchor to support collaborative computing and consistency management within the region. The resulting regional layer anchors can cover a certain geographical area while ensuring efficient inter-node communication and avoiding excessive concentration of computing resources, providing a balanced and efficient intermediate layer structure for subsequent global-level processing.

[0101] Step S203: Use a graph aggregation algorithm to perform network topology aggregation on the regional layer anchor points to generate global layer anchor points.

[0102] In some embodiments, a graph aggregation algorithm is used to aggregate the network topology of regional layer anchor points to generate global layer anchor points to achieve cross-regional collaborative management. Specifically, each regional layer anchor point is regarded as a graph node, and edge weights are established based on communication delay and data interaction frequency to form a network topology graph. The graph aggregation algorithm is applied to identify high-connectivity areas and merge closely related regional layer anchor points into global layer anchor points. For example, in a cross-regional power grid, multiple closely interconnected regional anchor points are aggregated into provincial or national global anchor points. The generated global layer anchor points cover key nodes across the entire network, ensuring cross-regional consistency and coordination capabilities while reducing the computational complexity of global management.

[0103] Through a multi-level topological clustering architecture, a semantic anchor system covering the physical, regional, and global layers was constructed, achieving cross-layer collaborative optimization from bit-level data synchronization to semantic-level semantic alignment. Physical-layer anchors are precisely clustered based on geographic location, communication latency, and computing power to ensure the physical proximity and basic communication efficiency of heterogeneous nodes. Regional-layer anchors further optimize the collaborative efficiency of nodes within a region through hierarchical aggregation, effectively reducing cross-node communication overhead. Global-layer anchors utilize graph aggregation algorithms to construct cross-regional semantic associations, eliminating the semantic gap between different regions. This solution breaks through the limitations of traditional consistency protocols that only handle bit-level consistency. For the first time, semantic-level semantic mapping and verification are implemented in distributed systems, accurately capturing the deep connections between data in time, space, and logical dimensions. Through multi-level collaborative design, not only is the accuracy and efficiency of data fusion between distributed nodes improved, but the system's adaptability and fault tolerance in dynamic environments are also significantly enhanced, providing an innovative solution for distributed collaboration in complex scenarios such as the Industrial Internet of Things and smart grids.

[0104] Reference Figure 3 As shown, in some embodiments of the present application, the above-mentioned step S103: extracting spatiotemporal features from the general data stream to obtain time features and spatial features, and jointly encoding the time features and spatial features to obtain spatiotemporal joint features, may include but is not limited to the following steps S301 to S303.

[0105] Step S301 : Use a gated recurrent unit to perform time series modeling on a general data stream, and set a time window to obtain the time characteristics of data changes in the time series.

[0106] In some embodiments, a Gated Recurrent Unit (GRU) is used to model the general data stream as a time series to extract temporal features that reflect dynamic changes. Specifically, the time window length can be set to 10 power frequency cycles (corresponding to 200ms at 50Hz in power system scenarios) to ensure that the complete voltage / current fluctuation cycle is captured.

[0107] Furthermore, the hidden layer dimension of the gated recurrent unit (GRU) is set to 64, and the gating weights are initialized using Xavier to ensure stable gradient propagation and avoid gradient vanishing / explosion issues in the early stages of training. The data sequence within the time window (such as voltage amplitude and current phase) is input into the GRU, and the hidden state is updated time-step by time. The final output is a vector representing the temporal dynamics, resulting in the corresponding temporal features.

[0108] Step S302: Use a deformable convolutional neural network to extract spatial features from the general data stream, and obtain spatial features of data changes based on the learnable offset.

[0109] In some embodiments, a deformable convolutional neural network (CNN) is used to extract spatial features from general data streams to accurately capture the distribution characteristics and abnormal areas of the data in the spatial dimension. Specifically, a standard 3×3 convolution kernel is used as the basic structure, and each convolution kernel introduces a learnable offset, which enables the convolution kernel to dynamically adjust the sampling position and enhance the perception of spatial deformation. The offset is constrained to support local deformation of ±3 pixels, ensuring computational stability while being sufficient to cover abnormal areas such as voltage waveform distortion.

[0110] For example, in response to abnormal operating conditions such as voltage sags and waveform distortion in power systems, the Deformable Convolutional Network adaptively adjusts sampling positions to automatically focus on the abnormal area. For example, when a sag is detected in a phase voltage waveform, the network dynamically adjusts the convolution kernel sampling points to enhance feature extraction of that abnormal area and obtain the corresponding spatial features.

[0111] In step S303, the temporal features and spatial features are cross-modally fused according to the multi-head self-attention mechanism to obtain a spatiotemporal joint feature.

[0112] In some embodiments, a multi-head self-attention mechanism (MHSA) is used to perform cross-modal fusion of temporal features and spatial features to obtain spatiotemporal joint features. Specifically, temporal features (such as voltage fluctuation trends) and spatial features (such as the location of abnormal waveform areas) are spliced to form a multimodal input. The multi-head self-attention mechanism is used to automatically learn the interaction between temporal and spatial features, and different attention heads focus on different semantic dimensions (such as temporal dynamics and spatial associations). The outputs of each attention head are then weighted and summed to obtain the final spatiotemporal joint features, which also include temporal dynamic changes and spatial distribution characteristics.

[0113] By extracting spatial features using a deformable CNN and modeling temporal data using a GRU, and utilizing a multi-head self-attention mechanism to achieve spatiotemporal feature fusion, the distributed system significantly improves its processing efficiency and semantic consistency analysis capabilities for heterogeneous data. The GRU accurately captures temporal dynamics such as voltage fluctuations and frequency changes, while the deformable CNN uses adaptive offsets to focus on spatial anomalies such as voltage drops and waveform distortion, enabling precise extraction of spatiotemporal features. Furthermore, the multi-head self-attention mechanism enables cross-modal fusion of temporal and spatial features, significantly enhancing the distributed system's ability to collaboratively process heterogeneous data in both temporal and spatial order.

[0114] Reference Figure 4 As shown, in some embodiments of the present application, the above-mentioned step S104: mapping the spatiotemporal joint features corresponding to the semantic anchor point to the shared semantic space to obtain a mapping feature vector, and using the multi-head attention mechanism to calculate the multi-level feature similarity based on the mapping feature vector, may include but is not limited to the following steps S401 to S404.

[0115] In step S401 , the spatiotemporal joint features of the physical layer anchor point, the regional layer anchor point, and the global layer anchor point are mapped to a shared semantic space to obtain corresponding mapping feature vectors of preset dimensions.

[0116] In some embodiments, a fully connected layer (FC) is used to reduce or increase the dimensionality of the spatiotemporal joint features of semantic anchor points at each level, uniformly mapping them to a 256-dimensional shared semantic space to obtain a corresponding 256-dimensional mapped feature vector. This mapping process preserves the key semantic information of the original features while eliminating dimensional differences between different levels to ensure consistency in subsequent calculations.

[0117] Step S402: Using a multi-head attention mechanism, cosine similarity is calculated on the mapping feature vector of the physical layer anchor point and the mapping feature vector of the regional layer anchor point to obtain a first similarity.

[0118] In some embodiments, a multi-head attention mechanism is used to perform deep semantic association analysis on the physical layer anchor points and the regional layer anchor points based on a quantification method of cosine similarity. First, the 256-dimensional mapping feature vectors of the physical layer and regional layer anchor points are input into the multi-head attention module, and multiple attention heads are used to capture semantic associations of different dimensions in parallel, such as the degree of matching between the temporal dynamic change trend and the spatial topological distribution characteristics. The attention mechanism adaptively adjusts the weight of each feature to highlight key similar information and suppress irrelevant noise. Based on the weighted feature representation, the cosine similarity between the two is calculated to obtain the first similarity, which quantifies their proximity in the shared semantic space. It can be understood that the higher the similarity value, the stronger the consistency between the physical layer anchor point and the regional layer anchor point in spatiotemporal features, and the ability to accurately identify the semantic association strength of anchor points at different levels, ensuring the coordination and consistency of distributed node data.

[0119] Step S403: Using a multi-head attention mechanism, cosine similarity is calculated on the mapping feature vector of the regional layer anchor point and the mapping feature vector of the global layer anchor point to obtain a second similarity.

[0120] Similarly, a multi-head attention mechanism is employed to perform deep semantic correlation analysis on the mapping feature vectors of regional and global anchor points, quantifying their degree of match in a shared semantic space using cosine similarity. Compared to traditional distance metrics, cosine similarity focuses more on vector direction rather than absolute differences, enabling more accurate capture of cross-level logical correlations in distributed systems. This provides an important basis for subsequent global consistency checks, ensuring efficient data collaboration in complex network environments.

[0121] Step S404: The first similarity and the second similarity are merged to obtain feature similarity.

[0122] In some embodiments, a weighted summation approach is used to combine the two similarity metrics, with the weights dynamically adjusted based on the specific application scenario. For example, in scenarios with strong timing dependencies, the weight of the first similarity can be increased to strengthen the temporal continuity constraints between the physical and regional layers; while in wide-area collaboration scenarios, the weight of the second similarity can be increased to optimize global consistency across regions.

[0123] The fusion process not only simply adds similarity values but also incorporates an attention mechanism to dynamically evaluate the importance of associations at different levels, ensuring that key levels have a more pronounced impact on the final similarity. This layered fusion strategy ensures that feature similarity maintains close connections between local anchor points while also taking into account global semantic consistency, providing a comprehensive and reliable basis for subsequent consistency checks. This fusion method effectively balances local detail with global perspective, enabling precise semantic alignment and efficient data collaboration in complex distributed environments.

[0124] Reference Figure 5 As shown, in some embodiments of the present application, the above-mentioned step S105: the reversibility verification network includes a generator and a discriminator; using the preset reversibility verification network to perform a reverse consistency check on the mapping feature vector to obtain a reverse verification parameter, it may also include but not be limited to the following steps S501 to S504.

[0125] Step S501: Use a generator to perform encoding-decoding transformation mapping on the mapped feature vector to obtain a reconstructed feature.

[0126] In some embodiments, the reversibility check network generator uses a U-Net architecture to perform encoding-decoding transformation mapping on the mapped feature vectors to obtain reconstructed features. Specifically, during the encoding phase, four layers of downsampling (max pooling) are used to gradually compress the feature dimensions, extracting multi-scale semantic information while retaining key features. During the decoding phase, four layers of upsampling (transposed convolution) are used to gradually restore the original feature dimensions. Each decoding layer incorporates skip connections from the corresponding encoding layer to enhance detail recovery.

[0127] Step S502 : Calculate the difference index between the reconstructed feature and the spatiotemporal joint feature to obtain the reconstruction error.

[0128] In some embodiments, the mean square error (MSE) of the reconstructed features and the spatiotemporal joint features is calculated to obtain the reconstruction error. That is, the difference in each feature dimension is squared and averaged to obtain an error value in scalar form. The error value intuitively reflects the accuracy of the generator's reconstructed features - the smaller the error, the closer the reconstructed features are to the original features, the more complete the information retained in the mapping process, and the stronger the reversibility; conversely, it indicates that the features may have information loss or distortion during the mapping process, and the network structure or parameters need to be optimized. In addition, the directional consistency of the feature vector can also be further analyzed in combination with cosine similarity to ensure that not only the numerical values are close, but also the semantic directions match. This embodiment does not limit this.

[0129] Step S503: Use a discriminator to perform authenticity discrimination on the reconstructed features to obtain discrimination parameters.

[0130] In some embodiments, the authenticity of the reconstructed features is evaluated by a discriminator to determine whether they conform to the distribution characteristics of the spatiotemporal joint features. Specifically, the discriminator using the PatchGAN architecture analyzes the feature map in a locally perceptual manner, divides the input reconstructed features into multiple 5×5 small areas (patches), and calculates the authenticity score block by block. Each patch is independently extracted through a multi-layer convolutional network, and finally outputs a 5×5 matrix, where each element represents the authenticity probability of the corresponding area. This design enables the discriminator to capture global consistency while paying attention to local detail features, avoiding ignoring subtle differences due to overall judgment. By calculating the average value or weighted score of the entire matrix, the final discriminant parameter is obtained to measure the similarity between the reconstructed features and the real features.

[0131] It can be understood that the output results of the discriminator can be used as the basis for backpropagation of generator optimization on the one hand, and on the other hand provide an auxiliary evaluation indicator of feature mapping quality to ensure the accuracy and robustness of the reverse verification process.

[0132] In some embodiments, the training of the discriminator adopts the loss function of Wasserstein distance + gradient penalty term, and optimizes the reversibility of feature transformation through adversarial training. Specifically, the discriminator learns to distinguish between the reconstructed feature F2 and the spatiotemporal joint feature F1, while the generator optimizes the reconstruction process to make F2 as close to F1 as possible, satisfying the constraint condition G(F(x))≈x. The Wasserstein distance ensures that the discriminator provides a smoother gradient signal, while the gradient penalty term enhances training stability and prevents mode collapse. Finally, the authenticity matrix output by the discriminator is globally pooled to obtain a scalar discriminant parameter, which is used to measure the reconstruction quality and guide the continuous optimization of the generator, thereby ensuring the accuracy and robustness of the reverse verification process.

[0133] Step S504: Obtain reverse verification parameters according to the reconstruction error and the discrimination parameter.

[0134] In some embodiments, the reconstruction error and the discriminant parameter are combined using weighted or normalized fusion. For example, the reconstruction error can be normalized and then added to the discriminant parameter with a fixed weight, or the contribution ratio of the two can be dynamically adjusted through a gating mechanism. The final reverse verification parameter contains both the absolute difference information of the reconstruction error and the relative distribution consistency assessment provided by the discriminator, thereby more accurately measuring the reversibility quality of the feature map.

[0135] Reference Figure 6 As shown, in some embodiments of the present application, the distributed heterogeneous data processing method based on bidirectional consistency distillation may also include but is not limited to the following steps S601 to S603.

[0136] Step S601: Using a consistent memory buffer to store a preset number of historical synchronization strategy records to obtain a plurality of decision storage items.

[0137] In some embodiments, a consistency memory buffer (CMB) stores historical synchronization policy records, providing experience replay support for bidirectional consistency distillation. This buffer uses a ring buffer structure with a fixed capacity of the synchronization policy records of the most recent 100 decision cycles, ensuring the system efficiently utilizes historical data within limited storage space. Each decision item is stored as a four-tuple, including the state vector, action space, immediate reward, and next state.

[0138] Furthermore, to improve sample utilization efficiency, the buffer uses a prioritized experience replay mechanism, using TD error (Temporal Difference Error) as a priority indicator to prioritize the replay of historical records that contribute more to policy optimization. For example, when a synchronization strategy leads to a significant improvement in consistency, its stored items are given a higher sampling weight, accelerating the reproduction and learning of key experiences. This allows for dynamic adaptation to changes in the network environment, guiding the selection of current synchronization strategies through historical experience, and significantly improving the decision-making efficiency and robustness of bidirectional consistency distillation.

[0139] In step S602, the decision storage items are optimized using the Q-learning decider, a synchronization strategy in the action space is selected based on the state vector, and the immediate reward is calculated accordingly to obtain the next state and update it to the consistency memory buffer.

[0140] In some embodiments, the state vector is composed of multi-dimensional features, including reverse verification parameters, network latency, computational load, and layer similarity: feature similarity matrices of the physical layer, regional layer, and global layer (dimension is number of layers × feature similarity). Synchronization strategies in the action space include: forced synchronization strategy, asynchronous compensation strategy, and local rollback strategy. The immediate reward is calculated according to the reward function R:

[0141] R = α·Consistency Accuracy - β·Delay - γ·Energy Consumption

[0142] Among them, α=0.6, β=0.3, and γ=0.1 are adjustable hyperparameters that can be adjusted according to scenario requirements.

[0143] Specifically, the Q-learning algorithm optimizes historical policies in the consistent memory buffer and dynamically selects the optimal synchronization policy for execution. After executing an action, it enters the next state and stores the four-tuple (state vector, action space, immediate reward, and next state) in the consistent memory buffer for subsequent training and optimization. Through continuous iteration, the Q-learning decision maker gradually learns a strategy pattern that adapts to the dynamic network environment, achieving efficient and adaptive synchronous decision-making.

[0144] Step S603 : Dynamically adjust the hyperparameters of the Q-learning decision maker according to the optimized decision storage items to obtain a dynamic arbitration engine.

[0145] In some embodiments, based on the optimized decision storage items, a dynamic arbitration engine with adaptive capabilities is constructed by adjusting the hyperparameters of the Q-learning decision maker online, thereby improving the optimization effect of the synchronization strategy. Specifically, a dynamic learning rate adjustment mechanism is adopted to adaptively adjust the learning rate of the Q value according to the policy update frequency and amplitude recorded in the decision storage item. At the same time, the reward function weight is dynamically calibrated, and the weight coefficients of consistency accuracy, network latency, and computing energy consumption are adjusted in real time by analyzing historical reward feedback data. In high-load or critical mission scenarios, the weights of latency and energy consumption are increased accordingly, giving priority to ensuring response speed and energy efficiency.

[0146] In addition, the arbitration engine fully utilizes the priority replay function in the consistency memory buffer to conduct intensive training on high-value decision samples (such as records with large TD errors), allowing key experiences to more quickly influence Q-value updates, thereby improving the learning efficiency of the overall strategy. Through the above mechanism, the dynamic arbitration engine can intelligently select the optimal synchronization strategy based on the real-time monitored state vector during operation, including actions such as forced synchronization, asynchronous compensation, or local rollback. It can not only effectively coordinate the trade-offs between consistency, latency, and energy consumption, but also maintain efficient and stable decision-making capabilities in complex and changing distributed environments.

[0147] Reference Figure 7 As shown, in some embodiments of the present application, in some embodiments of the present application, the above step S106: selecting a corresponding synchronization strategy to process heterogeneous data based on feature similarity and reverse verification parameters may include but is not limited to the following steps S701 to S705.

[0148] Step S701: construct a state vector based on feature similarity and reverse verification parameters.

[0149] In some embodiments, a multi-dimensional state vector is constructed based on feature similarity, reverse check parameters, and the real-time operating status of the system to provide a comprehensive decision-making basis for the dynamic selection of synchronization strategies. Specifically, feature similarity and reverse check parameters serve as the core semantic features of the state vector. At the same time, two real-time system indicators, network latency and computational load, are introduced. Network latency is obtained by periodically detecting the communication delay between nodes, reflecting the current network transmission efficiency; computational load is dynamically calculated based on the CPU / GPU occupancy rate of each node to represent resource constraints.

[0150] Therefore, the state vector not only captures the semantic consistency characteristics at the data level, but also incorporates the environmental constraints during system runtime, so that the selection of subsequent synchronization strategies can comprehensively consider the dynamic balance between data quality and system resources, thereby improving the robustness and efficiency of distributed heterogeneous data processing.

[0151] Step S702: Using a dynamic arbitration engine, a synchronization strategy in the action space is selected and executed according to the state vector.

[0152] In some embodiments, a dynamic arbitration engine intelligently selects the optimal synchronization strategy in the action space based on the state vector. Specifically, the forced synchronization strategy strictly aligns heterogeneous data across all distributed nodes to ensure global consistency, suitable for high-precision scenarios such as financial transactions and medical data synchronization. The asynchronous compensation strategy allows for delayed synchronization of some distributed nodes, prioritizing system throughput and real-time performance. It is suitable for scenarios such as large-scale IoT monitoring that tolerate temporary inconsistencies. The local rollback strategy performs data rollback on abnormal nodes, isolating the impact of the failure and ensuring overall system stability.

[0153] It's understandable that during the decision-making process, the engine dynamically calculates the expected benefits of each strategy (such as consistency improvement, latency cost, and energy consumption) based on the quantitative indicators of the state vector, and selects the strategy with the best overall benefits for execution. For example, when network latency is high, asynchronous compensation strategies are prioritized to reduce communication overhead; if a serious decrease in data consistency is detected, a forced synchronization strategy is triggered to ensure data reliability. This mechanism enables intelligent and adaptive synchronization strategies, significantly improving the collaborative efficiency and fault tolerance of distributed systems in complex environments.

[0154] Step S703: When the forced synchronization strategy is selected, all distributed nodes in the entire network are forced to synchronously update all heterogeneous data.

[0155] In some embodiments, when the dynamic arbitration engine determines to execute the forced synchronization strategy, it immediately starts the full data synchronization operation across the entire network to ensure that the data status of all distributed nodes is strictly consistent. Specifically, the coordination node broadcasts the forced synchronization instruction to the entire network, and carries the latest full heterogeneous data snapshot and related metadata (such as version number, timestamp). After each distributed node receives the instruction, it first verifies the integrity and consistency of the received data based on the spatiotemporal joint characteristics and reversibility verification results. If the verification passes, the data update is executed. In this way, the strong consistency guarantee mechanism effectively solves the problem of data inconsistency in a distributed environment, which is particularly suitable for scenarios with extremely high requirements for data accuracy, such as financial transactions, industrial control, etc.

[0156] Step S704: When the asynchronous compensation strategy is selected, an incremental update mechanism is used to update the changed heterogeneous data on the distributed nodes.

[0157] In some embodiments, when the dynamic arbitration engine determines to adopt an asynchronous compensation strategy, an incremental update mechanism is initiated to synchronize data on distributed nodes to minimize overhead while ensuring data consistency. Unlike forced synchronization, the asynchronous compensation strategy only performs incremental updates on heterogeneous data that has changed. Specifically, the data that has changed since the last synchronization is first identified, including newly added, modified, or deleted records, and these changes are encapsulated into incremental data packets. Subsequently, the coordinating node asynchronously distributes this incremental data to each target node. Each node independently applies these changes locally, while maintaining a version number or timestamp mechanism to track the data status. To ensure eventual consistency, a verification process can be triggered periodically or under specific conditions to check whether the data status of each node is consistent with the global status. It is suitable for scenarios that have high real-time requirements but allow temporary inconsistencies, such as large-scale IoT monitoring and real-time data analysis. It can effectively improve system throughput and response speed while ensuring data reliability.

[0158] Step S705 , when a local rollback strategy is selected, abnormal distributed nodes are identified and rolled back to the last stable state, triggering an abnormality diagnosis mechanism to repair abnormal heterogeneous data.

[0159] In some embodiments, when the dynamic arbitration engine determines to execute a local rollback strategy, it identifies the anomaly in the distributed node and rolls back its data state to the last stable version. This triggers an anomaly diagnosis mechanism to repair the anomaly. Specifically, through feature similarity analysis and reverse validation parameter evaluation, it locates anomaly nodes whose data consistency deviation exceeds a threshold and marks them as rollback targets. The coordinating node then sends a rollback instruction to these target nodes. Each node, based on its recorded historical version information, restores its local data to the most recently verified state, ensuring that the anomaly does not affect the consistency of the overall system. Simultaneously, the system activates an anomaly diagnosis mechanism, leveraging log analysis, data validation, and pattern recognition to identify the root cause of the anomaly, such as a network transmission error, a compute node failure, or external interference. For issues that can be automatically fixed, the system generates a remediation plan and resynchronizes the affected data. For complex anomalies, a diagnostic report is generated and operations personnel are notified to intervene. This strategy ensures system stability while precisely isolating and repairing anomaly data. It is suitable for addressing sudden failures or data contamination scenarios, effectively preventing the spread of anomalies and further impacting the entire network, and ensuring the continued reliable operation of distributed systems in complex environments.

[0160] In one embodiment of the present application, a distributed heterogeneous data processing method based on bidirectional consistency distillation can correspond to a three-level architecture. Specifically, the first level is a dynamic distributed abstraction module (data perception layer), the second level is a bidirectional consistency distillation tower (feature verification layer), and the third level is a dynamic arbitration engine (decision execution layer). The following is an example of substation voltage anomaly detection as a complete example:

[0161] First, a 110kV substation collected voltage signals using a PMU (synchronized phasor measurement unit) at a sampling rate of 256 points per cycle (corresponding to a resolution of 5.12ms at a 50Hz power frequency), generating a high-density time-series data stream. Spatiotemporal feature extraction was then performed. The GRU model captured a voltage dip lasting five cycles (25.6ms) and predicted its recovery trend (estimated to reach 95% of the rated value by the eighth cycle). A deformable CNN detected a localized dip in the phase B voltage waveform (with a trough depth of 15% of the rated value), localizing the anomaly to occur in cycles 3-7. It also identified that the anomaly affected only phase B and was not a global three-phase fault. Consistency arbitration was then performed, and the physical layer anchor reported a similarity of 0.82 for the phase B voltage anomaly (exceeding the preset threshold of 0.8), indicating cross-node consistency. A reversibility check network verified the feature map was lossless (reconstruction error <3%), confirming data quality. Based on the current network delay (50ms) and computing load (70% CPU usage), the arbitration engine selects an asynchronous compensation strategy, synchronizing only the voltage waveform data of the abnormal phase (phase B) to the associated node, rather than the full three-phase data.

[0162] Compared with the traditional BFT consensus algorithm: all network nodes are required to reach strong consistency on the complete voltage data (three-phase × 256 points / cycle), the communication volume reaches 1.2MB / s, and the decision delay is 120ms. The embodiment of the present application uses asynchronous compensation to synchronize only the abnormal phase data (single-phase × 256 points / cycle), the communication volume is reduced to 450KB / s, the decision delay is optimized to 45ms, and the accuracy of abnormal identification is guaranteed (98.7% vs 97.3% of the traditional solution). This example verifies the efficiency and accuracy of the system in power system anomaly detection, significantly reduces communication overhead and improves real-time response capabilities.

[0163] This approach significantly improves the data consistency and processing efficiency of distributed systems in complex heterogeneous environments through a deep bidirectional verification mechanism and multi-level feature alignment technology, demonstrating significant technical advantages and application value. First, in terms of data alignment capabilities, this method achieves precise representation of heterogeneous data in terms of temporal dynamics and spatial distribution through multi-level semantic anchor construction and spatiotemporal feature extraction. This allows for efficient alignment of features from different nodes and layers in a shared semantic space, making it particularly suitable for addressing the heterogeneity of sensor data in scenarios such as the Industrial Internet of Things. Second, in terms of consistency verification, a combined forward and backward consistency distillation mechanism is employed. The forward distillation quantifies multi-level feature similarities through a multi-head attention mechanism, while the backward distillation verifies the reliability of feature mappings through a reversibility verification network. This ensures that data consistency is accurately maintained even in complex network environments, avoiding the calibration failure caused by semantic gaps in traditional methods. Furthermore, through a dynamic decision-making framework, the method adaptively selects synchronization strategies based on real-time feature similarity and backward verification parameters, achieving a balance between high performance and low overhead, significantly improving the system's robustness and fault tolerance. Finally, the technology's multi-granularity consistency verification capability (bit level, semantic level) breaks through the limitations of traditional consistency algorithms, allowing the system to operate stably in complex scenarios such as dynamic topology and network latency, providing innovative solutions for high-reliability scenarios such as smart grids and cross-cloud database synchronization.

[0164] The present application also provides a distributed heterogeneous data processing system based on bidirectional consistency distillation, which can implement the above-mentioned distributed heterogeneous data processing method based on bidirectional consistency distillation. Figure 8 As shown, in some embodiments of the present application, a distributed heterogeneous data processing system based on bidirectional consistency distillation includes:

[0165] An acquisition module 100 is used to acquire heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a common data stream;

[0166] A clustering module 200 is configured to perform multi-level clustering on the multiple distributed nodes based on the topological information of the multiple distributed nodes to generate multi-level semantic anchor points;

[0167] The feature extraction module 300 is used to extract spatiotemporal features from the general data stream to obtain temporal features and spatial features, and to jointly encode the temporal features and spatial features to obtain spatiotemporal joint features;

[0168] Forward distillation module 400, for performing forward consistency distillation, mapping the spatiotemporal joint features corresponding to the semantic anchor points to a shared semantic space to obtain a mapping feature vector, and using a multi-head attention mechanism to calculate multi-level feature similarity based on the mapping feature vector;

[0169] The reverse distillation module 500 is used to perform reverse consistency distillation, and use a preset reversibility verification network to perform reverse consistency verification on the mapping feature vector to obtain reverse verification parameters;

[0170] The processing module 600 is used to select a corresponding synchronization strategy to process heterogeneous data according to feature similarity and reverse verification parameters.

[0171] The specific implementation of the distributed heterogeneous data processing system based on bidirectional consistency distillation in this embodiment is basically consistent with the specific implementation of the distributed heterogeneous data processing method based on bidirectional consistency distillation mentioned above, and will not be repeated here.

[0172] Figure 9 The electronic device 1000 provided in an embodiment of the present application is shown. The electronic device 1000 includes: a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the computer program is executed, it is used to execute the above-mentioned distributed heterogeneous data processing method based on bidirectional consistency distillation.

[0173] The processor 1001 and the memory 1002 may be connected via a bus or other means.

[0174] Memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs, such as the distributed heterogeneous data processing method based on bidirectional consistency distillation described in the embodiments of this application. Processor 1001 implements the distributed heterogeneous data processing method based on bidirectional consistency distillation by executing the non-transitory software program and instructions stored in memory 1002.

[0175] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store and execute the above-mentioned distributed heterogeneous data processing method based on bidirectional consistency distillation. In addition, the memory 1002 may include a high-speed random access memory 1002, and may also include a non-volatile memory 1002, such as at least one storage device memory device, a flash memory device or other non-volatile solid-state memory device. In some embodiments, the memory 1002 may optionally include a memory 1002 remotely arranged relative to the processor 1001, and these remote memories 1002 may be connected to the electronic device 1000 via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0176] The non-transient software programs and instructions required to implement the above-mentioned distributed heterogeneous data processing method based on bidirectional consistency distillation are stored in the memory 1002. When executed by one or more processors 1001, the above-mentioned distributed heterogeneous data processing method based on bidirectional consistency distillation is executed, for example, Figure 1 Steps S101 to S106 of the method, Figure 2 Steps S201 to S203 of the method, Figure 3 Steps S301 to S303 of the method, Figure 4 Steps S401 to S404 of the method, Figure 5 Steps S501 to S504 of the method, Figure 6 Steps S601 to S603 of the method, Figure 7 Method steps S701 to S705.

[0177] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, which implements the above-mentioned distributed heterogeneous data processing method based on bidirectional consistency distillation when the computer program is executed by the processor. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0178] The distributed heterogeneous data processing method and system based on bidirectional consistency distillation provided in the embodiment of the present application constructs a multi-level semantic anchor system, which can not only capture the direct correlation of heterogeneous data at the physical layer, but also deeply explore the deep connection at the semantic level, and can maintain semantic consistency across nodes and levels in a heterogeneous environment. The multi-granularity consistency verification mechanism mainly realizes the deep fusion of data in time and space dimensions through forward consistency distillation, while the reverse consistency distillation uses a reversible verification network for bidirectional verification, ensuring all-round consistency verification from micro bit changes to macro semantic expressions. This bidirectional, multi-granularity processing paradigm not only significantly improves the accuracy and reliability of heterogeneous data processing, but also realizes the leap of data processing from the bit level to the semantic level, and realizes multi-granularity consistency verification.

[0179] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0180] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, storage device storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0181] It should also be understood that the various implementations provided in the embodiments of this application can be arbitrarily combined to achieve different technical effects. The above is a specific description of the preferred implementation of this application, but this application is not limited to the above implementation. Those skilled in the art can also make various equivalent modifications or substitutions under the conditions that do not violate the spirit of this application.

Claims

1. A distributed heterogeneous data processing method based on bidirectional consistency distillation, characterized in that: include: Obtaining heterogeneous data from multiple distributed nodes and converting the heterogeneous data into a common data stream; Performing multi-level clustering on the plurality of distributed nodes based on topological information of the plurality of distributed nodes to generate multi-level semantic anchor points; Extracting spatiotemporal features from the general data stream to obtain a temporal feature and a spatial feature, and jointly encoding the temporal feature and the spatial feature to obtain a spatiotemporal joint feature; Perform forward consistency distillation to map the spatiotemporal joint features corresponding to the semantic anchor points to a shared semantic space to obtain a mapping feature vector, and use a multi-head attention mechanism to calculate multi-level feature similarity based on the mapping feature vector; Perform reverse consistency distillation, using a preset reversibility verification network to perform reverse consistency verification on the mapping feature vector to obtain reverse verification parameters; According to the feature similarity and the reverse verification parameter, a corresponding synchronization strategy is selected to process the heterogeneous data.

2. The distributed heterogeneous data processing method based on bidirectional consistency distillation according to claim 1 is characterized in that: The topology information includes geographic location information, communication delay parameters and computing power indicators; the semantic anchor points include physical layer anchor points, regional layer anchor points and global layer anchor points; The performing multi-level clustering on the plurality of distributed nodes based on the topological information of the plurality of distributed nodes to generate multi-level semantic anchor points includes: Clustering the distributed nodes using a clustering algorithm according to the geographic location information, the communication delay parameter, and the computing capability indicator to generate the physical layer anchor point; Clustering the physical layer anchor points using a hierarchical clustering algorithm according to the geographic location information, the communication delay parameter, and the computing capability indicator to generate the regional layer anchor points; A graph aggregation algorithm is used to perform network topology aggregation on the regional layer anchor points to generate the global layer anchor points.

3. The distributed heterogeneous data processing method based on bidirectional consistency distillation according to claim 1 is characterized in that: The extracting spatiotemporal features from the general data stream to obtain a temporal feature and a spatial feature, and jointly encoding the temporal feature and the spatial feature to obtain a spatiotemporal joint feature, includes: Using a gated recurrent unit to perform time series modeling on the general data stream, and setting a time window to obtain the temporal characteristics of data changes in the time series; Performing spatial feature extraction on the general data stream using a deformable convolutional neural network, and obtaining spatial features of data changes based on a learnable offset; The temporal features and the spatial features are cross-modally fused according to a multi-head self-attention mechanism to obtain the spatiotemporal joint features.

4. The distributed heterogeneous data processing method based on bidirectional consistency distillation according to claim 2 is characterized in that: Mapping the spatiotemporal joint features corresponding to the semantic anchor points to a shared semantic space to obtain a mapping feature vector, and calculating multi-level feature similarities based on the mapping feature vector using a multi-head attention mechanism, including: Mapping the spatiotemporal joint features of the physical layer anchor point, the regional layer anchor point, and the global layer anchor point to the shared semantic space respectively to obtain corresponding mapping feature vectors of preset dimensions; Using a multi-head attention mechanism, a cosine similarity calculation is performed on the mapping feature vector of the physical layer anchor point and the mapping feature vector of the regional layer anchor point to obtain a first similarity; Using a multi-head attention mechanism, a cosine similarity calculation is performed on the mapping feature vector of the regional layer anchor point and the mapping feature vector of the global layer anchor point to obtain a second similarity; The first similarity and the second similarity are fused to obtain the feature similarity.

5. The distributed heterogeneous data processing method based on bidirectional consistency distillation according to claim 1 is characterized in that: The reversibility check network includes a generator and a discriminator; the reverse consistency check of the mapping feature vector using the preset reversibility check network to obtain the reverse check parameter includes: Using the generator to perform encoding-decoding transformation mapping on the mapped feature vector to obtain a reconstructed feature; Calculating a difference index between the reconstructed feature and the spatiotemporal joint feature to obtain a reconstruction error; Using the discriminator to perform authenticity discrimination on the reconstructed features to obtain discrimination parameters; The reverse check parameter is obtained according to the reconstruction error and the discrimination parameter.

6. The distributed heterogeneous data processing method based on bidirectional consistency distillation according to any one of claims 1 to 5, characterized in that: The method further comprises: Using a consistent memory buffer to store a preset number of historical synchronization strategy records to obtain a plurality of decision storage items; wherein the decision storage items are four-tuples including a state vector, an action space, an immediate reward, and a next state; Utilizing a Q-learning decider to optimize the decision storage item, selecting a synchronization strategy in the action space to execute according to the state vector, and correspondingly calculating the immediate reward, obtaining a next state, and updating the next state to the consistency memory buffer; According to the optimized decision storage items, the hyperparameters of the Q-learning decision maker are dynamically adjusted to obtain a dynamic arbitration engine.

7. The distributed heterogeneous data processing method based on bidirectional consistency distillation according to claim 6 is characterized in that: The selecting a corresponding synchronization strategy to process the heterogeneous data according to the feature similarity and the reverse verification parameter includes: Constructing a state vector according to the feature similarity and the reverse verification parameter; Utilizing the dynamic arbitration engine to select a synchronization strategy in the action space according to the state vector for execution; wherein the synchronization strategy includes a forced synchronization strategy, an asynchronous compensation strategy, and a local rollback strategy; When the forced synchronization strategy is selected, all distributed nodes in the entire network are forced to synchronously update all heterogeneous data; When the asynchronous compensation strategy is selected, an incremental update mechanism is used to update the changed heterogeneous data on the distributed nodes; When the local rollback strategy is selected, abnormal distributed nodes are identified and rolled back to the last stable state, triggering the abnormal diagnosis mechanism to repair the abnormal heterogeneous data.

8. A distributed heterogeneous data processing system based on bidirectional consistency distillation, characterized in that: include: An acquisition module, configured to acquire heterogeneous data from multiple distributed nodes and convert the heterogeneous data into a common data stream; A clustering module, configured to perform multi-level clustering on the plurality of distributed nodes based on topological information of the plurality of distributed nodes to generate multi-level semantic anchor points; a feature extraction module, configured to extract spatiotemporal features from the general data stream to obtain temporal features and spatial features, and jointly encode the temporal features and the spatial features to obtain spatiotemporal joint features; A forward distillation module is used to perform forward consistency distillation, map the spatiotemporal joint features corresponding to the semantic anchor points to a shared semantic space to obtain a mapping feature vector, and calculate multi-level feature similarity based on the mapping feature vector using a multi-head attention mechanism; A reverse distillation module is used to perform reverse consistency distillation, using a preset reversibility verification network to perform reverse consistency verification on the mapping feature vector to obtain reverse verification parameters; A processing module is used to select a corresponding synchronization strategy to process the heterogeneous data according to the feature similarity and the reverse verification parameter.

9. An electronic device, characterized in that: It includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the distributed heterogeneous data processing method based on bidirectional consistency distillation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the distributed heterogeneous data processing method based on bidirectional consistency distillation as described in any one of claims 1 to 7.

Images