A manifold learning-based distributed real-time data processing method and device

Abstract

The present application relates to the technical field of electric digital data processing, and particularly relates to a distributed real-time data processing method and device based on manifold learning, which comprises the following steps: accessing multi-source heterogeneous data, mapping the preprocessed data into a node feature matrix and outputting the data in a streaming manner; constructing a dynamic graph based on the node features, generating an adjacency matrix and updating the topology structure in real time; taking the dynamic graph structure and manifold embedding as inputs, using a graph neural network to aggregate multi-layer information, and outputting stable node-level or graph-level inference results; abstracting each computing unit as a collaborative agent, monitoring the inference deviation and the degree of epidemic mismatch, actively triggering a consensus process, and realizing distributed consistent inference by exchanging low-frequency structure features; dynamically adjusting the aggregation range and spectral weight of the graph neural network according to the dynamic graph size, manifold dimension and model mismatch degree; summarizing and evaluating the confidence of the inference results after the consensus; and thereby realizing stable, robust and scalable data processing and intelligent decision-making.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a distributed real-time data processing method and a distributed real-time data processing device based on manifold learning. Background Technology

[0002] With the rapid development of streaming media technology and artificial intelligence, real-time data processing is becoming increasingly important across various industries, particularly in financial surveillance, smart manufacturing, and the Internet of Things (IoT). These industries typically need to process data from diverse heterogeneous sources, including not only sensor data and text information, but also complex structural relationships and dynamically changing environments. Since this data is often in unstructured form, containing rich semantic information and logical connections, efficiently parsing and utilizing this data has become crucial for achieving intelligent decision-making.

[0003] Existing real-time data processing methods often focus on rule-based parsing or local feature extraction, such as classifying and predicting data using traditional machine learning algorithms. While these methods are effective in simple scenarios, they fall short when dealing with dynamic changes and complex structures. Specifically, on the one hand, existing methods struggle to effectively capture long-term dependencies and global semantic information when processing streaming media data, limiting their information processing capabilities. On the other hand, noise and missing values ​​in the data stream often affect the accuracy of the analysis, and traditional methods lack effective monitoring and correction mechanisms for data quality, often limiting the parsing results to the identification of surface features. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a distributed real-time data processing method based on manifold learning. This method utilizes manifold learning, graph neural networks, and adaptive algorithms to achieve deep semantic understanding and structural analysis of dynamic data, thereby improving the robustness, accuracy, and generalization ability of the system and providing more reliable data support for intelligent decision-making.

[0005] The second objective of this invention is to propose a distributed real-time data processing device based on manifold learning.

[0006] To achieve the above objectives, the first embodiment of this invention proposes a distributed real-time data processing method based on manifold learning, comprising the following steps: acquiring real-time data sets from different data sources, and performing time synchronization, noise suppression, missing data completion, and scale normalization on the real-time data sets; simultaneously extracting node-level feature vectors based on data type and business semantics to obtain a node feature matrix; dynamically generating and updating a graph structure based on the similarity, spatial proximity, and semantic association between node features in the node feature matrix, mapping high-dimensional observation data to a low-dimensional continuous manifold space, characterizing the inherent association structure and its evolutionary characteristics of the data, to obtain a dynamic graph structure, manifold embedding results, and a set of perturbation labels; processing the dynamic graph structure and manifold embedding results, and using a graph neural network with controlled spectral response to perform multi-layer information aggregation on the dynamic graph structure, enhancing low-frequency structural information and suppressing node perturbation during the aggregation process. High-frequency instability caused by dynamic and edge changes yields node-level and graph-level inference results under model mismatch conditions. In scenarios with multiple data sources or multiple computing nodes, each subsystem is abstracted into a collaborative agent. Based on the local node-level and graph-level inference results of each agent, inference bias and manifold mismatch are monitored, and a consensus process is actively triggered. Distributed consistent inference is achieved by exchanging low-frequency structural features or intermediate embedded information to obtain the consensus-based intermediate representation and inference results. The model aggregation range and spectral weights are dynamically adjusted according to the dynamic graph size, manifold dimension, distribution changes, perturbation label set, and model mismatch degree to achieve an adaptive balance between generalization and discriminative capabilities. The control parameters obtained from the adaptive adjustment are fed back. The consensus-based intermediate representation and inference results are summarized and confidence is evaluated to generate a globally consistent output. The system operating status and error information are fed back to form a closed-loop optimization mechanism.

[0007] The distributed real-time data processing method based on manifold learning proposed in this application has the following advantages: by dynamically constructing graphs and embedding manifolds into multi-source heterogeneous real-time data, a robust graph neural network with a controlled spectral response is introduced to achieve stable inference. Combined with a consensus-driven collaborative mechanism and an adaptive generalization control strategy, multi-node inference consistency and dynamic self-adjustment are achieved without sharing the original data. This effectively improves the robustness, generalization ability, and reliability of the system in the face of model mismatch, data drift, and complex operating environments, and has good prospects for widespread application.

[0008] In addition, the distributed real-time data processing method based on manifold learning proposed in the above embodiments of the present invention may also have the following additional technical features:

[0009] Optionally, the real-time dataset is subjected to time synchronization, noise suppression, missing data completion, and scale normalization processing, including: converting the original timestamp of any data into a unified system time using a time alignment mapping function to align data from different data sources to the same time axis; processing the time-synchronized data using a noise suppression operator to obtain a denoised data representation; estimating the denoised data representation using a missing data completion function to obtain a completed data representation; and performing scale normalization processing on the completed data representation.

[0010] Optionally, the graph structure is dynamically generated and updated based on the similarity, spatial proximity, and semantic association between node features in the node feature matrix, including: calculating the association degree between any two nodes based on the node feature matrix; establishing a connection between the nodes when the association degree between any two nodes meets a preset condition to form a dynamic graph structure; calculating the corresponding edge weight for each edge in the dynamic graph based on the association degree between nodes, and constructing a weighted adjacency matrix corresponding to the dynamic graph based on the edge weight; dynamically updating the node set and edge set as real-time data continues to arrive, and updating the weighted adjacency matrix incrementally.

[0011] Optionally, a graph neural network with controlled spectral response is used to perform multi-layer information aggregation on the dynamic graph structure, including: in the first... In the layered graph neural network, node information is updated by a propagation operator with a controlled spectral response; in the graph filtering function, low-frequency components are preferentially retained by weighting the graph spectrum; high-frequency unstable components caused by node perturbations and edge changes are suppressed by using node feature perturbation terms and edge structure perturbation terms in the perturbation label set; the above steps are repeated to perform multi-layer information aggregation on the dynamic graph structure.

[0012] Optionally, monitoring inference bias and popularity mismatch can proactively trigger a consensus process, including: continuously monitoring the inference outputs of different agents; calculating the inference bias for any two agents; monitoring the structural differences between the manifold embedding results of each agent to obtain the popularity mismatch; weightedly fusing the inference bias and popularity mismatch to construct a comprehensive bias index; and proactively triggering a consensus collaboration process when the comprehensive bias index of any two agents exceeds a preset threshold.

[0013] Optionally, the control parameters include: the aggregation range of each layer of the graph neural network, the spectral response control parameters, the discrimination enhancement parameters, and the consistency constraint strength in the consensus collaboration process.

[0014] Optionally, the intermediate representations and inference results after consensus are summarized and confidence is evaluated to generate globally consistent outputs, including: obtaining the inference output set of each collaborative agent under consistency constraints; generating globally consistent inference results and corresponding confidence based on the inference output set using a weighted fusion function and a result confidence evaluation function; and generating node-level, event-level, and global-level outputs based on the globally consistent inference results and corresponding confidence.

[0015] To achieve the above objectives, a second aspect of the present invention proposes a distributed real-time data processing device based on manifold learning, comprising: a real-time data acquisition and preprocessing module, used to acquire real-time data sets from different data sources, and perform time synchronization, noise suppression, missing data completion, and scale normalization on the real-time data sets, while extracting node-level feature vectors based on data type and business semantics to obtain a node feature matrix; a dynamic graph construction and manifold modeling module, used to dynamically generate and update a graph structure based on the similarity, spatial proximity, and semantic association between node features in the node feature matrix, mapping high-dimensional observation data to a low-dimensional continuous manifold space, characterizing the inherent association structure of the data and its evolutionary characteristics, to obtain a dynamic graph structure, manifold embedding results, and a set of perturbation labels; and a manifold learning inference module, used to process the dynamic graph structure and manifold embedding results, employing a graph neural network with controlled spectral response to perform multi-layer information aggregation on the dynamic graph structure, enhancing low-frequency structural information and suppressing node-related information during the aggregation process. The high-frequency instability components caused by perturbations and edge changes yield node-level and graph-level inference results under model mismatch conditions. The consensus-driven collaborative module abstracts each subsystem into a collaborative agent in scenarios with multiple data sources or multiple computing nodes. Based on the local node-level and graph-level inference results of each agent, it monitors inference bias and manifold mismatch, proactively triggering the consensus process. Distributed consistent inference is achieved by exchanging low-frequency structural features or intermediate embedded information to obtain the consensus-based intermediate representation and inference results. The adaptive generalization control module dynamically adjusts the model aggregation range and spectral weights based on the dynamic graph size, manifold dimension, distribution changes, perturbation marker set, and model mismatch degree, achieving an adaptive balance between generalization and discriminative capabilities. It also provides feedback on the control parameters obtained from the adaptive adjustment. The result output and feedback module summarizes and evaluates the confidence level of the consensus-based intermediate representation and inference results, generating a globally consistent output and providing feedback on the system's operating status and error information to form a closed-loop optimization mechanism. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a distributed real-time data processing method based on manifold learning according to an embodiment of the present invention.

[0017] Figure 2This is a block diagram of a distributed real-time data processing device based on manifold learning according to an embodiment of the present invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0019] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0020] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0021] Figure 1 This is a flowchart illustrating a distributed real-time data processing method based on manifold learning according to an embodiment of the present invention, as shown below. Figure 1 As shown, this distributed real-time data processing method based on manifold learning includes the following steps:

[0022] S101: Obtain real-time data sets from different data sources, and perform time synchronization, noise suppression, missing data completion, and scale normalization on the real-time data sets. At the same time, extract node-level feature vectors based on data types and business semantics to obtain node feature matrices.

[0023] As one example, time synchronization, noise suppression, missing data completion, and scale normalization are performed on a real-time dataset, including: converting the original timestamp of any data into a unified system time using a time alignment mapping function to align data from different data sources to the same time axis; processing the time-synchronized data using a noise suppression operator to obtain a denoised data representation; estimating the denoised data representation using a missing data completion function to obtain a completed data representation; and performing scale normalization on the completed data representation.

[0024] As a specific embodiment, real-time data acquisition and preprocessing includes the following detailed steps:

[0025] S11 continuously receives real-time data from multi-source heterogeneous environments and abstracts it into a unified, processable data representation: During system operation, it first receives data in real-time from the multi-source heterogeneous environment through a unified data access interface, and at any given moment... Retrieving data collections from different data sources:

[0026]

[0027] in, Indicates the first Data sources (such as sensor data, business data, log data, or streaming data) at any given time The original observations; each type of data source Including the data source at time Multiple raw data records were generated:

[0028]

[0029] in, This indicates that the data source is at time [time]. The number of valid data entries.

[0030] The system achieves continuous, low-latency acquisition of multi-source data through caching and scheduling mechanisms, and performs unified abstraction of data from different sources to provide standardized input for subsequent processing.

[0031] S12, Time Synchronization Processing of Multi-Source Data: Due to inconsistencies in sampling frequencies and timestamp offsets between different data sources, time synchronization processing of multi-source data is necessary. For the original timestamp of any data entry... Through time alignment mapping function Convert to unified system time:

[0032]

[0033] This aligns data from different data sources to the same timeline, ensuring time consistency during subsequent processing.

[0034] S13, Noise Suppression and Abnormal Interference Elimination: After time synchronization is completed, noise suppression processing is performed on each data entry; let the original observation data after synchronization be... Through noise suppression operator After processing, the denoised data representation is obtained:

[0035]

[0036] in, It can be used as a smoothing filter, outlier suppression, or statistical robustness operator to reduce the impact of high-frequency noise, outliers, or unstable disturbances on data quality.

[0037] S14, Missing Data Completion: For cases where real-time data is missing or incomplete, a missing data completion operation is performed on the denoised data; let the data with missing components be... By using the missing completion function The estimation yields the completed data representation:

[0038]

[0039] This ensures the integrity of the data structure during subsequent feature extraction.

[0040] S15, Feature Scale Normalization: To eliminate dimensional and numerical range differences between different feature dimensions, scale normalization is performed on the completed data. Let the... The statistical mean and standard deviation of each feature dimension are respectively and Then the normalized feature representation is:

[0041]

[0042] The above processing ensures the consistency and comparability of data from different sources in terms of numerical scale.

[0043] S16, Node-level feature vector construction: Based on data type and business semantics, map the normalized data into node-level feature vectors; let the feature mapping function be... Then the first A data entity at time The node feature vectors are represented as follows:

[0044]

[0045] in, , Represents the dimension of node features.

[0046] The feature vectors of all nodes are summarized to form the time step. Node feature matrix:

[0047]

[0048] The node feature matrix serves as the basic input for subsequent dynamic graph construction and manifold modeling modules.

[0049] S17, Streaming Output and Timing Consistency Maintenance: Node Feature Matrix The data is output sequentially in a streaming manner to the dynamic graph construction and manifold modeling module, maintaining the temporal order and integrity of the data to form a feature sequence that evolves over time.

[0050]

[0051] This enables continuous characterization of data changes in dynamic environments, providing a stable and unified data foundation for subsequent real-time modeling and inference based on graph structures.

[0052] Through the above steps, a systematic processing of multi-source heterogeneous real-time data is achieved in terms of time alignment, noise suppression, missing data completion, scale unification, and feature representation. This effectively reduces the impact of data inconsistency and random disturbances on system performance and improves the stability and generalization ability of subsequent dynamic graph construction and inference in real-time scenarios.

[0053] S102 dynamically generates and updates the graph structure based on the similarity, spatial proximity, and semantic association between node features in the node feature matrix, maps high-dimensional observation data to a low-dimensional continuous manifold space, characterizes the inherent association structure of the data and its evolution characteristics, and obtains the dynamic graph structure, manifold embedding results, and perturbation label set.

[0054] As one embodiment, a graph structure is dynamically generated and updated based on the similarity, spatial proximity, and semantic association between node features in the node feature matrix. This includes: calculating the association degree between any two nodes based on the node feature matrix; establishing a connection between the nodes when the association degree between any two nodes meets a preset condition, thus forming a dynamic graph structure; calculating the corresponding edge weight for each edge in the dynamic graph based on the association degree between nodes, and constructing a weighted adjacency matrix corresponding to the dynamic graph based on the edge weight; and dynamically updating the node set and edge set as real-time data continues to arrive, and updating the weighted adjacency matrix incrementally.

[0055] As a specific example, dynamic graph construction and manifold modeling include the following detailed steps:

[0056] S21, Receive the node feature matrix as graph construction input: The system at time... Receive the node feature matrix output by the real-time data acquisition and preprocessing module:

[0057]

[0058] in, Indicates the first Each node at time... eigenvectors, This indicates the current number of nodes.

[0059] S22, Calculate the multidimensional correlation degree between nodes: Based on the node feature matrix, the system performs multidimensional correlation calculations on any node. With nodes Calculate the correlation degree to determine whether a connection relationship has been established between nodes.

[0060] The degree of association between nodes is defined as:

[0061]

[0062] in, This is a function for calculating the degree of association, which can be defined based on the similarity, spatial proximity, or semantic association between node features.

[0063] S23, Generate dynamic graph nodes and edge structure: when a node With nodes correlation When preset conditions are met, connections are established between nodes to form a dynamic graph structure:

[0064]

[0065] in, Represents a set of nodes. Represents the set of edges.

[0066] S24, Calculate edge weights and construct a weighted adjacency matrix: for each edge in the dynamic graph. Calculate the corresponding edge weights based on the degree of association between nodes:

[0067]

[0068] in, This represents the weight mapping function.

[0069] The system constructs a weighted adjacency matrix corresponding to the dynamic graph based on the edge weights:

[0070]

[0071] S25, Real-time Update of Dynamic Graph Topology: As real-time data continues to arrive, the system updates the node set... and edge set Perform dynamic updates and use an incremental approach to update the weighted adjacency matrix:

[0072]

[0073] in, It represents the information of newly added, deleted, or reweighted edges, thereby enabling the real-time evolution of the dynamic graph topology.

[0074] S26, Constructing the Laplacian operator for dynamic graphs: To characterize the structural properties of dynamic graphs, the system constructs the degree matrix based on the weighted adjacency matrix.

[0075]

[0076] And further construct the dynamic graph Laplace matrix:

[0077]

[0078] Alternatively, construct a normalized Laplace matrix:

[0079]

[0080] Used to characterize the local smoothness and global geometric features of a graph structure.

[0081] S27, Perform manifold embedding modeling based on dynamic graphs: The system assumes that node features change over time on a low-dimensional continuous manifold. To obtain this hidden manifold structure, manifold embedding mapping is performed on the high-dimensional node features. Let the embedding function be... Then the node The low-dimensional manifold is represented as:

[0082]

[0083] in, .

[0084] All node embeddings constitute a manifold representation matrix:

[0085]

[0086] S28, Embedding Optimization Maintaining Graph Structure Consistency: In the manifold embedding process, embedding consistency is achieved by maintaining the structural relationships between nodes. The optimization objective is expressed as:

[0087]

[0088] This preserves the geometric structure characteristics of the dynamic graph in a low-dimensional space.

[0089] S29, Marking Node Feature Perturbations and Edge Structure Changes: Since data distribution and relationships may change in the real-time environment, the system detects changes in node features and edge weights at adjacent time points. The characteristic change (node ​​characteristic perturbation) between adjacent time steps is defined as:

[0090]

[0091] The change in edge weight is defined as:

[0092]

[0093] When the change exceeds a preset threshold, the corresponding node or edge is marked as a disturbance, forming a disturbance set:

[0094]

[0095] S210, Output dynamic graph and manifold modeling results: Output the constructed dynamic graph structure, manifold embedding results and perturbation labeling information to the subsequent manifold learning inference module and consensus collaboration module as the basic input for robust inference and consistency control.

[0096] Through the above steps, dynamic graph modeling and low-dimensional manifold representation of the intrinsic correlation structure of real-time data are realized. While ensuring the integrity of structural information, the modeling complexity is reduced. Furthermore, the perturbation labeling mechanism provides structural support for robust inference under the conditions of model mismatch and distribution drift.

[0097] S103 processes the dynamic graph structure and manifold embedding results, and uses a graph neural network with controlled spectral response to perform multi-layer information aggregation on the dynamic graph structure. During the aggregation process, low-frequency structural information is enhanced and high-frequency unstable components caused by node perturbations and edge changes are suppressed. Under the condition of model mismatch, node-level inference results and graph-level inference results are obtained.

[0098] As one embodiment, a graph neural network with controlled spectral response is used to perform multi-layer information aggregation on a dynamic graph structure, including: in the first... In the layered graph neural network, node information is updated through a propagation operator with a controlled spectral response; in the graph filtering function, low-frequency components are preferentially preserved by weighting the graph spectrum; node feature perturbation terms and edge structure perturbation terms in the perturbation label set are used to suppress high-frequency unstable components caused by node perturbations and edge changes; the above steps are repeated to perform multi-layer information aggregation on the dynamic graph structure.

[0099] As a specific embodiment, manifold learning inference includes the following detailed steps:

[0100] S31, Receive dynamic graph structure and manifold embedding as inference input: The system at time... Receives the dynamic graph structure output by the Dynamic Graph Construction and Manifold Modeling module:

[0101]

[0102] The corresponding node embeddings constitute the manifold representation matrix:

[0103]

[0104] in, The system represents the low-dimensional representation of nodes in the implicit manifold space; the system represents the manifold representation matrix. As the initial state of the node, it serves together with the dynamic graph structure as the input to the graph neural network inference module.

[0105] S32, Constructing the spectral representation of the dynamic graph: To improve generalization robustness under model mismatch conditions, a graph neural network with controlled spectral response is used for information propagation and feature updating; based on the weighted adjacency matrix of the dynamic graph, the system constructs a normalized Laplacian matrix. And perform feature decomposition on it:

[0106]

[0107] in, Represents the set of eigenvalues ​​of the graph spectrum. This represents the corresponding eigenvector matrix, used to characterize the frequency domain structure of the dynamic graph.

[0108] S33, Initialize the node states of the graph neural network: The system represents the matrix in a manifold. As the initial layer input of a graph neural network, it is defined as follows:

[0109]

[0110] S34, Perform graph information propagation for controlled spectral response: in the... In a layered graph neural network, the system updates node information through a propagation operator with a controlled spectral response. The update process is represented as follows:

[0111]

[0112] in, Indicates the first Node state representation in a layered graph neural network. For learnable parameter matrix, It is a nonlinear mapping function. This is a parameterized graph filtering function.

[0113] S35, perform low-frequency priority weighting control on the graph spectrum: in the graph filtering function In the middle, the system analyzes the graph spectrum. Weighted control is applied to preferentially preserve low-frequency components. Its frequency domain expression is:

[0114]

[0115] in:

[0116]

[0117] And satisfy: Follow It increases and then decreases monotonically.

[0118] This effectively suppresses high-frequency components (corresponding to node perturbations and edge changes) during information propagation, enhancing the ability to model global structure and manifold consistency.

[0119] S36, Perform robust inference under model mismatch conditions: when the manifold model during the training phase... Manifold models in the real-time inference phase When mismatches exist, the node and edge structure is represented as follows:

[0120]

[0121] in, and These represent the node feature perturbation term and the edge structure perturbation term, respectively.

[0122] Since the low-frequency-priority graph filtering mechanism has a suppressive effect on high-frequency disturbances, its inference error satisfies:

[0123]

[0124] Where, constant The value decreases as the number of nodes increases, thus ensuring stable inference performance of the system under model mismatch conditions.

[0125] S37. Perform multi-layer graph aggregation and generate higher-order representations: By repeatedly executing steps S34 to S36, the system completes the information aggregation of the multi-layer graph neural network, gradually obtaining node representations that take into account both local information and global structure (node ​​representations of the final layer):

[0126]

[0127] This representation integrates local neighborhood information with global manifold structure features.

[0128] Step S38: Generate node-level or graph-level inference output: Based on application requirements, the system generates inference results at different levels based on the node representation of the final layer (outputting inference results at different levels according to application requirements):

[0129] Node-level reasoning results:

[0130]

[0131] Graph-level reasoning results:

[0132]

[0133] in, Represents the node-level prediction function. This represents graph-level aggregation functions.

[0134] S39. Output intermediate representations for subsequent modules: Output intermediate node representations and inference results to the consensus initiative collaboration module and the adaptive generalization control module as important inputs for consistency adjustment and parameter adaptation.

[0135] Through the above steps, the system prioritizes the retention of low-frequency, stable manifold structure information during dynamic graph inference, effectively suppressing the instability caused by node disturbances and edge changes. This enables the system to maintain stable output and good generalization ability even when there is a mismatch between the training manifold and the real-time manifold, providing a highly reliable inference foundation for distributed consensus and adaptive control.

[0136] S104, in scenarios with multiple data sources or multiple computing nodes, abstracts each subsystem into a collaborative intelligent agent. Based on the local node-level reasoning results and graph-level reasoning results of each intelligent agent, it monitors the degree of reasoning deviation and popular mismatch, actively triggers the consensus process, and achieves distributed consistent reasoning by exchanging low-frequency structural features or intermediate embedded information to obtain the intermediate representation and reasoning results after consensus.

[0137] As an example, monitoring inference bias and popularity mismatch and proactively triggering the consensus process includes: continuously monitoring the inference output of different agents; calculating the inference bias of any two agents; monitoring the structural differences between the manifold embedding results of each agent to obtain the popularity mismatch; weightedly fusing the inference bias and popularity mismatch to construct a comprehensive bias index; and proactively triggering the consensus collaboration process when the comprehensive bias index of any two agents is greater than a preset threshold.

[0138] As a specific example, consensus-driven proactive collaboration includes the following detailed steps:

[0139] S41, abstracting distributed computing units as collaborative agents: In distributed scenarios with multiple data sources, multiple computing nodes, or multiple subgraphs, the system abstracts each independent computing unit as a set of collaborative agents. Let the system contain... The set of *n* intelligent agents is represented as:

[0140]

[0141] Each intelligent agent Associate a local dynamic graph:

[0142]

[0143] and its corresponding node manifold representation:

[0144]

[0145] Each agent independently performs manifold robust graph inference based on a graph neural network model.

[0146] S42. Each agent performs local manifold inference and generates intermediate representations: at time... The system controls each agent to perform inference based on its local dynamic graph and manifold embedding. The local inference results of an agent can be represented as:

[0147]

[0148] in, This represents intermediate embeddings at the node or graph level.

[0149] And further generate the corresponding local prediction output:

[0150]

[0151] in, This is a task-related prediction mapping function.

[0152] S43. Monitoring inference bias between agents: To determine whether a consensus process needs to be initiated, the system continuously monitors the inference outputs of different agents, and for any two agents... and Calculate its inference bias:

[0153]

[0154] This reflects the degree of consistency in the decisions made by different agents at the same moment.

[0155] S44. Monitoring the degree of manifold mismatch between agents: In addition to the inference output, the system further monitors the structural differences between the manifold representations of each agent, and the manifold embedding bias is defined as:

[0156]

[0157] in, It represents a distance metric in the manifold space, used to measure the consistency of different agents in modeling data structures.

[0158] S45. Construct a comprehensive deviation index and determine whether consensus is triggered: The system weights and fuses inference bias with manifold mismatch to construct a comprehensive deviation index:

[0159]

[0160] in, and These are the weighting coefficients.

[0161] Unlike passive synchronization, an active triggering mechanism is introduced; when any pair of agents satisfies:

[0162]

[0163] The system proactively triggers the consensus and coordination process without waiting for periodic synchronization or external instructions. This is a preset threshold.

[0164] This mechanism can intervene early before the bias spreads on a large scale, effectively avoiding the cumulative amplification of model mismatch in a distributed environment.

[0165] S46. Extracting Low-Frequency Consistency Representations from Each Agent: After consensus is triggered, the system controls each agent to extract low-frequency components related to global structural stability from its local intermediate representation. The low-frequency representation of an agent is defined as:

[0166]

[0167] in, This indicates the preceding part of the Laplacian matrix of the corresponding graph. 1 eigenvector.

[0168] Low-frequency information represents the set of information that constitutes the consensus mechanism for the intelligent agent.

[0169]

[0170] S47. Exchanging consensus information without sharing original data: During the consensus phase, the system only allows agents to exchange the aforementioned low-frequency consensus information. It does not share raw data, complete model parameters, or high-frequency features, thereby achieving structural consistency alignment while ensuring data privacy and communication efficiency.

[0171] S48. Perform consensus update based on consistency constraints: After receiving low-frequency consistency information from other agents, the first... Each agent adjusts its local representation according to the following consensus update rules (updates its local representation according to consensus):

[0172]

[0173] in, Indicates the consensus step size, Representation and intelligent agent The set of neighbors for communication.

[0174] S49. Iterative updates until a globally consistent state is reached: The system iterates through S48 multiple times, causing the intermediate representations of each agent to gradually converge; when the following conditions are met:

[0175]

[0176] It is assumed that all agents reach a globally consistent reasoning representation at the current moment, where, Represents globally consistent reasoning.

[0177] S410, Output consensus results and enter subsequent control process: The system outputs the intermediate representation and inference results after consensus to the result output and feedback module and the adaptive generalization control module as the basis for subsequent decision output and system parameter adjustment.

[0178] Through the above steps, based on multi-agent parallel reasoning, real-time monitoring of reasoning bias and manifold mismatch and proactive consensus triggering are realized. By exchanging only low-frequency structural features and intermediate embeddings, high-frequency instability caused by model mismatch and structural drift is effectively suppressed. While ensuring communication efficiency and data privacy, the reasoning results in the distributed system are quickly converged to a globally consistent state, significantly reducing the risk of decision disagreement.

[0179] S105 dynamically adjusts the model aggregation range and spectral weights based on the dynamic graph size, manifold dimension, distribution changes, disturbance marker set, and model mismatch degree, achieving an adaptive balance between generalization and discriminative capabilities, and provides feedback on the control parameters obtained from the adaptive adjustment.

[0180] As an example, the control parameters include: the aggregation range of each layer of the graph neural network, the spectral response control parameters, the discrimination enhancement parameters, and the consistency constraint strength in the consensus collaboration process.

[0181] As a specific embodiment, adaptive generalization control includes the following detailed steps:

[0182] S51, Joint Evaluation of System Operating State and Structural Complexity: Used for continuous perception and quantitative evaluation of system operating state to characterize key factors affecting the model's generalization and discriminative abilities. Let's assume at time... The node size of the dynamic graph is The corresponding manifold embedding dimension is Meanwhile, by comparing the differences in manifold structure between the training and online inference phases, the degree of model mismatch is defined as:

[0183]

[0184] in This represents the distance metric function in the manifold space. Furthermore, considering both node feature perturbations and edge structure changes, a comprehensive mismatch index is introduced:

[0185]

[0186] in, and These represent the node attribute perturbation term and the topology perturbation term, respectively. This is the weighting coefficient; this indicator serves as the core control variable for subsequent adaptive adjustment.

[0187] S52, Adaptive adjustment mechanism for aggregation range based on dynamic graph size: Considering the impact of node size changes on model generalization error and oversmoothing risk, the aggregation range is adjusted according to the number of nodes in the dynamic graph. The aggregation range of the graph neural network is dynamically adjusted; let the first... The effective aggregation radius of the layer is Its update rule is defined as follows:

[0188]

[0189] in, Indicates the initial aggregation radius. This represents the adjustment coefficient; through this mechanism, the scope of structure perception is expanded as the number of nodes increases, the generalization error is mitigated by the sample size effect, and over-aggregation is avoided in small-scale scenarios.

[0190] S53, High-frequency response suppression and low-frequency consistency enhancement control strategy: When the system detects a comprehensive mismatch index... During enhancement, the spectral response characteristics of the graph neural network are adaptively controlled to suppress high-frequency instability components caused by structural perturbations; the spectral weighting function of the graph filter is given by:

[0191]

[0192] in, Eigenvalues ​​of the graph Laplacian operator; spectral response control parameters Dynamically adjusts according to the degree of mismatch:

[0193]

[0194] when As the frequency increases, the response corresponding to the high-frequency component is rapidly attenuated, thereby strengthening the consistency of the low-frequency structure and improving the inference robustness of the system under model mismatch conditions.

[0195] S54. Stability triggering mechanism for enhanced discrimination capability: When the system detects that the data distribution change tends to stabilize, i.e., the comprehensive mismatch index... Below the preset threshold At this time, the model's discriminative power is appropriately released to improve its performance in recognizing fine-grained structural differences. Specifically, the spectral response function is adjusted by introducing a discriminative enhancement term:

[0196]

[0197] in, To determine the enhancement parameters, they are gradually increased during the stable phase, so that the model maintains an appropriate response in the mid-to-high frequency range, thereby improving recognition accuracy without significantly sacrificing generalization ability.

[0198] S55. Cross-module feedback and collaborative adjustment of control parameters: The control parameters obtained from adaptive adjustment are fed back to the manifold learning inference module and the consensus initiative collaboration module to form a system-level collaborative control mechanism. The feedback parameters include: the aggregation range of each layer of the graph neural network. ; Spectrum response control parameters With discriminative enhancement parameters Consistency constraint strength in the consensus coordination process The consensus step size can be updated in conjunction with the degree of model mismatch.

[0199]

[0200] This accelerates the convergence speed of consensus among multiple agents during model mismatch enhancement and reduces the risk of over-constraint during the stable phase.

[0201] Through the above steps, the system continuously perceives the size of the dynamic graph, manifold complexity, and model mismatch during operation, and dynamically adjusts the aggregation range and spectral response characteristics of the graph neural network accordingly. When model mismatch is significant, the robustness of inference generalization is prioritized, and the model's discriminative ability is gradually released when the data distribution is stable. This ensures that the optimal balance between system performance and stability is maintained at different stages of operation, providing reliable adaptive control and long-term stable operation guarantee for real-time intelligent inference systems in distributed and dynamic environments.

[0202] S106 summarizes and evaluates the confidence level of the intermediate representations and inference results after consensus, generates globally consistent output, and provides feedback on system operating status and error information to form a closed-loop optimization mechanism.

[0203] As an example, the intermediate representations and inference results after consensus are summarized and confidence is evaluated to generate globally consistent outputs, including: obtaining the inference output set of each collaborative agent under consistency constraints; generating globally consistent inference results and corresponding confidence based on the inference output set using a weighted fusion function and a result confidence evaluation function; and generating node-level, event-level, and global-level outputs based on the globally consistent inference results and corresponding confidence.

[0204] As a specific implementation, the result output and feedback are used to uniformly summarize, reliably evaluate, and structure the system inference results after consensus-driven proactive collaboration. Furthermore, a state feedback mechanism drives the dynamic optimization of the front-end inference module, adaptive generalization control module, and consensus collaboration module, thereby constructing a closed-loop real-time inference system with continuous operation and self-optimization capabilities. The result output and feedback include the following detailed steps:

[0205] S61, Consensus Inference Result Summary and Confidence Assessment: After completing multi-agent consensus collaboration, the system obtains the inference output set of each collaborating agent under consistency constraints:

[0206]

[0207] in, Indicates the first The reasoning results of an agent under consensus constraints are represented as node embeddings, event-level decision results, or decision vectors.

[0208] First, a globally consistent inference result is generated using a weighted fusion function:

[0209]

[0210] Among them, weight The determination is adaptively based on the historical stability, consistency error, and long-term reliability of the corresponding intelligent agent.

[0211] Meanwhile, to quantify the reliability level of the current system output, a result confidence evaluation function is introduced:

[0212]

[0213] This confidence level is used to characterize the consistency of multi-agent reasoning results in the consensus space and serves as an important reference for subsequent decision outputs and control feedback.

[0214] S62. Multi-granularity reasoning result and decision output mechanism: Based on different application scenarios and interface requirements, it supports multi-granularity structured output of reasoning results, including node-level, event-level, and global-level decision forms.

[0215] Node-level output:

[0216]

[0217] Used to describe the state determination, attribute estimation, or anomaly identification results of a single node;

[0218] Event-level output:

[0219]

[0220] in, Representation and event The relevant set of nodes;

[0221] Global output:

[0222]

[0223] Used to generate system-level decision results or control instructions.

[0224] The above outputs are sent to upper-layer business systems, monitoring platforms, or decision engines through standardized interfaces to ensure consistency, interpretability, and engineering scalability of the results.

[0225] S63. Feedback Mechanism for Operating Status, Error, and Distribution Changes: To form a closed-loop self-optimizing structure, key status indicators during system operation are continuously monitored, and a unified feedback signal is constructed, including:

[0226] Consensus Residual:

[0227]

[0228] Model mismatch index:

[0229]

[0230] Measurement of data distribution change:

[0231]

[0232] in, This represents a function that measures the difference in distribution.

[0233] The above indicators are uniformly mapped to a system state vector:

[0234]

[0235] It is then fed back to the manifold learning inference module, the adaptive generalization control module, and the consensus initiative coordination module to dynamically adjust the aggregation range, spectral response parameters, and consistency constraint strength.

[0236] S64. External System Interface and Collaborative Control Support: This module reserves a standardized bidirectional interface for collaborative interaction with external control systems, scheduling platforms, or management systems. External systems can generate control commands based on global decision outputs and confidence information.

[0237]

[0238] The execution results or environmental feedback are then transmitted back to the system as important inputs for the construction of dynamic graphs, manifold modeling, and control adjustment in the next moment, thereby enabling collaborative decision-making and linkage control across systems.

[0239] Through the above steps, a unified and reliable aggregation of distributed inference results, multi-granular structured output, and system-level state feedback are achieved. While ensuring the interpretability and engineering usability of the inference results, a closed-loop optimization mechanism that runs through "inference-consensus-control-re-inference" is constructed, enabling the system to have continuous operation capability, adaptive optimization capability, and engineering-level stability in complex dynamic environments, providing complete and reliable output and control support for distributed real-time intelligent inference systems.

[0240] In summary, this application's distributed real-time data processing method based on manifold learning firstly continuously feeds in real-time data from different data sources, performing time synchronization, noise suppression, missing data completion, and scale normalization. Simultaneously, node-level feature vectors are extracted to provide a consistent data description for subsequent graph modeling. Next, the graph structure is dynamically generated and updated based on the similarity, spatial proximity, or semantic relationships between node features, mapping high-dimensional observation data to a low-dimensional continuous manifold space to characterize the inherent relational structure and evolutionary characteristics of the data. Then, a graph neural network with controlled spectral response is used to perform multi-layer information aggregation on the dynamic graph. During the aggregation process, low-frequency structural information is enhanced, and high-frequency unstable components caused by node perturbations and edge changes are suppressed, thereby improving the output quality even under model mismatch conditions. The system generates stable node-level or graph-level inference results. Then, in scenarios with multiple data sources or multiple computing nodes, each subsystem is abstracted into a collaborative intelligent agent. Based on monitoring inference bias and manifold mismatch, this agent proactively triggers a consensus process, achieving distributed consistent inference by exchanging low-frequency structural features or intermediate embedded information. Next, the model aggregation range and spectral weights are dynamically adjusted according to the dynamic graph size, manifold dimension, and distribution changes, achieving an adaptive balance between generalization and discriminative capabilities. Finally, the collaborative inference results are summarized and confidence levels are evaluated, and the system's operating status and error information are fed back to the front-end module and external platform, forming a closed-loop optimization mechanism. Thus, stable, robust, and scalable data processing and intelligent decision-making can be achieved under dynamic, multi-source, and distribution drift conditions.

[0241] To implement the above embodiments, this invention also proposes a distributed real-time data processing device based on manifold learning, such as... Figure 2 As shown, the distributed real-time data processing device based on manifold learning includes: a real-time data acquisition and preprocessing module 10, a dynamic graph construction and manifold modeling module 20, a manifold learning inference module 30, a consensus initiative collaboration module 40, an adaptive generalization control module 50, and a result output and feedback module 60.

[0242] The real-time data acquisition and preprocessing module 10 is used to acquire real-time data sets from different data sources and perform time synchronization, noise suppression, missing data completion, and scale normalization on the real-time data sets. Simultaneously, it extracts node-level feature vectors based on data type and business semantics to obtain a node feature matrix. The dynamic graph construction and manifold modeling module 20 is used to dynamically generate and update the graph structure based on the similarity, spatial proximity, and semantic association between node features in the node feature matrix. This maps high-dimensional observation data to a low-dimensional continuous manifold space, characterizing the inherent association structure and evolutionary characteristics of the data to obtain the dynamic graph structure, manifold embedding results, and perturbation label set. The manifold learning and inference module 30 is used to process the dynamic graph structure and manifold embedding results. It employs a graph neural network with controlled spectral response to perform multi-layer information aggregation on the dynamic graph structure. During the aggregation process, it enhances low-frequency structural information and suppresses high-frequency unstable components caused by node perturbations and edge changes, under model mismatch conditions. The system obtains node-level and graph-level inference results. The consensus-driven collaborative module 40, in scenarios with multiple data sources or multiple computing nodes, abstracts each subsystem into a collaborative agent. Based on the local node-level and graph-level inference results of each agent, it monitors inference bias and manifold mismatch, proactively triggers the consensus process, and achieves distributed consistent inference by exchanging low-frequency structural features or intermediate embedded information to obtain the consensus-based intermediate representation and inference results. The adaptive generalization control module 50 dynamically adjusts the model aggregation range and spectral weights according to the dynamic graph size, manifold dimension, distribution changes, perturbation label set, and manifold mismatch, achieving an adaptive balance between generalization and discriminative capabilities, and providing feedback on the control parameters obtained through adaptive adjustment. The result output and feedback module 60 summarizes and evaluates the confidence level of the consensus-based intermediate representation and inference results, generates a globally consistent output, and provides feedback on the system's operating status and error information to form a closed-loop optimization mechanism.

[0243] It should be noted that the above description and examples of the distributed real-time data processing method based on manifold learning also apply to the distributed real-time data processing device based on manifold learning in this embodiment, and will not be repeated here.

[0244] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0245] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0246] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0247] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0248] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0249] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0250] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0251] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0252] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0253] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0254] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0255] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A distributed real-time data processing method based on manifold learning, characterized in that, Includes the following steps: Acquire real-time data sets from different data sources, and perform time synchronization, noise suppression, missing data completion, and scale normalization on the real-time data sets. At the same time, extract node-level feature vectors based on data types and business semantics to obtain node feature matrices. The graph structure is dynamically generated and updated based on the similarity, spatial proximity and semantic association between node features in the node feature matrix. This maps high-dimensional observation data to a low-dimensional continuous manifold space, characterizes the inherent association structure of the data and its evolution characteristics, and obtains dynamic graph structure, manifold embedding results and perturbation label set. The dynamic graph structure and manifold embedding results are processed, and a graph neural network with controlled spectral response is used to perform multi-layer information aggregation on the dynamic graph structure. During the aggregation process, low-frequency structural information is enhanced and high-frequency unstable components caused by node perturbation and edge changes are suppressed. Under the condition of model mismatch, node-level inference results and graph-level inference results are obtained. In scenarios with multiple data sources or multiple computing nodes, each subsystem is abstracted into a collaborative intelligent agent. Based on the local node-level reasoning results and graph-level reasoning results of each collaborative intelligent agent, the reasoning deviation and manifold mismatch are monitored, and the consensus process is actively triggered. Distributed consistent reasoning is achieved by exchanging low-frequency structural features or intermediate embedded information to obtain the intermediate representation and reasoning results after consensus. The model aggregation range and spectral weights are dynamically adjusted based on the size of the dynamic graph, manifold dimension, distribution changes, disturbance marker set, and model mismatch degree to achieve an adaptive balance between generalization ability and discriminative ability, and the control parameters obtained by the adaptive adjustment are fed back. The intermediate representations and inference results after consensus are summarized and confidence is evaluated to generate a globally consistent output. The system's operating status and error information are fed back to form a closed-loop optimization mechanism. The method employs a graph neural network with controlled spectral response to perform multi-layer information aggregation on the dynamic graph structure, including: In the l-th layer of the graph neural network, node information is updated by a propagation operator with a controlled spectral response, where l represents the layer number of the graph neural network; In the graph filtering function, the low-frequency components are preferentially retained by weighting the graph spectrum. The node feature perturbation term and edge structure perturbation term in the perturbation label set are used to suppress the high-frequency unstable components caused by node perturbation and edge changes; Repeat the above steps to perform multi-layer information aggregation on the dynamic graph structure; This includes monitoring inference bias and manifold mismatch, and proactively triggering the consensus process, including: Continuously monitor the inference output of different cooperative agents, and calculate the inference deviation between any two cooperative agents; Monitor the structural differences between the manifold embedding results of each cooperative agent to obtain the degree of manifold mismatch; The inference bias and the degree of manifold mismatch are weighted and fused to construct a comprehensive bias index; When the combined deviation index of any two collaborative agents exceeds a preset threshold, the consensus collaboration process is proactively triggered.

2. The distributed real-time data processing method based on manifold learning as described in claim 1, characterized in that, The real-time dataset undergoes time synchronization, noise suppression, missing data completion, and scale normalization processes, including: For any data's original timestamp, a time alignment mapping function is used to convert it into a unified system time, so that data from different data sources can be aligned to the same timeline; The time-synchronized data is processed using a noise suppression operator to obtain a denoised data representation; The denoised data representation is estimated using a missing data completion function to obtain the completed data representation; The completed data representation is then subjected to scale normalization.

3. The distributed real-time data processing method based on manifold learning as described in claim 1, characterized in that, The graph structure is dynamically generated and updated based on the similarity, spatial proximity, and semantic association between node features in the node feature matrix, including: Based on the node feature matrix, calculate the correlation degree between any two nodes; When the correlation between any two nodes meets the preset conditions, a connection relationship is established between the nodes to form a dynamic graph structure; For each edge in the dynamic graph, the corresponding edge weight is calculated based on the correlation between nodes, and a weighted adjacency matrix corresponding to the dynamic graph is constructed based on the edge weight. As real-time data continues to arrive, the node set and edge set are dynamically updated, and the weighted adjacency matrix is ​​updated incrementally.

4. The distributed real-time data processing method based on manifold learning as described in claim 1, characterized in that, The control parameters include: the aggregation range of each layer of the graph neural network, the spectral response control parameters, the discrimination enhancement parameters, and the consistency constraint strength in the consensus collaboration process.

5. The distributed real-time data processing method based on manifold learning as described in claim 1, characterized in that, The intermediate representations and inference results after consensus are summarized and their confidence is evaluated to generate a globally consistent output, including: Obtain the set of inference outputs of each collaborative agent under consistency constraints; Based on the inference output set, a weighted fusion function and a result confidence evaluation function are used to generate globally consistent inference results and corresponding confidence levels; Node-level, event-level, and global-level outputs are generated based on the globally consistent inference results and the corresponding confidence levels.

6. A distributed real-time data processing device based on manifold learning, characterized in that, include: The real-time data acquisition and preprocessing module is used to acquire real-time data sets from different data sources, and to perform time synchronization, noise suppression, missing data completion and scale normalization on the real-time data sets. At the same time, it extracts node-level feature vectors according to data type and business semantics to obtain node feature matrix. The dynamic graph construction and manifold modeling module is used to dynamically generate and update the graph structure based on the similarity, spatial proximity and semantic association between the node features in the node feature matrix. It maps high-dimensional observation data to a low-dimensional continuous manifold space, characterizes the inherent association structure of the data and its evolution characteristics, so as to obtain the dynamic graph structure, manifold embedding results and perturbation label set. The manifold learning and inference module is used to process the dynamic graph structure and manifold embedding results. It uses a graph neural network with controlled spectral response to perform multi-layer information aggregation on the dynamic graph structure. During the aggregation process, it enhances low-frequency structural information and suppresses high-frequency unstable components caused by node perturbations and edge changes. Under the condition of model mismatch, it obtains node-level inference results and graph-level inference results. The consensus initiative collaboration module is used to abstract each subsystem into a collaborative agent in scenarios with multiple data sources or multiple computing nodes. Based on the local node-level reasoning results and graph-level reasoning results of each collaborative agent, it monitors the reasoning deviation and manifold mismatch, and actively triggers the consensus process. It achieves distributed consistent reasoning by exchanging low-frequency structural features or intermediate embedded information to obtain the intermediate representation and reasoning results after consensus. The adaptive generalization control module is used to dynamically adjust the model aggregation range and spectral weights based on the dynamic graph size, manifold dimension, distribution changes, disturbance label set, and model mismatch degree, to achieve an adaptive balance between generalization ability and discriminative ability, and to provide feedback on the control parameters obtained from the adaptive adjustment. The results output and feedback module is used to summarize and evaluate the confidence of the intermediate representations and inference results after consensus, generate globally consistent output, and provide feedback on the system's operating status and error information to form a closed-loop optimization mechanism. The manifold learning inference module is further used to update node information in the l-th layer of the graph neural network using a propagation operator with a controlled spectral response, where l represents the layer number of the graph neural network; in the graph filtering function, low-frequency components are preferentially preserved by weighting the graph spectrum; high-frequency unstable components caused by node perturbations and edge changes are suppressed by using node feature perturbation terms and edge structure perturbation terms in the perturbation label set; and the above steps are repeated to perform multi-layer information aggregation on the dynamic graph structure. The consensus initiative collaboration module is also used to continuously monitor the inference output of different collaborative agents, calculate the inference bias of any two collaborative agents, monitor the structural differences between the manifold embedding results of each collaborative agent to obtain the degree of manifold mismatch, weight and fuse the inference bias and the degree of manifold mismatch to construct a comprehensive bias index, and actively trigger the consensus collaboration process when the comprehensive bias index of any two collaborative agents is greater than a preset threshold.

Images