Event-driven distributed information grid microcirculation network data processing method

Abstract

The application discloses an event-driven distributed information grid microcirculation network data processing method, and belongs to the technical field of distributed information grid system data processing.S1, data encapsulation step;S2, arbitration step;S3, intelligent data routing step;S4, hybrid processing mode step;S5, standardized output step;S6, information consumption step.Multi-role node dynamic switching mechanism: the node is no longer fixed as a certain functional role (such as encapsulation, arbitration, consumption), but dynamically switches the identity according to real-time events in the grid, and has multiple identities (both producer and consumer, and participates in contradiction resolution).This mechanism breaks the linear pipeline or master-slave architecture of the traditional distributed system, and realizes true peer collaboration and flexible resource scheduling.Multi-layer self-correction and data integrity protection: prevent data tampering and block the spread of error data through the read-only design of the sealed information package (timestamp, hash value, source label).

Description

Technical Field

[0001] This invention relates to the field of data processing technology for distributed information grid systems, and in particular to a data processing method for distributed information grid micro-circulation networks based on event-driven principles. Background Technology

[0002] In a distributed information grid system, events and tasks need to be effectively processed in one or more neighboring nodes. Therefore, the nodes in a distributed system are different from those in a centralized system. The relationship between nodes is not a linear pipeline and there is no absolute upstream or downstream relationship. Instead, it is a peer-to-peer or multi-master relationship. The role and function of each node are dynamically switched in response to the events occurring in the grid.

[0003] In distributed information grid systems, existing data processing methods mostly employ fixed roles, linear pipelines, or centralized coordination mechanisms, making it difficult to cope with complex environments characterized by dynamic node changes, strong data heterogeneity, and frequent conflicts. Specifically, existing technologies suffer from the following technical problems: Fixed node roles lead to poor system adaptability: In traditional systems, node roles are fixed (such as only acting as producers or consumers), and the functions cannot be dynamically switched according to events, resulting in low resource utilization and difficulty in coping with sudden tasks or local failures.

[0004] Risk of conflict between data fusion path and original data path: After the introduction of data fusion mechanism, if there is a lack of intelligent routing and standardized output, it is easy to conflict with existing services that only rely on original data, which may cause downstream systems to be unable to correctly parse or consume data.

[0005] Existing distributed system data processing methods (such as Paxos / Raft consensus algorithms, CRDT conflict resolution strategies, and traditional pipelined processing systems) employ fixed roles for nodes (such as producers, consumers, and storage nodes), with task allocation relying on central scheduling or static configuration; linear pipelines or fixed data paths are used for processing paths, with merged paths being mutually exclusive or difficult to coexist with original paths; error handling and defense rely on data replication, fault recovery, or node removal, lacking proactive defense capabilities; conflict resolution mechanisms employ voting, weighted averaging, CRDT merging, and replica replication, which cannot truly "resolve" conflicts, only making selections or merging; and system collaboration methods employ fixed or semi-fixed topologies, with nodes relying on strong consistency or eventual consistency protocols. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a data processing method for a distributed information grid micro-circulation network based on event-driven principles; it can solve the problems of poor system adaptability caused by the rigidity of node roles in existing technologies and the risk of conflict between data fusion paths and original data paths.

[0007] Technical Solution: To solve the above-mentioned technical problems, according to one aspect of the present invention, more specifically, a data processing method for a distributed information grid micro-circular network based on event-driven principles, comprising the following steps: S1. Data encapsulation steps: The node receives the raw data, adds a timestamp, source tag and hash value, encapsulates it in place into a read-only encapsulated information packet and broadcasts it. S2. Arbitration Steps: Idle nodes listen to the broadcast to claim tasks, read the sealed information packets and pull relevant data, and determine according to the conflict detection rules: if there is no conflict, aggregate to generate a consensus context packet; if there is a conflict, initiate an arbitration task request and broadcast it. Multiple related nodes work together to execute pluggable data fusion and conflict resolution logic to generate a consensus context packet and broadcast it. S3, Intelligent Data Routing Steps: Based on data tags and business rules, dynamically direct the data flow to the corresponding nodes; S4. Hybrid processing mode steps: Data travels along two paths simultaneously. One path is sent to the interference node to generate fusion sensing results for real-time control, and the other path is sent to the storage node to retain the original data for post-analysis. The two paths run in parallel and do not conflict with each other. S5. Standardized output steps: Encapsulate the consensus context packet in a standard format and carry metadata for downstream services to identify and process; S6. Information consumption steps: Business nodes listen to broadcasts and actively pull consensus context packets to perform business operations.

[0008] Furthermore, the node has multiple identities and can dynamically switch functional roles based on events.

[0009] Furthermore, in step S5, the metadata includes: data type data_type and source identifier source_ids; the downstream service checks the data type in the metadata. If it is raw data, the raw data is parsed according to the original logic; if it is fused context, the corresponding fields are directly extracted and the processing logic is called.

[0010] Furthermore, in step S2, the pluggable data fusion and conflict resolution logic executed collaboratively by multiple associated nodes includes conflict resolution based on weighted credibility, multi-source consistency conversion, or other fusion rules preset by the system.

[0011] Furthermore, in step S2, the consensus context packet can output an uncertainty index.

[0012] Furthermore, the micro-loop process is a closed-loop process driven by events triggered between nodes: the original data is encapsulated into a sealed information package by the data encapsulation node, which then verifies it through arbitration nodes or generates a consensus context package through intelligent data fusion, and is then invoked by the information consumption node; the process triggers other nodes to automatically respond and execute the next action by broadcasting events and issuing requests within the grid, thereby achieving distributed collaboration; multiple micro-loop processes occur simultaneously throughout the entire grid, each micro-loop is executed by different node groups, and they do not block each other. Nodes dynamically join or leave different micro-loops based on events.

[0013] Furthermore, the sealed information package is designed to be read-only and includes a timestamp, hash value, and source tag to prevent data tampering and block the spread of erroneous data; the arbitration node initiates re-verification and multi-party collaborative revision of contradictory data; the system can dynamically isolate abnormal nodes or request redundant data sources to achieve self-healing capabilities.

[0014] According to another aspect of the present invention, an event-driven distributed information grid micro-loop network data processing system is provided. This system is used to implement the above-described event-driven distributed information grid micro-loop network data processing method, comprising: The data encapsulation module, configured at any node within the grid, receives raw data, adds timestamps, source tags, and hash values, encapsulates it in-situ into a read-only encapsulated information packet, and broadcasts it. The arbitration module, configured on idle nodes within the grid, is used to listen for broadcasts and claim tasks, read sealed information packets and pull relevant data, and determine the conflict according to the conflict detection rules: if there is no conflict, it aggregates and generates a consensus context packet; if there is a conflict, it initiates an arbitration task request and broadcasts it. Multiple related nodes work together to execute pluggable data fusion and conflict resolution logic to generate a consensus context packet and broadcast it. The intelligent data routing module is used to dynamically direct data streams to the corresponding nodes based on data tags and business rules; it supports a hybrid processing mode, allowing data to simultaneously follow the original data fidelity path and the data fusion path, with the two paths running in parallel without conflict. The standardized output module is used to encapsulate the consensus context packet in a standard format and carry metadata for downstream services to identify and process; The information consumption module is configured on any business node that needs information. It is used to listen to broadcasts and actively pull consensus context packets to perform business operations. The nodes within the system have multiple identities and dynamically switch functional roles driven by events.

[0015] Furthermore, the system is a decentralized micro-loop network where nodes are in a peer-to-peer or multi-master relationship, without any absolute upstream or downstream relationship; nodes collaborate through event-driven micro-loops, supporting parallel multi-micro-loops.

[0016] Furthermore, the events include: grid-wide broadcasts, task requests, and responses.

[0017] Beneficial effects: Multi-role node dynamic switching mechanism: Nodes are no longer fixed to a single functional role (such as encapsulation, arbitration, or consumption), but dynamically switch identities based on real-time events within the grid, possessing multiple identities (being both a producer and a consumer, and also participating in conflict resolution). This mechanism breaks the linear pipeline or master-slave architecture of traditional distributed systems, achieving true peer-to-peer collaboration and elastic resource scheduling.

[0018] Event-driven micro-loop network: The system automatically triggers nodes to execute closed-loop processes such as encapsulation, arbitration, fusion, and consumption through event mechanisms such as intra-mesh broadcasting, task requests, and responses, forming multiple parallel, decentralized micro-loops. Unlike traditional pipelines that rely on a central scheduler, this invention enables on-site processing of tasks at neighboring nodes, significantly improving system response speed and resource utilization efficiency.

[0019] Intelligent data routing and hybrid processing mode: The system dynamically selects data paths based on data tags and business rules, simultaneously supporting both "raw data fidelity paths" and "data fusion paths." These two paths run in parallel without conflict, achieving incremental enhancement rather than replacement of the existing system. Downstream services automatically identify data types through metadata, ensuring compatibility and consumability.

[0020] Distributed conflict detection and self-healing mechanism: Arbitration nodes automatically identify data conflicts by pulling related data and executing conflict detection rules. When a conflict is detected, an arbitration task request is initiated, and multiple related nodes collaboratively execute pluggable data fusion and conflict resolution logic (such as conflict resolution based on weighted credibility, multi-source consistency conversion, etc.) to generate a consensus context packet with verified or resolved conflicts. This mechanism does not rely on centralized scheduling and supports uncertainty measurement of output results.

[0021] Multi-layered self-correction and data integrity protection: The read-only design of the sealed information packet (timestamp, hash value, source tag) prevents data tampering and blocks the spread of erroneous data; the arbitration node initiates re-verification and multi-party collaborative revision of contradictory data; the system can dynamically isolate abnormal nodes or request redundant data sources to achieve self-healing capability.

[0022] The hybrid processing mode of intelligent data routing strategy and micro-circulation process ensures that data fusion and conflict resolution mechanisms can be seamlessly integrated into the existing system ecosystem. It does not replace traditional data processing methods, but rather incrementally enhances them, enabling the system to simultaneously possess both raw data fidelity and high-level information fusion capabilities, which can be invoked on demand according to the scenario.

[0023] The micro-loop process triggers responses from other nodes by broadcasting events and publishing requests within the grid, thus executing the next action and truly achieving the effect of distributed collaboration.

[0024] Data encapsulation nodes encapsulate raw data into read-only encapsulated information packets, which can prevent the information content from being tampered with later and block the spread of erroneous data within the system.

[0025] Arbitration nodes automatically initiate "arbitration task requests," and other nodes respond and collaboratively revise the data, automatically correcting erroneous data and outputting a high-dimensional consensus context packet that has resolved the contradictions, which can ensure that information consumption nodes obtain valid information that is usable for business purposes. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the method. Detailed Implementation

[0027] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Example 1

[0029] Scenario Description: An autonomous vehicle is equipped with multiple sensors (cameras, millimeter-wave radar, lidar). The raw data generated by these sensors needs to be processed in a distributed information grid for real-time obstacle detection and decision-making. Simultaneously, the raw data needs to be retained for post-event analysis.

[0030] System Configuration: The grid contains multiple nodes, each with multiple roles including data encapsulation, arbitration, and information consumption, dynamically switching based on events. The system pre-sets the following business rules: For high real-time control tasks: prioritize the use of the merged consensus context pack.

[0031] Data archiving task: Retain a copy of the original data.

[0032] Conflict detection threshold: If the difference between the measurement values ​​of different sensors of the same object exceeds the preset threshold (e.g., distance difference > 2 meters), it is determined to be a conflict and an arbitration task request is triggered.

[0033] Step 1: Data Encapsulation The camera node (current role: data encapsulation node) captures a frame of image and generates a raw byte stream.

[0034] The encapsulation node adds a timestamp (t=1633024800), a source tag (source=camera_01), and a hash value (sha256=0x3a4b...) to the data, and encapsulates it into a read-only "SealedPacket#001".

[0035] The encapsulation node stores SealedPacket#001 in its local cache and broadcasts a "New Sealed Packet Pending Processing" event to the grid.

[0036] Meanwhile, radar nodes and lidar nodes generate SealedPacket#002 and SealedPacket#003 in the same way and broadcast the event.

[0037] Step 2: Arbitration Node Acknowledgment and Conflict Detection An idle node (ID=Node_A) in the grid hears a broadcast event, switches its role to "arbitration node", and competes to claim three sealed information packets.

[0038] Node_A reads the contents of three packets: Camera: Obstacle distance 25.0 meters Radar: 24.5 meters LiDAR: 8.0 meters (abnormal due to reflection) Node_A retrieves relevant historical data from its local cache and neighboring nodes (e.g., the distance to the previous frame was approximately 24.8 meters) and executes the conflict detection rules: The difference between camera and radar data is 0.5 meters < threshold → consistent.

[0039] If the difference between the lidar data and the camera data is 17.0 meters or more, it is considered a contradiction.

[0040] Step 3: Initiate an arbitration request and collaborative revision Node_A generates an "Arbitration Task Request" (ArbitrationRequest#001), which includes the conflict data source identifier (laser_01), related data (24.5 meters, 25.0 meters, 8.0 meters), and request type ("Distance Fusion and Conflict Resolution").

[0041] The request will be broadcast to at least one other node associated with the conflicting data (e.g., Node_B, Node_C, which have historical data or redundant sensors, respectively).

[0042] The response nodes (Node_B, Node_C) and Node_A collaboratively execute pluggable data fusion logic. This embodiment employs weighted reliability fusion. The reliability of the camera is 0.9, radar is 0.85, and lidar is 0.2 (due to a high historical false alarm rate).

[0043] The fusion result = (25.0×0.9+24.5×0.85+8.0×0.2) / (0.9+0.85+0.2)≈24.1 meters.

[0044] Meanwhile, the uncertainty index (variance) was calculated to be 0.12 (low uncertainty).

[0045] Once the three nodes reach a consensus on the fusion result (e.g., through majority confirmation or a pre-defined fast confirmation protocol), they generate a "consensus context packet" (ConsensusPacket#001), which contains: Data fusion: distance=24.1m Metadata: data_type="fused_context", source_ids=["camera_01", "radar_01", "laser_01"], uncertainty=0.12, timestamp=1633024805 ConsensusPacket#001 is written to the Distributed Hash Table (DHT), and a "Consensus Context Packet is Consumable" event is broadcast.

[0046] Step 4: Intelligent Data Routing and Hybrid Processing According to the system's business rules: Real-time control path: The vehicle decision node (role switched to "information consumption node") listens to the event, actively pulls ConsensusPacket#001, and performs braking / obstacle avoidance decisions based on the fusion distance of 24.1 meters.

[0047] Raw data archiving path: At the same time, the intelligent routing strategy directs copies of SealedPacket#001, #002, and #003 to storage nodes based on the tags of the sealed information packets (data_type="raw" and preserve_for_analysis=true) for post-event model training or incident investigation.

[0048] Step 5: Standardize output and ensure compatibility with downstream services After receiving the data packet, the downstream service (such as a third-party obstacle warning module) checks the data_type in the metadata: If it is raw, then the original data will be parsed according to the original logic.

[0049] If it is fused_context, then the distance field is extracted directly, and the new efficient processing logic is called.

[0050] Example 2

[0051] Scenario Description: An industrial IoT system monitors the temperature of a production line. Three temperature sensors (T1, T2, T3) are deployed in different locations. At a certain moment, T3 experiences a sudden temperature jump (150°C) due to electromagnetic interference, while T1 and T2 remain around 100°C.

[0052] System execution flow: Encapsulation: Each of the three sensor nodes generates an encapsulated information packet, which includes a timestamp and a hash value to prevent tampering.

[0053] Arbitration node detection: The arbitration node reads three packets and finds that the deviation between T3 and T1 and T2 exceeds the threshold (±5°C), which is determined to be a contradiction.

[0054] Initiate an arbitration task request: Broadcast the request to related nodes (such as historical database nodes, other sensor nodes in the neighboring area).

[0055] Collaborative revision: Multiple related nodes pull historical data (the average T3 value over the past 10 minutes is 101°C), and use multi-source consistency transformation logic (e.g., removing obvious outliers and taking the median) to generate a consensus context packet: temperature=101°C, uncertainty=0.05, anomaly_flag="T3_fault_suspected".

[0056] Standardized output: The consensus context packet is encapsulated in a standard format, carrying metadata data_type="fused_context",source_ids=["T1","T2","T3_history"],uncertainty=0.05, for direct consumption by downstream temperature control services.

[0057] Output and self-healing: The control node consumes the consensus context packet and executes normal temperature control commands. Simultaneously, the system automatically isolates the T3 node and requests maintenance.

[0058] Example 3

[0059] Within the same grid, multiple independent event-driven micro-loop processes occur simultaneously: Micro-loop A: Processing data fusion from forward sensors for autonomous driving (as in Example 1).

[0060] Micro-loop B: Processes data from the backward ultrasonic radar (no contradictions, directly verifies and generates consensus packets).

[0061] Micro-circulation C: Handles conflicting data between vehicle GPS and inertial navigation (initiates arbitration task).

[0062] Each microcycle is executed by a different group of nodes, without blocking each other. Nodes can dynamically join or leave different microcycles based on events. For example, the same node may participate in microcycle A as an arbitrator node at time t1, and consume the output of microcycle B as a consumer node at time t2.

[0063] Summary of Implementation Methods Specific embodiments of the present invention include, but are not limited to, the following key operations: Data encapsulation: Standardize and read-only the raw data to prevent the propagation of errors.

[0064] Conflict detection: Determine data conflicts based on rules or thresholds.

[0065] Arbitration task request and collaborative revision: Utilize multiple related nodes to execute pluggable fusion logic (such as weighted credibility, outlier removal, historical comparison, etc.) to generate consensus context packets.

[0066] Intelligent routing: Based on data tags and business rules, it supports both raw data paths and fused data paths in parallel.

[0067] Standardized output: Carry metadata (data type, source, uncertainty indicators) to ensure compatibility with downstream services.

[0068] Event-driven micro-loop: Automatically triggers node role switching and task execution through events such as broadcasts, requests, and responses, without the need for central scheduling.

[0069] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An event-driven distributed information grid micro-loop network data processing method, applied to a distributed information grid system including multiple peer or multi-master relationship nodes, characterized in that, Includes the following steps: S1. Data encapsulation steps: The node receives the raw data, adds a timestamp, source tag and hash value, encapsulates it in place into a read-only encapsulated information packet and broadcasts it. S2. Arbitration Steps: Idle nodes listen to the broadcast to claim tasks, read the sealed information packets and pull relevant data, and determine according to the conflict detection rules: if there is no conflict, aggregate to generate a consensus context packet; if there is a conflict, initiate an arbitration task request and broadcast it. Multiple related nodes work together to execute pluggable data fusion and conflict resolution logic to generate a consensus context packet and broadcast it. S3, Intelligent Data Routing Steps: Based on data tags and business rules, dynamically direct the data flow to the corresponding nodes; S4. Hybrid processing mode steps: Data travels along two paths simultaneously. One path is sent to the interference node to generate fusion sensing results for real-time control, and the other path is sent to the storage node to retain the original data for post-analysis. The two paths run in parallel and do not conflict with each other. S5. Standardized output steps: Encapsulate the consensus context packet in a standard format and carry metadata for downstream services to identify and process; S6. Information consumption steps: Business nodes listen to broadcasts and actively pull consensus context packets to perform business operations.

2. The event-driven distributed information grid micro-circular network data processing method according to claim 1, characterized in that: The node has multiple identities and its functional roles are dynamically switched based on events.

3. The event-driven distributed information grid micro-circular network data processing method according to claim 1, characterized in that: In step S5, the metadata includes: data type data_type and source identifier source_ids. The downstream service checks the data type in the metadata. If it is raw data, it parses the raw data according to the original logic. If it is fused context, it directly extracts the corresponding fields and calls the processing logic.

4. The event-driven distributed information grid micro-circular network data processing method according to claim 1, characterized in that: In step S2, the pluggable data fusion and conflict resolution logic executed collaboratively by multiple associated nodes includes conflict resolution based on weighted credibility, multi-source consistency conversion, or other fusion rules preset by the system.

5. The event-driven distributed information grid micro-circular network data processing method according to claim 1, characterized in that: In step S2, the consensus context packet can output an uncertainty index.

6. The event-driven distributed information grid micro-circular network data processing method according to claim 1, characterized in that: The micro-loop process is a closed-loop process driven by events triggered between nodes: the raw data is encapsulated into a sealed information package by the data encapsulation node, and after being detected and verified by the arbitration node or intelligently fused to generate a consensus context package, it is then invoked by the information consumption node; the process triggers other nodes to automatically respond and execute the next action by broadcasting events and issuing requests within the grid, thus achieving distributed collaboration; multiple micro-loop processes occur in the entire grid at the same time, each micro-loop is executed by different node groups, and they do not block each other. Nodes dynamically join or leave different micro-loops based on events.

7. The event-driven distributed information grid micro-circular network data processing method according to claim 1, characterized in that: The sealed information package is designed to be read-only and includes a timestamp, hash value, and source tag to prevent data tampering and block the spread of erroneous data; the arbitration node initiates re-verification and multi-party collaborative revision of contradictory data; the system can dynamically isolate abnormal nodes or request redundant data sources to achieve self-healing capabilities.

8. An event-driven distributed information grid micro-circular network data processing system, characterized in that: This system is used to implement the event-driven distributed information grid micro-circular network data processing method according to any one of claims 1-7, comprising: The data encapsulation module, configured at any node within the grid, receives raw data, adds timestamps, source tags, and hash values, encapsulates it in-situ into a read-only encapsulated information packet, and broadcasts it. The arbitration module, configured on idle nodes within the grid, is used to listen for broadcasts and claim tasks, read sealed information packets and pull relevant data, and determine the conflict according to the conflict detection rules: if there is no conflict, it aggregates and generates a consensus context packet; if there is a conflict, it initiates an arbitration task request and broadcasts it. Multiple related nodes work together to execute pluggable data fusion and conflict resolution logic to generate a consensus context packet and broadcast it. The intelligent data routing module is used to dynamically direct data streams to the corresponding nodes based on data tags and business rules; it supports a hybrid processing mode, allowing data to simultaneously follow the original data fidelity path and the data fusion path, with the two paths running in parallel without conflict. The standardized output module is used to encapsulate the consensus context packet in a standard format and carry metadata for downstream services to identify and process; The information consumption module is configured on any business node that needs information. It is used to listen to broadcasts and actively pull consensus context packets to perform business operations. The nodes within the system have multiple identities and dynamically switch functional roles driven by events.

9. The event-driven distributed information grid micro-circular network data processing system according to claim 8, characterized in that: The system is a decentralized micro-circulation network where nodes are in a peer-to-peer or multi-master relationship, without any absolute upstream or downstream relationship. Nodes collaborate through event-driven micro-loops, supporting parallel multi-micro-loops.

10. The event-driven distributed information grid micro-circular network data processing system according to claim 9, characterized in that: The events include: grid-wide broadcasts, task requests, and responses.

Images