Information collaborative processing method and system in distributed environment

Abstract

The application relates to the technical field of distributed computing, and discloses an information collaborative processing method and system in a distributed environment, which comprises the following steps: acquiring node connection relations to form a global connection dataset, calculating node connection degrees, average path lengths and comprehensive load scores; screening high-stability nodes as candidate backbones, and selecting a minimum backbone set to cover all non-backbones; periodically collecting local state characteristic vectors of the non-backbones, collecting global state information, and redundantly storing the global state information in the non-backbones after encoding by using a Vandermonde matrix; monitoring backbone abnormalities to trigger reconstruction, and acquiring encoded blocks from the non-backbones by a new backbone node to recover the global state information through error correction code decoding; and publishing an updated task list and routing instruction according to the recovered information to complete collaborative relationship reconstruction. The method can realize reliable storage and rapid recovery of global state information, and significantly improves the self-adaptability, fault tolerance and collaborative efficiency of a distributed system.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology, and in particular to a method and system for collaborative information processing in a distributed environment. Background Technology

[0002] Distributed systems consist of multiple interconnected nodes that work together to complete complex computing tasks and are widely used in fields such as cloud computing, the Internet of Things, and edge computing.

[0003] In one existing technology, information coordination in a distributed system mainly relies on pre-defined backbone nodes or fixed communication paths, which are responsible for collecting global state information and scheduling tasks.

[0004] However, node load and link status in a distributed environment are highly dynamic. Nodes may join or leave at any time, and link latency and bandwidth fluctuate in real time. Existing technologies, due to the need for big data analysis of network-wide operational data, struggle to accurately identify dynamic patterns from massive amounts of status information. This results in the inability to optimize backbone node selection in real time, often leading to some nodes being overloaded while others are idle. Therefore, existing technologies suffer from the problem of failing to accurately identify dynamic patterns from massive amounts of status information through big data analysis. Summary of the Invention

[0005] This invention provides a method and system for collaborative information processing in a distributed environment to solve the problem of accurately identifying dynamic change patterns from massive amounts of state information through big data analysis.

[0006] Firstly, in order to solve the above-mentioned technical problems, the present invention provides an information collaborative processing method in a distributed environment, comprising: Obtain the current connection relationships of all nodes in the distributed network to form a global connection dataset. Calculate the node connectivity of the node under test and the average path length to other nodes in the global connection dataset. Calculate the comprehensive load score based on the CPU utilization and memory usage of the node under test. Calculate the predicted node lifetime of the node under test that has a node connectivity greater than a preset effective connection threshold and a comprehensive load score less than a preset load threshold. If the predicted lifetime is in a preset high stability range, then mark the node under test as a candidate backbone node. Select the smallest subset from the candidate backbone nodes that can cover all non-backbone nodes, and determine it as the backbone node set; Collect local state feature vectors from the non-backbone nodes connected to the backbone node set; The global state information is obtained by summarizing the local state feature vectors. The global state information is encoded using a preset Vandermonde matrix. If the encoded segmented sequence meets the preset encoding redundancy requirement, the storage mapping index is locked. If there are abnormal nodes in the backbone node set, the reconstruction process of the backbone node set is triggered, and a new backbone node set is regenerated based on the latest link status information and node load fed back by the topology probe. Based on the new backbone node set, the encoded data blocks are re-obtained from the non-backbone nodes, and the encoded data blocks are decoded using the inverse of the preset Vandermonde matrix to recover the complete global state information. Based on the complete global state information, the updated task execution list and routing control instruction package are published to all non-backbone nodes to obtain the reconstructed system collaboration relationship.

[0007] Secondly, the present invention provides an information collaborative processing system in a distributed environment, comprising: The network status awareness and load assessment module obtains the current connection relationships of all nodes in the distributed network, forms a global connection dataset, calculates the node connectivity of the node under test and the average path length to other nodes in the global connection dataset, and calculates a comprehensive load score based on the CPU utilization and memory usage of the node under test. The candidate backbone node screening module calculates the predicted node lifecycle of the node to be tested if the node connectivity is greater than a preset effective connection threshold and the comprehensive load score is less than a preset load threshold. If the predicted lifecycle is in a preset high stability range, the node to be tested is marked as a candidate backbone node. The backbone node election and coverage optimization module selects the smallest subset from the candidate backbone nodes that can cover all non-backbone nodes and determines it as the backbone node set. The local state acquisition module collects local state feature vectors from the non-backbone nodes connected to the backbone node set; The global state encoding and redundant storage module summarizes the local state feature vectors to obtain global state information, and encodes the global state information using a preset Vandermonde matrix. If the encoded fragmented sequence meets the preset encoding redundancy requirements, then a storage mapping index is constructed and locked. The backbone node dynamic reconstruction module, if there are abnormal nodes in the backbone node set, triggers the reconstruction process of the backbone node set, and regenerates a new backbone node set based on the latest link status information and node load fed back by the topology probe. The global state recovery module re-obtains encoded data blocks from the non-backbone nodes based on the new backbone node set, and decodes the encoded data blocks using the inverse of the preset VanderMonde matrix to recover the complete global state information. The collaboration relationship reconstruction and instruction issuance module publishes the updated task execution list and routing control instruction package to all non-backbone nodes based on the complete global state information, thereby obtaining the reconstructed system collaboration relationship.

[0008] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention constructs a global connectivity dataset and calculates a comprehensive load score by collecting multi-dimensional data such as node connectivity, communication latency, CPU and memory usage in real time. It then dynamically selects candidate backbone nodes by combining connectivity thresholds and lifetime prediction values. The dynamic election mechanism based on big data analysis enables backbone node selection to respond in real time to network topology and load fluctuations, avoiding node overload or idleness, and achieving efficient and balanced utilization of network resources.

[0009] (2) This invention selects the smallest subset of candidate nodes that can cover all non-backbone nodes as the backbone node set by using a weighted shortest path tree, which greatly reduces the number of core nodes. The backbone nodes periodically collect local state information, which is redundantly stored in multiple non-backbone nodes after being encoded by a Vandermonde matrix. This makes the global state information no longer dependent on a single node, and the complete state can still be restored even if some nodes fail, which significantly enhances the reliability and resilience of state management.

[0010] (3) By establishing a dynamic reconfiguration mechanism for backbone nodes, when a backbone node is detected to be disconnected or overloaded, the system quickly re-elects a new backbone node based on the latest topology probe and node load data. The new node obtains coded blocks from non-backbone nodes through a distributed hash mapping table, restores the global state information through erasure coding decoding, and issues update instructions, thereby achieving seamless reconfiguration of the collaborative relationship. This reduces the fault recovery time from minutes to seconds, significantly improving the system's fault self-healing capability and business continuity. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the information collaborative processing method in a distributed environment provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an information collaborative processing system in a distributed environment provided in the second embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Reference Figure 1 The first embodiment of the present invention provides an information collaborative processing method in a distributed environment, comprising the following steps: S11, obtain the current connection relationship of all nodes in the distributed network to form a global connection dataset, calculate the node connectivity of the node under test and the average path length to other nodes in the global connection dataset, and calculate the comprehensive load score based on the CPU utilization and memory usage of the node under test. S12, calculate the predicted node lifetime of the node to be tested if the node connectivity is greater than a preset effective connection threshold and the comprehensive load score is less than a preset load threshold. If the predicted lifetime is in a preset high stability range, then mark the node to be tested as a candidate backbone node. S13, Select the smallest subset from the candidate backbone nodes that can cover all non-backbone nodes, and determine it as the backbone node set; S14, collect local state feature vectors from the non-backbone nodes connected to the backbone node set; S15, summarize the local state feature vectors to obtain global state information, encode the global state information using a preset Vandermonde matrix, and if the encoded segmented sequence meets the preset encoding redundancy requirement, then construct and lock the storage mapping index. S16. If there are abnormal nodes in the backbone node set, the reconstruction process of the backbone node set is triggered, and a new backbone node set is regenerated based on the latest link status information and node load fed back by the topology probe. S17. Based on the new backbone node set, re-obtain the encoded data blocks from the non-backbone nodes, and decode the encoded data blocks using the inverse of the preset Vandermonde matrix to recover the complete global state information. S18, based on the complete global state information, publish the updated task execution list and routing control instruction package to all non-backbone nodes to obtain the reconstructed system collaboration relationship.

[0014] In step S11, the current connection relationships of all nodes in the distributed network are obtained to form a global connectivity dataset. The node connectivity degree of the node under test and the average path length to other nodes in the global connectivity dataset are calculated. Based on the CPU utilization and memory usage of the node under test, a comprehensive load score is calculated, including: The node under test in the distributed network is controlled to send probe messages to neighboring nodes. The difference between the receiving timestamp and the sending timestamp of the probe message is calculated to obtain the one-way communication delay value. If the one-way communication delay value is less than a preset effective time threshold, a local neighbor list of the node under test is generated, and all local neighbor lists are aggregated to form a global connection dataset. Based on the global connectivity dataset, determine the node connectivity of the node under test and the average path length to other nodes; The CPU utilization and memory usage of the node under test are obtained, and the node connectivity, the average path length, the CPU utilization and the memory usage are encapsulated into a multi-dimensional resource consumption vector. The multi-dimensional resource consumption vector is normalized to generate a load normalized value. A preset node weight coefficient is matched to the normalized load value, and the comprehensive load score of each node is calculated.

[0015] It should be noted that the node under test in the control distributed network sends probe messages to neighboring nodes. These messages embed precise sending timestamp information. After receiving the message, the neighboring node records the receiving timestamp and returns a response data packet containing the receiving timestamp and the original sending timestamp. After receiving the response, the node under test extracts the receiving timestamp and the sending timestamp, and calculates the difference between the receiving timestamp and the sending timestamp to obtain the one-way communication delay value.

[0016] For example, node A sends a probe message to node B at local time T1. After receiving the message at local time T2, node B immediately encapsulates T2 and T1 in a response and returns it to A. A receives the response at time T3. The one-way delay from A to B can be obtained by subtracting T1 from T2. To eliminate the impact of clock asynchrony between nodes, a network time protocol can be used for clock synchronization or the time can be estimated by taking the average of multiple measurements.

[0017] It should be noted that the preset effective time threshold is determined based on statistical analysis of historical network latency data. The average latency is 30 milliseconds and the standard deviation is 10 milliseconds. Therefore, the threshold is set as the average latency plus twice the standard deviation, which is 50 milliseconds. In systems with high real-time requirements, the threshold can be reduced to 20 milliseconds to meet stricter latency requirements. When the measured latency is 15 milliseconds, which is less than 50 milliseconds, it is determined that there is an active connection between node A and node B. Node B is recorded in the local neighbor list of node A. Each node generates its own local neighbor list and reports it to the central coordinator. The central coordinator performs deduplication and merging of all lists. For example, if node A's list contains B and C, node B's list contains A and C, and node C's list contains A and B, the merged list will yield a global connection dataset containing all effective connection relationships.

[0018] It should be noted that, based on the global connectivity dataset, the node connectivity is obtained by counting the number of directly connected neighboring nodes of each node; at the same time, the one-way communication delay values ​​from the node to all other nodes are extracted from the link state matrix, and the arithmetic mean of these delays is calculated as the average path length. This value comprehensively reflects the reachability efficiency and centrality of the node in the network, and the lower the value, the better its network location.

[0019] For example, in a cluster consisting of six server nodes, if node A has a valid connection with nodes B, C, and D, then its connectivity is 3; if the delays from node A to nodes B, C, D, E, and F are 10 milliseconds, 15 milliseconds, 20 milliseconds, 50 milliseconds, and 55 milliseconds respectively, then the average path length is the sum of the five delay values ​​divided by 5, which gives 30 milliseconds.

[0020] It should be noted that the CPU utilization and memory usage are collected by the local performance monitoring agent on each node. Combined with the node connectivity and average path length to other nodes obtained from the global connectivity dataset, these four metrics are encapsulated into a multi-dimensional resource consumption vector. Since the dimensions and numerical ranges of each metric differ, max-min normalization is performed, linearly mapping it to the interval between 0 and 1. After performing this operation on all four metrics, a weighted average is used to generate a load normalization value between 0 and 1, which integrates network and hardware load.

[0021] For example, if the cluster's historical maximum connection degree is 5 and the minimum is 1, then the connection degree of node A is 3, which is normalized to 0.5. Similarly, the normalization of CPU utilization of 65% needs to be based on the historical CPU utilization range. If the historical maximum is 90% and the minimum is 10%, then it is normalized to 0.6875. The memory utilization of 40% is normalized based on the historical range, such as the maximum being 80% and the minimum being 20%, which is normalized to 0.333.

[0022] It should be noted that preset node weight coefficients are used to match the load normalization value. In communication-intensive networks, the preset node weight coefficients place higher weights on the connectivity and average path length of core routing nodes, while computational nodes are weighted more heavily on CPU and memory. The overall load score of a node is obtained by multiplying the load normalization value by its corresponding node weight coefficient. The overall load score quantifies the combined pressure on the node in processing and forwarding data.

[0023] For example, if the load normalization value of node A is 0.6 and the weight coefficient of the core routing node is set to 1.2 based on the analysis of the historical load balancing effect of the network, then its comprehensive load score is 0.6 multiplied by 1.2 equals 0.72. This score quantifies the comprehensive pressure of the node in processing and forwarding data and is used to measure the real-time load level of the node.

[0024] In step S12, the predicted node lifetime of the node under test that has a node connectivity greater than a preset effective connection threshold and a comprehensive load score less than a preset load threshold is calculated. If the predicted lifetime is within a preset high stability range, the node under test is marked as a candidate backbone node, including: If the node connectivity is greater than a preset effective connection threshold and the overall load score is less than a preset load threshold, then a preliminary candidate node index is generated. The historical data of the online duration of the node and the link stability coefficient corresponding to the node to be tested are obtained from the preliminary candidate node index and input into the preset link failure probability model to calculate the predicted life cycle value of the node to be tested. If the predicted node lifecycle value is within a preset high stability range, the corresponding node will be marked as a candidate backbone node.

[0025] It should be noted that if the node connectivity is greater than a preset effective connection threshold and the comprehensive load score is less than a preset load threshold, a preliminary candidate node index is generated. The preset effective connection threshold is set according to the network size and the desired connectivity. For example, in a distributed network consisting of hundreds of nodes, by statistically analyzing the distribution of the connectivity of all nodes in the network, the median or average value is taken as the benchmark, and the connection threshold is set to 4 to ensure that the candidate node has enough neighbors to undertake backbone functions. The preset load threshold is set according to the node's processing capacity and historical load data. For example, by collecting the comprehensive load score of the node under typical load, its 80th percentile value of 0.7 is taken as the threshold to ensure that the candidate node is in a relatively idle state. When a node has a connectivity of 5 and a comprehensive load score of 0.65 that is lower than the threshold, the node is identified and included in the preliminary candidate node index.

[0026] For example, the preset effective connection threshold might be set to 4, and the preset load threshold might be set to 0.7. When a sensor node is found to have a connectivity score of 5, indicating that it maintains effective connections with all five neighboring nodes, and its overall load score is only 0.65, which is lower than the threshold of 0.7, then the node meets the preliminary criteria, and its identifier will be included in the preliminary candidate node index.

[0027] It should be noted that the historical online duration data and link stability coefficient of the nodes to be tested that meet the aforementioned conditions are obtained from the preliminary candidate node index. The historical online duration data can be obtained by recording the continuous stable running time of the node since its last startup or fault recovery; for example, node A has been running stably for 30 consecutive days. The link stability coefficient is calculated by statistically analyzing the average packet loss rate and signal strength fluctuation of the communication link between this node and all its neighboring nodes over the past week. A stability coefficient of 0.95 is assigned when the average packet loss rate is less than 1% and the signal strength fluctuation is less than 3dB. The historical online duration data and link stability coefficient are then input into a preset link failure probability model, which is built based on historical data statistics. Historical online duration data, combined with historical node failure rates, can be used to estimate the expected remaining uptime. For example, based on a node's operational records over the past year, if its mean time between failures (MTBF) is 1000 hours, and it has already been running for 720 hours, the estimated remaining expected uptime is 280 hours. The link failure model is derived by fitting historical failure data based on the mapping relationship between stability coefficients and historical failure rates. For example, a stability coefficient of 0.95 corresponds to a failure rate of 0.01 failures per hour, estimating an MTBF of 100 hours. The smaller value between the estimated remaining online time and the link's MTBF is taken as the predicted lifetime. If this predicted value is greater than the minimum service duration required by the system for backbone nodes, the node is considered to be in a high-stability range.

[0028] It is worth noting that the preset link failure probability model can be constructed using a statistical regression model or a machine learning model based on historical data. For example, in the initial stage of system deployment, historical operational data of all network nodes over the past three months is collected, including the continuous online time of nodes (in hours), the link stability coefficient (ranging from 0 to 1, calculated based on packet loss rate and signal strength fluctuations), and the actual time of node failure. This data is used as training samples, with online time and stability coefficient as input features, and the remaining time from the current moment to the occurrence of failure as the prediction target. A random forest regression algorithm is used to train the failure probability model. In actual operation, the current time of the target node is used to... Historical online duration data, such as 500 hours of continuous operation, and a link stability coefficient, such as 0.95, are input into the model. The model output is the predicted lifetime of the node, for example, predicting that it can still operate stably for another 300 hours. This predicted value comprehensively reflects the possibility of the node leaving the network due to hardware aging, energy depletion, or link degradation. The predicted lifetime is mainly based on the node's software operation stability and network link reliability, and is applicable to various distributed computing nodes. For nodes with energy constraints, such as IoT devices, additional remaining energy parameters can be introduced for comprehensive evaluation. For continuously powered server nodes, the focus is on evaluating runtime and link quality.

[0029] It should be noted that if the predicted node lifecycle value is within a preset high stability range, the corresponding node will be marked as a candidate backbone node. This preset high stability range is set according to the system's requirement for the minimum continuous service time of backbone nodes. For example, if the backbone node is required to run continuously and stably for at least 24 hours based on business continuity requirements, then the lower bound of the range is set to 24 hours, and the upper bound can be set to infinity or according to the maximum theoretical lifespan of the node.

[0030] For example, when the predicted lifespan of a node is 36 hours, which is greater than 24 hours, it is determined that it falls into the high stability range and is marked as a candidate backbone node.

[0031] In step S13, the smallest subset that can cover all non-backbone nodes is selected from the candidate backbone nodes and determined as the backbone node set, including: The link transmission rate between the candidate backbone node and its neighboring nodes is obtained through the global connectivity dataset, and the weighted communication path cost matrix between the nodes is calculated based on the link transmission rate. Based on the weighted communication path cost matrix, a weighted shortest path tree is constructed from the candidate backbone nodes to non-backbone nodes, and a node coverage index is generated. Establish a dominance mapping relationship between the candidate backbone nodes and the non-backbone nodes based on the node coverage index, and calculate the independent coverage contribution based on the dominance mapping relationship; Candidate backbone nodes are removed in order of their independent coverage contribution from low to high. If the remaining candidate backbone nodes can still cover all the non-backbone nodes after removal, the removal is performed until no further removal is possible. The final remaining candidate backbone nodes are then used as the backbone node set.

[0032] It should be noted that the link transmission rate between candidate backbone nodes and neighboring nodes is obtained through the global connectivity dataset. The weighted communication path cost matrix between nodes is calculated based on the transmission rate. The cost is inversely proportional to the transmission rate. The baseline transmission rate is set to 1 Mbps. Then the link cost weight between any two nodes is the baseline rate divided by the actual transmission rate. For example, when the actual transmission rate of a link is 2 Mbps, its cost is 0.5. When the rate is 0.5 Mbps, the cost is 2. The path cost between nodes is the sum of the costs of each link along the path. After the system traverses all possible paths, it selects the path with the minimum cost as the optimal path. Then, the cost value of the path is filled into the corresponding position in the matrix to form the weighted communication path cost matrix.

[0033] For example, there are two paths from node A to node B. Path 1 passes through three nodes and each link has a speed of 2Mbps, so the total cost is 0.5 + 0.5 + 0.5 = 1.5. Path 2 passes through two nodes, but one link has a speed of 0.8Mbps, so the total cost is 0.5 + 1.25 = 1.75. The system fills the cost values ​​of the two paths into the corresponding positions in the matrix to form a weighted communication path cost matrix.

[0034] It should be noted that, based on the weighted communication path cost matrix, starting from each candidate backbone node, the Dijkstra algorithm is used to construct a weighted shortest path tree from it to all non-backbone nodes. The sum of the costs of each link on the path is the cumulative cost from the non-backbone node to the candidate backbone node. The link cost is calculated according to the link transmission rate using a preset formula. For example, with a reference transmission rate of 1Mbps, the link cost is equal to the reference rate divided by the actual transmission rate. The link cost for an actual rate of 2Mbps is 0.5, and the link cost for an actual rate of 0.5Mbps is 2. The non-backbone nodes that each candidate backbone node can cover and their corresponding minimum cumulative costs are summarized to form a node coverage index. This index records the set of non-backbone nodes covered by each candidate backbone node and the total cumulative cost of all nodes in the set.

[0035] For example, candidate backbone node X covers 15 surrounding non-backbone nodes, with cumulative costs of 0.2, 0.4, 0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 0.5, 0.6, 0.7, 0.8, and 0.9 respectively. The sum of these costs yields a total cumulative weight of 8.2. Another candidate backbone node Y covers 10 non-backbone nodes, with a total cumulative weight of 6.5. This weight comprehensively reflects the communication cost of the network covered by the candidate backbone node. The smaller the weight, the lower the latency that the node can cover the non-backbone nodes under its jurisdiction.

[0036] It should be noted that a dominance mapping relationship between candidate backbone nodes and non-backbone nodes is established based on the node coverage index. This mapping records the set of non-backbone nodes covered by each candidate backbone node and which candidate backbone nodes cover each non-backbone node. Based on this, a greedy strategy is used to calculate the independent coverage contribution of each candidate backbone node. The method for calculating the independent coverage contribution is to traverse each candidate backbone node in the set of non-backbone nodes that are not currently covered by any selected backbone nodes and count the number of nodes that it can cover. This statistical value is the current contribution of the node.

[0037] For example, initially, all non-backbone nodes are uncovered. Candidate backbone node A covers nodes 1, 2, and 3, so its contribution is 3. Candidate backbone node B covers nodes 2, 3, and 4, so its contribution is 3. Candidate backbone node C covers nodes 4 and 5, so its contribution is 2. After selecting A to add to the backbone set according to the contribution from largest to smallest, the uncovered nodes are updated to 4 and 5. At this time, the contribution of the candidate backbone nodes is re-traversed and recalculated. B can only cover node 4, so its contribution becomes 1. C can cover nodes 4 and 5, so its contribution is 2. Then C is selected to complete the selection of backbone nodes. This iterative calculation process is based on the principle of maximizing the number of newly added nodes each time, ensuring that the finally selected backbone node can cover all non-backbone nodes with the minimum number of nodes.

[0038] It should be noted that the independent coverage contribution is defined as the number of nodes that the node can cover in the current set of non-backbone nodes that are not covered by any selected backbone nodes. This number is obtained from the node coverage range index. For example, if initially all non-backbone nodes are not covered, and candidate node X covers 5 of them, the contribution is 5. Nodes are selected and added to the backbone set in descending order of contribution. Each time a node is selected, the non-backbone nodes it covers are marked as covered, and these nodes are excluded from the contribution calculation of subsequent candidate nodes. This process is repeated until all non-backbone nodes are covered, and the backbone node set at the current time is finally obtained. This greedy selection method is based on maximizing the number of newly added nodes each time it covers, and can minimize the number of backbone nodes while ensuring full network coverage.

[0039] For example, the initial sequence contains 10 candidate backbone nodes. The system starts by extracting from the node with the highest contribution and removes the non-backbone nodes it covers each time until all non-backbone nodes are covered. In the end, only 4 backbone nodes may be needed to complete the full network coverage.

[0040] In step S14, local state feature vectors are collected from the non-backbone nodes connected to the backbone node set, including: A status request command is sent to the non-backbone nodes through the polling clock signal configured in the backbone node set, and a status feedback data packet is obtained. Extract task execution progress values ​​and hardware resource usage metrics from the status feedback data packets to construct the local state feature vector of the non-backbone node; Calculate the dynamic deviation value of the local state feature vector. If the dynamic deviation value is higher than the preset abnormal fluctuation threshold, generate a resource bottleneck early warning sign. Update the node status record of the local state feature vector based on the resource bottleneck early warning identifier.

[0041] It should be noted that a polling clock signal configured by the backbone node set periodically sends status request commands to directly connected non-backbone nodes. The polling period is set according to the network size and the rate of status change. For example, in a network composed of hundreds of sensor nodes, by statistically analyzing the average interval of node status changes and combining the system's real-time requirements, the polling period is set to 30 seconds to ensure timely perception of node status while avoiding excessive consumption of network resources. After receiving the command, the non-backbone nodes send back status feedback data packets containing task execution progress and hardware resource usage. The task execution progress is expressed as the percentage of completed tasks out of the total tasks, such as 75%. Hardware resource usage includes values ​​such as CPU utilization (e.g., 60%) and memory usage. The backbone nodes receive and parse these data packets to obtain the real-time operating status of each non-backbone node, providing basic data for subsequent resource scheduling and task allocation.

[0042] For example, in a network consisting of hundreds of sensor nodes, the backbone node sends a status request command every 30 seconds. Upon receiving the command, the non-backbone nodes will send back a data packet containing task execution progress and hardware resource usage. This data packet includes task completion percentage data, such as a node's current task progress being 75%, and resource usage data, such as CPU utilization being 60%.

[0043] It should be noted that the task execution progress value, CPU utilization value, and memory usage value are parsed from the status feedback data packet. The task execution progress is represented by the percentage of completed tasks out of the total tasks, the CPU utilization is the percentage of current usage, and the memory usage is the percentage of used memory out of the total memory. These values ​​are arranged in a preset order to form the local state feature vector of the non-backbone node.

[0044] For example, if a node reports a task progress of 80%, a CPU utilization of 60%, and a memory usage of 70%, then its feature vector is [80, 60, 70]. This vector is used for subsequent load assessment and anomaly detection.

[0045] It should be noted that the dynamic deviation value of the local state feature vector is calculated by comparing the current feature vector with the historical normal state baseline vector. The baseline vector is the average value of historical data within a sliding window. For example, the mean of each dimension of the local state feature vector is calculated by collecting the local state feature vectors of each hour over the past 24 hours as the baseline. The deviation value is measured by Euclidean distance or Mahalanobis distance to measure the degree of deviation between the current vector and the baseline vector. If the deviation value is higher than the preset abnormal fluctuation threshold, a resource bottleneck warning sign is generated. This threshold is set according to the statistical distribution of historical deviation values. For example, the deviation value data of all nodes within a week is collected, and the 95th percentile is taken as the threshold. When the deviation value of a node exceeds this threshold, it is judged as abnormal.

[0046] For example, based on historical data statistical analysis, the normal fluctuation range is determined to be within 0.2, the abnormal fluctuation threshold is 0.3, and when the node deviation value reaches 0.5, the system generates a resource bottleneck warning sign.

[0047] It should be noted that, based on the generated resource bottleneck warning flag, the status entry of the corresponding non-backbone node in the locally maintained neighbor node status table is updated, the node is marked as a resource bottleneck and its current task execution progress value and hardware resource usage metric value are stored.

[0048] For example, the entry for node NODE_123 is updated to show a task progress of 80%, CPU utilization of 70%, and memory usage of 65%, and a high load label is added. The backbone nodes then use a weighted round-robin algorithm to adjust task allocation based on this status table, prioritizing the allocation of new tasks to nodes with loads below a preset threshold, such as 60%, and avoiding the allocation of tasks to nodes marked as resource bottlenecks, thereby achieving dynamic balancing of network load.

[0049] In step S15, the local state feature vectors are summarized to obtain global state information. The global state information is then encoded using a preset Vandermonde matrix. If the encoded segmented sequence meets the preset encoding redundancy requirement, a storage mapping index is constructed and locked, including: Obtain the global state information summarizing the local state feature vectors, and perform a linear transformation on the global state information using a preset Vandermonde matrix to generate an encoded segment sequence; The encoded fragment sequence is sent to a preset candidate storage node list via a distribution routing path; Receive write confirmation signals from the preset candidate storage node list, and count the number of valid fragments successfully stored based on the write confirmation signals; If the number of effective fragments reaches the preset coding redundancy requirement, then the storage mapping index is constructed and locked.

[0050] It should be noted that the local state feature vectors of the backbone nodes are obtained, and the data such as the task execution progress, CPU utilization, and memory usage of all non-backbone nodes are integrated to form global state information. This information is organized in the form of data blocks, for example, containing 100 raw data units. The global state information is then linearly transformed and encoded using a preset Vandermonde matrix. The Vandermonde matrix is ​​constructed based on a finite field, with the number of rows corresponding to the number of encoded fragments n and the number of columns corresponding to the number of raw data blocks k. The k raw data blocks are linearly combined through matrix multiplication to generate n encoded fragment sequences, where n is greater than k, and the redundancy r is equal to n minus k, set according to the system reliability requirements.

[0051] For example, if the requirement is to be able to recover data even when any 50 nodes fail simultaneously, then k is set to 100, n to 150, and redundancy to 50. After encoding, 150 encoded fragments are generated, each fragment carrying part of the original data information, ensuring that any 100 fragments can be decoded to recover the complete global state information.

[0052] It should be noted that the encoded fragment sequence is sent to a preset candidate storage node list through a preset distribution routing path. This candidate storage node list is generated by the backbone node from non-backbone nodes in the entire network based on the node's remaining storage capacity and current network latency.

[0053] For example, 10 candidate nodes are selected based on the condition that the remaining storage capacity is greater than 500MB and the network latency is less than 50 milliseconds. The 150 encoded fragments are then allocated to each node using an average distribution strategy, with 15 fragments distributed. The distribution routing path is determined based on the minimum cost path between each node calculated in the weighted communication path cost matrix, ensuring the efficiency of fragment transmission and load balancing.

[0054] It should be noted that the backbone nodes receive write confirmation signals from each node in the preset candidate storage node list after completing the coded fragment writing. Each write confirmation signal carries the fragment identifier of the corresponding fragment and the node identifier of the storage node. Based on the received confirmation signals, the backbone nodes count the number of successfully stored valid fragments and construct a storage mapping index based on the fragment identifier and node identifier carried in the confirmation signals. This index records the correspondence between each coded fragment and its storage node. For example, if 135 fragments are successfully written and confirmation signals are received, these 135 mapping relationships are recorded in the index table. Then, the backbone nodes determine whether the counted number of valid fragments meets the preset coding redundancy requirement. The preset redundancy requirement is set according to the minimum number of fragments required for data recovery. To ensure that data can still be recovered even if some nodes fail, the redundancy requirement is set to no less than 120 valid fragments. If the actual number of valid fragments meets this requirement, the storage mapping index is locked, prohibiting subsequent modifications, thus completing the distributed redundant storage of global state information.

[0055] For example, if 135 shards are successfully written and acknowledgment signals are received, the backbone node will organize this location information into an index for easy data retrieval later. The preset redundancy requirement is 120 valid shards, but since 135 shards have actually been successfully stored, the backbone node will lock the index table to prevent subsequent modifications.

[0056] In step S16, if there are abnormal nodes in the backbone node set, the reconstruction process of the backbone node set is triggered. Based on the latest link status information and node load feedback from the topology probe, a new backbone node set is regenerated, including: Collect the heartbeat frequency and task processing queue length of the backbone node set and the adjacent non-backbone nodes to generate a real-time load status vector table. If there are abnormal nodes in the real-time load status vector table, generate a network reconstruction trigger command. In response to the network reconstruction trigger command, a topology probe is sent to the entire network through the backbone node set, and a network link status topology map is constructed based on the link status information fed back by the topology probe; Input the network link status topology map into a preset multi-dimensional weighted evaluation model and output a comprehensive score list of candidate backbone nodes. Nodes that meet the preset coverage conditions are selected from the comprehensive score list of candidate backbone nodes to form a new backbone node set.

[0057] It should be noted that the heartbeat frequency and task processing queue length of the backbone node set and adjacent non-backbone nodes are collected. The heartbeat frequency is obtained by counting the number of heartbeat packets received per unit time, and the task processing queue length is read from the node's local task scheduler. These data are combined to form a real-time load status vector table. Each node in the table has an entry recording its current heartbeat frequency and queue length. An anomaly judgment threshold is set. The heartbeat frequency threshold is set according to the statistical distribution of historical heartbeat data when the node is running normally. For example, the lowest value under normal conditions is taken as the threshold. By analyzing the heartbeat frequency data of all nodes within a week, the average value is calculated to be 2 times per second, and the standard deviation is 0.3. The threshold is set to the average value minus twice the standard deviation, which is 1.4 times per second. Considering that the heartbeat counter is usually designed as an integer, in order to simplify the judgment logic and reserve a certain detection margin, the actual anomaly judgment threshold is rounded to 1 time per second. That is, when the heartbeat frequency is lower than 1 time per second, it is judged as an anomaly.

[0058] For example, if a node has 4 CPU cores and the average processing time for each task is 100 milliseconds, then the maximum concurrent processing capacity is 40 tasks. Considering the buffer capacity, the threshold is set to 100 tasks. If a node's heartbeat frequency is less than once per second or the task processing queue length exceeds 100 tasks, and this abnormal state lasts for three sampling periods (the sampling period is set to 10 seconds based on the network state change rate), then the node is determined to be an abnormal node and a network reconstruction trigger instruction is generated.

[0059] It should be noted that after responding to the network reconfiguration trigger command, the current backbone node set broadcasts topology probe data packets to all nodes in the network. Each probe packet carries a sending timestamp and a unique sequence number. After receiving the probe, non-backbone nodes record the receiving timestamp and immediately send back a response packet. The response packet contains information such as the original sending timestamp, the receiving timestamp of this node, the current link round-trip time, the packet loss rate within the statistical period, and the available bandwidth. After collecting all feedback data, the backbone nodes calculate the comprehensive quality weight of each link based on the round-trip time and packet loss rate. For example, the link quality value is obtained by weighted summing of round-trip time and packet loss rate, where the round-trip time weight is set to 0.7 and the packet loss rate weight is set to 0.3. The weights are determined by the analytic hierarchy process (AHP). A comparison matrix is ​​constructed with link quality as the objective and latency and packet loss rate as the criteria. After a consistency check (CR < 0.1), the calculated eigenvector is the weight value. When the round-trip latency of a link increases from the historical baseline average of 50 milliseconds to 800 milliseconds and the packet loss rate reaches 15%, its comprehensive quality weight increases significantly, reflecting link degradation. Finally, a network link status topology map is constructed with nodes as vertices and link quality weights as edges, which intuitively presents the real-time network connectivity status and congested areas.

[0060] It should be noted that the entire network link status topology map is input into a preset multi-dimensional weighted evaluation model. This model first extracts four dimensions of indicators for each node from the topology map: remaining energy, computing power, link stability, and historical load balancing. Remaining energy is read from the node's power module and expressed as a percentage. Computing power is normalized by combining the number of CPU cores and clock speed. Link stability is calculated by statistically analyzing the average packet loss rate and round-trip delay variance between the node and all its neighbors. Historical load balancing is measured based on the variance of the node's comprehensive load score over the past ten sampling periods. Each indicator is mapped to the 0-1 range using the max-min normalization method. After normalization, the node's comprehensive score is obtained by weighted summation according to the weights determined by the analytic hierarchy process. The weights for remaining energy, computing power, link stability, and historical load balancing are 0.2, 0.2, 0.3, and 0.3, respectively. The weight settings are determined by expert scoring and consistency checks based on the network's requirements for reliability and performance.

[0061] For example, the original backbone node A has a lower link stability index due to link congestion, resulting in a comprehensive score of 0.62. On the other hand, the new node B has a comprehensive score of 0.89 due to sufficient remaining power and excellent link quality. Finally, a list of comprehensive scores of all candidate backbone nodes is output for subsequent selection.

[0062] It should be noted that the candidate backbone nodes are sorted in descending order of their comprehensive scores from the comprehensive score list. A greedy algorithm is used to select nodes sequentially. For each selected node, the non-backbone nodes it can cover are marked as covered. This process is repeated until all non-backbone nodes are covered, forming a new backbone node set. This coverage condition is set based on the system's basic requirements for network connectivity, ensuring that each non-backbone node is managed by at least one backbone node. For example, by statistically analyzing historical data on network size and the average coverage capability of backbone nodes, it is determined that several nodes with the highest scores need to be selected until 100% coverage is achieved. In actual operation, selecting the first five nodes may be sufficient to cover more than 90% of all non-backbone nodes, but to achieve complete coverage, selection must continue until the condition is met. Finally, the new backbone node set is reliably broadcast to all nodes in the network. Each node updates its local role and routing table accordingly, completing the dynamic backbone election and coverage set selection. For example, the top five nodes with the highest scores are selected as the new backbone node set. These nodes can cover more than 90% of the original non-backbone node communication range. The new backbone node set is then distributed to the entire network via reliable broadcast, and each node updates its local role and routing table, thereby completing the dynamic backbone election and coverage set selection.

[0063] In step S17, based on the new backbone node set, encoded data blocks are re-obtained from the non-backbone nodes, and the encoded data blocks are decoded using the inverse of the preset VanderMonde matrix to recover the complete global state information, including: A data retrieval request queue is generated based on the distributed hash mapping table maintained by the new backbone node set, and the encoded data blocks fed back by the non-backbone nodes are received by traversing the data retrieval request queue. The encoded data that has passed the preset cyclic redundancy check is divided into blocks and stored in a preset temporary reassembly buffer. If the preset temporary reassembly buffer meets the preset minimum recovery threshold, a data matrix to be decoded is generated. The data matrix to be decoded is input into a preset erasure coding decoding engine to restore the original binary state data stream. The original binary state data stream is then parsed to obtain the network topology and node load data. The network topology and node load data are written into a preset global state database to obtain complete global state information.

[0064] It should be noted that a data retrieval request queue is generated based on the distributed hash mapping table maintained by the new backbone node set. This distributed hash mapping table records the correspondence between each encoded data block and the storage node. For example, blocks 1 to 5 are stored on nodes A to E, and blocks 6 to 10 are stored on nodes F to J. The backbone nodes traverse this mapping table to generate a retrieval request for each block that needs to be retrieved, and arrange them in order of block number to form a queue. The queue is traversed to send retrieval instructions to the target non-backbone nodes in turn, receive the encoded data blocks fed back by each node, and record the reception time and size of each block.

[0065] For example, a 2MB block 1 is received from node A and a 1.5MB block 2 is received from node B. At the same time, a cyclic redundancy check code is used to initially verify the data integrity, preparing the data for subsequent decoding and recovery.

[0066] It should be noted that the encoded data that passes the preset cyclic redundancy check is stored in a preset temporary reassembly buffer. The capacity of this buffer is set according to the maximum amount of data required for system recovery. For example, if the average size of each block is 1MB and at least 120 blocks are required to meet the recovery needs, the buffer capacity is set to 128MB to leave a margin. If a block fails the cyclic redundancy check, a pull request is resent to the original storage node. The request is retried up to three times. If the retrieval still fails, the system queries other nodes storing the same block from the distributed hash map to continue pulling until all blocks pass the check or the retry limit is reached. When the number of verified blocks reaches a preset minimum recovery threshold, a data matrix to be decoded is generated. This threshold is set according to the erasure coding parameters. For example, during encoding, the original data is divided into k equal to 100 blocks and the total number of encoded blocks n equals 150. To ensure that the original data can be decoded and recovered, at least any k blocks, i.e., 100 blocks, must be obtained. Considering network latency and node failure factors, the minimum recovery threshold is set to 120 blocks to provide 20 blocks of redundancy. When the number of blocks in the buffer reaches 120, they are arranged into a k-row n-column data matrix to be decoded according to the block number order. Each column in the matrix corresponds to an encoded block, providing input data for the subsequent erasure coding decoding algorithm.

[0067] For example, the threshold is set to receive 80% of the block data, that is, after 8 blocks are in place, the backbone node arranges these blocks into a matrix according to a predetermined rule to facilitate subsequent decoding processing.

[0068] It should be noted that the data matrix to be decoded is input into a preset erasure coding decoding engine. This decoding engine uses the inverse of the Vandermonde matrix, which is the same as that used during encoding, for linear transformation. According to the principle of erasure coding, when the number of effective fragments in the matrix reaches the number of original data blocks k set during encoding, the original binary state data stream can be restored by solving a system of linear equations. The value of k is set according to the system's trade-off between data reliability and storage overhead. For example, to tolerate a maximum of 50 fragment losses, k is set to 100, and the total number of encoded fragments n is set to 150. The number of rows in the data matrix to be decoded is k, and the number of columns is the number of effective fragments actually received. The decoding engine selects any k linearly independent fragments to calculate the original data stream. For example, if 120 fragments are actually received, including 100 linearly independent fragments, the original binary data stream of size 10MB is successfully restored. Then, the data stream is parsed according to a predefined data structure to extract network topology information such as the connection relationship between nodes and link latency, as well as node load data such as the CPU utilization and memory usage of each node.

[0069] For example, after parsing, it was found that there are 15 nodes in the network, of which 5 nodes have a CPU utilization of 80%, and some link delays are found to be beyond the normal range, providing a basis for subsequent task allocation and route adjustment.

[0070] It should be noted that the parsed network topology and node load data are written into a preset global state database. This database adopts a distributed key-value storage structure and can store the state information of at least 1,000 nodes according to the network size. Each node record includes fields such as node identifier, neighbor list, connectivity, average path length, CPU utilization, memory usage, and comprehensive load score.

[0071] For example, after the update, the database records that node A has a connectivity score of 3 and a load score of 0.72, and node B has a connectivity score of 5 and a load score of 0.65. The backbone nodes can then read the latest global status information from this database for task scheduling and routing decisions, thereby completing the full recovery and storage of the global status information.

[0072] In step S18, based on the complete global state information, an updated task execution list and routing control instruction packet are published to all non-backbone nodes to obtain the reconstructed system collaboration relationship, including: Extract node computing power and link communication bandwidth data from the complete global state information, and calculate the load balancing factor of each node by combining the topological adjacency relationship; The load balancing factor is input into a preset multi-objective optimization model to generate a task allocation matrix, and the task allocation matrix is ​​mapped to form a task execution list; The routing table is updated according to the task execution list and encapsulated into a routing control instruction packet, which is then transmitted to the non-backbone node. The system receives confirmation feedback signals from the non-backbone nodes, updates the collaboration status snapshot based on the confirmation feedback signals, and obtains the final reconstructed system collaboration relationship.

[0073] It should be noted that the computing power and link communication bandwidth data of each node are extracted based on complete global state information. The computing power is quantified by the number of CPU cores, clock speed, and memory capacity of the node and expressed as the maximum number of concurrent tasks. For example, benchmark tests determine that the maximum number of concurrent tasks for node 1 is 5000, and for node 2 it is 3000. The link communication bandwidth is obtained from the link state matrix. For example, the bandwidth from node 1 to node 2 is 100Mbps, and from node 2 to node 3 it is 80Mbps. The number of neighbors of each node and the link bandwidth utilization are calculated by combining the topological adjacency relationship. The current task queue length is extracted from the local state feature vector. A weighted average formula is used when calculating the load balancing factor. in, For nodes The current length of the task queue. For nodes Maximum number of concurrent tasks, and The ratio of these values ​​is used as the node load factor. For nodes The average bandwidth utilization of all neighboring links is used as the link load factor. The preset balance coefficient, with a value between 0 and 1, is determined using the analytic hierarchy process (AHP) based on the network's sensitivity to computing and communication resources.

[0074] For example, in a compute-intensive scenario, set Equal to 0.6, 1 minus The load factor for node 1 is 0.4. If node 1 currently has 3000 tasks, its load factor is 3000 divided by 5000, which equals 0.6. If the average link bandwidth utilization is 0.5, the link load factor is 0.5. The overall load balancing factor is 0.6 multiplied by 0.6 plus 0.4 multiplied by 0.5, which equals 0.56. If node 2 currently has 2400 tasks and a maximum concurrent task count of 3000, its load factor is 0.8. If the average link bandwidth utilization is 0.7, the link load factor is 0.7. The overall factor is 0.6 multiplied by 0.8 plus 0.4 multiplied by 0.7, which equals 0.76. A higher factor indicates a greater node load pressure, providing a basis for subsequent task allocation.

[0075] It should be noted that the calculated load balancing factor and the distance data between nodes are input into a preset multi-objective optimization model. This model uses a weighted summation method to construct the objective function, minimizing the overall system load imbalance and communication latency. The load imbalance is represented by the sum of the products of the task allocation amount of each node and its load balancing factor, and the communication latency is represented by the sum of the products of the task allocation amount and the shortest path cost between nodes. The weight coefficients of the two objectives are determined by the analytic hierarchy process based on business priorities. For example, the load balancing weight is set to 0.6 and the communication latency weight is set to 0.4. The constraints are that the task allocation amount of each node does not exceed its maximum processing capacity and the total number of tasks is equal to the number of tasks to be allocated. The task allocation matrix is ​​obtained by solving linear programming, and each element in the matrix represents the proportion or number of tasks allocated to each node.

[0076] It is worth noting that the preset multi-objective optimization model is specifically a constrained linear programming model; the distance data between nodes in the model refers to the shortest path cost value between nodes extracted from the weighted communication path cost matrix constructed in step S13. This cost value comprehensively reflects the communication delay and bandwidth usage costs; the objective function of the model is to minimize the total cost, which is composed of a weighted average of node processing cost and inter-node communication cost. The node processing cost is the sum of the products of the load balancing factor and the allocated task volume for each node, and the inter-node communication cost is the sum of the products of the path cost and communication traffic between each pair of nodes. Both costs are multiplied by their corresponding weighting coefficients before being summed. The total cost is obtained by adding the weight coefficients, which sum to 1. The weight coefficients are dynamically adjusted based on real-time feedback of the system's operating status. For example, the weight of communication cost is increased when the network is congested, and the weight of processing cost is increased when the node load is uneven, to adapt to different operating conditions. The constraints of the model include that the amount of tasks assigned to each node does not exceed its maximum processing capacity; the total communication traffic on each link does not exceed its available bandwidth; and the sum of the task assignments of all nodes equals the total amount of tasks to be assigned. By solving this linear programming model, the optimal task assignment matrix is ​​obtained, where each element represents the amount of tasks that the corresponding node should undertake, thereby generating a specific task execution list.

[0077] For example, the solution results show that node 1 is allocated 40% of the task workload, node 2 is allocated 20%, node 3 is allocated 25%, and node 4 is allocated 15%. Multiplying the proportions in the matrix by the total number of tasks to be allocated and rounding down, a specific task execution list is formed, in which the task identifier and quantity to be executed for each node are clearly specified.

[0078] It should be noted that, based on the number of tasks assigned to each node in the task execution list, the routing table is updated. First, the current link latency and available bandwidth data are obtained from the global status information. The weighted shortest path algorithm is used to recalculate the optimal path between each node. The link weight is set based on a combination of latency and bandwidth. For example, the latency weight is 0.7 and the bandwidth weight is 0.3. The weight coefficient is determined based on historical data statistical analysis. For every 1 millisecond increase in latency, the weight increases by 0.1. For every 1 Mbps decrease in bandwidth, the weight increases by 0.05. This ensures that when a node with a large task load, such as node 1, is assigned 40% of the tasks, its forwarding path will prioritize the link with the lowest total weight. The updated routing table entries are encapsulated into routing control command packets. The command packets contain information such as the target node identifier, next-hop address, and path weight value. These packets are then distributed in batches to all non-backbone nodes through a reliable transmission protocol such as TCP, ensuring that each node updates its local routing table synchronously.

[0079] For example, if node 1 is assigned more tasks, the forwarding table will prioritize assigning it links with shorter paths, and then encapsulate them into routing control command packets and send them to non-backbone nodes.

[0080] It should be noted that when receiving confirmation feedback signals from non-backbone nodes, if no confirmation is received within a preset timeout period, such as 5 seconds, a retransmission will be performed. The maximum number of retransmissions is set to three based on network reliability requirements. The collaboration status snapshot is updated based on all received confirmation signals. Confirmed nodes are marked as having effective collaboration relationships and their latest task and routing information is recorded. Unconfirmed nodes are marked as pending processing and their abnormal status is recorded, thus obtaining the final reconstructed system collaboration relationship.

[0081] For example, if seven out of eight non-backbone nodes return confirmation, then in the snapshot, the status of seven nodes is synchronized, one node is pending and triggers the subsequent exception handling process.

[0082] It should be noted that the weighting coefficients involved in this scheme, such as the preset node weighting coefficient in step S11, the multi-dimensional weighted evaluation model weight in step S16, the balance coefficient and multi-objective optimization weight in step S18, are all determined based on the business scenario using the analytic hierarchy process (AHP) or the entropy weighting method. Taking the balance coefficient in step S18 as an example, its value depends on the network's sensitivity to computational and communication loads. In computationally intensive distributed computing scenarios, node CPU and memory resources are more strained, and the balance coefficient can be set to 0.7~0.8, focusing more on balancing the computational load. In communication-intensive scenarios, link bandwidth is more critical, and the balance coefficient can be set to 0.3~0.4, focusing more on balancing the link load. The specific values ​​can be obtained during system initialization by analyzing historical task execution data, with the goal of minimizing the average task completion time, and by using a grid search method to traverse and optimize within the 0~1 range.

[0083] In summary, this invention discloses an information collaborative processing method in a distributed environment, including: acquiring node connection relationships to form a global connection dataset; calculating node connectivity, average path length, and comprehensive load score; selecting highly stable nodes as candidate backbones; selecting a minimum backbone set to cover all non-backbone nodes; periodically collecting local state feature vectors of non-backbone nodes, summarizing them to obtain global state information, and redundantly storing it in non-backbone nodes after encoding with a Vandermonde matrix; triggering reconstruction when backbone nodes are detected to be abnormal, with new backbone nodes obtaining encoded blocks from non-backbone nodes and restoring global state information through erasure coding; and publishing an updated task list and routing instructions based on the restored information to complete the reconstruction of collaborative relationships. This invention dynamically organizes backbone nodes through real-time data analysis, achieving reliable storage and rapid recovery of global state information, significantly improving the adaptive capability, fault tolerance, and collaborative efficiency of distributed systems.

[0084] Reference Figure 2 The second embodiment of the present invention provides an information collaborative processing system in a distributed environment, comprising: The network status awareness and load assessment module obtains the current connection relationships of all nodes in the distributed network, forms a global connection dataset, calculates the node connectivity of the node under test and the average path length to other nodes in the global connection dataset, and calculates a comprehensive load score based on the CPU utilization and memory usage of the node under test. The candidate backbone node screening module calculates the predicted node lifecycle of the node to be tested if the node connectivity is greater than a preset effective connection threshold and the comprehensive load score is less than a preset load threshold. If the predicted lifecycle is in a preset high stability range, the node to be tested is marked as a candidate backbone node. The backbone node election and coverage optimization module selects the smallest subset from the candidate backbone nodes that can cover all non-backbone nodes and determines it as the backbone node set. The local state acquisition module collects local state feature vectors from the non-backbone nodes connected to the backbone node set; The global state encoding and redundant storage module summarizes the local state feature vectors to obtain global state information, and encodes the global state information using a preset Vandermonde matrix. If the encoded fragmented sequence meets the preset encoding redundancy requirements, then a storage mapping index is constructed and locked. The backbone node dynamic reconstruction module, if there are abnormal nodes in the backbone node set, triggers the reconstruction process of the backbone node set, and regenerates a new backbone node set based on the latest link status information and node load fed back by the topology probe. The global state recovery module re-obtains encoded data blocks from the non-backbone nodes based on the new backbone node set, and decodes the encoded data blocks using the inverse of the preset VanderMonde matrix to recover the complete global state information. The collaboration relationship reconstruction and instruction issuance module publishes the updated task execution list and routing control instruction package to all non-backbone nodes based on the complete global state information, thereby obtaining the reconstructed system collaboration relationship.

[0085] It should be noted that the information collaborative processing system in a distributed environment provided in this embodiment of the invention is used to execute all the process steps of the information collaborative processing method in a distributed environment described in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0086] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0087] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for collaborative information processing in a distributed environment, characterized in that, include: Obtain the current connection relationships of all nodes in the distributed network to form a global connection dataset. Calculate the node connectivity of the node under test and the average path length to other nodes in the global connection dataset. Calculate the comprehensive load score based on the CPU utilization and memory usage of the node under test. Calculate the predicted node lifetime of the node under test that has a node connectivity greater than a preset effective connection threshold and a comprehensive load score less than a preset load threshold. If the predicted lifetime is in a preset high stability range, then mark the node under test as a candidate backbone node. Select the smallest subset from the candidate backbone nodes that can cover all non-backbone nodes, and determine it as the backbone node set; Collect local state feature vectors from the non-backbone nodes connected to the backbone node set; The global state information is obtained by summarizing the local state feature vectors. The global state information is encoded using a preset Vandermonde matrix. If the encoded segmented sequence meets the preset encoding redundancy requirement, the storage mapping index is constructed and locked. If there are abnormal nodes in the backbone node set, the reconstruction process of the backbone node set is triggered, and a new backbone node set is regenerated based on the latest link status information and node load fed back by the topology probe. Based on the new backbone node set, the encoded data blocks are re-obtained from the non-backbone nodes, and the encoded data blocks are decoded using the inverse of the preset Vandermonde matrix to recover the complete global state information. Based on the complete global state information, the updated task execution list and routing control instruction package are published to all non-backbone nodes to obtain the reconstructed system collaboration relationship.

2. The information collaborative processing method in a distributed environment according to claim 1, characterized in that, Obtain the current connection relationships of all nodes in the distributed network to form a global connectivity dataset. Calculate the node connectivity degree of the node under test and the average path length to other nodes in the global connectivity dataset. Based on the CPU utilization and memory usage of the node under test, calculate a comprehensive load score, including: The node under test in the distributed network is controlled to send probe messages to neighboring nodes. The difference between the receiving timestamp and the sending timestamp of the probe message is calculated to obtain the one-way communication delay value. If the one-way communication delay value is less than a preset effective time threshold, a local neighbor list of the node under test is generated, and all local neighbor lists are aggregated to form a global connection dataset. Based on the global connectivity dataset, determine the node connectivity of the node under test and the average path length to other nodes; The CPU utilization and memory usage of the node under test are obtained, and the node connectivity, the average path length, the CPU utilization and the memory usage are encapsulated into a multi-dimensional resource consumption vector. The multi-dimensional resource consumption vector is normalized to generate a load normalized value. A preset node weight coefficient is matched to the normalized load value, and the comprehensive load score of each node is calculated.

3. The information collaborative processing method in a distributed environment according to claim 2, characterized in that, Calculate the predicted lifetime value of the node under test for nodes whose node connectivity is greater than a preset effective connection threshold and whose comprehensive load score is less than a preset load threshold. If the predicted lifetime value is within a preset high stability range, then the node under test is marked as a candidate backbone node, including: If the node connectivity is greater than a preset effective connection threshold and the overall load score is less than a preset load threshold, then a preliminary candidate node index is generated. The historical data of the online duration of the node and the link stability coefficient corresponding to the node to be tested are obtained from the preliminary candidate node index and input into the preset link failure probability model to calculate the predicted life cycle value of the node to be tested. If the predicted node lifecycle value is within a preset high stability range, the corresponding node will be marked as a candidate backbone node.

4. The information collaborative processing method in a distributed environment according to claim 3, characterized in that, The smallest subset that can cover all non-backbone nodes is selected from the candidate backbone nodes and determined as the backbone node set, including: The link transmission rate between the candidate backbone node and its neighboring nodes is obtained through the global connectivity dataset, and the weighted communication path cost matrix between the nodes is calculated based on the link transmission rate. Based on the weighted communication path cost matrix, a weighted shortest path tree is constructed from the candidate backbone nodes to non-backbone nodes, and a node coverage index is generated. Establish a dominance mapping relationship between the candidate backbone nodes and the non-backbone nodes based on the node coverage index, and calculate the independent coverage contribution based on the dominance mapping relationship; Candidate backbone nodes are removed in order of their independent coverage contribution from low to high. If the remaining candidate backbone nodes can still cover all the non-backbone nodes after removal, the removal is performed until no further removal is possible. The final remaining candidate backbone nodes are then used as the backbone node set.

5. The information collaborative processing method in a distributed environment according to claim 4, characterized in that, Local state feature vectors are collected from the non-backbone nodes connected to the backbone node set, including: A status request command is sent to the non-backbone nodes through the polling clock signal configured in the backbone node set, and a status feedback data packet is obtained. Extract task execution progress values ​​and hardware resource usage metrics from the status feedback data packets to construct the local state feature vector of the non-backbone node; Calculate the dynamic deviation value of the local state feature vector. If the dynamic deviation value is higher than the preset abnormal fluctuation threshold, generate a resource bottleneck early warning sign. Update the node status record of the local state feature vector based on the resource bottleneck early warning identifier.

6. The information collaborative processing method in a distributed environment according to claim 5, characterized in that, The global state information is obtained by summarizing the local state feature vectors. The global state information is then encoded using a preset Vandermonde matrix. If the encoded segmented sequence meets a preset encoding redundancy requirement, a storage mapping index is constructed and locked, including: Obtain the global state information summarizing the local state feature vectors, and perform a linear transformation on the global state information using a preset Vandermonde matrix to generate an encoded segment sequence; The encoded fragment sequence is sent to a preset candidate storage node list via a distribution routing path; Receive write confirmation signals from the preset candidate storage node list, and count the number of valid fragments successfully stored based on the write confirmation signals; If the number of effective fragments reaches the preset coding redundancy requirement, then the storage mapping index is constructed and locked.

7. The information collaborative processing method in a distributed environment according to claim 6, characterized in that, If there are abnormal nodes in the backbone node set, the backbone node set reconstruction process is triggered. Based on the latest link status information and node load feedback from the topology probe, a new backbone node set is regenerated, including: Collect the heartbeat frequency and task processing queue length of the backbone node set and the adjacent non-backbone nodes to generate a real-time load status vector table. If there are abnormal nodes in the real-time load status vector table, generate a network reconstruction trigger command. In response to the network reconstruction trigger command, a topology probe is sent to the entire network through the backbone node set, and a network link status topology map is constructed based on the link status information fed back by the topology probe; Input the network link status topology map into a preset multi-dimensional weighted evaluation model and output a comprehensive score list of candidate backbone nodes. Nodes that meet the preset coverage conditions are selected from the comprehensive score list of candidate backbone nodes to form a new backbone node set.

8. The information collaborative processing method in a distributed environment according to claim 7, characterized in that, Based on the new backbone node set, coded data blocks are re-obtained from the non-backbone nodes, and the coded data blocks are decoded using the inverse of the preset VanderMonde matrix to recover complete global state information, including: A data retrieval request queue is generated based on the distributed hash mapping table maintained by the new backbone node set, and the encoded data blocks fed back by the non-backbone nodes are received by traversing the data retrieval request queue. The encoded data that has passed the preset cyclic redundancy check is divided into blocks and stored in a preset temporary reassembly buffer. If the preset temporary reassembly buffer meets the preset minimum recovery threshold, a data matrix to be decoded is generated. The data matrix to be decoded is input into a preset erasure coding decoding engine to restore the original binary state data stream. The original binary state data stream is then parsed to obtain the network topology and node load data. The network topology and node load data are written into a preset global state database to obtain complete global state information.

9. The information collaborative processing method in a distributed environment according to claim 8, characterized in that, Based on the complete global state information, an updated task execution list and routing control instruction package are published to all non-backbone nodes to obtain the reconstructed system collaboration relationships, including: Extract node computing power and link communication bandwidth data from the complete global state information, and calculate the load balancing factor of each node by combining the topological adjacency relationship; The load balancing factor is input into a preset multi-objective optimization model to generate a task allocation matrix, and the task allocation matrix is ​​mapped to form a task execution list; The routing table is updated according to the task execution list and encapsulated into a routing control instruction packet, which is then transmitted to the non-backbone node. The system receives confirmation feedback signals from the non-backbone nodes, updates the collaboration status snapshot based on the confirmation feedback signals, and obtains the final reconstructed system collaboration relationship.

10. An information collaborative processing system in a distributed environment, characterized in that, include: The network status awareness and load assessment module obtains the current connection relationships of all nodes in the distributed network, forms a global connection dataset, calculates the node connectivity of the node under test and the average path length to other nodes in the global connection dataset, and calculates a comprehensive load score based on the CPU utilization and memory usage of the node under test. The candidate backbone node screening module calculates the predicted node lifecycle of the node to be tested if the node connectivity is greater than a preset effective connection threshold and the comprehensive load score is less than a preset load threshold. If the predicted lifecycle is in a preset high stability range, the node to be tested is marked as a candidate backbone node. The backbone node election and coverage optimization module selects the smallest subset from the candidate backbone nodes that can cover all non-backbone nodes and determines it as the backbone node set. The local state acquisition module collects local state feature vectors from the non-backbone nodes connected to the backbone node set; The global state encoding and redundant storage module summarizes the local state feature vectors to obtain global state information, and encodes the global state information using a preset Vandermonde matrix. If the encoded fragmented sequence meets the preset encoding redundancy requirements, then a storage mapping index is constructed and locked. The backbone node dynamic reconstruction module, if there are abnormal nodes in the backbone node set, triggers the reconstruction process of the backbone node set, and regenerates a new backbone node set based on the latest link status information and node load fed back by the topology probe. The global state recovery module re-obtains encoded data blocks from the non-backbone nodes based on the new backbone node set, and decodes the encoded data blocks using the inverse of the preset VanderMonde matrix to recover the complete global state information. The collaboration relationship reconstruction and instruction issuance module publishes the updated task execution list and routing control instruction package to all non-backbone nodes based on the complete global state information, thereby obtaining the reconstructed system collaboration relationship.

Images