Link reliability tracking and optimizing method combined with real-time monitoring

Abstract

The invention discloses a link reliability tracking and optimizing method combined with real-time monitoring, and relates to the technical field of data transmission. Comprising the following steps: constructing a topological graph of a full link based on component information and a data flow relationship of a data medium station, and deploying a data acquisition probe to acquire link data in real time; a potential fault or an abnormal link is automatically identified; based on the data map, sorting the data dependency relationship of the upstream and downstream of the link, recording the relevance and dependency path of each component, and forming an accurate link blood relationship map; link abnormal conditions are captured and backtracked, log information is summarized in a unified mode, and centralized analysis and problem positioning are carried out; and on the basis of operation historical data of the link, according to the time sequence prediction model, identifying a bottleneck and a high-risk area in the link, dynamically adjusting a transmission strategy of the link, and optimizing load balance and a link priority of a station in the data in real time. According to the invention, the influence on the service when the fault occurs is reduced, and the abnormal problem positioning efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a link reliability tracking and optimization method combined with real-time monitoring. Background Art

[0002] With the rapid development of the information age, the complexity and scale of modern communication networks are constantly expanding, and the stability and reliability of network links have become important indicators for ensuring efficient data transmission. Current network monitoring technologies are mostly focused on passive link status detection, often relying on regular status reports or traditional fault diagnosis methods.

[0003] These methods usually require obvious network failures to trigger alarms and repair mechanisms, which often results in problems not being handled in a timely manner. In particular, in scenarios with complex data transmission and multi-level dependencies, existing methods are difficult to deal with efficiently. At the same time, with the popularization of the concept of data middleware, multiple business systems have formed a complex dependency network through links. How to monitor and track the health status of links in real time in scenarios with massive data flow has become the focus of current technical research. Summary of the invention

[0004] In view of the problems existing in the existing link monitoring and optimization technologies, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is how to identify potential problems and perform dynamic optimization by real-time collection and analysis of link data.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In the first aspect, an embodiment of the present invention provides a link reliability tracking and optimization method combined with real-time monitoring, which includes: building a full-link topology map based on component information and data flow relationships of the data middle station, and deploying data collection probes at key nodes to collect link data in real time; analyzing the real-time collected link data to automatically identify potential faulty or abnormal links; based on the data map, sorting out the data dependencies between upstream and downstream links, recording the correlation and dependency paths of each component, and forming an accurate link blood relationship map; capturing and tracing link abnormalities, and unifying the log information for centralized analysis and problem location; based on the link's operating history data, according to the time series prediction model, identifying bottlenecks and high-risk areas in the link, and dynamically adjusting the link's transmission strategy, and optimizing the load balancing and link priority of the data middle station in real time.

[0007] As a preferred solution of the link reliability tracking and optimization method combined with real-time monitoring described in the present invention, the construction of the topology diagram of the entire link includes: extracting component information based on the data middle station, identifying the data transmission link between components :When data is received from a component Transfer to another component , if there are M other components in the middle, and , then mark the intermediate transmission component as a transit layer; if , then it is recorded as a direct transmission relationship, where M is a non-negative integer; each component is represented as a node, and the data transmission relationship between components is represented by an edge. For indirect transmission, the transit layer is added to the graph as a virtual node to construct a full-link topology graph.

[0008] As a preferred solution of the link reliability tracking and optimization method combined with real-time monitoring of the present invention, the automatic identification of potential faulty or abnormal links includes: constructing a fault propagation graph , using the PageRank algorithm to calculate the fault propagation graph The importance score of each key node in ; According to the importance score and abnormal state, abnormal links are identified, including: The set of key abnormal nodes is defined as: ; in, is the PageRank threshold; is the key node; is the abnormal state of the key node. It is abnormal if It is normal; The abnormal link set is expressed as: ; in, For data transmission link.

[0009] As a preferred solution of the link reliability tracking and optimization method combined with real-time monitoring described in the present invention, wherein: the fault propagation graph is calculated using the PageRank algorithm The importance score of each key node in The calculation formula is: ; Where d is the damping factor, Represents the probability of jumping to any other node; For key nodes The PageRank score, PageRank score for common nodes, used to calculate key nodes The influence of its adjacent nodes needs to be considered when scoring; For ordinary nodes and key nodes The weight of the edge between them; For ordinary nodes The sum of the edge weights to all its neighboring nodes.

[0010] As a preferred solution of the link reliability tracking and optimization method combined with real-time monitoring described in the present invention, wherein: the data dependency relationship between the upstream and downstream of the link is sorted out, and the correlation and dependency path of each component are recorded, including the following steps: for each link, the dependency relationship between its upstream and downstream nodes is determined, and by analyzing the transmission order and direction of the data flow, the dependency path of each data processing link in the link is gradually sorted out; wherein, the upstream node represents the source of the data, and the downstream node represents the consumer of the data; the computing node in the link parses its code through an automated code parsing tool to identify the input dependency and output, and maps back to the data transmission in the link; based on the above data dependency and code parsing, the correlation between each component is further clarified, and for each component, the upstream component it depends on and the type of data it depends on are recorded; the sorted out data dependency, code parsing results and component correlation are integrated to construct a complete link blood relationship map.

[0011] As a preferred solution of the link reliability tracking and optimization method combined with real-time monitoring described in the present invention, the unified aggregation of log information for centralized analysis and problem location includes: unified organization of log snapshots of different links and different time periods to generate a complete abnormal log chain; classification and labeling of logs, and archiving by time, node, and abnormality type; using an abnormality analysis system to trace back and centrally analyze logs: for a single abnormality, analyze the system status at the time of triggering, find the log snapshots before and after the abnormality, and determine the root cause of the problem; for multiple abnormalities in the link, by analyzing the time series information of the log link, trace back and analyze the impact of each abnormal node and the path of abnormal propagation.

[0012] As a preferred solution of the link reliability tracking and optimization method combined with real-time monitoring described in the present invention, the dynamic adjustment of the link transmission strategy includes: according to the priority allocation formula, for each key node in the link To prioritize: ; in, For key nodes Priority; For key nodes The delay indicates the transmission delay of data at this key node; For key nodes The throughput, which indicates the transmission capacity of key nodes; For key nodes Utilization rate; For key nodes Memory usage.

[0013] As a preferred solution of the link reliability tracking and optimization method combined with real-time monitoring described in the present invention, the dynamic adjustment of the link transmission strategy also includes: to prevent bottleneck nodes from being overloaded, data traffic is evenly distributed to different links according to the load balancing strategy, and the link load is calculated as follows: ; in, For key nodes The current throughput of For key nodes Maximum throughput; If the load on a link exceeds a preset threshold , then dynamically allocate its transmission traffic to the link with low load and adjust the load balancing strategy of the link.

[0014] In a second aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the link reliability tracking and optimization method combined with real-time monitoring as described in the first aspect of the present invention are implemented.

[0015] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the link reliability tracking and optimization method combined with real-time monitoring as described in the first aspect of the present invention are implemented.

[0016] The beneficial effects of the present invention are as follows: through the construction of topology graphs and dependency analysis, the system can accurately grasp the overall picture and key nodes of the link, thereby improving the transparency and controllability of link management; and by using the PageRank algorithm to calculate the importance score of each key node in the fault propagation graph, and combining the abnormal state to judge the link failure, the automatic identification of potential failures or abnormal links is realized, reducing the impact of failures on the business. The present invention also improves the efficiency of problem location, reduces the workload of operation and maintenance personnel, and can reduce potential failure losses through early warning and problem location. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1Flowchart of the link reliability tracking and optimization method combined with real-time monitoring.

[0019] Figure 2 FIG. 1 is a schematic diagram of the structure of a terminal in the first embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1: Reference Figure 1~Figure 2 , which is the first embodiment of the present invention, and provides a link reliability tracking and optimization method combined with real-time monitoring, including: S1: Based on the component information and data flow relationship of the data center, a full-link topology diagram is constructed, and data collection probes are deployed at key nodes to collect link data in real time.

[0024] Preferably, based on the data middle platform, component information is extracted and each component is assigned a unique identifier.

[0025] Identify data transfer links between components :When data is received from a component Transfer to another component , if there are M other components in the middle, and if , mark the intermediate transmission component as a transit layer; if , then it is recorded as a direct transmission relationship, where M is a non-negative integer.

[0026] Each component is represented as a node, and the data transmission relationship between components is represented by edges. For indirect transmission, the transit layer is added to the graph as a virtual node to construct a full-link topology graph.

[0027] Based on factors such as data traffic, processing capacity or business importance, a series of key nodes are selected from the topology map, and k is the key node number.

[0028] Deploy data collection probes at each key node and configure the probes to collect the following indicators in real time: Transfer Status : Such as connection status, delay, etc.; Data flow : Such as throughput, number of packets, etc.; Key Performance Indicators : Such as CPU usage, memory usage, etc.

[0029] Establish a data collection mechanism so that the probes can regularly collect the indicators Transmitted to the central monitoring system.

[0030] S2: Analyze the link data collected in real time to automatically identify potential faulty or abnormal links.

[0031] The indicators collected in step S1 are preprocessed, including denoising, normalization and other operations. Suppose the processed data of the key node at time t is: .

[0032] Use the set thresholds and rules to analyze the pre-processed data, automatically identify potential abnormal links, and process the data of key nodes. The status mark at time t is compared with the preset abnormal condition to determine whether an abnormality is triggered. If there is no abnormality, the key node status at that time is marked as ; If some nodes are detected to trigger an exception, the key node status at that moment is marked as , and further analyze the adjacent nodes of these nodes.

[0033] Constructing a fault propagation graph , used to represent possible fault propagation paths. If both adjacent nodes trigger anomalies, then Add the above similar nodes to form edges.

[0034] Calculate the fault propagation graph using the PageRank algorithm The importance score of each key node in : ; Where d is the damping factor, which is usually taken as 0.85; Represents the probability of jumping to any other node; For key nodes The PageRank score, PageRank score for common nodes, used to calculate key nodes The influence of its adjacent nodes needs to be considered when scoring; For ordinary nodes and key nodes The weight of the edge between them; For ordinary nodes The sum of the edge weights to all its neighboring nodes.

[0035] Furthermore, based on PageRank importance scores and abnormal status, abnormal links are identified: The key abnormal node set is defined as: ; in, is the PageRank threshold, indicating that the importance of the node exceeds a certain critical value; the abnormal link set is expressed as: ; Considering the different weights of key nodes and common nodes, the severity of the anomaly is calculated for each abnormal link: ; in For ordinary nodes The PageRank score is calculated in the same way as above. and Respectively reflect whether the node is in an abnormal state.

[0036] It can be seen that this step evaluates the severity of the link abnormality by comprehensively considering the PageRank scores of the nodes at both ends of the link, the link weights, and the abnormal states of the nodes. Common fault detection methods are mostly based on static thresholds or simple link monitoring, while the dynamic weight evaluation method of the present invention significantly improves the accuracy of fault warning. Moreover, after combining the PageRank scores and abnormal states, by defining and identifying abnormal links, the system can accurately lock the critical path of fault propagation and effectively distinguish between minor faults and major faults, thereby achieving hierarchical response.

[0037] S3: Based on the data map, sort out the data dependencies of the upstream and downstream links, combine code analysis and link mapping technology, record the correlation and dependency path of each component, and form an accurate link blood relationship map.

[0038] Preferably, based on the abnormal links and normal links identified in step S2, a global data map is constructed to display all data flow paths. The global data map reflects the data interaction of each node in the system and records the transmission relationship between each link, including information such as data source, transmission path, and target node.

[0039] For each link, determine the dependency relationship between its upstream and downstream nodes, and gradually sort out the dependency path of each data processing link in the link by analyzing the transmission order and direction of the data flow.

[0040] The upstream node represents the source of the data, which may be a database, API, or other input source; the downstream node represents the consumer of the data, such as the downstream computing module, processing engine, or storage node, etc., which records the data dependency relationship between the nodes to form a dependency matrix.

[0041] Furthermore, the data read and write operations in each module are analyzed through automated code parsing tools. During the code parsing process, the input and output parameters of key functions are extracted and their corresponding relationship with the data flow is identified. For example, for a computing node in a link, its input dependency and output can be identified by parsing its code, and this information can be mapped back to the data transmission in the link.

[0042] Based on the above data dependencies and code analysis, the association between each component (such as nodes or modules) is further clarified. For each component, the upstream components it depends on and the data types it depends on are recorded to ensure that the association relationship between all components is clear and accurate.

[0043] Integrate the sorted data dependencies, code analysis results, and component associations to build a complete link blood relationship map. This map shows the dependency links of each node, data source, processing module, and terminal. Through the map, you can track the upstream and downstream relationships of any abnormal node or link to help quickly locate the source of the problem. Among them, the blood relationship map can be represented by a directed graph, in which the nodes represent the processing links of each component or data flow in the system, and the edges represent the dependencies.

[0044] After constructing the link lineage graph, the dependency path analysis is performed to identify the critical path, that is, the longest or most important path from the data source to the abnormal node.

[0045] The depth-first search (DFS) or breadth-first search (BFS) algorithm can be used to find all possible paths in the blood relationship graph and evaluate the impact of each path on link performance and data processing. Combined with the PageRank score in step S2, the key nodes and their dependent paths can be focused on.

[0046] The resulting link bloodline graph is a directed graph that accurately displays the upstream and downstream dependencies of the link. The graph not only records the relevance and dependency of each component, but also marks the transmission direction of the data flow, link abnormality status, and the scores of important nodes.

[0047] S4: Combined with log snapshot technology, link abnormalities are captured and traced back, and relevant log information is summarized for centralized analysis and problem location.

[0048] During the link operation process, combined with log snapshot technology, the key logs of each module of the system are captured in real time; when a link or node in the system triggers an exception, the exception information is captured in real time.

[0049] Furthermore, exception capture includes the following key steps: Capture the log information of abnormal nodes, including link status, input and output, node configuration, etc.; capture the log data of the upstream and downstream nodes of the abnormality to ensure that the logs of all related nodes in the process of abnormal propagation are captured; if the number of abnormal occurrences exceeds the preset threshold within a certain time window (such as 1 hour or 24 hours) or the impact of the abnormality on the system is large, such as serious abnormalities that cause key nodes to fail to work normally, then additionally capture the operating status of the abnormal node before a period of time in order to retrospectively analyze the cause and consequence of the abnormality; the core data contained in the abnormal log helps to understand the root cause of the abnormality and provides a basis for retrospective analysis.

[0050] The log snapshot information of each abnormal node and related links is uniformly aggregated into the abnormal analysis system. The log aggregation operation includes: unifying the log snapshots of different links and different time periods to generate a complete abnormal log chain; classifying and labeling the logs, and archiving them by time, node, and abnormality type to facilitate subsequent rapid retrieval and analysis.

[0051] The aggregated log information should include all key data related to the anomaly, such as the status changes of the abnormal node, dependency information of upstream and downstream links, system performance data, etc.

[0052] Use the exception analysis system to backtrack and centrally analyze the logs. Through backtracking analysis, you can locate the time point when the exception occurred, the triggering conditions, and the scope of its impact. For example: For a single anomaly, analyze the system status when it is triggered, find the log snapshots before and after the anomaly, and determine the root cause of the problem. For multiple anomalies in a link, analyze the time series information of the log link, trace back and analyze the impact of each abnormal node and the path of anomaly propagation.

[0053] The log backtracking process should be based on the timeline, gradually checking each link before the anomaly is triggered to locate possible system defects or data errors; in the backtracking analysis, combined with the blood relationship map in step S3, track how the anomaly propagates through the upstream and downstream links.

[0054] S5: Based on the historical operation data of the link and the timing prediction model, the bottlenecks and high-risk areas in the link are identified, and the transmission strategy of the link is dynamically adjusted to optimize the load balancing and link priority of the data center in real time.

[0055] First, key performance data is collected from the link's operation history, including but not limited to link delay time, packet loss rate, data throughput, resource usage of each node, number and type of abnormal occurrences, and link dependencies and link lineage graphs. Use ARIMA, LSTM or regression models (such as linear regression, random forest regression) to train the model using the historical data of the link. The input of the time series prediction model is the preprocessed link historical data set to identify link bottlenecks and high-risk areas.

[0056] Furthermore, based on the results of link bottleneck identification and high-risk area analysis, the transmission strategy of the link is dynamically adjusted, including: Give higher priority to links with bottlenecks or high risks to ensure that these links can handle key business traffic first. The adjustment strategy is as follows: ; in, For key nodes Priority; For key nodes The delay indicates the transmission delay of data at this key node; For key nodes The throughput, which indicates the transmission capacity of key nodes; For key nodes Utilization rate; For key nodes Memory usage.

[0057] According to the priority allocation formula, each key node in the link Prioritize.

[0058] To prevent bottleneck nodes from being overloaded, the system will evenly distribute data traffic to different links based on the load balancing strategy and monitor link loads in real time using the following formula: ; in, For key nodes The current throughput of For key nodes The maximum throughput indicates the maximum transmission capacity of the node.

[0059] If the load on a link exceeds a preset threshold , the transmission traffic is dynamically allocated to the link with lower load and the load balancing strategy of the link is adjusted.

[0060] When a link bottleneck occurs or the load is too high, the system can dynamically adjust the data transmission path and select a better link transmission path. The path optimization strategy is calculated using the following formula: ; in, is the optimized path, indicating the data transmission path after dynamic adjustment; is the risk value of the link; is the delay of the link; is the throughput of the link.

[0061] It can be seen that during the link operation, the system monitors the performance indicators of each link in real time, such as latency, throughput, and resource usage. The load balancing system will dynamically adjust the data traffic distribution strategy according to the current link status to ensure the overall load balance of the data center. By setting priorities for bottleneck links and high-risk areas, the system will give priority to ensuring the data transmission needs of critical businesses. Non-critical business traffic on low-risk links can be temporarily slowed down or delayed to ensure the stability and performance of critical links.

[0062] This embodiment also provides a computer device, which is suitable for the link reliability tracking and optimization method combined with real-time monitoring, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the link reliability tracking and optimization method combined with real-time monitoring proposed in the above embodiment.

[0063] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0064] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the link reliability tracking and optimization method combined with real-time monitoring proposed in the above embodiment is implemented.

[0065] In summary, through the construction of topology graphs and dependency analysis, the system can accurately grasp the overall picture and key nodes of the link, improving the transparency and controllability of link management; and the importance score of each key node in the fault propagation graph is calculated by the PageRank algorithm, and the link fault is judged in combination with the abnormal state to realize the automatic identification of potential faults or abnormal links, reducing the impact on the business when the fault occurs. The present invention improves the efficiency of abnormal problem location, reduces the workload of operation and maintenance personnel, and can reduce potential fault losses through abnormal judgment and location.

[0066] Example 2: Referring to Table 1, this is the second example of the present invention. This example provides a link reliability tracking and optimization method combined with real-time monitoring. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0067] The experimental environment consists of six key nodes, namely data source A, processing node B, transfer node C, computing node D, storage node E and target node F. These nodes are interconnected through a high-speed network to form a complex data processing link.

[0068] First, based on the component information and data flow relationship of the data center, a full-link topology diagram was constructed. In order to achieve this goal, distributed tracing technology was used to assign a unique identifier to each component. By analyzing the packet header and network traffic, the data transmission link between components was identified. In this process, it was found that the data transmission from data source A to target node F passed through multiple intermediate nodes, so these nodes were marked as transit layers.

[0069] A full-link topology graph was constructed using the graph database Neo4j, where each component is represented as a node and the data transmission relationship between components is represented by edges. Considering the data flow, processing power, and business importance, the above six nodes were selected as key nodes, and high-performance data collection probes were deployed on these nodes. These probes are configured to collect transmission status (such as connection status, latency), data flow (such as throughput, number of packets), and key performance indicators (such as CPU usage, memory usage) in real time. The distributed stream processing framework Apache Flink is used to process and analyze these real-time data streams.

[0070] Based on preset thresholds and rules, anomaly detection algorithms (such as Isolation Forest) are used to automatically identify potential abnormal links. In order to construct the fault propagation graph, a fault propagation model based on a graph algorithm is implemented, which takes into account the dependencies between nodes and the probability of fault propagation.

[0071] When calculating PageRank scores, we used the distributed graph computing framework Apache Giraph to improve the efficiency of large-scale graph data processing. By adjusting the damping factor d and PageRank threshold, we optimized the accuracy of identifying abnormal links.

[0072] In addition, a time series-based anomaly severity assessment model is introduced, which not only considers the PageRank score and anomaly status of the node, but also incorporates historical anomaly data, thereby providing a more accurate anomaly severity assessment.

[0073] Based on the results of the first two steps, a global data map was constructed. The open source data lineage analysis tool Apache Atlas was used, combined with a self-developed code static analysis tool, to deeply analyze the code structure and data processing logic of each component.

[0074] In order to form an accurate link blood relationship map, a visualization tool based on a graph database was developed, which can dynamically display data flow paths, component dependencies, and abnormal propagation trajectories.

[0075] During the log aggregation and analysis phase, an intelligent log correlation analysis engine was developed that can automatically establish correlations between logs from different nodes and time periods, thereby forming a complete abnormal log chain.

[0076] Finally, based on the historical operation data of the link, a composite time series prediction model was constructed, which can more accurately predict link performance and potential risks. When identifying link bottlenecks and high-risk areas, not only traditional performance indicators are considered, but also multi-dimensional factors such as data quality and business importance are incorporated. Some experimental data are as follows: Table 1 Part of experimental data ; Through in-depth analysis of the above table data, it can be seen that the delay distribution of the entire link is relatively uniform, and the delay of most nodes is controlled within 10ms, which shows that the link optimization method proposed in the present invention can effectively balance the load of each node and avoid serious delay hot spots. It is particularly noteworthy that although the delay of computing node D is relatively high (12ms), considering its complex computing task, this delay level is still within an acceptable range. This fine delay control benefits from the dynamic link transmission strategy adjustment mechanism proposed in step S5 of the present invention.

[0077] In terms of throughput, each node maintains a high level, and the throughput of most nodes exceeds 850Mbps. This result fully demonstrates the effectiveness of the load balancing strategy proposed in step S5 of the present invention. In particular, the throughput of data source A and target node F reaches 950Mbps and 930Mbps respectively, which not only ensures the high efficiency of data input and output, but also reflects the excellent performance of the present invention in end-to-end performance optimization.

[0078] In addition, the number of abnormal occurrences is one of the key indicators for evaluating link reliability. It can be seen from the data that the number of abnormal occurrences of most nodes is low, which fully proves the effectiveness of the abnormal identification and processing mechanism proposed in steps S2 and S4 of the present invention, especially that no abnormality occurs at the data source A and the target node F, which ensures the stability of the starting point and end point of data transmission.

[0079] Although the number of exceptions on computing node D was relatively high (4 times), it was still within a controllable range considering its complex computing tasks and high resource utilization. More importantly, these exceptions were captured and processed in a timely manner and did not lead to overall system failure.

[0080] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A link reliability tracking and optimization method combined with real-time monitoring, characterized in that: include, Based on the component information and data flow relationship of the data center, a full-link topology diagram is constructed, and data collection probes are deployed at key nodes to collect link data in real time. Analyze the real-time collected link data and automatically identify potential faults or abnormal links; Based on the data map, sort out the data dependency relationships of the upstream and downstream links, record the relevance and dependency paths of each component, and form an accurate link blood relationship map; Capture and trace link anomalies, aggregate log information, and conduct centralized analysis and problem location. Based on the link's historical operation data and the timing prediction model, bottlenecks and high-risk areas in the link are identified, and the link's transmission strategy is dynamically adjusted to optimize the load balancing and link priority of the data center in real time.

2. The link reliability tracking and optimization method combined with real-time monitoring as claimed in claim 1, characterized in that: The construction of the full-link topology diagram includes: Based on the data center, extract component information and identify data transmission links between components : When data is received from a component Transfer to another component , if there are M other components in the middle, and , then mark the intermediate transmission component as the transit layer; if , then the record is a direct transmission relationship, where M is a non-negative integer; Each component is represented as a node, and the data transmission relationship between components is represented by edges. For indirect transmission, the transit layer is added to the graph as a virtual node to construct a full-link topology graph.

3. The link reliability tracking and optimization method combined with real-time monitoring as claimed in claim 2, characterized in that: The automatic identification of potential faults or abnormal links includes: Constructing a fault propagation graph , using the PageRank algorithm to calculate the fault propagation graph The importance score of each key node in ; Identify abnormal links based on importance scores and abnormal status, including: The key abnormal node set is defined as: ; in, is the PageRank threshold; is the key node; is the abnormal state of the key node. It is abnormal if It is normal; The abnormal link set is expressed as: ; in, For data transmission link.

4. The link reliability tracking and optimization method combined with real-time monitoring as claimed in claim 3, characterized in that: The PageRank algorithm is used to calculate the fault propagation graph The importance score of each key node in The calculation formula is: ； Where d is the damping factor, Represents the probability of jumping to any other node; For key nodes The PageRank score, PageRank score for common nodes, used to calculate key nodes The influence of its adjacent nodes needs to be considered when scoring; For ordinary nodes and key nodes The weight of the edge between them; For ordinary nodes The sum of the edge weights to all its neighboring nodes.

5. The link reliability tracking and optimization method combined with real-time monitoring as claimed in claim 4, characterized in that: The step of combing the data dependency relationship between upstream and downstream links and recording the relevance and dependency path of each component includes the following steps: For each link, determine the dependency relationship between its upstream and downstream nodes, and gradually sort out the dependency path of each data processing link in the link by analyzing the transmission order and direction of the data flow; Among them, the upstream node represents the source of data, and the downstream node represents the consumer of data; The computing nodes in the link parse their codes through automated code parsing tools to identify input dependencies and outputs, and map them back to data transmission in the link; Based on the above data dependencies and code analysis, clarify the association between each component. For each component, record the upstream components it depends on and the data types it depends on. Integrate the sorted data dependencies, code analysis results and component associations to build a complete link blood relationship map.

6. The link reliability tracking and optimization method combined with real-time monitoring as claimed in claim 5, characterized in that: The log information is aggregated and analyzed centrally to locate the problem, including: Unify the log snapshots of different links and different time periods to generate a complete abnormal log chain; classify and label the logs, and archive them by time, node, and abnormal type; Use the exception analysis system to trace back and centrally analyze logs: For a single exception, analyze the system status when it is triggered, find the log snapshots before and after the exception, and determine the root cause of the problem; For multiple anomalies in the link, by analyzing the time series information of the log link, we can trace back and analyze the impact of each abnormal node and the path of anomaly propagation.

7. The link reliability tracking and optimization method combined with real-time monitoring as claimed in claim 6, characterized in that: The transmission strategy of dynamically adjusting the link includes: According to the priority allocation formula, each key node in the link To prioritize: ； in, For key nodes Priority; For key nodes The delay indicates the transmission delay of data at this key node; For key nodes The throughput, which indicates the transmission capacity of key nodes; For key nodes Utilization rate; For key nodes Memory usage.

8. The link reliability tracking and optimization method combined with real-time monitoring as claimed in claim 7, characterized in that: The transmission strategy of dynamically adjusting the link also includes: To prevent bottleneck nodes from being overloaded, data traffic is evenly distributed to different links according to the load balancing strategy. The link load is calculated as follows: ； in, For key nodes The current throughput of For key nodes Maximum throughput; If the load on a link exceeds a preset threshold , then dynamically allocate its transmission traffic to the link with low load and adjust the load balancing strategy of the link.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the link reliability tracking and optimization method combined with real-time monitoring described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the link reliability tracking and optimization method combined with real-time monitoring described in any one of claims 1 to 8 are implemented.

Images