Business level network visualized monitoring method for failure prediction and health management

Abstract

The application relates to the technical field of fault prediction and health management, and particularly discloses a business-level network visual monitoring method for fault prediction and health management, which restores the round trip process of a business session through a bypass probe, splits network access time consumption into three parts of client transmission, server transmission and application processing, and identifies an abnormal delay position by using a historical baseline. For an abnormal session, instead of directly taking the CPU rise collected during diagnosis as fault evidence, a diagnosis disturbance instruction with calibrated resource consumption is first sent, and then the processor utilization deviation amplitude and recovery time length of the target device are observed to judge the pressure margin. Meanwhile, the system determines the fault direction in combination with indexes such as retransmission, zero window, packet loss and same-type business delay, outputs a unique root cause and confidence, and is visually displayed in a business topology.

Description

Technical Field

[0001] This invention relates to the field of fault prediction and health management technology, and more specifically to a business-level network visualization monitoring method for fault prediction and health management. Background Technology

[0002] During the operation of a business-level network, increased session latency can be caused by client-side, network link, server-side processing, and application interaction processes. Existing network monitoring systems typically obtain packet timestamps, device operating status, and application performance metrics through bypass probes, log collection, or metric retrieval, and then use these metrics to determine the root cause of faults.

[0003] However, during fault diagnosis, in order to verify a fault hypothesis, the system often temporarily increases the frequency of collecting indicators from the target device or sends additional probe requests to the target device. This diagnostic behavior itself consumes the processing resources of the target device. When the target device is already in a high-load critical state, the resource consumption generated by the diagnostic behavior may further increase the processor utilization, causing the operating status indicators subsequently collected by the system to simultaneously include the original fault factors and the disturbance factors caused by the diagnostic behavior itself. This leads to the root cause reasoning process using the abnormal data generated by its own diagnostic actions as evidence to confirm the fault, thus forming a self-reinforcing diagnostic loop and affecting the objectivity and confidence accuracy of the root cause judgment. Summary of the Invention

[0004] The purpose of this invention is to provide a business-level network visualization monitoring method for fault prediction and health management, thereby solving the aforementioned technical problems.

[0005] The objective of this invention can be achieved through the following technical solutions: A business-grade network visualization monitoring method for fault prediction and health management includes the following steps: S1: Deploy probes in a bypass mode in the network to be monitored, collect each data packet flowing through the network to be monitored, and record the timestamp of each data packet passing through the probe; S2: Calculate the latency segment of the session based on the timestamps of each data packet in the same session, and mark the latency segment that deviates from the historical baseline as an abnormal segment. The latency segment includes client latency, service latency and application latency. S3: For abnormal segments, send a diagnostic disturbance command with known resource consumption characteristics to the target device corresponding to the session; S4: Collect the operating status indicators of the target device after receiving the diagnostic disturbance command, and determine the disturbance response characteristics of the target device to the diagnostic disturbance command based on the operating status indicators; S5: Based on the disturbance response characteristics, determine the root cause from the preset set of anomaly-cause mapping relationships, and visualize the root cause conclusions with confidence levels.

[0006] As a further aspect of the present invention: the process of calculating the time delay segment is as follows: Deploy probes on both the client and server sides of the network to be monitored; Data packets from the same session are divided into a first set and a second set according to their transmission direction. The transmission directions of data packets in the first set and the second set are from the client to the server and from the server to the client, respectively. Select the data packet with the smallest timestamp recorded by the probe on the client side from the first set, and record the timestamp of the data packet recorded by the probe on the client side as A1, and record the timestamp of the data packet recorded by the probe on the server side as A2. The difference between A2 and A1 is taken as the client latency. Select the data packet with the largest timestamp recorded by the probe on the server side from the second set, and record the timestamp of the data packet recorded by the probe on the server side as A3, and record the timestamp of the data packet recorded by the probe on the client side as A4. The difference between A4 and A3 is used as the service latency. Subtracting the customer latency and service latency from the difference between A4 and A1 gives the application latency.

[0007] As a further aspect of the present invention: the process of determining the abnormal segment is as follows: Obtain historical values ​​of a single latency segment for the same IP within a preset time period, sort the obtained historical values ​​in ascending order according to their numerical values, and use the values ​​at the preset quantiles in the sort as the historical baseline for that type of latency segment. The current latency segment is compared with the corresponding historical baseline, and the latency segment that is greater than the historical baseline is marked as an abnormal segment.

[0008] As a further aspect of the present invention: the process of sending a diagnostic disturbance command is as follows: The target device is determined based on the type of the abnormal segment. When the abnormal segment is client latency, the target device is the client; when the abnormal segment is service latency, the target device is the server; and when the abnormal segment is application latency, the target device is the server. When the network under monitoring is running normally, a diagnostic disturbance command sample is sent to the target device. The average processor utilization of the target device before receiving the diagnostic disturbance command sample and the peak processor utilization after receiving the diagnostic disturbance command sample are collected. The resource consumption calibration value is determined based on the difference between the peak processor utilization and the average processor utilization. The diagnostic disturbance command sample carrying the resource consumption calibration value is used as the diagnostic disturbance command. The diagnostic disturbance command is consistent with the normal service request in terms of protocol format. Before sending the diagnostic disturbance command, a self-test status query command is sent to the target device, and the self-test status declaration returned by the target device is received. The self-test status declaration is generated by the target device when it determines that its own processor utilization is lower than the self-test threshold. When the self-test status declaration is received within the waiting time, the diagnostic disturbance command is sent to the target device.

[0009] As a further aspect of the present invention: the process of collecting disturbance response characteristics is as follows: The processor utilization rate of the target device is collected as an indicator of its operating status. Taking the time of sending the diagnostic disturbance command as the zero point, the processor utilization rate in the first time period before the diagnostic disturbance command is sent is recorded as the baseline time period data, and the processor utilization rate in the second time period after the diagnostic disturbance command is sent is recorded as the response time period data. Calculate the average processor utilization rate in the baseline time period data, and use it as the baseline average. Extract the maximum value of processor utilization from the response time period data as the response peak; use the difference between the response peak and the baseline mean as the deviation. Extract the moment when processor utilization first drops from the peak response time to the baseline average from the response time data, and use the difference between this moment and the sending time as the recovery time; The deviation magnitude and recovery time are used as characteristics of the disturbance response.

[0010] As a further aspect of the present invention, the process of outputting root cause conclusions is as follows: Based on the type of abnormal segment, collect the associated metrics corresponding to the abnormal segment: When the abnormal segment is customer latency, the associated metrics include terminal retransmission rate and TCP zero window count. When the abnormal segment is service latency, the associated metrics include server TCP timestamp difference and packet loss rate. When the abnormal segment is application latency, the associated metrics include application layer protocol type and application latency of other sessions of the same type of business during the same period. The pressure margin assessment result of the target equipment in response to the diagnostic disturbance command is determined based on the disturbance response characteristics. Determine the direction of the fault based on relevant indicators; When the pressure margin determination result is from the same source as the fault direction, the confidence level of the corresponding root cause is increased; when the pressure margin determination result is from a different source than the fault direction, the confidence level of the corresponding root cause is decreased.

[0011] As a further aspect of the present invention: the process of establishing the anomaly-cause mapping relationship set is as follows: Define the exception segment types, which include customer latency exceptions, service latency exceptions, and application latency exceptions. Terminal retransmission rate and TCP zero window count are used as correlation indicators for abnormal customer latency. The difference between server-side TCP timestamps and packet loss rate are used as indicators related to service latency anomalies. The application latency of application layer protocol type and other sessions of the same type of business during the same period is used as the correlation indicator for application latency anomalies; Store the abnormal segment types and associated indicators as a set of abnormal-cause mapping relationships.

[0012] As a further aspect of the present invention, the process of outputting root cause conclusions further includes: Based on the type of abnormal segment, the corresponding associated indicators are read from the abnormal-cause mapping relationship set, the current value of each associated indicator is collected, and the fault direction is determined based on the current value. When the recovery time exceeds the historical baseline of the recovery time and the deviation is greater than the upper limit of the allowable fluctuation range corresponding to the resource consumption calibration value, it is determined that the target equipment has insufficient pressure margin to withstand the diagnostic disturbance command. When the recovery time does not exceed the historical baseline of the recovery time, or the deviation does not exceed the upper limit of the allowable fluctuation range corresponding to the resource consumption calibration value, the target equipment is judged to have normal pressure margin to the diagnostic disturbance command; When the pressure margin determination result is insufficient and the fault direction points to the abnormality of the target equipment side, the pressure margin determination result and the fault direction are from the same source. Otherwise, the pressure margin determination result is determined to be from a different source than the fault direction; When determining the same source, the cause corresponding to the fault direction is taken as the root cause, and the initial confidence score is added to the same source weight to obtain the confidence score before outputting it; When different sources are identified, the cause corresponding to the fault direction is taken as the root cause, and the initial confidence score is added to the weight of different sources to obtain the confidence score before outputting it. The initial confidence level, same-source weight, and different-source weight are all preset.

[0013] The beneficial effects of this invention compared to the prior art are as follows: This invention deploys probes on both the client and server sides to segment and calculate client latency, service latency, and application latency for the same service session. It then identifies anomalous segments by combining these with historical baselines, enabling the system to first determine the location of the service link where the anomalous latency occurs. Simultaneously, this invention no longer directly uses processor utilization changes collected during the diagnostic process as root cause evidence. Instead, it sends diagnostic disturbance commands with known resource consumption characteristics to the target device. The disturbance response characteristics are constructed using the baseline average before the disturbance, the peak response after the disturbance, and the recovery time, thereby determining the target device's tolerance margin for controlled diagnostic actions. Furthermore, this invention determines the fault direction based on the correlation indicators corresponding to the anomalous segments and adjusts the confidence level of the unique root cause based on the tolerance margin determination results. This ensures that resource disturbances caused by the diagnostic actions themselves are included in the analysis process, rather than being mistakenly identified as original fault evidence. This reduces the contamination of root cause conclusions by diagnostic actions and improves the credibility and interpretability of service-level network fault location results. Attached Figure Description

[0014] The invention will now be further described with reference to the accompanying drawings.

[0015] Figure 1 This is a flowchart illustrating the business-level network visualization monitoring method for fault prediction and health management according to the present invention. Figure 2 This is a flowchart illustrating the process of outputting root cause conclusions in this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figures 1-2 As shown, this invention provides a business-level network visualization monitoring method for fault prediction and health management, comprising the following steps: S1: Deploy probes in a bypass mode in the network to be monitored, collect each data packet flowing through the network to be monitored, and record the timestamp of each data packet passing through the probe; S2: Calculate the latency segment of the session based on the timestamps of each data packet in the same session, and mark the latency segment that deviates from the historical baseline as an abnormal segment. The latency segment includes client latency, service latency and application latency. In a preferred embodiment of the present invention, the process of calculating the time delay segment is as follows: Deploy probes on both the client and server sides of the network to be monitored; Data packets from the same session are divided into a first set and a second set according to their transmission direction. The transmission directions of data packets in the first set and the second set are from the client to the server and from the server to the client, respectively. Select the data packet with the smallest timestamp recorded by the probe on the client side from the first set, and record the timestamp of the data packet recorded by the probe on the client side as A1, and record the timestamp of the data packet recorded by the probe on the server side as A2. The difference between A2 and A1 is taken as the client latency. Select the data packet with the largest timestamp recorded by the probe on the server side from the second set, and record the timestamp of the data packet recorded by the probe on the server side as A3, and record the timestamp of the data packet recorded by the probe on the client side as A4. The difference between A4 and A3 is used as the service latency. Subtracting the customer latency and service latency from the difference between A4 and A1 gives the application latency.

[0018] It should be noted that when identifying data packets within the same session, the source IP address, destination IP address, source port, destination port, and transport layer protocol type are extracted from each data packet. Data packets with identical information and corresponding transmission directions are grouped into the same candidate session. For TCP connections, the start and end range of the session is determined based on the connection establishment message, connection release message, and the order of data packets during the connection's duration. If no data packets belonging to the same candidate session are collected within a preset waiting time, the data packets collected before the waiting time expires are identified as data packets within the same session. For cases where application layer protocols carry transaction identifiers, request identifiers, or message sequence identifiers, these identifiers are extracted further within the data packet range of the same session. Data packets with identical identifiers and belonging to the same request-response process are identified as the set of data packets corresponding to the same business transaction, and the client latency, service latency, and application latency are calculated primarily based on the set of data packets corresponding to the same business transaction.

[0019] Before calculating client latency, service latency, and application latency, the timestamps recorded by the client-side probe and the server-side probe are processed to unify the time base. The client-side probe and the server-side probe are connected to the same clock source, and the local acquisition time is corrected according to the time information output by the same clock source. When two probes cannot be directly connected to the same clock source, a synchronization calibration message is sent between the client-side probe and the server-side probe. The reception time of the synchronization calibration message at the two probes is recorded. The probe time deviation is obtained based on the time difference recorded by the two probes for the same synchronization calibration message. Before calculating the differences between A2 and A1, A4 and A3, and A4 and A1, the timestamp recorded by one of the probes is corrected according to the probe time deviation so that A1, A2, A3, and A4 are under the same time base.

[0020] The same session used to calculate client latency, service latency, and application latency refers to the set of data packets that can correspond to a single request-response process. When the underlying connection is a short connection, the data packets in the short connection are used as the data packets of the same session. When the underlying connection is a long connection, not all data packets in the entire long connection are used as a latency calculation object. Instead, the set of data packets corresponding to the same business transaction is divided within the long connection according to the application layer transaction identifier, request identifier, message sequence identifier, or request-response boundary, and A1, A2, A3, and A4 are selected based on the set of data packets corresponding to the same business transaction.

[0021] When determining the timestamp of the same data packet at the client-side probe and the server-side probe, the source IP address, destination IP address, source port, destination port, transport layer protocol type, transmission direction, message length, and transport layer sequence number are extracted from the data packet records collected by the two probes. For data packets carrying payloads, a digest is also calculated to obtain the payload digest value. When the source IP address, destination IP address, source port, destination port, transport layer protocol type, transmission direction, message length, and transport layer sequence number in the two data packet records are consistent, and the payload digest values ​​are consistent or there is no payload to compare, it is determined that the two data packet records correspond to the same data packet, and the timestamp of the data packet recorded at the client-side probe and the server-side probe are read respectively.

[0022] In a preferred embodiment, the process of determining the abnormal segment is as follows: Obtain historical values ​​of a single latency segment for the same IP within a preset time period, sort the obtained historical values ​​in ascending order according to their numerical values, and use the values ​​at the preset quantiles in the sort as the historical baseline for that type of latency segment. The current latency segment is compared with the corresponding historical baseline, and the latency segment that is greater than the historical baseline is marked as an abnormal segment.

[0023] When establishing historical baselines for latency segments, different IP statistical objects are used for different types of latency segments. Client latency reflects the request transmission process from the client side to the server side, and the client IP is used as the statistical object for the client latency historical baseline. Service latency reflects the response transmission process from the server side to the client side, and the server IP is used as the statistical object for the service latency historical baseline. Application latency reflects the processing process from when the server receives a request to when it sends a response, and the server IP and service type are used as the statistical objects for the application latency historical baseline. When obtaining the historical value of the same IP in a single latency segment within a preset time period, the corresponding historical value is read according to the above statistical objects.

[0024] The remaining historical baselines in this invention were also obtained using the above process.

[0025] S3: For abnormal segments, send a diagnostic disturbance command with known resource consumption characteristics to the target device corresponding to the session; In another preferred embodiment of the present invention, the process of sending a diagnostic disturbance command is as follows: The target device is determined based on the type of the abnormal segment. When the abnormal segment is client latency, the target device is the client; when the abnormal segment is service latency, the target device is the server; and when the abnormal segment is application latency, the target device is the server. When the network under monitoring is running normally, a diagnostic disturbance command sample is sent to the target device. The average processor utilization of the target device before receiving the diagnostic disturbance command sample and the peak processor utilization after receiving the diagnostic disturbance command sample are collected. The resource consumption calibration value is determined based on the difference between the peak processor utilization and the average processor utilization. The diagnostic disturbance command sample carrying the resource consumption calibration value is used as the diagnostic disturbance command. The diagnostic disturbance command is consistent with the normal service request in terms of protocol format. Before sending the diagnostic disturbance command, a self-test status query command is sent to the target device, and the self-test status declaration returned by the target device is received. The self-test status declaration is generated by the target device when it determines that its own processor utilization is lower than the self-test threshold. When the self-test status declaration is received within the waiting time, the diagnostic disturbance command is sent to the target device.

[0026] If no self-test status declaration is received from the target device within the waiting time, no diagnostic disturbance command is sent to the target device, and the pressure margin judgment result corresponding to the abnormal segment is marked as undetermined. The system continues to read related indicators based on the abnormal segment type, collects the current values ​​of each related indicator, and determines the fault direction based on the comparison between the current values ​​of the related indicators and the historical baseline. The root cause is determined according to the one-to-one correspondence between the fault direction and the cause, and the initial confidence level corresponding to the root cause is read and output as the confidence level of the root cause conclusion. At this time, since no diagnostic disturbance command is executed, the initial confidence level is not adjusted using same-source weights or different-source weights.

[0027] S4: Collect the operating status indicators of the target device after receiving the diagnostic disturbance command, and determine the disturbance response characteristics of the target device to the diagnostic disturbance command based on the operating status indicators; In a preferred embodiment of the present invention, the process of collecting disturbance response characteristics is as follows: The processor utilization rate of the target device is collected as an indicator of its operating status. Taking the time of sending the diagnostic disturbance command as the zero point, the processor utilization rate in the first time period before the diagnostic disturbance command is sent is recorded as the baseline time period data, and the processor utilization rate in the second time period after the diagnostic disturbance command is sent is recorded as the response time period data. Calculate the average processor utilization rate in the baseline time period data, and use it as the baseline average. Extract the maximum value of processor utilization from the response time period data as the response peak; use the difference between the response peak and the baseline mean as the deviation. Extract the moment when processor utilization first drops from the peak response time to the baseline average from the response time data, and use the difference between this moment and the sending time as the recovery time; The deviation magnitude and recovery time are used as characteristics of the disturbance response.

[0028] When calculating the recovery time, the moment when the processor utilization first drops from the peak response time to ≤ the baseline average is taken as the recovery time. If the processor utilization does not drop to ≤ the baseline average within the response period, the end of the response period is taken as the recovery time, and the difference between the end of the response period and the time when the diagnostic disturbance command is sent is taken as the recovery time. In this way, even if the target device does not complete the recovery within the response period, the recovery time used for pressure margin determination can still be obtained.

[0029] S5: Based on the disturbance response characteristics, determine the root cause from the preset set of anomaly-cause mapping relationships, and visualize the root cause conclusions with confidence levels.

[0030] In a preferred embodiment of the present invention, the process of outputting root cause conclusions is as follows: Based on the type of abnormal segment, collect the associated metrics corresponding to the abnormal segment: When the abnormal segment is customer latency, the associated metrics include terminal retransmission rate and TCP zero window count. When the abnormal segment is service latency, the associated metrics include server TCP timestamp difference and packet loss rate. When the abnormal segment is application latency, the associated metrics include application layer protocol type and application latency of other sessions of the same type of business during the same period. The pressure margin assessment result of the target equipment in response to the diagnostic disturbance command is determined based on the disturbance response characteristics. Determine the direction of the fault based on relevant indicators; When the pressure margin determination result is from the same source as the fault direction, the confidence level of the corresponding root cause is increased; when the pressure margin determination result is from a different source than the fault direction, the confidence level of the corresponding root cause is decreased.

[0031] In a preferred embodiment, the process of establishing the anomaly-cause mapping set is as follows: Define the exception segment types, which include customer latency exceptions, service latency exceptions, and application latency exceptions. Terminal retransmission rate and TCP zero window count are used as correlation indicators for abnormal customer latency. The difference between server-side TCP timestamps and packet loss rate are used as indicators related to service latency anomalies. The application latency of application layer protocol type and other sessions of the same type of business during the same period is used as the correlation indicator for application latency anomalies; Store the abnormal segment types and associated indicators as a set of abnormal-cause mapping relationships.

[0032] The anomaly-cause mapping set stores the relationships between anomaly segment types, associated indicators, associated indicator judgment rules, fault directions, causes corresponding to fault directions, initial confidence levels, same-source weights, and different-source weights. When establishing the anomaly-cause mapping set, customer latency anomalies, service latency anomalies, and application latency anomalies are first defined. Then, corresponding associated indicators are configured for each anomaly segment type, and comparison rules between the current value and the historical baseline are configured for each associated indicator. The fault directions corresponding to different associated indicator states are determined based on the comparison rules, and a unique cause is configured for each fault direction. Furthermore, the initial confidence level, same-source weight, and different-source weight corresponding to the cause are associated and stored with that cause, so that after reading the anomaly segment type, the associated indicators, fault directions, causes, and parameters used to calculate the confidence level can be obtained sequentially.

[0033] In another preferred embodiment, the process of outputting root cause conclusions further includes: Based on the type of abnormal segment, the corresponding associated indicators are read from the abnormal-cause mapping relationship set, the current value of each associated indicator is collected, and the fault direction is determined based on the current value. When the recovery time exceeds the historical baseline of the recovery time and the deviation is greater than the upper limit of the allowable fluctuation range corresponding to the resource consumption calibration value, it is determined that the target equipment has insufficient pressure margin to withstand the diagnostic disturbance command. When the recovery time does not exceed the historical baseline of the recovery time, or the deviation does not exceed the upper limit of the allowable fluctuation range corresponding to the resource consumption calibration value, the target equipment is judged to have normal pressure margin to the diagnostic disturbance command; When the pressure margin determination result is insufficient and the fault direction points to the abnormality of the target equipment side, the pressure margin determination result and the fault direction are from the same source. Otherwise, the pressure margin determination result is determined to be from a different source than the fault direction; When determining the same source, the cause corresponding to the fault direction is taken as the root cause, and the initial confidence score is added to the same source weight to obtain the confidence score before outputting it; When different sources are identified, the cause corresponding to the fault direction is taken as the root cause, and the initial confidence score is added to the weight of different sources to obtain the confidence score before outputting it. The initial confidence level, same-source weight, and different-source weight are all preset.

[0034] When calculating the final confidence level, the initial confidence level corresponding to the root cause is first read, and the same-source weight or different-source weight is selected according to whether the pressure margin determination result and the fault direction are from the same source. When the same source is determined, the initial confidence level and the same-source weight are added to obtain the candidate confidence level. When different sources are determined, the initial confidence level and the different-source weight are added to obtain the candidate confidence level. If the candidate confidence level is greater than the preset confidence level upper limit, the preset confidence level upper limit is used as the final confidence level. If the candidate confidence level is less than the preset confidence level lower limit, the preset confidence level lower limit is used as the final confidence level. If the candidate confidence level is between the preset confidence level lower limit and the preset confidence level upper limit, the candidate confidence level is used as the final confidence level.

[0035] It should be noted that when determining the historical baseline of recovery time, for the same target device and the same diagnostic disturbance command, within a historical period when the target device is in normal operation and meets the self-test status declaration conditions, diagnostic disturbance command samples are sent multiple times, and the processor utilization change process of the target device is recorded after each transmission of the diagnostic disturbance command sample. For each historical transmission process, the transmission time of the diagnostic disturbance command sample is taken as the zero point of time, the average processor utilization within the calibration reference time before transmission is calculated, and this average processor utilization is used as the historical reference average for this historical transmission process. The peak processor utilization within the calibration response time after transmission is extracted, and the moment when the processor utilization first ≤ the historical reference average is found from the processor utilization sequence after transmission. The difference between this moment and the transmission time of the diagnostic disturbance command sample is used as the historical value of a single recovery time. If the processor utilization within the calibration response time after transmission is always greater than the historical reference average, then the calibration response time is used as the historical value of a single recovery time for this historical transmission process.

[0036] It should be noted that when determining the direction of the fault, the current values ​​of each associated indicator corresponding to the abnormal segment are first obtained, and the historical baseline formed by each associated indicator during the normal operation period is read. The current value of the associated indicator is then compared with its historical baseline. Among them, the terminal retransmission rate refers to the proportion of retransmission of the business data packets sent by the client in the corresponding session, the TCP zero window count refers to the number of times the client announces to the other end that the receive window is zero in the corresponding session, and the server TCP timestamp difference refers to the difference between the timestamp of the request data packet recorded by the server probe arriving at the server side and the timestamp of the server response data packet leaving the server side in the same business transaction.

[0037] When it is necessary to compare the deviation of two related indicators from their respective historical baselines, the deviation value of each related indicator is calculated separately. If the historical baseline of the related indicator is not zero, the difference between the current value of the related indicator and the historical baseline of the related indicator is divided by the historical baseline of the related indicator to obtain the deviation value of the related indicator. If the historical baseline of the related indicator is zero, the current value of the related indicator is used as the deviation value of the related indicator.

[0038] When the abnormal segment is a client latency abnormality, if the current value of the terminal retransmission rate exceeds the historical baseline of the terminal retransmission rate and the current value of the TCP zero window count does not exceed the historical baseline of the TCP zero window count, then the fault direction is determined to be a client link transmission abnormality. If the current value of the TCP zero window count exceeds the historical baseline of the TCP zero window count and the current value of the terminal retransmission rate does not exceed the historical baseline of the terminal retransmission rate, then the fault direction is determined to be a client reception and processing abnormality. If both the current value of the terminal retransmission rate and the current value of the TCP zero window count exceed their respective historical baselines, then the deviation values ​​of the two are calculated separately, and the direction corresponding to the correlation indicator with the larger deviation value is taken as the fault direction. If the deviation values ​​of the two are the same, then the fault direction is determined to be a client reception and processing abnormality.

[0039] When the abnormal segment is a service latency abnormality, if the current value of the server's TCP timestamp difference exceeds the historical baseline of the server's TCP timestamp difference and the current value of the packet loss rate does not exceed the historical baseline of the packet loss rate, then the fault direction is determined to be a server-side processing abnormality. If the current value of the packet loss rate exceeds the historical baseline of the packet loss rate and the current value of the server's TCP timestamp difference does not exceed the historical baseline of the server's TCP timestamp difference, then the fault direction is determined to be a network path transmission abnormality. If both the current value of the server's TCP timestamp difference and the current value of the packet loss rate exceed their respective historical baselines, then the deviation values ​​of the two are calculated separately, and the direction corresponding to the correlation indicator with the larger deviation value is taken as the fault direction. If the deviation values ​​of the two are the same, then the fault direction is determined to be a server-side processing abnormality.

[0040] When the abnormal segment is an application latency abnormality, the service type to which the current session belongs is determined according to the application layer protocol type. Then, the application latency of other sessions of the same type of service during the same period is read. If the application latency of other sessions of the same type of service during the same period exceeds its historical baseline, the fault direction is determined to be an application processing abnormality on the server side. If the application latency of other sessions of the same type of service during the same period does not exceed its historical baseline, or there are no other sessions of the same type of service, the fault direction is determined to be an application interaction abnormality in the current session.

[0041] When collecting application latency data for other sessions of the same type of service during the same period, first read the service type of the current session and the session time window corresponding to the abnormal segment. Then, search for other sessions in the network to be monitored that have the same service type, time windows that overlap with the current session's time window, and different session identifiers. Calculate the application latency of these other sessions respectively, and sort the calculated application latency data in ascending order of value. Use the median of the sorted data as the current value of the application latency for other sessions of the same type of service during the same period. If no other sessions that meet the above conditions are found, it is determined that there are no other sessions of the same type of service.

[0042] When determining the origin of the problem, the target device already identified before sending the diagnostic disturbance command is used, and the fault direction determined based on the correlation indicators is read: when the abnormal segment is a client latency abnormality and the fault direction is a client reception and processing abnormality, the fault direction is determined to point to the processing abnormality on the target device side; when the abnormal segment is a service latency abnormality and the fault direction is a server processing abnormality, the fault direction is determined to point to the processing abnormality on the target device side; when the abnormal segment is an application latency abnormality and the fault direction is a server application processing abnormality, the fault direction is determined to point to the processing abnormality on the target device side; when the fault direction is a client link transmission abnormality, network path transmission abnormality, or current session application interaction abnormality, the fault direction is determined to not point to the processing abnormality on the target device side.

[0043] When collecting the current values ​​of related indicators, the session time window corresponding to the abnormal segment is used as the statistical window. The session time window corresponding to the abnormal segment is determined by the start data packet timestamp and end data packet timestamp of the session or business transaction to which the abnormal segment belongs. For the terminal retransmission rate, the number of business data packets sent by the client within the statistical window and the number of data packets judged as retransmissions are counted. The ratio of the number of retransmitted data packets to the number of business data packets sent by the client is used as the current value of the terminal retransmission rate. For the TCP zero window count, the number of messages from the client announcing that the receive window is zero to the peer within the statistical window is counted. This number is used as the current value of the TCP zero window count. For the packet loss rate, the number of data packets lost due to missing sequence numbers, duplicate acknowledgments, or retransmission events within the statistical window is counted. The ratio of the number of lost data packets to the number of data packets that should have been transmitted within the statistical window is used as the current value of the packet loss rate.

[0044] When determining the business type of the current session based on the application layer protocol type, a mapping table between application layer protocol types and business types is pre-established, and the application layer protocol name, application layer port, request path, interface identifier, database operation type, or remote call method name are used as matching fields in the mapping table. The application layer protocol type is parsed from the data packets of the current session, and the corresponding matching fields are extracted. The extracted matching fields are compared item by item with the fields in the mapping table. When a matching record exists, the business type corresponding to the matching record is determined as the business type of the current session. When no matching record exists, the current session is classified into an unclassified business type, and subsequent judgments are made according to the historical baseline and similar session set corresponding to the unclassified business type.

[0045] When establishing a pre-defined set of anomaly-cause mapping relationships, historical fault samples that have completed manual confirmation or repair loop closure are read, and initial confidence, same-source weight, and different-source weight are calculated for the cause corresponding to each fault direction: For any cause, the number of samples in the historical fault samples that are determined to be the direction of the fault by the corresponding correlation index is recorded as the number of triggering samples, the number of samples in the triggering samples that are finally confirmed as the root cause of the cause is recorded as the number of hit samples, and the ratio of the number of hit samples to the number of triggering samples is used as the initial confidence level of the cause. For each triggered sample, the resource consumption calibration value corresponding to the diagnostic disturbance command in the triggered sample is read, and the corresponding allowable fluctuation range is determined according to the resource consumption calibration value. If the recovery time in the triggered sample exceeds the historical baseline of the recovery time and the deviation exceeds the upper limit of the allowable fluctuation range corresponding to the resource consumption calibration value, the pressure margin judgment result of the triggered sample is determined to be insufficient pressure margin; otherwise, the pressure margin judgment result of the triggered sample is determined to be normal pressure margin.

[0046] In the triggered samples, samples whose pressure margin determination results are from the same source as the fault direction are selected. The initial confidence level is subtracted from the ratio of the number of samples whose root cause is finally confirmed to be this cause to the number of samples from the same source to obtain the same source weight. In the triggered samples, samples whose pressure margin determination results are different from the fault direction are screened. The initial confidence level is subtracted from the ratio of the number of samples whose root cause is finally confirmed to be this cause to the number of samples with different sources, and the weights of different sources are obtained.

[0047] Same-source weights are used to increase the initial confidence level. Same-source weights are non-negative values. When the same-source weight calculated based on historical fault samples is less than zero, the same-source weight is reset to zero. Different-source weights are used to decrease the initial confidence level. Different-source weights are non-positive values. When the different-source weight calculated based on historical fault samples is greater than zero, the different-source weight is reset to zero. When the number of historical fault samples corresponding to a certain cause is insufficient to calculate the above ratio, the maintenance personnel enter the initial confidence level, same-source weight, and different-source weight according to the frequency of occurrence and confirmation results of the cause in the historical maintenance records. The entered parameters are stored together with the cause in the preset anomaly-cause mapping relationship set.

[0048] When calculating the weights of the same source and the weights of different sources based on historical fault samples, the number of samples from the same source and the number of samples from different sources are counted separately. When the number of samples from the same source is not zero, the initial confidence level is subtracted from the ratio of the number of samples from the same source whose root cause is finally confirmed to be the corresponding cause to the number of samples from the same source, to obtain the weight of the same source. When the number of samples from the same source is zero, the above ratio calculation is not performed, and the weight of the same source is set to the preset default weight of the same source or entered by the operation and maintenance personnel. The above issues also need to be considered and solved by analogy during the calculation of the weight of different sources. The weights of the same source and the weights of different sources after being entered or set are stored together with the corresponding causes in the anomaly-cause mapping relationship set.

[0049] A one-to-one correspondence is established between the fault direction and its cause. Once a fault direction is determined, only the cause corresponding to that fault direction is output as the root cause. When the fault direction is client link transmission abnormality, the corresponding cause is client link transmission abnormality; when the fault direction is client reception and processing abnormality, the corresponding cause is client reception and processing abnormality; when the fault direction is server processing abnormality, the corresponding cause is server processing bottleneck; when the fault direction is network path transmission abnormality, the corresponding cause is network path transmission abnormality; when the fault direction is server application processing abnormality, the corresponding cause is server application processing abnormality; when the fault direction is current session application interaction abnormality, the corresponding cause is current session application interaction abnormality. If multiple related indicators corresponding to the same abnormal segment exceed their respective historical baselines, the fault direction corresponding to the related indicator with the largest deviation value is determined as the fault direction of this abnormal segment. Then, a unique corresponding cause is determined according to this fault direction. Subsequent same-source or different-source determinations are only used to adjust the confidence of this unique cause and do not change the already determined root cause category.

[0050] In this embodiment, the resource consumption calibration information includes a resource consumption calibration value and an allowable fluctuation range. For the same target device and the same diagnostic disturbance command, diagnostic disturbance command samples are repeatedly sent within multiple historical time periods when the target device is in normal operation and meets the self-test status declaration conditions. For each sending process, the average processor utilization within the calibration reference time before sending is recorded, and the peak processor utilization within the calibration response time after sending is recorded. The difference between the peak processor utilization and the average processor utilization is used as the single resource consumption sample value. Multiple single resource consumption sample values ​​are sorted in ascending order of numerical value. The median of the sorted values ​​is used as the resource consumption calibration value, the minimum value in the sorted values ​​is used as the lower limit of the allowable fluctuation range, and the maximum value in the sorted values ​​is used as the upper limit of the allowable fluctuation range. The lower limit of the allowable fluctuation range, the resource consumption calibration value, and the upper limit of the allowable fluctuation range are stored together as the resource consumption calibration information of the diagnostic disturbance command on the target device.

[0051] When the deviation obtained during subsequent fault diagnosis is within the allowable fluctuation range, it is determined that the deviation belongs to the resource consumption change under normal calibration. When the deviation obtained during subsequent fault diagnosis is greater than the upper limit of the allowable fluctuation range, it is determined that the deviation exceeds the resource consumption change under normal calibration, and the judgment result is sent to the pressure margin judgment process.

[0052] When visualizing root cause conclusions with confidence levels, a business-level network topology diagram is generated based on the client IP, server IP, service type, and transmission path corresponding to the abnormal session. The client nodes, network path nodes, and server nodes traversed by the abnormal session are marked in the business-level network topology diagram. The abnormal segment type, target device, fault direction, root cause name, and confidence level are displayed as visualization fields associated with the abnormal session. When the abnormal segment is a client latency anomaly, an anomaly marker is displayed on the client side or client link side; when the abnormal segment is a service latency anomaly, an anomaly marker is displayed on the server side or network path side; when the abnormal segment is an application latency anomaly, an anomaly marker is displayed on the server application side or the current session application interaction side. The confidence level of the root cause conclusion is also displayed as a text value, progress bar, or color gradient, enabling operations personnel to view the anomaly location, root cause category, and confidence level in the business-level network topology diagram.

[0053] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A business-level network visualization monitoring method for fault prediction and health management, characterized in that, Includes the following steps: S1: Deploy probes in a bypass mode in the network to be monitored, collect each data packet flowing through the network to be monitored, and record the timestamp of each data packet passing through the probe; S2: Calculate the latency segment of the session based on the timestamps of each data packet in the same session, and mark the latency segment that deviates from the historical baseline as an abnormal segment. The latency segment includes client latency, service latency and application latency. S3: For abnormal segments, send a diagnostic disturbance command with known resource consumption characteristics to the target device corresponding to the session; S4: Collect the operating status indicators of the target device after receiving the diagnostic disturbance command, and determine the disturbance response characteristics of the target device to the diagnostic disturbance command based on the operating status indicators; S5: Based on the disturbance response characteristics, determine the root cause from the preset set of anomaly-cause mapping relationships, and visualize the root cause conclusions with confidence levels.

2. The business-level network visualization monitoring method for fault prediction and health management according to claim 1, characterized in that, The process of calculating the delay segment is as follows: Deploy probes on both the client and server sides of the network to be monitored; Data packets from the same session are divided into a first set and a second set according to their transmission direction. The transmission directions of data packets in the first set and the second set are from the client to the server and from the server to the client, respectively. Select the data packet with the smallest timestamp recorded by the probe on the client side from the first set, and record the timestamp of the data packet recorded by the probe on the client side as A1, and record the timestamp of the data packet recorded by the probe on the server side as A2. The difference between A2 and A1 is taken as the client latency. Select the data packet with the largest timestamp recorded by the probe on the server side from the second set, and record the timestamp of the data packet recorded by the probe on the server side as A3, and record the timestamp of the data packet recorded by the probe on the client side as A4. The difference between A4 and A3 is used as the service latency. Subtracting the customer latency and service latency from the difference between A4 and A1 gives the application latency.

3. The business-level network visualization monitoring method for fault prediction and health management according to claim 1, characterized in that, The process of identifying abnormal segments is as follows: Obtain historical values ​​of a single latency segment for the same IP within a preset time period, sort the obtained historical values ​​in ascending order according to their numerical values, and use the values ​​at the preset quantiles in the sort as the historical baseline for that type of latency segment. The current latency segment is compared with the corresponding historical baseline, and the latency segment that is greater than the historical baseline is marked as an abnormal segment.

4. The business-level network visualization monitoring method for fault prediction and health management according to claim 1, characterized in that, The process of sending diagnostic disturbance commands is as follows: The target device is determined based on the type of the abnormal segment. When the abnormal segment is client latency, the target device is the client; when the abnormal segment is service latency, the target device is the server; and when the abnormal segment is application latency, the target device is the server. When the network under monitoring is running normally, a diagnostic disturbance command sample is sent to the target device. The average processor utilization of the target device before receiving the diagnostic disturbance command sample and the peak processor utilization after receiving the diagnostic disturbance command sample are collected. The resource consumption calibration value is determined based on the difference between the peak processor utilization and the average processor utilization. The diagnostic disturbance command sample carrying the resource consumption calibration value is used as the diagnostic disturbance command. The diagnostic disturbance command is consistent with the normal service request in terms of protocol format. Before sending the diagnostic disturbance command, a self-test status query command is sent to the target device, and the self-test status declaration returned by the target device is received. The self-test status declaration is generated by the target device when it determines that its own processor utilization is lower than the self-test threshold. When the self-test status declaration is received within the waiting time, the diagnostic disturbance command is sent to the target device.

5. The business-level network visualization monitoring method for fault prediction and health management according to claim 1, characterized in that, The process of collecting disturbance response characteristics is as follows: The processor utilization rate of the target device is collected as an indicator of its operating status. Taking the time of sending the diagnostic disturbance command as the zero point, the processor utilization rate in the first time period before the diagnostic disturbance command is sent is recorded as the baseline time period data, and the processor utilization rate in the second time period after the diagnostic disturbance command is sent is recorded as the response time period data. Calculate the average processor utilization rate in the baseline time period data, and use it as the baseline average. Extract the maximum value of processor utilization from the response time period data as the response peak; The difference between the peak response and the baseline mean is used as the deviation magnitude. Extract the moment when processor utilization first drops from the peak response time to the baseline average from the response time data, and use the difference between this moment and the sending time as the recovery time; The deviation magnitude and recovery time are used as characteristics of the disturbance response.

6. The business-level network visualization monitoring method for fault prediction and health management according to claim 1, characterized in that, The process of outputting root cause conclusions is as follows: Based on the type of abnormal segment, collect the associated metrics corresponding to the abnormal segment: When the abnormal segment is customer latency, the associated metrics include terminal retransmission rate and TCP zero window count. When the abnormal segment is service latency, the associated metrics include server TCP timestamp difference and packet loss rate. When the abnormal segment is application latency, the associated metrics include application layer protocol type and application latency of other sessions of the same type of business during the same period. The pressure margin assessment result of the target equipment in response to the diagnostic disturbance command is determined based on the disturbance response characteristics. Determine the direction of the fault based on relevant indicators; When the pressure margin determination result is from the same source as the fault direction, the confidence level of the corresponding root cause is increased; when the pressure margin determination result is from a different source than the fault direction, the confidence level of the corresponding root cause is decreased.

7. The business-level network visualization monitoring method for fault prediction and health management according to claim 6, characterized in that, The process of establishing the exception-cause mapping set is as follows: Define the exception segment types, which include customer latency exceptions, service latency exceptions, and application latency exceptions. Terminal retransmission rate and TCP zero window count are used as correlation indicators for abnormal customer latency. The difference between server-side TCP timestamps and packet loss rate are used as indicators related to service latency anomalies. The application latency of application layer protocol type and other sessions of the same type of business during the same period is used as the correlation indicator for application latency anomalies; Store the abnormal segment types and associated indicators as a set of abnormal-cause mapping relationships.

8. The business-level network visualization monitoring method for fault prediction and health management according to claim 7, characterized in that, The process of deriving root cause conclusions also includes: Based on the type of abnormal segment, the corresponding associated indicators are read from the abnormal-cause mapping relationship set, the current value of each associated indicator is collected, and the fault direction is determined based on the current value. When the recovery time exceeds the historical baseline of the recovery time and the deviation is greater than the upper limit of the allowable fluctuation range corresponding to the resource consumption calibration value, it is determined that the target equipment has insufficient pressure margin to withstand the diagnostic disturbance command. When the recovery time does not exceed the historical baseline of the recovery time, or the deviation does not exceed the upper limit of the allowable fluctuation range corresponding to the resource consumption calibration value, the target equipment is judged to have normal pressure margin to the diagnostic disturbance command; When the pressure margin determination result is insufficient and the fault direction points to the abnormality of the target equipment side, the pressure margin determination result and the fault direction are from the same source. Otherwise, the pressure margin determination result is determined to be from a different source than the fault direction; When determining the same source, the cause corresponding to the fault direction is taken as the root cause, and the initial confidence score is added to the same source weight to obtain the confidence score before outputting it; When different sources are identified, the cause corresponding to the fault direction is taken as the root cause, and the initial confidence score is added to the weight of different sources to obtain the confidence score before outputting it. The initial confidence level, same-source weight, and different-source weight are all preset.

Images