Network communication monitoring method and device, electronic equipment and medium

Abstract

Embodiments of the present application relate to the technical field of network communication monitoring, and provide a network communication monitoring method and device, electronic equipment and a medium. The method comprises: collecting meta-information of each data block of a network card export when the network card performs a set communication operation in real time; performing data processing on the meta-information to generate an evolution rate sequence vector set of all queue pairs of all communication endpoints under the same set communication operation; inputting the evolution rate sequence vector set into a fault detection model to obtain a predicted rate of each queue pair output by the fault detection model; comparing the predicted rate of each queue pair with a rate error of the corresponding actual traffic; calculating an error mean and a standard deviation of all queue pairs based on the rate error of each queue pair; performing network communication anomaly judgment based on the error mean and the standard deviation, and determining an abnormal point position based on a queue pair number. Thus, efficient and real-time monitoring of network communication status is achieved, and a fault node is quickly identified.

Description

Technical Field

[0001] This invention relates to the field of network communication monitoring technology, and in particular to a network communication monitoring method, device, electronic equipment, and medium. Background Technology

[0002] Existing C4 Diagnosis technology uses aggregated communication library logs to check network communication status in real time. However, the monitoring information it collects is complex, the anomaly detection process is cumbersome, and it delays the detection of faults, making its real-time performance inferior to some proactive detection methods.

[0003] Alternatively, network communication status can be monitored using Event Tracking within a Large Language Model (LLM) training framework (CUDA), visualizing the 3D parallel training process along a timeline and pinpointing faults to specific steps in the training process in real time. However, this proactive monitoring method itself impacts model training efficiency and fails to identify specific fault nodes.

[0004] Alternatively, the LLaMA3 (Large Language Model Meta AI) model incorporates the NVIDIA Collective Communications Library (NCCL) Extended, which tracks network activity and provides snapshots of faults by sending small packets. However, these redundant packets can slow down data flow and even cause unnecessary congestion. Therefore, efficiently and in real-time monitoring of network communication status and quickly identifying faulty nodes has become a pressing issue. Summary of the Invention

[0005] This invention provides a network communication monitoring method, device, electronic device, and medium to address the shortcomings of existing technologies in the untimely and inefficient detection of network communication anomalies, thereby achieving efficient and real-time monitoring of network communication status and rapid identification of faulty nodes.

[0006] This invention provides a network communication monitoring method, comprising:

[0007] The metadata of each data block at the network interface card (NIC) egress during real-time collection of network interface card (NIC) aggregate communication operations includes the queue pair number corresponding to each data block, and the queue pair includes a sending queue and a receiving queue.

[0008] The metadata is processed to generate a set of evolution rate sequence vectors for all queue pairs of all communication endpoints under the same set communication operation;

[0009] The evolution rate sequence vector set is input into the fault detection model to obtain the predicted rate of each queue pair output by the fault detection model.

[0010] Compare the predicted rate of each queue pair with the corresponding actual traffic rate error;

[0011] The mean and standard deviation of the errors for all queue pairs are calculated based on the rate error of each queue pair.

[0012] Network communication anomalies are determined based on the mean and standard deviation of the errors, and the location of anomalies is determined based on the queue pair number.

[0013] In one possible implementation, the method further includes:

[0014] The metadata is cleaned to obtain valid metadata;

[0015] The effective metadata is normalized, and the normalized effective metadata is aggregated to generate a set of evolution rate sequence vectors for all queue pairs of all communication endpoints under the same set communication operation.

[0016] In one possible implementation, the method further includes:

[0017] When the mean error is greater than the error threshold, it is determined that there is an anomaly in the network communication. The error threshold is dynamically adjusted by combining the mean error and the standard deviation with preset adjustment parameters.

[0018] When it is determined that there is an anomaly in network communication, the queue pair number corresponding to the abnormal traffic is queried, and the location of the anomaly point is determined based on the queue pair number.

[0019] When the mean error is less than or equal to the error threshold, network communication is determined to be normal.

[0020] In one possible implementation, the method further includes:

[0021] The fault detection model is trained through the following steps:

[0022] Obtain the set of sample evolution rate sequence vectors for all queue pairs created when the network interface card performs a set communication operation;

[0023] The set of sample evolution rate sequence vectors is input into the Transformer model for online learning, and the model training is terminated when a preset early stopping condition is met to obtain the fault detection model. The preset early stopping condition includes that the verification loss no longer improves, the verification accuracy no longer increases, and the gap between the training loss and the verification loss increases.

[0024] In one possible implementation, the method further includes:

[0025] The set of sample evolution rate sequence vectors is input into the Transformer model. When the Transformer model learns based on the set of sample evolution rate sequence vectors, an adaptive patch processing method is adopted. By dynamically adjusting the patch size, the predictive performance of the Transformer model is optimized according to the degree of data fluctuation in the time series.

[0026] In one possible implementation, the method further includes:

[0027] The rate of change of the time series data is determined based on the degree of data fluctuation in the time series.

[0028] When the rate of change of the data is greater than the first fluctuation threshold, the change of the sample evolution rate sequence vector set is captured by the minimum patch.

[0029] When the data change rate is less than the second fluctuation threshold, the maximum patch is used to capture the change of the sample evolution rate sequence vector set.

[0030] When the rate of change of the data is greater than the second fluctuation threshold and less than the first fluctuation threshold, the size of the patch is adaptively adjusted using a preset formula.

[0031] The present invention also provides a network communication monitoring device, comprising the following modules:

[0032] The data acquisition module is used to collect metadata of each data block at the network interface card (NIC) egress when the NIC performs a set of communication operations in real time. The metadata includes the queue pair number corresponding to each data block, and the queue pair includes a sending queue and a receiving queue. The metadata is processed to generate a set of evolution rate sequence vectors of all queue pairs of all communication endpoints under the same set of communication operations.

[0033] The prediction module is used to input the set of evolution rate sequence vectors into the fault detection model to obtain the predicted rate of each queue pair output by the fault detection model.

[0034] The comparison module is used to compare the predicted rate of each queue pair with the rate error of the corresponding actual traffic; and to calculate the mean and standard deviation of the error of all queue pairs based on the rate error of each queue pair.

[0035] The judgment module is used to judge network communication anomalies based on the mean and standard deviation of the error, and to determine the location of the anomaly point based on the queue pair number.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the network communication monitoring method as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the network communication monitoring method as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the network communication monitoring method as described above.

[0039] The network communication monitoring method, apparatus, electronic device, and medium provided by this invention collect metadata of each data block at the network interface card (NIC) egress during a cascaded communication operation in real time. This metadata includes the queue pair number corresponding to each data block, and each queue pair comprises a sending queue and a receiving queue. The metadata is processed to generate an evolution rate sequence vector set of all queue pairs across all communication endpoints under the same cascaded communication operation. This evolution rate sequence vector set is input into a fault detection model to obtain the predicted rate of each queue pair output by the fault detection model. The predicted rate of each queue pair is compared with the rate error of the corresponding actual traffic. The mean and standard deviation of the errors for all queue pairs are calculated based on the rate error of each queue pair. Network communication anomalies are judged based on the mean and standard deviation of the errors, and the location of the anomaly is determined based on the queue pair number. Compared to the shortcomings of existing technologies, such as untimely and inefficient detection of network communication anomalies, this solution, by collecting metadata information of cascaded communication operations at the NIC layer and using a deep learning model to predict network traffic and detect anomalies, can be applied to high-performance computing clusters in data centers. This method achieves efficient real-time network communication monitoring without affecting model training, and can quickly locate faulty nodes or links. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the architecture of the network communication monitoring method provided by the present invention.

[0042] Figure 2This is a flowchart illustrating the network communication monitoring method provided by the present invention.

[0043] Figure 3 This is a schematic diagram of the network communication monitoring device provided by the present invention.

[0044] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0047] Figure 1 This is a schematic diagram of the architecture of the network communication monitoring method provided by the present invention, as shown below. Figure 1 As shown, the architecture of this network communication monitoring method is used in the network environment for training large model networks. The figure includes the training network, storage network, graphics processing unit (GPU) server, and network switch.

[0048] The training network connects to multiple GPU servers via switches. It is responsible for transferring training data and model parameters between the GPU servers to support distributed training tasks.

[0049] The storage network is also connected to the GPU server via a switch. It is responsible for providing data storage and access services; the GPU server can read training data or write model parameters through the storage network.

[0050] The diagram shows multiple GPU servers, which are the main computing units for performing model training tasks. Each GPU server is connected to both the training and storage networks for data transfer and storage access.

[0051] The diagram shows multiple network switches, which are the core components of the network and responsible for forwarding data packets between different network devices. The switches connect the training network, storage network, and GPU servers, ensuring efficient data transmission across the network.

[0052] Training data can be transferred from the storage network to the GPU server and then shared among different GPU servers via the training network to support distributed training. Updated model parameters during training can be synchronized across GPU servers via the training network to maintain model consistency. GPU servers can access storage devices through the storage network to read training data or write training results.

[0053] Figure 2 This is a flowchart illustrating the network communication monitoring method provided by the present invention, as shown below. Figure 2 As shown, the method includes the following:

[0054] S21. Real-time acquisition of metadata for each data block exported by the network card when the network card performs aggregated communication operations.

[0055] S22. Perform data processing on the metadata to generate a set of evolution rate sequence vectors for all queue pairs of all communication endpoints under the same set communication operation.

[0056] This invention provides a fault detection tool based on a large model-trained network, which can ensure seamless training while monitoring the network status in real time and identifying faulty nodes.

[0057] Specifically, based on passive detection and aggregated communication operations, MegaMon monitoring tool is proposed: a large-scale model network monitoring tool that uses a Transformer model to detect temporal anomalies by retrieving real-time traffic data from the server's network interface card (NIC). MegaMon collects the evolution rate sequences of all basic communication endpoints (Queue Pairs, QPs) created by the sending NIC during aggregated communication operations, inputs them into a Transformer model deployed independently of the training network machine for online prediction, and if the predicted rate exceeds a pre-set error threshold compared to the actual traffic, an anomaly in network communication is identified, and an alarm is triggered.

[0058] First, the data acquisition and processing module in MegaMon collects metadata for each data block at the network interface card (NIC) egress in real time. This metadata includes the size of each data block, the transmission time window, and the queue pair number corresponding to each data block. The metadata is then processed to generate a set of evolution rate sequence vectors for all queue pairs across all communication endpoints under the same set of communication operations.

[0059] Specifically, when the network interface card (NIC) transmits data, both parties first exchange necessary communication information, including the Global Identifier (GID), Queue Pair Number (QPN), and First In First Out (FIFO) queue. The receiving end prepares the receive buffer and provides the sending end with the data receive address, data block size, and index information. After receiving this information, the sending end initiates data transmission by issuing a Work Request (WR). Whenever a WR is completed, the receiving end reclaims the request and obtains the transmission metadata based on the feedback from the Completion Queue (CQ). This metadata includes the size of each data block, the transmission time window, and its corresponding QPN. The data acquisition and processing module monitors and manages data transmission by recording this information multiple times. Optionally, the data acquisition and processing module can also perform data cleaning on the metadata to obtain effective metadata; normalize the effective metadata; and aggregate the normalized effective metadata to generate a set of evolution rate sequence vectors for all queue pairs of all communication endpoints under the same set of communication operations.

[0060] A Global Identifier (GID) is an identifier used to uniquely identify each endpoint node in a network. GIDs ensure that each device in a wide area network (WAN) or data center network can be uniquely identified, allowing data packets to be routed accurately to their destination.

[0061] A queue pair number (QPN) is a unique identifier assigned to each queue pair in the channel adapter to identify different communication channels. Each queue pair (QP) contains a transmit queue and a receive queue for processing incoming and outgoing data.

[0062] A First-In-First-Out (FIFO) queue is a data structure in which elements always enter from one end and leave from the other, with the first element to enter being the first to leave. This structure guarantees the sequential order of data processing.

[0063] S23. Input the set of evolution rate sequence vectors into the fault detection model to obtain the predicted rate of each queue pair output by the fault detection model.

[0064] The set of evolution rate sequence vectors is input into a pre-trained fault detection model. The model predicts the flow rate of each queue pair based on the set of evolution rate sequence vectors and outputs the predicted rate of each queue pair.

[0065] Specifically, MegaMon predicts the traffic of all queues at a future point in time and generates the first... The queue pairs are in time Expected rate .

[0066] S24. Compare the error between the predicted rate and the corresponding actual flow of all queue pairs to obtain the mean and standard deviation of the error.

[0067] S25. Calculate the mean and standard deviation of the errors of all queue pairs based on the rate error of each queue pair.

[0068] S26. Based on the mean and standard deviation of the error, network communication anomalies are judged, and the location of the anomaly point is determined based on the queue pair number.

[0069] Then, the predicted rate for each queue pair With actual rate The prediction errors of each queue pair are compared and calculated using the following formula:

[0070]

[0071] Furthermore, for all queue pairs in time The errors are aggregated, and the mean error of all queue pairs is calculated using the following formula:

[0072]

[0073] Where n is the number of queue pairs, based on a preset error threshold. If at a certain time step Aggregation error If the error exceeds this threshold, it is marked as abnormal. To adapt to dynamically changing data environments, the error threshold... The mean error over all time steps can be calculated. and standard deviation The formula for calculating the dynamic threshold is as follows:

[0074]

[0075] in, To adjust parameters, anomalies in aggregated communication operations can be detected by individually predicting the rate of each queue pair and aggregating these rate errors. This effectively identifies anomalies in the communication process and reduces false alarms caused by network fluctuations.

[0076] Furthermore, when an anomaly is detected in network communication, the location of the anomaly can be determined based on the queue pair number.

[0077] The specific communication channel is identified by the queue pair number (QPN). The device or node where the fault occurs is determined by combining this with the global identifier (GID).

[0078] For example, at a certain time step t The system detected the following:

[0079] Queue pair 1: Predicted rate is 100 Mbps, actual rate is 80 Mbps, error is 20 Mbps.

[0080] Queue pair 2: Predicted rate was 120 Mbps, actual rate was 118 Mbps, error was 2 Mbps.

[0081] Queue pair 3: Predicted rate was 90 Mbps, actual rate was 60 Mbps, and error was 30 Mbps.

[0082] Assuming the dynamic threshold is 10 Mbps:

[0083] The errors in queue pair 1 and queue pair 3 exceeded the threshold and were marked as abnormal.

[0084] Using the queue pair number (QPN) and global identifier (GID), the system located the devices and links containing queue pair 1 and queue pair 3. Further analysis of the transmission path determined that the fault might be located on links Link1−2 and Link3−4.

[0085] The advantages of this method are: it can quickly detect anomalies by collecting and analyzing network traffic data in real time; it can accurately identify fault points by dynamically adjusting the patch size and dynamic threshold; it is based on passive detection, so it does not affect the efficiency of model training; and it can adapt to complex network environments and dynamically changing data traffic.

[0086] When testing three typical network faults—slow NICs, abnormal congestion, and link jitter—MegaMon demonstrated excellent anomaly detection capabilities and exhibited differentiated response speeds for different types of faults. Slow NIC faults cause a significant drop in transmission rate, and MegaMon was able to quickly detect the anomaly through noticeable changes in the aggregate rate. Abnormal congestion manifests as sharp fluctuations within a short period, and MegaMon flexibly responds to data fluctuations by adaptively adjusting the patch size, achieving timely feedback. Link jitter, due to its small and irregular fluctuations, is more difficult to detect, but MegaMon was still able to effectively capture this fault through dynamic monitoring and adaptive adjustment mechanisms. Overall, MegaMon demonstrated its efficient and accurate anomaly detection capabilities when handling various faults in complex network environments.

[0087] The network communication monitoring method provided by this invention collects metadata of each data block at the network interface card (NIC) egress during a cascaded communication operation in real time. This metadata includes the queue pair number corresponding to each data block, and each queue pair contains a sending queue and a receiving queue. The metadata is processed to generate an evolution rate sequence vector set of all queue pairs for all communication endpoints under the same cascaded communication operation. This evolution rate sequence vector set is input into a fault detection model to obtain the predicted rate of each queue pair output by the fault detection model. The predicted rate of each queue pair is compared with the rate error of the corresponding actual traffic. Based on the rate error of each queue pair, the mean and standard deviation of the error for all queue pairs are calculated. Network communication anomalies are judged based on the mean and standard deviation of the error, and the location of the anomaly is determined based on the queue pair number. This method overcomes the shortcomings of existing technologies, such as untimely and inefficient network communication anomaly detection. This method collects metadata data from the network interface card (NIC) layer's collective communication operation layer, uses a deep learning model framework to predict network traffic, and then performs anomaly detection. It can be applied to high-performance computing network clusters in data centers, ensuring seamless model training while enabling efficient, real-time monitoring of network communication status and rapid identification of faulty nodes.

[0088] In this embodiment of the invention, a fault detection model needs to be pre-trained, which is obtained through the following steps:

[0089] Obtain the set of sample evolution rate sequence vectors for all queue pairs created when the network card performs ensemble communication operations; input the set of sample evolution rate sequence vectors into the Transformer model for online learning, and terminate model training when a preset early stopping condition is met to obtain the fault detection model. The preset early stopping conditions include no further improvement in validation loss, no further improvement in validation accuracy, and an increase in the gap between training loss and validation loss.

[0090] During model training, an adaptive patching method is used to optimize the prediction performance of the Transformer model by dynamically adjusting the patch size based on the degree of data fluctuation in the time series.

[0091] Specifically, to ensure network transmission stability and training efficiency, the model not only needs to make predictions quickly and accurately, but also needs to be able to adapt to dynamic network environments. Traditional patch processing methods divide time-series data into fixed-length patches and input them into the Transformer model for processing, but this has limitations when dealing with uneven network traffic. Therefore, this invention proposes an adaptive patch processing method (AdaptivePatch-Transformer), which dynamically adjusts the patch size to optimize prediction performance based on the severity of data fluctuations in the time series.

[0092] Specifically, Patch size Rate of change of time series The decision is made using the following formula:

[0093]

[0094] in, It is the smallest patch size. It is the maximum patch size. It is the first fluctuation threshold (high fluctuation threshold). It is the second fluctuation threshold (low fluctuation threshold). To adjust the coefficient, It is a tiny constant that prevents the denominator from being zero.

[0095] when Exceed When using the minimum patch ( Finely capture changes; when they are less than Using the maximum patch ( It focuses on global trends and reduces computational burden. The patch size is adaptively adjusted according to a formula, striking a balance between these two objectives. This mechanism effectively controls the patch size through a fixed threshold, ensuring the model balances accuracy and efficiency when processing complex time-series data. Furthermore, the AdaptivePatch-Transformer does not use a fixed batch of data for training in each iteration; instead, it dynamically adjusts weights through online learning, enabling the model to capture data changes in real time, improving prediction accuracy and real-time performance, making it particularly suitable for complex network traffic time-series prediction tasks.

[0096] During model training, online learning is performed based on data from the time series, and training is terminated when the early stopping condition is met.

[0097] Early stopping is a judgment technique used to prevent model overfitting. It monitors the model's performance on the validation set during training and stops training early when model performance begins to decline. Judgment conditions may include: validation loss no longer improving, validation accuracy no longer increasing, and the gap between training loss and validation loss widening.

[0098] By collecting metadata of network interface card (NIC) layer communication operation layer, and using a deep learning model framework to predict network traffic and perform anomaly detection, this method can be applied to high-performance computing network clusters in data centers, ensuring seamless model training.

[0099] The network communication monitoring device provided by the present invention is described below. The network communication monitoring device described below can be referred to in correspondence with the network communication monitoring method described above.

[0100] Figure 3 This is a schematic diagram of the network communication monitoring device provided by the present invention, specifically including:

[0101] The data acquisition module 301 is used to collect metadata of each data block at the network interface card (NIC) egress during a cascaded communication operation in real time. The metadata includes the queue pair number corresponding to each data block, and each queue pair contains a sending queue and a receiving queue. The metadata is processed to generate a set of evolution rate sequence vectors for all queue pairs across all communication endpoints under the same cascaded communication operation. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0102] The prediction module 302 is used to input the set of evolution rate sequence vectors into the fault detection model to obtain the predicted rate of each queue pair output by the fault detection model. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0103] The comparison module 303 is used to compare the predicted rate of each queue pair with the rate error of the corresponding actual traffic; based on the rate error of each queue pair, the mean and standard deviation of the errors of all queue pairs are calculated. For detailed explanations, please refer to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0104] The judgment module 304 is used to judge network communication anomalies based on the mean and standard deviation of the errors, and to determine the location of the anomaly point based on the queue pair number. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0105] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logical instructions in the memory 830 to execute a network communication monitoring method. This method includes: real-time acquisition of metadata for each data block at the network interface card (NIC) egress during a cascaded communication operation, the metadata including the queue pair number corresponding to each data block, the queue pair comprising a sending queue and a receiving queue; data processing of the metadata to generate an evolution rate sequence vector set of all queue pairs for all communication endpoints under the same cascaded communication operation; inputting the evolution rate sequence vector set into a fault detection model to obtain the predicted rate of each queue pair output by the fault detection model; comparing the predicted rate of each queue pair with the corresponding actual traffic rate error; calculating the mean and standard deviation of the error for all queue pairs based on the rate error of each queue pair; determining network communication anomalies based on the mean and standard deviation of the error, and determining the location of the anomaly point based on the queue pair number.

[0106] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the network communication monitoring method provided by the above methods. The method includes: real-time acquisition of metadata of each data block at the network interface card (NIC) exit when the NIC performs a set of communication operations, the metadata including the queue pair number corresponding to each data block, the queue pair including a sending queue and a receiving queue; data processing of the metadata to generate an evolution rate sequence vector set of all queue pairs of all communication endpoints under the same set of communication operations; inputting the evolution rate sequence vector set into a fault detection model to obtain the predicted rate of each queue pair output by the fault detection model; comparing the predicted rate of each queue pair with the rate error of the corresponding actual traffic; calculating the mean and standard deviation of the error of all queue pairs based on the rate error of each queue pair; judging network communication anomalies based on the mean and standard deviation of the error, and determining the location of the anomaly point based on the queue pair number.

[0108] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the network communication monitoring method provided by the above methods. The method includes: real-time acquisition of metadata of each data block at the network interface card (NIC) egress when the NIC performs a set of communication operations, the metadata including the queue pair number corresponding to each data block, the queue pair including a sending queue and a receiving queue; data processing of the metadata to generate an evolution rate sequence vector set of all queue pairs of all communication endpoints under the same set of communication operations; inputting the evolution rate sequence vector set into a fault detection model to obtain the predicted rate of each queue pair output by the fault detection model; comparing the predicted rate of each queue pair with the rate error of the corresponding actual traffic; calculating the mean and standard deviation of the error of all queue pairs based on the rate error of each queue pair; judging network communication anomalies based on the mean and standard deviation of the error, and determining the location of the anomaly point based on the queue pair number.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network communication monitoring method, characterized in that, include: The metadata of each data block at the network interface card (NIC) egress during real-time collection of network interface card (NIC) aggregate communication operations includes the size of each data block, the transmission time window, and the queue pair number corresponding to each data block. Each queue pair contains a sending queue and a receiving queue. The queue pair number is a unique identifier assigned to each queue pair to identify different communication channels. The metadata is processed to generate a set of evolution rate sequence vectors for all queue pairs of all communication endpoints under the same set communication operation; The evolution rate sequence vector set is input into the fault detection model to obtain the predicted rate of each queue pair output by the fault detection model. The predicted rate and the corresponding actual rate of each queue pair are compared to obtain the rate error; The mean and standard deviation of the errors for all queue pairs are calculated based on the rate error of each queue pair. Network communication anomalies are determined based on the mean and standard deviation of the errors, and the location of anomalies is determined based on the queue pair number. The process of determining network communication anomalies based on the mean and standard deviation of the errors, and determining the location of anomalies based on the queue pair numbers, includes: When the mean error is greater than the error threshold, it is determined that there is an anomaly in the network communication. The error threshold is dynamically adjusted by combining the mean error and the standard deviation with preset adjustment parameters. When it is determined that there is an anomaly in network communication, the queue pair number corresponding to the abnormal traffic is queried, and the location of the anomaly point is determined based on the queue pair number. When the mean error is less than or equal to the error threshold, network communication is determined to be normal.

2. The method according to claim 1, characterized in that, The step of processing the metadata to generate a set of evolution rate sequence vectors for all queue pairs of all communication endpoints under the same set of communication operations includes: The metadata is cleaned to obtain valid metadata; The effective metadata is normalized, and the normalized effective metadata is aggregated to generate a set of evolution rate sequence vectors for all queue pairs of all communication endpoints under the same set communication operation.

3. The method according to claim 1 or 2, characterized in that, The fault detection model is trained through the following steps: Obtain the set of sample evolution rate sequence vectors for all queue pairs created when the network interface card performs a set communication operation; The set of sample evolution rate sequence vectors is input into the Transformer model for online learning, and the model training is terminated when a preset early stopping condition is met to obtain the fault detection model. The preset early stopping condition includes that the verification loss no longer improves, the verification accuracy no longer increases, or the gap between the training loss and the verification loss increases.

4. The method according to claim 3, characterized in that, The step of inputting the set of sample evolution rate sequence vectors into the Transformer model for online learning includes: The set of sample evolution rate sequence vectors is input into the Transformer model. When the Transformer model learns based on the set of sample evolution rate sequence vectors, an adaptive patch processing method is adopted. By dynamically adjusting the patch size, the predictive performance of the Transformer model is optimized according to the degree of data fluctuation in the time series.

5. The method according to claim 4, characterized in that, The adaptive patch processing method, which dynamically adjusts the patch size to optimize the predictive performance of the Transformer model based on the degree of data fluctuation in the time series, includes: The rate of change of the time series data is determined based on the degree of data fluctuation in the time series. When the rate of change of the data is greater than the first fluctuation threshold, the change of the sample evolution rate sequence vector set is captured by the minimum patch. When the data change rate is less than the second fluctuation threshold, the maximum patch is used to capture the change of the sample evolution rate sequence vector set. When the rate of change of the data is greater than the second fluctuation threshold and less than the first fluctuation threshold, the size of the patch is adaptively adjusted using a preset formula.

6. A network communication monitoring device, characterized in that, include: The data acquisition module is used to collect metadata of each data block at the network interface card (NIC) egress during a cascaded communication operation in real time. The metadata includes the size of each data block, the transmission time window, and the queue pair number corresponding to each data block. Each queue pair contains a sending queue and a receiving queue. The queue pair number is a unique identifier assigned to each queue pair to identify different communication channels. The module processes the metadata to generate a set of evolution rate sequence vectors for all queue pairs of all communication endpoints under the same cascaded communication operation. The prediction module is used to input the set of evolution rate sequence vectors into the fault detection model to obtain the predicted rate of each queue pair output by the fault detection model. The comparison module is used to compare the predicted rate and the corresponding actual rate of each queue pair to obtain the rate error; based on the rate error of each queue pair, the mean and standard deviation of the error of all queue pairs are calculated. The judgment module is used to judge network communication anomalies based on the mean and standard deviation of the error, and to determine the location of the anomaly point based on the queue pair number. The method of determining network communication anomalies based on the mean and standard deviation of the errors, and determining the location of anomalies based on the queue pair numbers, includes: determining that network communication is abnormal when the mean error is greater than an error threshold, wherein the error threshold is dynamically adjusted by combining the mean and standard deviation of the errors with preset adjustment parameters; when network communication is determined to be abnormal, querying the queue pair numbers corresponding to the abnormal traffic, and determining the location of the anomaly based on the queue pair numbers; and determining that network communication is normal when the mean error is less than or equal to the error threshold.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the network communication monitoring method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the network communication monitoring method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the network communication monitoring method as described in any one of claims 1 to 5.

Images