Adaptive traffic optimization method and system for a supercomputing center storage area network

Abstract

The application provides a kind of self-adaptive traffic optimization method and system of intelligent center storage area network, comprising: collecting the traffic data of switch in storage area network by full-stack telemetry data acquisition layer;The collected traffic data is input to the embedded AI inference engine deployed in the switch for analysis, the embedded AI inference engine includes a series of anomaly detection model and time series prediction model;When the congestion prediction result output by the time series prediction model meets the first preset condition, perform dynamic load balancing operation through adaptive traffic optimizer;When the anomaly score output by the anomaly detection model meets the second preset condition, perform automatic isolation operation on abnormal traffic source through abnormal traffic isolation subsystem.The application enables FC switch to have self-learning, self-adaptive, self-optimizing capability, eliminates time-consuming manual process, and is suitable for the characteristics of AI work load of intelligent center, strong burst.

Description

Technical Field

[0001] This invention relates to the field of data center storage network technology, and specifically to an adaptive traffic optimization method and system for a smart computing center storage area network (SAN). Background Technology

[0002] With the explosive growth of large-scale AI model training, intelligent computing centers are placing unprecedented demands on the performance, stability, and intelligent management of storage networks. Storage Area Networks (SANs), as the core infrastructure connecting servers and storage devices, directly impact the efficiency of upper-layer AI training tasks and the reliability of data access through their traffic management.

[0003] Currently, traffic management and operation and maintenance of SAN networks face the following prominent problems: (1) Manual operation and maintenance processes are time-consuming and have slow response. In traditional SAN networks, network administrators need to manually configure parameters such as zoning, quality of service (VQOS) policies, and link load balancing to cope with business changes. When congestion, performance degradation, or changes in application requirements occur, manual intervention is required for analysis, location, and adjustment, which not only consumes a lot of manpower but also has a response speed that lags far behind the dynamically changing business needs of the intelligent computing center. (2) Lack of end-to-end full network visualization capabilities. Traditional SAN management tools usually only provide isolated monitoring views at the port level or device level, making it difficult to present the end-to-end traffic path and performance bottlenecks between virtual machines and physical storage devices from a global perspective. When application performance problems occur, administrators often need to switch between multiple management interfaces, making it difficult to quickly locate the root cause. (3) Lack of automatic isolation mechanism for abnormal traffic. In SAN networks, slow devices, oversubscription links, or abnormal traffic behavior can seriously interfere with the transmission of normal traffic, causing the "noisy neighbor" effect. Traditional solutions require administrators to manually identify abnormal traffic and perform isolation operations, which is inefficient and prone to errors. (4) Static strategies cannot adapt to dynamically changing AI workloads. AI training tasks in intelligent computing centers are characterized by varied traffic patterns, large differences in I / O characteristics, and strong bursts. Traditional traffic management strategies based on static rules or predefined thresholds are difficult to cope with this highly dynamic environment and can easily lead to a decline in the performance of critical applications.

[0004] In summary, there is an urgent need for an intelligent SAN traffic optimization solution that can automatically learn SAN network traffic patterns, dynamically optimize performance, automatically isolate abnormal traffic, and provide end-to-end visualization. Summary of the Invention

[0005] This invention provides an adaptive traffic optimization method for intelligent computing center storage area networks, comprising the following steps: Traffic data from switches in the storage area network is collected through a full-stack telemetry data acquisition layer. The collected traffic data is input into the embedded AI inference engine deployed in the switch for analysis. The embedded AI inference engine includes a series of anomaly detection models and time series prediction models. When the congestion prediction result output by the time-series prediction model meets the first preset condition, dynamic load balancing operation is performed through the adaptive traffic optimizer. When the anomaly score output by the anomaly detection model meets the second preset condition, the anomaly traffic source is automatically isolated through the anomaly traffic isolation subsystem.

[0006] Optionally, the anomaly detection model is a robust random pruning forest model, and the temporal prediction model is a quantized compressed temporal convolutional network model. The anomaly detection model is used to calculate the feature vector of the streaming input in real time and output anomaly score; The time-series prediction model is used to infer the feature sequence within a time window and output the probability of congestion occurring within a preset time period and the expected duration of congestion.

[0007] Optionally, the dynamic load balancing operation performed through the adaptive traffic optimizer specifically includes: Calculate the expected utility function for all candidate performance groups and select the non-congestion performance group with the highest utility value as the target performance group. Calculate the migration cost of migrating the traffic source to be migrated from the current performance group to the target performance group; If the migration cost is lower than a preset threshold, the traffic of the traffic source to be migrated will be migrated to the target performance group without interruption by issuing a shadow weight update instruction to the switching ASIC.

[0008] Optionally, before performing the dynamic load balancing operation via the adaptive traffic optimizer, the method further includes: Monitor the speed and protocol type of devices within each performance group; If the rate ratio of devices within a performance group exceeds the preset splitting threshold, or if there are multiple protocol types of traffic with a proportion exceeding the preset ratio, the performance group will be automatically split or reorganized.

[0009] Optionally, the automatic isolation operation of abnormal traffic sources through the abnormal traffic isolation subsystem specifically includes: When the abnormal traffic source is determined to be a slow device, the interface of the switching ASIC is invoked to clamp the available buffer credit value of the port where the abnormal traffic source is located to a preset low threshold. The abnormal traffic source is given the lowest priority and mapped to a dedicated isolated virtual channel.

[0010] Optionally, after performing the automatic isolation operation, an automatic recovery step is also included: Send probe frames to the isolated abnormal traffic source at a preset period; If the round-trip delay of N consecutive probes is lower than the delay threshold, and no credit depletion occurs again during the probe period, then the isolation of the abnormal traffic source is lifted.

[0011] Optionally, while collecting the traffic data, a mapping relationship between virtual machines and traffic data can be established through the interface integration between the data acquisition layer and the virtualization platform.

[0012] Optionally, establishing the mapping relationship between virtual machines and traffic data specifically includes: An extended hook is implanted in the Fabric login service of the switch. When a login request is received from the host bus adapter, the virtualization platform is queried to obtain the virtual machine UUID of the initiator. Establish a triplet mapping relationship between Fibre Channel address, global port name and virtual machine UUID, and inject this mapping relationship into the switch's hardware forwarding table to achieve virtual machine-level traffic tracing.

[0013] This invention also provides an adaptive traffic optimization system for a smart computing center storage area network, comprising: The full-stack telemetry data acquisition subsystem is used to collect traffic data from the switch. An embedded AI inference engine, deployed within the switch, is used to receive the traffic data and output analysis results, including an anomaly detection model and a time-series prediction model. An adaptive traffic optimizer is used to perform dynamic load balancing operations when the analysis results of the time-series prediction model meet a first preset condition. The abnormal traffic isolation subsystem is used to perform automatic isolation operations when the analysis results of the abnormal detection model meet the second preset condition.

[0014] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the adaptive traffic optimization method for the intelligent computing center storage area network as described above.

[0015] The present invention provides a non-transitory computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the adaptive traffic optimization method for the intelligent computing center storage area network as described above.

[0016] The adaptive traffic optimization method and system for intelligent computing center storage area networks provided by this invention, through closed-loop management using embedded AI technology, enables FC switches to possess self-learning, self-adaptive, and self-optimizing capabilities, eliminating time-consuming manual processes. Automated traffic management and anomaly handling capabilities significantly reduce the frequency of manual intervention by network administrators. The AI-driven learning mechanism allows the system to continuously learn and adapt to ever-changing application requirements, making it particularly suitable for the variable and sudden nature of AI workloads in intelligent computing centers. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the adaptive traffic optimization method for the intelligent computing center storage area network provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the workflow of the embedded AI inference engine in this invention.

[0020] Figure 3 This is a flowchart of an adaptive load balancing algorithm.

[0021] Figure 4 This is the state transition diagram of the automatic isolation mechanism for abnormal traffic in this invention.

[0022] Figure 5 This is a schematic diagram illustrating the key steps of the adaptive flow optimization method in this invention, labeled along a time axis.

[0023] Figure 6 This is a hardware structure diagram of the adaptive traffic optimization system for the intelligent computing center storage area network provided by the present invention.

[0024] Figure 7 This is a software architecture diagram of the adaptive traffic optimization system for the intelligent computing center storage area network provided by the present invention.

[0025] Figure 8 A schematic diagram illustrating the interaction relationships among modules in the adaptive traffic optimization system for intelligent computing center storage area networks provided by this invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] This invention provides an adaptive traffic optimization method for intelligent computing center storage area networks, such as... Figure 1 As shown, it includes the following steps: S1. Collect traffic data of switches in the storage area network through the full-stack telemetry data acquisition layer; S2. The collected traffic data is input into the embedded AI inference engine deployed in the switch for analysis. The embedded AI inference engine includes a series of anomaly detection models and time series prediction models. S3. When the congestion prediction result output by the time series prediction model meets the first preset condition, a dynamic load balancing operation is performed through the adaptive traffic optimizer; S4. When the abnormal score output by the abnormal detection model meets the second preset condition, the abnormal traffic source is automatically isolated through the abnormal traffic isolation subsystem.

[0028] In step S1, the full-stack telemetry data acquisition layer is responsible for constructing an end-to-end observable data foundation with millisecond-level precision, providing high-quality, labeled training and inference feature data for the upper-layer AI model. The full-stack telemetry data acquisition layer utilizes the telemetry capabilities of the FC switching ASIC to collect traffic statistics, buffer credit utilization, congestion alarm events, etc., for each port with a precision of 100ms. Specifically, it uses the built-in RPC (Remote Procedure Call) diagnostic interface and hardware performance monitoring counter (PMC) of the FC switching ASIC to collect the following fine-grained indicators with a micro-period of sampling period T_s = 100ms: 1. Virtual channel (VC) queue status information, including VC_RDY (virtual channel ready) normalized value, VC_CREDIT_AVAIL (virtual channel credit availability) real-time snapshot, and VC_TX_WAIT_TIME (virtual channel send waiting time) histogram statistics; 2. Frame-level timing characteristics, including Frame_Pacing_Rate deviation and EOF_ABORT count increment; 3. NVMe over FC protocol awareness fields: These include the CS_CTL / Priority field in the FC frame header and the NVMe command capsule. By parsing the above fields and command capsule, the information Command_Submission_Queue_Identifier and Dataset_Management_Range can be extracted.

[0029] For NVMe over Fabrics traffic, in addition to the fine-grained metrics mentioned above, protocol-specific metrics such as NVMe command queue depth and I / O completion latency can also be collected.

[0030] As a preferred implementation, by integrating the full-stack telemetry data acquisition layer with the interface of the virtualization platform (such as the VMware vCenter API), virtual machine identification information is obtained, and a mapping relationship between virtual machines and storage traffic is established, thereby achieving end-to-end visualization at the virtual machine level. Specifically, an FDMI (Network Device Management Interface) extended hook is implanted into the switch's Fabric login service daemon. When the host bus adapter (HBA) initiates an FLOGI (Fabric Login) / PLOGI (Port Login) request, it actively queries the virtualization platform for the initiator's NPIV (N-Port ID Virtualization) sub-instance UUID to establish an FCID (Fibre Channel Address). WWPN (Global Port Name) The hardware forwarding table entry of VM_UUID (Virtual Machine Universal Unique Identifier) ​​enables hardware tag injection of traffic at the ASIC level.

[0031] Step S1 constructs an end-to-end observable data foundation with millisecond-level precision, providing high-quality, labeled training and inference feature data for upper-layer AI models.

[0032] In step S2, the embedded AI inference engine is implemented based on a lightweight inference runtime environment of the switch control plane CPU, and adopts a dual-cascade architecture of front-end anomaly detector and back-end timing predictor. The dual-cascade architecture will be described in detail below.

[0033] The front-end anomaly detector adopts a lightweight streaming anomaly detection model—Robust Randomly Sheared Forest (RRCF). The structural parameters of this model include: forest size N_tree = 100 trees, maximum depth of each tree D_max = 15, and reservoir sampling window size W = 256 sampling points.

[0034] like Figure 2 As shown, the input feature vector (six dimensions) of this model includes: - x1: Buffer credit consumption rate (unit: Credit / ms) - x2: I / O delay variance (unit: μs) 2 ) - x3: Frame length distribution entropy (normalized to [0, 1]) - x4: Class 3 timeout drop count increment (in seconds) - x5: Link reset event flag (boolean) - x6: PFC (Priority Flow Control) pause frame duration (unit: 1024-bit time).

[0035] The model uses the following formula to calculate the outlier score: S_a = (1 / N_tree) Σ Collusive_Displacement(x_t, T_i) Where S_a represents the collusive displacement score, N_tree represents the forest size, x_t represents the new sample, and Collusive_Displacement is defined as the expected change in model structure depth caused by sample insertion.

[0036] The model is trained using a strategy of pre-training and online incremental update fine-tuning. During the pre-training phase, offline pre-training can be employed, where the dataset is collected from 30 consecutive days of FC telemetry logs in the intelligent computing center's production environment, with start and end timestamps of congestion events manually labeled. The loss function L combines Focal Loss (to handle imbalanced positive and negative samples, γ=2) and Huber Loss (to regress duration), i.e., L = λ1 · FocalLoss(P_c, Y_event) + λ2 · HuberLoss(T_cong, T_true).

[0037] The relevant parameters, meanings, and values ​​of this loss function are explained in the table below: Table 1

[0038] The AdamW optimizer was used for training, with an initial learning rate of lr = 1e-3, weight decay of wd = 1e-4, and 100 training iterations. During the online incremental update fine-tuning phase, the trigger condition was that the switch was in a low I / O period (port throughput was less than 20% of the peak value of the past 24 hours) and CPU idle rate > 70%. The EWC (Elastic Weight Consolidation) algorithm was used, and a regularization term was added to the fine-tuning loss function. , Where β=10 3 , The relevant parameters, meanings, and values ​​of this loss function are explained in the table below: Table 2

[0039] At this stage, it can be started from a size of 10 4 256 samples are randomly selected from the local circular buffer for mini-batch gradient descent.

[0040] The backend temporal predictor employs a quantized compressed temporal convolutional network (TCN) model, which structurally includes an input layer, hidden layers, and an output layer. The input layer preferably has a time window of length L=30 and a feature dimension D=12 (including the aforementioned 6-dimensional features and their first and second differences). The hidden layer preferably consists of 4 residual blocks, each containing a dilated convolutional layer (kernel size k=3, dilation coefficient...). The output layer is preferably a fully connected layer with an output dimension of O=2, representing the probability P_c of a buffer credit exhaustion event occurring within the next 60 seconds and the expected congestion duration T_cong, respectively. (i=0,1,2,3), weight normalization (WeightNorm), ReLU activation, and spatial dropout (ratio p=0.1).

[0041] The 12-dimensional feature is composed of 3 sets of features, each set consisting of 4 dimensions. The specific details of each dimension are shown in the table below.

[0042] Table 3

[0043] The Temporal Convolutional Network (TCN) model preferably uses post-training dynamic quantization to compress the weights from FP32 to INT8. The model file size is approximately 1.2MB, and the single inference latency does not exceed 45μs (based on actual measurements using ARM Cortex-A72 @ 1.8GHz).

[0044] like Figure 2 As shown, as a specific implementation, the RRCF model is designed as follows: 100 trees, depth 15, window 256 → collusion displacement fraction S_a → decision S_a>0.95? RRCF output connection "immediately trigger anomaly isolation"; The TCN model is designed as follows: sliding window L=30, D=12 → residual block ×4 dilated convolution → INT8 quantization model 1.2MB → output P_c, T_cong → decision P_c∈[0.70,0.95]? TCN output connection "periodically output P_c, T_cong → trigger dynamic load balancing".

[0045] As one specific implementation, the embedded AI inference engine runs on the switch's control plane processor, such as... Figure 3 As shown, the specific reasoning process includes: Step S201: Read data directly from the ASIC shadow register via PCIe BAR space mapping. Specifically, the switch daemon reads the ASIC Shadow Register directly at 100ms intervals via PCIe BAR space mapping (PCIe Base Address Register space mapping) to obtain raw data such as Port_Frame_Pacing_Rate and VC_Buffer_Fill_Level.

[0046] Step S202: Construct a multi-dimensional feature vector in the data plane based on the read data. Construct a feature vector V_t in the data plane, which includes 12-dimensional features such as instantaneous throughput, credit change acceleration, frame length distribution entropy, and the ratio of NVMe_CMD to SCSI_CMD, as well as their differences.

[0047] Step S203: Receive the vector stream using the RRCF model, calculate the collusion displacement score S_a, and perform streaming anomaly scoring. For example, in a certain detection, a slow device caused the anomaly score to reach 0.97, exceeding the preset threshold of 0.95.

[0048] Step S204: Perform causal convolution calculation on the sequence within the window using the TCN model to predict congestion time series. After the sliding window accumulates 30 sampling points (i.e., 3 seconds of data), the model outputs the probability of credit depletion within the next 60 seconds, P_c=0.82.

[0049] Step S205: Trigger the adaptive load balancing process based on the credit depletion probability, and trigger the anomaly isolation process based on the anomaly score event. Preferably, the adaptive load balancing process is triggered when P_c ∈ [0.70, 0.95], and the anomaly isolation process is triggered when S_a > 0.95 and the C3_Timeout_Discard counter increases.

[0050] In step S3, the adaptive traffic optimizer is the core execution mechanism. This module interacts directly with the switch's ASIC forwarding pipeline, responsible for transforming the AI's predictions into deterministic hardware behavior. The adaptive traffic optimizer employs a dynamic performance group adaptive reconfiguration algorithm. In this algorithm, a performance group set G = {g_1, g_2, ..., g_m} is defined, and each group g_i corresponds to an independent virtual channel resource pool. The following definitions apply: - Definition 1 (Performance Group Resource Vector): For performance group g_i, the resource state vector R(g_i) = [C_avail, Q_depth, B_util], where: - C_avail: The number of available buffer credits (normalized to [0, 1]) - Q_depth: Average queuing depth of the virtual channel (normalized to [0, 1]) - B_util: Virtual channel bandwidth utilization (%) - Definition 2 (Traffic Source Feature Vector): For a traffic source d to be migrated (identified by FCID), the feature vector F(d) = [Speed, Protocol, P_c(d)], where: - Speed: Link speed (values ​​8 / 16 / 32 / 64 / 128 Gbps, mapped to 1~5) - Protocol: Protocol type (SCSI=0, NVMe=1) - P_c(d): The predicted congestion probability of this traffic source. like Figure 3 As shown, the algorithm performs the following steps: 1. Triggered detection: The monitoring module detects that the bandwidth utilization B_util of a certain performance group g_src is greater than 85% for 3 seconds.

[0051] 2. Target group selection: Calculate the expected utility function for all candidate groups g_i ∈ G: U(g_i) = w1 · C_avail(g_i) - w2 · Q_depth(g_i) + w3 · (1 - B_util(g_i)) The empirical weight values ​​are w1=0.5, w2=0.3, and w3=0.2. The non-congested group with the largest U value is selected as the target g_dst.

[0052] 3. Migration Cost Assessment: Calculate the cost function for switching traffic source d from g_src to g_dst: Cost(d) = α · Protocol_Mismatch + β · (1 - Speed_Ratio) + γ · P_c(d) Where α=10, β=5, γ=20. If Cost(d)<15, migration is allowed.

[0053] 4. Shadow Weight Update: Issue a sequence of instructions to the ASIC, first set a temporary weight of 0.2 for the target VC, add a traffic redirection rule, sleep for 500ms to wait for the old queue to be emptied, and then reset the source VC weight to 0 and the target VC weight to 1.0 to achieve uninterrupted traffic migration.

[0054] In this algorithm, dynamic performance group adaptive reconstruction is performed according to the following performance group adaptive customization rules: 1) Speed ​​matching rule: If there are devices with a speed ratio ≥ 4:1 (such as 64G and 16G) in the same group, the split will be automatically triggered. Split threshold: Speed_Variance_Ratio = 2.0.

[0055] 2) Protocol isolation rules: If the proportion of NVMe traffic in the group is >30% and the proportion of SCSI traffic is >30%, an NVMe-specific performance group will be automatically created, and newly logged-in NVMe devices will be added to this group first.

[0056] The following is a detailed explanation of an embodiment using a typical predictive load balancing example. In this embodiment, virtual channel 3 carries 64G NVMe traffic and 16G SCSI traffic. The TCN model predicts that the probability of congestion occurring in this virtual channel within the next 60 seconds is 0.78. The execution flow of the adaptive load balancing algorithm in this embodiment includes: Step S301: Calculate the expected utility function of each candidate virtual channel and select a migration path based on the utility value. The optimizer calculates the expected utility function of each candidate virtual channel. Currently, virtual channel 5 has an available credit count C_avail = 0.8, a queuing depth Q_depth = 0.2, and bandwidth utilization B_util = 40%. The calculated utility value U(g5) = 0.5 × 0.8 - 0.3 × 0.2 + 0.2 × (1 - 0.4) = 0.52, which is higher than other candidate groups.

[0057] Step S302: Assess migration feasibility based on protocol compatibility, rate ratio, and congestion probability. Specifically, the target group and the traffic source to be migrated need to be protocol compatible, have similar rates, and have low congestion risk. In this example, the traffic source to be migrated is a 16G SCSI device, and there is already a 16G SCSI device in the target group. The protocol is compatible, the rate ratio is 1:1, and the predicted congestion probability P_c = 0.3. The migration cost Cost = 10×0 + 5×0 + 20×0.3 = 6 is calculated, which is less than the threshold of 15, so migration is deemed feasible. The migration cost Cost(d) is a multi-factor comprehensive evaluation function. Before deciding whether to migrate traffic source d from the current performance group g_src to the target performance group g_dst, it quantifies and evaluates the potential negative impact of the migration operation itself. Migration is only performed when the benefits outweigh the costs.

[0058] The following table explains the physical quantities and corresponding parameters involved in the migration cost Cost(d).

[0059] Table 4

[0060] Step S303: Perform uninterrupted traffic migration by issuing a two-cycle shadow weight update instruction to the ASIC. Issuing the two-cycle shadow weight update instruction to the ASIC includes: 1) Call the SDK interface to set a temporary weight of 0.2 for the target VC5.

[0061] 2) Add a traffic redirection rule: bfal_flow_redirect_add(fcid=0x123456, target_vc=5).

[0062] 3) Sleep for 500ms, waiting for the queued frames of this traffic in the source VC3 to be sent.

[0063] 4) Adjust the weight of the traffic corresponding to the FCID in the source VC3 to 0.

[0064] 5) Adjust the weight of this traffic in the target VC5 to 1.0.

[0065] Step S304: Migration complete. Verify the effect based on telemetry data. In this embodiment, telemetry data 100ms after migration shows that the bandwidth utilization of source VC3 decreased from 86% to 58%, and the predicted congestion probability decreased to 0.35. The system records the optimization event via syslog: SANDRAIN-6-FLOW_OPTIMIZED.

[0066] In step S4, as Figure 4 As shown, the abnormal traffic automatic isolation subsystem preferably executes the following step-by-step containment strategy when the AI ​​engine outputs an abnormal score S_a > 0.95 and identifies it as a slow-consuming anomaly, namely... The system includes an automatic isolation mechanism for slow devices (Slow Drain) and an automatic recovery detection and release mechanism. These two mechanisms will be explained in detail below.

[0067] The automatic isolation mechanism for slow devices (Slow Drain) involves three aspects: credit depletion duration, impact quantification, and anomaly score assistance. Regarding credit depletion duration, a port is considered a primary slow device when its BB_Sc counter remains active for more than T_stall = 200ms. For impact quantification, the throughput decrease percentage Δ_TP of other ports within the same Trunk group is monitored. If Δ_TP > 30% and coincides with the aforementioned slow event within a time window of ±2s, it is considered a harmful slow device. For anomaly score assistance, the RRCF anomaly score is defined as S_a > 0.95.

[0068] The automatic isolation mechanism for slow devices (Slow Drain) triggers isolation actions when all three conditions mentioned above are met simultaneously. The isolation execution strategy is preferably implemented in hardware, specifically by calling the ASIC SDK interface, setting the isolation mode to HARD, and setting the credit ceiling parameter credit_ceiling=15 (normal value is 256). This forces the available credit announcement value for the affected egress port to be clamped. Simultaneously, the internal priority of the port's traffic is reduced to 0x0 and mapped to an isolated virtual channel (Quarantine_VC), which allocates only 8% of the total buffer resources.

[0069] The automatic recovery detection and de-isolation mechanism is defined in two aspects: detection period and recovery conditions. Regarding the detection period, an FC-NOP (no-operation) detection frame is sent every 5 seconds after isolation. Regarding the recovery conditions, the round-trip time (RTT) for N=10 consecutive probes (i.e., lasting 50 seconds) is defined as <20μs, and the credit depletion duration does not exceed 50ms again during this period. The preferred de-isolation action of this mechanism is to call the interface to de-isolate and restore the port traffic priority to the historical configuration value before isolation.

[0070] The following describes the state transition process of the automatic isolation mechanism for abnormal traffic using slow device isolation as a specific example.

[0071] Step S401: Determine whether to trigger isolation (slow root cause locking) based on the duration of the credit depletion state, the throughput of ports within the same trunk group, and the anomaly score. In this embodiment, the telemetry acquisition layer detected that the credit depletion state of port 24 had lasted for 350ms (exceeding the threshold of 200ms). Simultaneously, the throughput of port 23 within the same trunk group decreased from 32Gbps to 18Gbps, a decrease of 43.75% (exceeding the 30% threshold). The AI ​​analysis layer outputs an anomaly score S_a=0.98. With all three conditions met simultaneously, isolation is triggered.

[0072] Step S402: If isolation is triggered, perform hardware-forced isolation. The decision execution layer calls the ASICSDK interface to perform the isolation action.

[0073] As a preferred embodiment, the adaptive traffic optimization method for intelligent computing center storage area networks provided by the present invention further includes step S5: implementing end-to-end topology visualization for each traversed Fabric through a management presentation layer.

[0074] In this step, based on the OpenTracing semantic extension, a globally unique Fabric Trace ID is assigned to each I / O request traversing Fabric. A three-layer topology view is provided: a physical view (switch port - cable - storage front port), a logical view (partition-based access control relationships and performance group affiliation), and a service view (full path mapping from VM hostname to virtual HBA WWPN to switch port to storage volume), along with a real-time heatmap of current I / O latency and throughput along this path.

[0075] By implementing a closed-loop policy northbound interface through the management presentation layer—that is, providing a RESTful API—this invention allows upper-layer AI scheduling platforms (such as Kubernetes schedulers) to pre-request bandwidth resources and performance group acceleration policies from the SAN network based on the SLA level of the upcoming training task. Thus, this invention extends the monitoring granularity from the traditional port level to the virtual machine level, achieving end-to-end visualization of all connected devices and components (including virtual machines) within the SAN.

[0076] As another preferred implementation, the adaptive traffic optimization method for intelligent computing center storage area networks provided by this invention offers a RESTful API, allowing upper-layer AI scheduling platforms (such as Kubernetes schedulers) to pre-request bandwidth resources and performance group acceleration strategies from the SAN network based on the SLA level of the upcoming training task. This enables resource pre-configuration before the embedded model training task within the intelligent computing center storage area network starts. The core value of this technology lies in achieving proactive and intelligent scheduling of storage resources, transforming the storage network from a passive "pipeline" into an intelligent partner that can work collaboratively with the training task.

[0077] The adaptive traffic optimization method for intelligent computing center storage area networks provided by this invention, such as... Figure 5As shown, in a preferred implementation, the timeline collects five key actions from T+0ms to T+50s: telemetry acquisition, AI inference (S_a=0.97, P_c=0.82), hardware isolation (Credit clamped to 15), load balancing (shadow weight migration), and self-healing recovery (released after 10 consecutive RTT<20μs). This invention, through closed-loop management using embedded AI technology, enables FC switches to possess self-learning, self-adaptive, and self-optimizing capabilities, eliminating time-consuming manual processes. The automated traffic management and anomaly handling capabilities significantly reduce the frequency of manual intervention by network administrators, greatly lowering operation and maintenance costs.

[0078] This invention provides an adaptive traffic optimization system for a smart computing center storage area network. The adaptive traffic optimization system is built and operated based on hardware devices such as GPU servers, FC switches, NVMe storage arrays and SCSI slow devices. The FC switches are internally labeled with VC3 (congestion) and VC_Quarantine (isolation channel). The aforementioned embedded AI inference engine predicts congestion and determines and restores isolation. Figure 6 The connection relationships between the various hardware devices and the related data collected are shown.

[0079] Figure 7 The system architecture and key technologies used to implement each part of the system are illustrated. The adaptive traffic optimization system includes: a full-stack telemetry data acquisition subsystem for acquiring traffic data from the switch; an embedded AI inference engine deployed within the switch for receiving the traffic data and outputting analysis results, including an anomaly detection model and a time-series prediction model; and an adaptive traffic optimizer for performing dynamic load balancing when the analysis results of the time-series prediction model meet a first preset condition, and performing automatic isolation when the analysis results of the anomaly detection model meet a second preset condition.

[0080] The anomaly detection model is a robust random pruning forest model, and the temporal prediction model is a quantized compressed temporal convolutional network model. The anomaly detection model is used to perform real-time calculation on the feature vector of the streaming input and output anomaly scores. The temporal prediction model is used to infer the feature sequence within the time window and output the probability of congestion occurring within a preset time and the expected duration of congestion.

[0081] The adaptive traffic optimizer performs dynamic load balancing through the dynamic load balancing subsystem and automatic isolation through the abnormal traffic automatic isolation subsystem.

[0082] The dynamic load balancing subsystem specifically includes: a target performance group selection module, used to calculate the expected utility function of all candidate performance groups and select the non-congested performance group with the highest utility value as the target performance group; a migration cost calculation module, used to calculate the migration cost of the traffic source to be migrated from the current performance group to the target performance group; and a traffic migration module, used to migrate the traffic of the traffic source to be migrated to the target performance group without interruption by issuing a shadow weight update instruction to the switching ASIC when the migration cost is lower than a preset threshold.

[0083] The dynamic load balancing subsystem also includes a performance group splitting or reorganization module, which monitors the rate and protocol type of devices in each performance group. If the rate ratio of devices in a performance group exceeds a preset splitting threshold, or if there are multiple protocol types of traffic with a proportion exceeding a preset ratio, the performance group splitting or reorganization will be automatically triggered.

[0084] The abnormal traffic isolation subsystem specifically includes: an interface calling module for the switching ASIC, used to call the interface of the switching ASIC to clamp the available buffer credit value of the port where the abnormal traffic source is located to a preset low threshold when the abnormal traffic source is determined to be a slow device; and a mapping module, used to reduce the traffic priority of the abnormal traffic source to the lowest level and map it to a dedicated isolation virtual channel.

[0085] The abnormal traffic isolation subsystem also includes an isolation release module, which sends probe frames to the isolated abnormal traffic source at a preset period. If the round-trip delay of N consecutive probes is lower than the delay threshold and no credit exhaustion occurs again during the probe period, the isolation of the abnormal traffic source is released.

[0086] The full-stack telemetry data acquisition subsystem, while acquiring the traffic data, establishes a mapping relationship between virtual machines and traffic data through the interface integration between the data acquisition layer and the virtualization platform. Specifically, establishing this mapping relationship involves: embedding an extended hook in the switch's Fabric login service; upon receiving a login request from the host bus adapter, querying the virtualization platform to obtain the initiator's virtual machine UUID; establishing a triplet mapping relationship between the Fibre Channel address, global port name, and virtual machine UUID; and injecting this mapping relationship into the switch's hardware forwarding table to achieve virtual machine-level traffic tracing.

[0087] Figure 8This diagram illustrates the interaction process between various modules in an adaptive traffic optimization system for a smart computing center's storage area network. The vertical axis represents time, marking key time points (T+0ms, T+5ms, etc.). The horizontal axis consists of five "swimlanes," each corresponding to one of the five functional modules in the adaptive traffic optimization system. Arrows indicate the direction of interaction, with the transmitted data / signal content labeled next to each arrow. Dashed arrows indicate return or callback. Through the interaction of these modules, this invention provides true end-to-end visualization, extending monitoring granularity from the traditional port level to the virtual machine level, achieving end-to-end visualization of all connected devices and components (including virtual machines) within the SAN. It also enables "autonomous driving" of the SAN network, achieving self-learning, self-adaptation, and self-optimization capabilities for FC switches through closed-loop management using embedded AI technology, eliminating time-consuming manual processes.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive traffic optimization method for a smart computing center storage area network, characterized in that, Includes the following steps: Traffic data from switches in the storage area network is collected through a full-stack telemetry data acquisition layer. The collected traffic data is input into the embedded AI inference engine deployed in the switch for analysis. The embedded AI inference engine includes a series of anomaly detection models and time series prediction models. When the congestion prediction result output by the time-series prediction model meets the first preset condition, dynamic load balancing operation is performed through the adaptive traffic optimizer. When the anomaly score output by the anomaly detection model meets the second preset condition, the anomaly traffic source is automatically isolated through the anomaly traffic isolation subsystem.

2. The method according to claim 1, characterized in that, The anomaly detection model is a robust random pruning forest model, and the temporal prediction model is a quantized compressed temporal convolutional network model. The anomaly detection model is used to calculate the feature vector of the streaming input in real time and output anomaly score; The time-series prediction model is used to infer the feature sequence within a time window and output the probability of congestion occurring within a preset time period and the expected duration of congestion.

3. The method according to claim 1, characterized in that, The dynamic load balancing operation performed through the adaptive traffic optimizer specifically includes: Calculate the expected utility function for all candidate performance groups and select the non-congestion performance group with the highest utility value as the target performance group. Calculate the migration cost of migrating the traffic source to be migrated from the current performance group to the target performance group; If the migration cost is lower than a preset threshold, the traffic of the traffic source to be migrated will be migrated to the target performance group without interruption by issuing a shadow weight update instruction to the switching ASIC.

4. The method according to claim 3, characterized in that, Before performing dynamic load balancing operations via the adaptive traffic optimizer, the following steps are also included: Monitor the speed and protocol type of devices within each performance group; If the rate ratio of devices within a performance group exceeds a preset splitting threshold, or if multiple protocol types of traffic exist simultaneously with a proportion exceeding a preset ratio, the performance group will be automatically split or reassembled.

5. The method according to claim 1, characterized in that, The automatic isolation operation of abnormal traffic sources through the abnormal traffic isolation subsystem specifically includes: When the abnormal traffic source is determined to be a slow device, the interface of the switching ASIC is invoked to clamp the available buffer credit value of the port where the abnormal traffic source is located to a preset low threshold. The abnormal traffic source is given the lowest priority and mapped to a dedicated isolated virtual channel.

6. The method according to claim 5, characterized in that, After performing automatic isolation, an automatic recovery step is also included: Send probe frames to the isolated abnormal traffic source at a preset period; If the round-trip delay of N consecutive probes is lower than the delay threshold, and no credit depletion occurs again during the probe period, then the isolation of the abnormal traffic source is lifted.

7. The method according to claim 1, characterized in that, While collecting the traffic data, a mapping relationship between virtual machines and traffic data is established through the interface integration between the data acquisition layer and the virtualization platform.

8. The method according to claim 7, characterized in that, The establishment of the mapping relationship between virtual machines and traffic data specifically includes: An extended hook is implanted in the Fabric login service of the switch. When a login request is received from the host bus adapter, the virtualization platform is queried to obtain the virtual machine UUID of the initiator. Establish a triplet mapping relationship between Fibre Channel address, global port name and virtual machine UUID, and inject this mapping relationship into the switch's hardware forwarding table to achieve virtual machine-level traffic tracing.

9. An adaptive traffic optimization system for a smart computing center storage area network, characterized in that, include: The full-stack telemetry data acquisition subsystem is used to collect traffic data from the switch. An embedded AI inference engine, deployed within the switch, is used to receive the traffic data and output analysis results, including an anomaly detection model and a time-series prediction model. An adaptive traffic optimizer is used to perform dynamic load balancing operations when the analysis results of the time-series prediction model meet a first preset condition. The abnormal traffic isolation subsystem is used to perform automatic isolation operations when the analysis results of the abnormal detection model meet the second preset condition.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the adaptive traffic optimization method for the intelligent computing center storage area network as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the adaptive traffic optimization method for the intelligent computing center storage area network as described in any one of claims 1 to 8.

Images