Method and system for testing performance of an ethernet switch

Abstract

This invention discloses a performance testing method and system for Ethernet switches, relating to the field of switch testing technology. The method includes setting up an Ethernet switch test scenario, fixing Ethernet switch parameters, using a tester to collect multi-source data transmitted by the Ethernet switch, classifying data packets into complete packets and missing / abnormal packets, further dividing the data into normal packets and abnormal packets, locating and tracing the source of abnormal data, analyzing the influencing factors of abnormal data, and comprehensively evaluating the Ethernet switch test results. This invention overcomes the limitations of traditional testing methods that rely solely on normal data for evaluation. By incorporating missing and abnormal data into the performance analysis system, it marks missing and abnormal packets through integrity verification and identifies outliers. This not only distinguishes between occasional noise and genuine anomalies but also improves the efficiency of root cause localization of abnormal data through a hierarchical progressive method and temporal correlation analysis, thereby enhancing the realism of Ethernet switch performance testing.

Description

Technical Field

[0001] This invention relates to the field of switch testing technology, and in particular to a performance testing method and system for Ethernet switches. Background Technology

[0002] Ethernet switches are network devices used in Ethernet networks to perform data exchange functions. They operate at the data link layer of the OSI model, while some high-end Layer 3 switches can operate at the network layer. Their core function is to provide a dedicated electrical signal path for connected network devices, enabling efficient and stable data transmission between devices within a local area network.

[0003] Ethernet switch performance testing is used to assess the quality of an Ethernet switch. Typically, testing equipment inputs test data into the switch, directly collects the data, and identifies missing and abnormal data. While normal data is used to evaluate the switch's performance, missing and abnormal data can also reflect problems encountered during operation. Directly discarding missing and abnormal data affects the comprehensiveness of the performance evaluation. Without combining the root cause analysis results of the abnormal data for a comprehensive judgment, it's impossible to fully reflect the switch's true performance in complex scenarios, and it's difficult to provide targeted optimization suggestions, thus resulting in certain deficiencies. Summary of the Invention

[0004] The purpose of this invention is to provide a performance testing method and system for Ethernet switches to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a performance testing method for an Ethernet switch, comprising:

[0006] Step S1: Set up an Ethernet switch test scenario to make the Ethernet switch, tester, environment and time constitute a multi-dimensional test control group, and perform performance testing on the Ethernet switch based on parameter characteristics.

[0007] Step S2: Solidify the Ethernet switch parameters. The solidified parameter list includes core parameters such as IFG settings, frame length distribution, traffic model, packet transmission rate, test duration, and acquisition frequency. Perform pre-calibration on the test control group, record the calibration data, and then start the test.

[0008] Step S3: Input the solidified test parameters into the test control group, set the acquisition frequency, use the tester to acquire multi-source data transmitted by the Ethernet switch, and realize the time correlation of multi-source data by uniformly timestamping through the clock synchronization network to form a single acquisition data packet, and store the single acquisition data packet in the database;

[0009] Step S4: Store the single-collection data packet in the database. Before writing the data packet to the database, the integrity and format of the data packet are automatically checked. Data packets that pass the check are divided into complete data packets and missing or abnormal data packets.

[0010] Step S5: After separating the missing and abnormal data in the database, the 3σ criterion and DBSCAN clustering algorithm are used to identify outliers, divide the data into normal data packets and abnormal data packets, and evaluate the performance of the device based on the normal data packets;

[0011] Step S6: Locate and trace the source of abnormal data, analyze the influencing factors of abnormal data, package them into abnormal data packets, and combine the analysis results of normal data and abnormal data to comprehensively evaluate the test results of Ethernet switches.

[0012] Preferably, the multi-dimensional test control group in step S1 includes a single-dimensional test control group, a two-dimensional test control group, a three-dimensional test control group, and a key scenario test control group. The single-dimensional test control group is formed by changing one of the following dimensions: Ethernet switch dimension, tester dimension, environment dimension, and time dimension. The two-dimensional test control group is formed by changing two of the following dimensions: Ethernet switch dimension, tester dimension, environment dimension, and time dimension. The three-dimensional test control group is formed by changing three of the following dimensions: Ethernet switch dimension, tester dimension, environment dimension, and time dimension. The key scenario test control group is a separate test control group used to verify core scenarios, and the number of different test control groups is at least three.

[0013] Preferably, the curing test parameters in step S2 must not be adjusted during the test to ensure that the parameters are consistent with the test plan. The pre-calibration of the test control group in step S2 includes pre-calibration steps such as tester self-loop calibration, link stability verification, and temperature stability check.

[0014] The recorded standard data establishes an electronic signature for calibration data. All prior calibration data must be automatically recorded and generate an unalterable electronic signature. Formal testing can only begin after confirmation by the test manager. The calibration data is archived as a necessary attachment to the test report, making the testing process traceable.

[0015] The tester self-loop calibration involves looping each port of the tester through the optical module cable, sending line-speed traffic for testing, determining whether there is zero packet loss, whether the packet transmission rate error is less than or equal to 0.1%, and whether the delay fluctuation is less than or equal to 10ns, and recording the calibration data as the link reference value.

[0016] The link stability verification is performed by connecting all devices according to the actual test topology, sending test traffic with 10% load, and running the verification standard time. The requirements are that there is no packet loss on the link, the port negotiation rate is normal, and the optical power is within the normal range.

[0017] The temperature stability check is performed by powering on the Ethernet switch and running it under full load for a standard time. The temperature of the chassis inlet, outlet, and chip core is monitored. If the temperature fluctuation is less than or equal to 2°C within a continuous verification unit time, it is considered to be temperature stable, and the values ​​of each temperature point are recorded as a benchmark.

[0018] Preferably, the set acquisition frequency in step S3 is achieved by using different acquisition frequencies to balance acquisition accuracy and acquisition overhead. The importance of factors such as Ethernet switch dimension, tester dimension, environment dimension and time dimension is considered, and different acquisition frequencies are used in a weighted combination. The multi-source data acquisition includes end-to-end performance data from the tester side, internal operating data from the switch side and environmental parameters from the environmental monitoring system. The data is timestamped uniformly through a clock synchronization network, and time correlation is performed on data from different sources. Tags are automatically added to each piece of acquired data, including time tag, device tag, tester tag and environment tag, for subsequent data retrieval and analysis.

[0019] Preferably, the single-collection data packet stored in the database in step S4 contains three parts: timestamp, test control group label information, and collected values. Before data is written, integrity and format checks are automatically performed. If the format check fails, the format is converted and the data is rewritten. Data with valid format and complete data is packaged into a complete data packet. Data that fails the integrity check is marked as missing data and triggers data re-collection. If the re-collected data fails the integrity check again, it is marked as missing abnormal data, packaged into a missing abnormal data packet, and the missing data information is recorded.

[0020] Preferably, the data processing in the database in step S5 includes the following steps:

[0021] S5.1: First, separate and extract the missing abnormal data packets from the database;

[0022] S5.2: Based on the 3σ criterion and DBSCAN clustering algorithm, identify abnormal data points in the remaining data of the database and mark outliers that exceed the normal fluctuation range;

[0023] S5.3: Based on the preprocessing results, the data is divided into two categories: normal data and abnormal data. Data packets with outlier labels are separated. Data packets after separating missing abnormal data packets and data packets with outlier labels from the database are classified as normal data packets. Missing abnormal data packets and data packets with outlier labels are collectively referred to as abnormal data. Data with different sampling frequencies are aligned according to a unified time axis.

[0024] S5.4: Perform performance index analysis based on normal data packets, statistically analyze the positional statistics of various core indicators, and generate a performance baseline;

[0025] S5.5: Based on normal data packets, perform scenario comparison analysis, compare performance differences under different devices, different testers, different environments, and different times, and analyze the impact weight of each factor on performance;

[0026] S5.6: Perform trend analysis based on normal data packets to analyze the trend of performance indicators changing with time and load, and predict the trend of performance degradation under long-term operation;

[0027] S5.7: Based on normal data packets, perform benchmarking analysis, compare the test results with the manufacturer's claimed values, industry standards, and the performance of similar products, and evaluate the equipment performance level.

[0028] Preferably, the abnormal data influencing factors in step S6 include equipment malfunctions, testing tool malfunctions, environmental factors malfunctions, and human error malfunctions.

[0029] The device malfunction is caused by a hardware failure, software bug, or configuration error in the switch, resulting in performance abnormalities.

[0030] The abnormal test tool was caused by tester malfunction, optical module compatibility issues, or clock synchronization errors, resulting in abnormal data.

[0031] The aforementioned environmental factors are due to performance abnormalities caused by excessively high temperatures, power fluctuations, and electromagnetic interference.

[0032] The aforementioned human error is abnormal data caused by incorrect test configuration, improper parameter settings, or non-standard operation.

[0033] Preferably, the abnormal data location and source tracing in step S6 is performed by combining the hierarchical progressive method and the time-series correlation analysis method to locate the root cause of the abnormality.

[0034] The layered progressive method is based on the abnormality of the equipment itself, the abnormality of the testing tool, the abnormality of environmental factors, and the abnormality of human operation. Abnormal data is divided into the root cause location layer in sequence: data layer verification layer, configuration layer check layer, environmental layer investigation layer, tool layer verification layer, and equipment layer diagnosis layer. The data layer verification layer compares the statistical data of the tester and the switch to determine whether the abnormality occurs inside the switch. The configuration layer check layer checks whether the switch configuration, tester configuration, and topology connection are correct and eliminates configuration errors. The environmental layer investigation layer checks whether environmental parameters such as temperature, humidity, and power supply are within the normal range and eliminates environmental factors. The tool layer verification layer replaces the tester port, optical module, and cable for cross-verification to eliminate test tool problems. The equipment layer diagnosis layer collects switch logs, core dump files, and chip counter information to locate the fault point of the equipment itself.

[0035] The time-series correlation analysis method automatically locates the root cause by calculating the time-series correlation between performance anomalies and the root cause layer. The combination of hierarchical progressive method and time-series correlation analysis for root cause location includes the following steps:

[0036] S6.11: The sliding window Pearson correlation coefficient algorithm is used to calculate the temporal correlation between the abnormal indicators and the root cause layer, and a correlation threshold is set to compare the calculated correlation coefficient and the correlation threshold.

[0037] S6.12: When the temporal correlation coefficients of the root cause layer are all less than the correlation threshold, the abnormal root cause is automatically located according to the hierarchical progressive method until the abnormal root cause is identified, and then the automatic location and identification stops.

[0038] S6.13: When the correlation coefficient is greater than or equal to the correlation threshold, the lag time of each root cause layer is output simultaneously to provide priority sorting for subsequent investigation. The abnormal root cause is automatically located according to the priority sorting. When an abnormal root cause is identified, the identification of the subsequent root cause layers is stopped. After the root cause layers with correlation coefficients greater than or equal to the correlation threshold are identified, and no abnormal root cause is identified, the automatic location identification is performed according to the hierarchical progressive method, and the root cause layers with temporal correlation coefficients greater than the correlation threshold are excluded.

[0039] Preferably, the comprehensive evaluation of the Ethernet switch test results in step S6 includes the following steps:

[0040] S6.21: Based on the design parameter characteristics of Ethernet switches, assign weights to the parameter characteristics of normal and abnormal data, and give a comprehensive score to each characteristic according to the ratio of the actual test value to the nominal value;

[0041] S6.22: Set a passing threshold. When the overall score is greater than or equal to the passing threshold, the Ethernet switch performance test is qualified. Conversely, when the overall score is less than the passing threshold, the Ethernet switch performance test is unqualified.

[0042] S6.23: Set a veto option. If the Ethernet switch has a packet loss rate greater than 0.1%, restarts during stability testing, or fails critical functions, the Ethernet switch will be directly judged as failing the performance test. This option takes precedence over the overall score.

[0043] S6.24: Output a standardized report, which includes a test overview, test plan, test results, anomaly analysis, and comprehensive evaluation;

[0044] S6.25: Output optimization suggestions based on test results. The optimization suggestions include hardware optimization suggestions, software optimization suggestions, configuration optimization suggestions, and test improvement suggestions.

[0045] Another objective of this invention is to provide a performance testing system for Ethernet switches, comprising:

[0046] The test benchmark calibration module is used for automatic calibration by the tester, link stability verification, and generation of tamper-proof electronic signatures for calibration data.

[0047] The multi-dimensional test scenario management module is used to build four types of test control groups: single-dimensional, dual-dimensional, three-dimensional, and key scenarios, covering combinations of Ethernet switches, test instruments, environment, and time variables.

[0048] The multi-source data acquisition and storage module is used to synchronously acquire data and inject time, device, test instrument, and environmental tags to save the data;

[0049] The intelligent data processing and analysis module is used to analyze normal data and trace the root causes of abnormal data, using a hierarchical progressive method and time-series correlation analysis to locate the root causes.

[0050] The comprehensive evaluation and report generation module is used to comprehensively evaluate Ethernet switch tests and generate test reports.

[0051] The technical effects and advantages of this invention are as follows:

[0052] (1) This invention breaks through the limitations of traditional testing methods that rely solely on normal data for evaluation. It incorporates missing and abnormal data into the performance analysis system, marks missing and abnormal data packets through integrity verification, and identifies abnormal values. It not only distinguishes between occasional noise and real anomalies, but also improves the efficiency of abnormal data root cause localization through hierarchical progressive method and time-series correlation analysis method, thereby improving the authenticity of Ethernet switch performance testing.

[0053] (2) This invention constructs a multi-dimensional test control group that includes Ethernet switches, testers, environment and time, covering a variety of variable combinations, eliminating individual differences and external interference, ensuring the universality of test results, and simultaneously collecting end-to-end performance data of the tester, internal operation data of the switch and environmental monitoring data, and uniformly timestamps and injects tags through a clock synchronization network to achieve time correlation and retrieval of cross-source data;

[0054] (3) This invention establishes a weighted scoring and veto system to give weighted scores to core indicators such as throughput, packet loss rate, latency, back-to-back frame capability and stability. At the same time, it sets a veto item and combines the root cause analysis results of abnormal data so that the evaluation results not only reflect the nominal performance of the equipment, but also reflect the real operating capability in complex scenarios. Attached Figure Description

[0055] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention, but do not constitute a limitation thereof. In the drawings:

[0056] Figure 1 This is a flowchart of the Ethernet switch performance testing method of the present invention;

[0057] Figure 2 This is a flowchart of the missing abnormal data judgment process of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] This invention provides, for example Figures 1-2 The method for testing the performance of an Ethernet switch, as shown, includes:

[0060] Step S1: Set up an Ethernet switch test scenario to form a multi-dimensional test control group consisting of Ethernet switches, testers, environment, and time. Perform performance tests on the Ethernet switches based on parameter characteristics. The parameter characteristics of the Ethernet switch test include throughput, packet loss rate, latency, back-to-back frames, and stability. The design parameters of the Ethernet switch include hardware parameters, software parameters, and core parameters of rated indicators. Hardware parameters include the switching chip model, buffer capacity, port speed type, and power module specifications. Software parameters include the operating system version, forwarding table capacity, supported protocol stacks, and QoS scheduling policies. Rated indicators include nominal throughput, maximum power consumption, and operating temperature range. All collected parameters need to be cross-verified with the manufacturer's datasheet, hardware BOM list, and software version description. If the error rate exceeds 1%, it needs to be reconfirmed to ensure that the test benchmark data is 100% accurate.

[0061] Step S2: Solidify the Ethernet switch parameters. The solidified parameter list includes core parameters such as IFG settings, frame length distribution, traffic model, packet transmission rate, test duration, and acquisition frequency. Perform pre-calibration on the test control group, record the calibration data, and then start the test.

[0062] Step S3: Input the solidified test parameters into the test control group, set the acquisition frequency, use the tester to acquire multi-source data transmitted by the Ethernet switch, and realize the time correlation of multi-source data by uniformly timestamping through the clock synchronization network to form a single acquisition data packet, and store the single acquisition data packet in the database;

[0063] Step S4: Store the single-collection data packet in the database. Before writing the data packet to the database, the integrity and format of the data packet are automatically checked. Data packets that pass the check are divided into complete data packets and missing or abnormal data packets.

[0064] Step S5: After separating the missing and abnormal data in the database, the 3σ criterion and DBSCAN clustering algorithm are used to identify outliers, divide the data into normal data packets and abnormal data packets, and evaluate the performance of the device based on the normal data packets;

[0065] Step S6: Locate and trace the source of abnormal data, analyze the influencing factors of abnormal data, package them into abnormal data packets, and combine the analysis results of normal data and abnormal data to comprehensively evaluate the test results of Ethernet switches.

[0066] Furthermore, the multi-dimensional test control group in step S1 includes a single-dimensional test control group, a two-dimensional test control group, a three-dimensional test control group, and a key scenario test control group. The equipment dimension includes Ethernet switches from different manufacturers, models, hardware versions, and software versions. At least three samples of the same model are selected for parallel testing to eliminate the influence of individual differences. The testing instrument dimension includes cross-validation using at least two different brands of testing instruments, including mainstream commercial testing instruments and self-developed testing platforms. Data is considered valid only if the test result deviation is less than 2%. The environmental dimension covers four typical environments: ambient temperature (25℃), low temperature (0℃), high temperature (40℃), and high humidity (90%RH), while also including simulations of special scenarios such as electromagnetic interference and power fluctuations. The time dimension includes tests conducted at the initial power-on stage of the device. Repeated tests were conducted over three time periods: -1 hour, stable operation period (1-24 hours), and long-term operation period (24-72 hours) to capture performance changes over time. The single-dimensional test control group was formed by changing one of the following dimensions: Ethernet switch, tester, environment, and time. The two-dimensional test control group was formed by changing two of these dimensions. The three-dimensional test control group was formed by changing three of these dimensions. The critical scenario test control group was a separate, specialized test control group used to verify core scenarios, and at least three different test control groups were required. Performance testing of Ethernet switches included:

[0067] Throughput testing covers the full frame length range of 64B-1518B, including IMIX mixed frame length scenarios, and tests the line-rate forwarding capability under different loads, namely, continuous forwarding stability under 30%, 50%, 80%, and 100% load respectively.

[0068] Packet loss rate testing involves continuous testing under different loads, and statistical analysis of the total packet loss rate and packet loss distribution by time period. The percentage of packet loss is calculated separately for different types of packet loss: congestion packet loss, error packet loss, and silent packet loss.

[0069] Latency testing uses hardware timestamp technology to measure key indicators such as end-to-end average latency, latency percentile, and latency jitter, enabling flow-by-flow latency statistics and pinpointing latency differences between different service flows.

[0070] Back-to-back frame testing uses the shortest IFG to continuously send burst frames of the maximum length to test the maximum capacity of the Ethernet switch's buffer, and introduces a buffer exhaustion recovery time metric to evaluate the service recovery speed after buffer overflow.

[0071] Stability testing involves running the system continuously at full load for one cycle, such as 72 hours, and collecting full performance metrics every unit of time, such as every hour. An abnormal fluctuation tolerance assessment is also included, meaning that if the fluctuation range of the metrics exceeds the fluctuation error value, the system is deemed to be unstable.

[0072] In particular, the fixed test parameters in step S2 must not be adjusted during the test to ensure that the parameters are consistent with the test plan. The IFG is set to 96-bit time, and the IMIX standard for frame length distribution is: 64B 7%, 128B 8%, 256B 12%, 512B 20%, 1024B 25%, and 1518B 28%. The traffic model is 60% TCP, 30% UDP, and 10% other protocols. The pre-calibration of the test control group in step S2 includes tester self-loop calibration, link stability verification, and temperature stability check.

[0073] Recording standard data is essential for establishing an electronic signature for calibration data. All prior calibration data must be automatically recorded and generate an unalterable electronic signature. Formal testing can only begin after confirmation by the test manager. Calibration data is archived as a necessary attachment to the test report, ensuring the traceability of the testing process.

[0074] The tester's loopback calibration involves looping each port of the tester through the optical module cable and sending line-speed traffic for testing. The tester determines whether there is zero packet loss, whether the packet transmission rate error is less than or equal to 0.1%, and whether the latency fluctuation is less than or equal to 10ns. If the test results meet the conditions of zero packet loss, packet transmission rate error less than or equal to 0.1%, and latency fluctuation less than or equal to 10ns, the tester's loopback calibration is qualified. If any of the conditions are not met, the tester's loopback calibration is unqualified, and the calibration data is recorded as the link reference value.

[0075] Link stability verification involves connecting all devices according to the actual test topology, sending test traffic with 10% load, and running the verification standard time. The requirements are no packet loss, normal port negotiation rate, and optical power within the normal range. The normal range of optical power is received optical power greater than or equal to -12dBm and optical module temperature between 0-70℃.

[0076] Temperature stability is checked by powering on the Ethernet switch and running it under full load for a standard time, such as 30 minutes, monitoring the temperature at the inlet and outlet of the chassis and the core temperature of the chip. If the temperature fluctuation is less than or equal to 2°C within a continuous verification unit time, it is considered to be temperature stable. The unit time is such as 5 minutes, and the values ​​of each temperature point are recorded as a baseline.

[0077] Furthermore, in step S3, setting the sampling frequency involves using different sampling frequencies to balance sampling accuracy and overhead. Considering the importance of factors such as Ethernet switch dimensions, test instrument dimensions, environmental dimensions, and time dimensions, different sampling frequencies are used in a weighted combination. For example, basic performance indicators such as throughput, packet loss rate, and average latency use a 1-second sampling frequency to ensure the continuity of macroscopic performance data, while key abnormal indicators such as packet loss events, latency spikes, and queue congestion are captured with microsecond-level high precision to detect instantaneous abnormal events. Internal device metrics such as CPU utilization, memory usage, and cache queue depth are sampled at a frequency of 10 seconds to monitor device operating status. Environmental parameters such as temperature, humidity, power supply voltage, and magnetic field are sampled at a frequency of 1 minute to record environmental changes. Multi-source data collection includes end-to-end performance data from the test instrument, internal operating data from the switch, and environmental parameters from the environmental monitoring system. Data is timestamped uniformly through a clock synchronization network with a time synchronization accuracy better than 10ns. Data from different sources is correlated by time, and each piece of collected data is automatically tagged with time tags, device tags, test instrument tags, and environmental tags for subsequent data retrieval and analysis.

[0078] Furthermore, the single-collection data packet stored in the database in step S4 contains three parts: timestamp, test and control group label information, and collected values. Before data is written, integrity and format checks are automatically performed. If the format check fails, the format is converted and the data is rewritten. Data with valid format and complete data is packaged into a complete data packet. Data that fails the integrity check is marked as missing data, and data is re-collected. If the re-collected data fails the integrity check again, it is marked as missing abnormal data, packaged into a missing abnormal data packet, and the missing data information is recorded to distinguish between complete data and missing data, which facilitates subsequent classification and processing of different types of data.

[0079] In particular, the data processing in the database in step S5 includes the following steps:

[0080] S5.1: First, separate and extract the missing abnormal data packets from the database;

[0081] S5.2: Based on the 3σ criterion and the DBSCAN clustering algorithm, outlier data points are identified in the remaining data of the database, and outliers exceeding the normal fluctuation range are marked. The 3σ criterion is an outlier detection method based on the normal distribution assumption. In a normal distribution, 99.73% of the data will fall within the range of mean ± 3 standard deviations, and only 0.27% of the data will exceed this range. Therefore, data exceeding this range are judged as outliers. For measured values, To calculate the sample mean, the sample size must be greater than or equal to 30. Calculate the sample standard deviation when When established, the judgment is made. These are outliers; the threshold can be adjusted based on the severity of the scenario. High-reliability scenarios should use [a more stringent threshold]. With a confidence level of 95.45%, typical scenarios were adopted. DBSCAN clustering algorithm is a density-based clustering algorithm that divides high-density connected regions in the data space into clusters and identifies points in low-density regions as noise points, i.e., outliers. The core of the algorithm is based on the neighborhood radius. and minimum sample size Two parameters are used to identify core points, boundary points, and noise points. No pre-specified number of clusters is required, and clusters of arbitrary shapes can be detected. This makes it suitable for anomaly detection in multi-dimensional performance data. The calculation formula is as follows:

[0082]

[0083] The domain is based on data points Center, radius The set of all points within the range, where For the dataset, point and points The distance between them, if The number of samples in the neighborhood satisfies ,but The core point is the boundary point, which is not a core point but falls on some core point. Points within the neighborhood that are neither core points nor boundary points are considered outliers.

[0084] S5.3: Based on the preprocessing results, the data is divided into two categories: normal data and abnormal data. Normal data refers to valid data that meets the threshold range of the indicator and fluctuates within a reasonable range. Abnormal data refers to data points that exceed the threshold and deviate significantly from the overall trend. When a single data point is abnormal but the data before and after it is normal, it is judged as occasional noise and included in the normal dataset. When three or more consecutive data points are abnormal, it is judged as real abnormality and included in the abnormal dataset. Data packets with outlier labels are separated. Data packets after separating missing abnormal data packets and data packets with outlier labels from the database are classified as normal data packets. Missing abnormal data packets and data packets with outlier labels are collectively referred to as abnormal data. Data with different sampling frequencies are aligned according to a unified time axis.

[0085] S5.4: Perform performance index analysis based on normal data packets, and statistically analyze the positional statistics of various core indicators, namely the average, maximum, minimum, and quantile values ​​of core indicators such as throughput, packet loss rate, latency, and jitter, to generate a performance baseline.

[0086] S5.5: Based on normal data packets, perform scenario comparison analysis, compare performance differences under different devices, different testers, different environments, and different times, and analyze the impact weight of each factor on performance;

[0087] S5.6: Perform trend analysis based on normal data packets to analyze the trend of performance indicators changing with time and load, and predict the trend of performance degradation under long-term operation;

[0088] S5.7: Based on normal data packets, perform benchmarking analysis, compare the test results with the manufacturer's claimed values, industry standards, and the performance of similar products, and evaluate the equipment performance level.

[0089] Specifically, the factors affecting abnormal data in step S6 include equipment malfunction, testing tool malfunction, environmental factor malfunction, and human operation malfunction.

[0090] Device malfunctions are caused by hardware failures, software bugs, or configuration errors in the switch, such as chip failures, memory leaks, or protocol processing errors.

[0091] Test tool malfunctions are caused by tester failures, optical module compatibility issues, or clock synchronization errors, resulting in abnormal data such as insufficient packet transmission rate of the tester or excessive timestamp errors.

[0092] Abnormal environmental factors are performance abnormalities caused by excessively high temperatures, power fluctuations, and electromagnetic interference, such as chip performance degradation at high temperatures and bit errors caused by excessive power ripple.

[0093] Human error is caused by abnormal data due to test configuration errors, improper parameter settings, or non-standard operation, such as incorrect IFG settings or incorrect traffic model configuration.

[0094] In particular, the method of locating and tracing the source of abnormal data in step S6 is to use a combination of hierarchical progressive method and time series correlation analysis method to locate the root cause of abnormal data. The combination of hierarchical progressive method and time series correlation analysis method can improve the efficiency of locating the root cause of abnormal data.

[0095] The layered progressive method is based on the abnormality of the equipment itself, the test tool, the environmental factors, and the human operation. Abnormal data is divided into the root cause layer of data verification, configuration check, environmental investigation, tool verification, and equipment diagnosis. The data verification layer compares the statistical data of the tester and the switch to determine whether the abnormality occurs inside the switch. The configuration check layer checks whether the switch configuration, tester configuration, and topology connection are correct to eliminate configuration errors. The environmental investigation layer checks whether environmental parameters such as temperature, humidity, and power supply are within the normal range to eliminate environmental factors. The tool verification layer replaces the tester port, optical module, and cable for cross-verification to eliminate test tool problems. The equipment diagnosis layer collects switch logs, core dump files, and chip counter information to locate the fault point of the equipment itself.

[0096] Temporal correlation analysis automatically locates the root cause by calculating the temporal correlation between performance anomalies and the root cause layer. The combination of hierarchical progressive methods and temporal correlation analysis for root cause localization includes the following steps:

[0097] S6.11: The sliding window Pearson correlation coefficient algorithm is adopted. This algorithm is a time-series optimized version of the traditional Pearson correlation coefficient. By sliding a fixed-size time window across the time series, it dynamically calculates the local correlation between two time-series variables within the window. This captures the dynamic changes in correlation within time-series data, making it suitable for locating the time points of abnormal events in switch testing and identifying transient, localized correlations. It overcomes the limitation of the global Pearson correlation coefficient in reflecting local time-series correlations. The algorithm calculates the time-series correlation between abnormal indicators and the root cause layer, sets a correlation threshold (which can be 0.7), and compares the calculated correlation coefficient with the correlation threshold. The calculation formula is as follows:

[0098]

[0099] in, This represents the correlation coefficient between abnormal indicators and root causes within a time window, with values ​​ranging from -1 to 1. A larger absolute value indicates a stronger correlation. For each time point within the window, the abnormal indicator value is... This represents the average value of the abnormal indicators within the window. For each time point within the window, the root cause value is... The average value of the root factors within the window when If so, it is determined to be a strong correlation, and it is highly suspected to be the root cause;

[0100] S6.12: When the temporal correlation coefficients of the root cause layer are all less than the correlation threshold, the abnormal root cause is automatically located according to the hierarchical progressive method until the abnormal root cause is identified, and then the automatic location and identification stops.

[0101] S6.13: When the correlation coefficient is greater than or equal to the correlation threshold, the lag time of each root cause layer is output simultaneously to provide priority sorting for subsequent investigation. The abnormal root cause is automatically located according to the priority sorting. When an abnormal root cause is identified, the identification of the subsequent root cause layers is stopped. After the root cause layers with correlation coefficients greater than or equal to the correlation threshold are identified, and no abnormal root cause is identified, the automatic location identification is performed according to the hierarchical progressive method, and the root cause layers with temporal correlation coefficients greater than the correlation threshold are excluded.

[0102] Furthermore, step S6, which involves comprehensively evaluating the Ethernet switch test results, includes the following steps:

[0103] S6.21: Based on the design parameter characteristics of Ethernet switches, assign weights to the evaluation parameter characteristics of normal and abnormal data. Each characteristic is comprehensively scored according to the ratio of the actual test value to the nominal value, such as throughput of 30%, packet loss rate of 25%, latency of 20%, back-to-back frame capability of 15%, and stability of 10%.

[0104] S6.22: Set a passing threshold. When the overall score is greater than or equal to the passing threshold, the overall score ranges from 0 to 100. The passing threshold can be set to 70 points. Then the Ethernet switch performance test is qualified. The higher the overall score, the better the Ethernet switch performance test. Conversely, if the overall score is less than the passing threshold, the Ethernet switch performance test is unqualified. For example, an overall score of 90 points or above is excellent, 80-89 points is good, 70-79 points is qualified, and below 70 points is unqualified.

[0105] S6.23: Set a veto option. If the Ethernet switch has a packet loss rate greater than 0.1%, restarts during stability testing, or fails critical functions, the Ethernet switch will be directly judged as failing the performance test. This option takes precedence over the overall score.

[0106] S6.24: Output a standardized report. The standardized report includes a test overview, test plan, test results, anomaly analysis, and comprehensive evaluation. The test overview includes the test purpose, test scope, reference standards, and test environment description. The test plan includes the test topology, test case design, and test parameter configuration. The test results include detailed test data, performance curves, and comparative analysis results for each test case. The anomaly analysis includes root cause analysis and impact assessment for all anomalies. The comprehensive evaluation includes the overall score, level determination, and compliance conclusion.

[0107] S6.25: Output optimization suggestions based on test results. These suggestions include hardware optimization suggestions, software optimization suggestions, configuration optimization suggestions, and test improvement suggestions. Hardware optimization suggestions address hardware bottlenecks, such as increasing cache capacity, optimizing heat dissipation design, and replacing with higher-performance chips. Software optimization suggestions address software issues, such as optimizing forwarding algorithms, fixing protocol bugs, and improving cache scheduling strategies. Configuration optimization suggestions provide configuration optimization guidelines for actual user deployment scenarios, such as adjusting buffer parameters, optimizing QoS policies, and providing port configuration suggestions. Test improvement suggestions address issues discovered during testing and propose optimization solutions for test methods, test tools, and test processes.

[0108] Another objective of this invention is to provide a performance testing system for Ethernet switches, including a test benchmark calibration module, a multi-dimensional test scenario management module, a multi-source data acquisition and storage module, an intelligent data processing and analysis module, and a comprehensive evaluation and report generation module. The test benchmark calibration module is used to automatically calibrate the test instrument, verify link stability, and generate an immutable electronic signature for the calibration data, providing a traceable and accurate benchmark for formal testing and ensuring the reliability of test data from the source. The multi-dimensional test scenario management module is used to construct four types of test control groups: single-dimensional, dual-dimensional, three-dimensional, and key scenarios, covering combinations of Ethernet switches, test instruments, environment, and time variables. This achieves full-scenario coverage under different loads, environments, and equipment conditions, ensuring the comprehensiveness and universality of test results. The multi-source data acquisition and storage module is used to synchronously acquire data and inject time, device, test instrument, and environment tags to save the data. To ensure data retrievability, the system also features data integrity verification and format conversion capabilities. An automatic re-sampling mechanism is triggered for abnormal data, ultimately storing standardized data in the database. The intelligent data processing and analysis module analyzes normal data and traces the root causes of abnormal data. It employs a hierarchical progressive method and temporal correlation analysis to locate the root cause, first checking the data layer, then the configuration layer, then the environment layer, then the tool layer, and finally the equipment layer. It then uses a sliding window Pearson correlation coefficient to calculate the temporal correlation between the anomaly and the root cause, quickly locating four types of abnormal factors: equipment hardware failure, tester error, environmental interference, and human error. The comprehensive evaluation and report generation module performs a comprehensive evaluation of Ethernet switch testing and generates a test report. The test benchmark calibration module, multi-dimensional test scenario management module, multi-source data acquisition and storage module, intelligent data processing and analysis module, and comprehensive evaluation and report generation module are connected.

[0109] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A performance testing method for an Ethernet switch, characterized in that: The specific steps include the following: Step S1: Set up an Ethernet switch test scenario to make the Ethernet switch, tester, environment and time constitute a multi-dimensional test control group, and perform performance testing on the Ethernet switch based on parameter characteristics. Step S2: Solidify the Ethernet switch parameters. The solidified parameter list includes core parameters such as IFG settings, frame length distribution, traffic model, packet transmission rate, test duration, and acquisition frequency. Perform pre-calibration on the test control group, record the calibration data, and then start the test. Step S3: Input the solidified test parameters into the test control group, set the acquisition frequency, use the tester to acquire multi-source data transmitted by the Ethernet switch, and realize the time correlation of multi-source data by uniformly timestamping through the clock synchronization network to form a single acquisition data packet, and store the single acquisition data packet in the database; Step S4: Store the single-collection data packet in the database. Before writing the data packet to the database, the integrity and format of the data packet are automatically checked. Data packets that pass the check are divided into complete data packets and missing or abnormal data packets. Step S5: After separating the missing and abnormal data in the database, the 3σ criterion and DBSCAN clustering algorithm are used to identify outliers, divide the data into normal data packets and abnormal data packets, and evaluate the performance of the device based on the normal data packets; Step S6: Locate and trace the source of abnormal data, analyze the influencing factors of abnormal data, package them into abnormal data packets, and combine the analysis results of normal data and abnormal data to comprehensively evaluate the test results of Ethernet switches.

2. The performance testing method for an Ethernet switch according to claim 1, characterized in that: The multi-dimensional test control group in step S1 includes a single-dimensional test control group, a two-dimensional test control group, a three-dimensional test control group, and a key scenario test control group. The single-dimensional test control group is formed by changing one of the following dimensions: Ethernet switch dimension, test instrument dimension, environment dimension, and time dimension. The two-dimensional test control group is formed by changing two of the following dimensions: Ethernet switch dimension, test instrument dimension, environment dimension, and time dimension. The three-dimensional test control group is formed by changing three of the following dimensions: Ethernet switch dimension, test instrument dimension, environment dimension, and time dimension. The key scenario test control group is a separate test control group designed to verify core scenarios, and the number of different test control groups is at least three.

3. The performance testing method for an Ethernet switch according to claim 1, characterized in that: The curing test parameters in step S2 must not be adjusted during the test, and the parameters must be consistent with the test plan. The pre-calibration of the test control group in step S2 includes the pre-calibration steps of tester self-loop calibration, link stability verification, and temperature stability check. The recorded standard data establishes an electronic signature for calibration data. All prior calibration data must be automatically recorded and generate an unalterable electronic signature. Formal testing can only begin after confirmation by the test manager. The calibration data is archived as a necessary attachment to the test report, making the testing process traceable. The tester self-loop calibration involves looping each port of the tester through the optical module cable, sending line-speed traffic for testing, determining whether there is zero packet loss, whether the packet transmission rate error is less than or equal to 0.1%, and whether the delay fluctuation is less than or equal to 10ns, and recording the calibration data as the link reference value. The link stability verification is performed by connecting all devices according to the actual test topology, sending test traffic with 10% load, and running the verification standard time. The requirements are that there is no packet loss on the link, the port negotiation rate is normal, and the optical power is within the normal range. The temperature stability check is performed by powering on the Ethernet switch and running it under full load for a standard time. The temperature of the chassis inlet, outlet, and chip core is monitored. If the temperature fluctuation is less than or equal to 2°C within a continuous verification unit time, it is considered to be temperature stable, and the values ​​of each temperature point are recorded as a benchmark.

4. The performance testing method for an Ethernet switch according to claim 1, characterized in that: The setting of the acquisition frequency in step S3 involves using different acquisition frequencies to balance acquisition accuracy and acquisition overhead. The importance of factors such as Ethernet switch, tester, environment, and time is considered, and different acquisition frequencies are used in a weighted combination. The acquisition of multi-source data includes end-to-end performance data from the tester, internal operating data from the switch, and environmental parameters from the environmental monitoring system. The data is timestamped uniformly through a clock synchronization network, and time correlation is performed on data from different sources. Tags are automatically added to each piece of acquired data, including time tags, device tags, tester tags, and environment tags, for subsequent data retrieval and analysis.

5. The performance testing method for an Ethernet switch according to claim 4, characterized in that: The single-collection data packet stored in the database in step S4 contains three parts: timestamp, test control group label information, and collected values. Before the data is written, integrity and format checks are automatically performed. If the format check fails, the format is converted and the data is rewritten. Data with a valid format and complete data are packaged into a complete data packet. Data that fails the integrity check is marked as missing data and triggers data re-collection. If the re-collected data fails the integrity check again, it is marked as missing abnormal data, packaged into a missing abnormal data packet, and the missing data information is recorded.

6. The performance testing method for an Ethernet switch according to claim 5, characterized in that: The data processing in the database in step S5 includes the following steps: S5.1: First, separate and extract the missing abnormal data packets from the database; S5.2: Based on the 3σ criterion and DBSCAN clustering algorithm, identify abnormal data points in the remaining data of the database and mark outliers that exceed the normal fluctuation range; S5.3: Based on the preprocessing results, the data is divided into two categories: normal data and abnormal data. Data packets with outlier labels are separated. Data packets after separating missing abnormal data packets and data packets with outlier labels from the database are classified as normal data packets. Missing abnormal data packets and data packets with outlier labels are collectively referred to as abnormal data. Data with different sampling frequencies are aligned according to a unified time axis. S5.4: Perform performance index analysis based on normal data packets, statistically analyze the positional statistics of various core indicators, and generate a performance baseline; S5.5: Based on normal data packets, perform scenario comparison analysis, compare performance differences under different devices, different testers, different environments, and different times, and analyze the impact weight of each factor on performance; S5.6: Perform trend analysis based on normal data packets to analyze the trend of performance indicators changing with time and load, and predict the trend of performance degradation under long-term operation; S5.7: Based on normal data packets, perform benchmarking analysis, compare the test results with the manufacturer's claimed values, industry standards, and the performance of similar products, and evaluate the equipment performance level.

7. The performance testing method for an Ethernet switch according to claim 6, characterized in that: The abnormal data in step S6 includes abnormal equipment, abnormal testing tools, abnormal environmental factors, and abnormal human operation. The device malfunction is caused by a hardware failure, software bug, or configuration error in the switch, resulting in performance abnormalities. The abnormal test tool was caused by tester malfunction, optical module compatibility issues, or clock synchronization errors, resulting in abnormal data. The aforementioned environmental factors are due to performance abnormalities caused by excessively high temperatures, power fluctuations, and electromagnetic interference. The aforementioned human error is abnormal data caused by incorrect test configuration, improper parameter settings, or non-standard operation.

8. The performance testing method for an Ethernet switch according to claim 7, characterized in that: The abnormal data location and source tracing in step S6 is performed by combining the hierarchical progressive method and the time-series correlation analysis method to locate the root cause of the abnormality. The layered progressive method is based on the abnormality of the equipment itself, the abnormality of the testing tool, the abnormality of environmental factors, and the abnormality of human operation. Abnormal data is divided into the root cause location layer in sequence: data layer verification layer, configuration layer check layer, environmental layer investigation layer, tool layer verification layer, and equipment layer diagnosis layer. The data layer verification layer compares the statistical data of the tester and the switch to determine whether the abnormality occurs inside the switch. The configuration layer check layer checks whether the switch configuration, tester configuration, and topology connection are correct and eliminates configuration errors. The environmental layer investigation layer checks whether environmental parameters such as temperature, humidity, and power supply are within the normal range and eliminates environmental factors. The tool layer verification layer replaces the tester port, optical module, and cable for cross-verification to eliminate test tool problems. The equipment layer diagnosis layer collects switch logs, core dump files, and chip counter information to locate the fault point of the equipment itself. The time-series correlation analysis method automatically locates the root cause by calculating the time-series correlation between performance anomalies and the root cause layer. The combination of hierarchical progressive method and time-series correlation analysis for root cause location includes the following steps: S6.11: The sliding window Pearson correlation coefficient algorithm is used to calculate the temporal correlation between the abnormal indicators and the root cause layer, and a correlation threshold is set to compare the calculated correlation coefficient and the correlation threshold. S6.12: When the temporal correlation coefficients of the root cause layer are all less than the correlation threshold, the abnormal root cause is automatically located according to the hierarchical progressive method until the abnormal root cause is identified, and then the automatic location and identification stops. S6.13: When the correlation coefficient is greater than or equal to the correlation threshold, the lag time of each root cause layer is output simultaneously to provide priority sorting for subsequent investigation. The abnormal root cause is automatically located according to the priority sorting. When an abnormal root cause is identified, the identification of the subsequent root cause layers is stopped. After the root cause layers with correlation coefficients greater than or equal to the correlation threshold are identified, and no abnormal root cause is identified, the automatic location identification is performed according to the hierarchical progressive method, and the root cause layers with temporal correlation coefficients greater than the correlation threshold are excluded.

9. The performance testing method for an Ethernet switch according to claim 8, characterized in that: The comprehensive evaluation of the Ethernet switch test results in step S6 includes the following steps: S6.21: Based on the design parameter characteristics of Ethernet switches, assign weights to the parameter characteristics of normal and abnormal data, and give a comprehensive score to each characteristic according to the ratio of the actual test value to the nominal value; S6.22: Set a passing threshold. When the overall score is greater than or equal to the passing threshold, the Ethernet switch performance test is qualified. Conversely, when the overall score is less than the passing threshold, the Ethernet switch performance test is unqualified. S6.23: Set a veto option. If the Ethernet switch has a packet loss rate greater than 0.1%, restarts during stability testing, or fails critical functions, the Ethernet switch will be directly judged as failing the performance test. This option takes precedence over the overall score. S6.24: Output a standardized report, which includes a test overview, test plan, test results, anomaly analysis, and comprehensive evaluation; S6.25: Output optimization suggestions based on test results. The optimization suggestions include hardware optimization suggestions, software optimization suggestions, configuration optimization suggestions, and test improvement suggestions.

10. A performance testing system for an Ethernet switch, characterized in that, The Ethernet switch performance testing system includes the Ethernet switch performance testing method as described in any one of claims 1-9, comprising: The test benchmark calibration module is used for automatic calibration by the tester, link stability verification, and generation of tamper-proof electronic signatures for calibration data. The multi-dimensional test scenario management module is used to build four types of test control groups: single-dimensional, dual-dimensional, three-dimensional, and key scenarios, covering combinations of Ethernet switches, test instruments, environment, and time variables. The multi-source data acquisition and storage module is used to synchronously acquire data and inject time, device, test instrument, and environmental tags to save the data; The intelligent data processing and analysis module is used to analyze normal data and trace the root causes of abnormal data, using a hierarchical progressive method and time-series correlation analysis to locate the root causes. The comprehensive evaluation and report generation module is used to comprehensively evaluate Ethernet switch tests and generate test reports.

Images