A model inference server pressure test data filtering method and device

Abstract

This application provides a method and apparatus for filtering load testing data for model inference servers, belonging to the field of load testing data filtering technology. The method provided in this application involves: acquiring multiple load testing data sets for the same test item from a model inference server to form a load testing dataset; calculating the first-word latency of each load testing data set in the dataset; calculating the coefficient of variation and relative range of the dataset based on the first-word latency; performing anomaly judgment on each load testing data set based on the coefficient of variation and relative range, removing abnormal data to obtain a candidate dataset; and selecting the optimal load testing data from the candidate dataset based on the throughput and latency of the load testing data. The method and apparatus provided in this application for filtering load testing data for model inference servers are used to improve the stability of model test item samples and the reliability of results.

Description

Technical Field

[0001] This application relates to the field of stress test data filtering technology, and in particular to a method and apparatus for filtering stress test data for model inference servers. Background Technology

[0002] The performance of the model inference service is directly related to user experience and system efficiency. Therefore, before deploying the model, it is necessary to conduct rigorous performance stress tests on the inference server to evaluate its throughput, response latency and other key performance indicators under different loads. To ensure the accuracy and reliability of the stress test results, it is usually necessary to perform multiple stress tests on the same test item and select the most representative data as the final evaluation result.

[0003] In existing technologies, after performing one or more load tests on the same test item of a model inference server, the data fluctuation is judged by calculating simple statistical indicators such as the mean and variance of the multiple sampled data. When the data fluctuation is determined to be too large, the entire set of data is often discarded and a retest is required. In terms of data storage, usually only the final results after filtering are saved, and not all the original load test data. Some solutions use a single indicator to judge the fluctuation of multiple samples in order to decide whether to adopt the set of load test data.

[0004] However, existing technologies have relatively simple rules for judging fluctuations in multiple sampled test data. This can easily lead to misjudging occasional single abnormal samples as unstable data sets, resulting in the complete discarding of usable data and increasing unnecessary retesting costs and time. Since only the filtered final results are saved and not all original load test data is retained, it is impossible to trace, verify, and recalculate historical data, resulting in insufficient transparency and auditability of data governance. Using a single statistical indicator for fluctuation judgment can easily lead to judgment distortion when faced with different types of performance data distributions, making it impossible to accurately distinguish between real performance fluctuations and occasional measurement noise, resulting in insufficient stability and reliability of test sample data. Summary of the Invention

[0005] In view of this, this application provides a method and apparatus for filtering load test data for model inference servers to solve the problem of insufficient stability and reliability of test item samples.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a method for filtering load test data for a model inference server, the method comprising:

[0008] Obtain multiple load test data for the same test item from the model inference server to form a load test dataset;

[0009] Calculate the first-word delay of each load test data in the load test dataset, and calculate the coefficient of variation and relative range of the load test dataset based on the first-word delay;

[0010] Anomaly detection is performed on each load test data in the load test dataset based on the coefficient of variation and the relative range. Abnormal data is removed to obtain candidate datasets. The optimal load test data is then selected from the candidate datasets based on the throughput and latency of the load test data.

[0011] A second aspect of this application provides a filtering device for stress test data of a model inference server, the device comprising a generation module and a processing module;

[0012] The generation module is used to obtain multiple load test data for the same test item in the model inference server to form a load test dataset;

[0013] The processing module is used to calculate the first character delay of each load test data in the load test dataset, and calculate the coefficient of variation and relative range of the load test dataset based on the first character delay;

[0014] The processing module is further configured to perform anomaly judgment on each load test data in the load test dataset based on the coefficient of variation and the relative range, remove abnormal data to obtain candidate datasets, and select the optimal load test data from the candidate datasets based on the throughput and latency of the load test data.

[0015] The filtering method and apparatus for load testing data of model inference servers provided in this application achieve high-precision filtering of load testing data by collecting multiple load testing data of the same test item, jointly calculating the coefficient of variation and relative range based on the first-letter delay, identifying anomalies, selectively removing abnormal data to obtain candidate datasets, and then selecting the optimal load testing data from the candidate datasets based on throughput and latency. This significantly improves the credibility of test item samples and the representativeness of the final evaluation results. Specifically, by collecting multiple load testing data of the same test item to form a dataset, multiple sampling replaces single measurement, reducing the interference of accidental factors such as environmental temperature fluctuations and hardware status jitter on the evaluation results from the source, ensuring the sufficiency of the samples. The first-letter delay is selected as the core dimension for fluctuation assessment. Utilizing the high sensitivity of this indicator to user-perceived response speed, the data stability judgment is more in line with the actual inference service quality. On this basis, the joint calculation of the coefficient of variation and relative range not only considers the relative dispersion of data distribution but also captures the impact of extreme values ​​on overall fluctuations, overcoming the limitations of a single indicator in the face of... To address the issue of distorted judgments when data distributions differ, datasets deemed abnormal are not discarded entirely. Instead, further anomaly identification and targeted removal are employed, removing only the single problematic record causing the fluctuations while preserving the remaining valid data within the group. This effectively avoids the false positive problem of invalidating the entire test results due to a single occasional fluctuation, reducing unnecessary retesting overhead. Within candidate datasets that ensure data stability, throughput and latency are introduced as selection criteria to choose the best-performing load test data as the final representative result for that test item. This ensures that the output conclusions are not only statistically reliable but also competitive in terms of performance metrics. Attached Figure Description

[0016] Figure 1 A flowchart of an embodiment of the filtering method for stress test data of a model inference server provided in this application;

[0017] Figure 2 A schematic diagram of the structure of Embodiment 2 of the filtering device for model inference server stress test data provided in this application. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0021] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0022] Figure 1 This is a flowchart of an embodiment of the filtering method for model inference server load testing data provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0023] S101. Obtain multiple load test data for the same test item from the model inference server to form a load test dataset.

[0024] Specifically, under fixed test conditions, multiple load tests are performed on the same test item, and performance index data generated in each load test are collected; key attribute fields are extracted from the load test data, and test item hash values ​​are generated by combining multiple fields; all load test data with the same test item hash value are merged into a single load test dataset.

[0025] Furthermore, fixed test conditions refer to keeping the test environment variables such as the hardware configuration, model version, request parameters, and concurrency of the inference server unchanged. Multiple load test requests are repeatedly initiated only under the same test item, with appropriate intervals between each test to allow the server state to recover. After each load test, the load testing tool outputs a set of performance metrics, including at least first-word latency, throughput, and end-to-end latency. Before the data is stored in the database or enters the filtering process, the hardware configuration information, model metadata, and test item identifiers corresponding to each load test are extracted. These fields are concatenated in a predetermined order and their hash values ​​are calculated as the grouping identifier for that load test. All load test records with the same hash value are grouped into the same load test dataset, ensuring that all records within the same dataset belong to the exact same test scenario, providing a comparable data basis for subsequent fluctuation comparisons. It should be noted that during a single test, the server temperature will rise due to data processing. To ensure that the test environment is consistent for each load test, the server temperature is measured before each load test, and the load test begins when the server temperature reaches the target temperature.

[0026] Furthermore, the steps to obtain multiple load test data points for the same test item from the model inference server and form a load test dataset include:

[0027] (1) Scan the load test directory structure and parse the hardware configuration information and model metadata in the load test files;

[0028] Specifically, the root directory where the load test results are stored is traversed, and all load test result files that conform to the preset format are identified and read. For each load test result file, hardware configuration information is extracted from its content or associated description file. For each load test result file, model metadata is extracted from its content or associated description file.

[0029] Furthermore, after completing each round of performance testing, the load testing tool outputs the performance metrics data generated by the load test, along with the test environment information, to a designated directory. The directory structure is typically organized hierarchically by test batch, test time, or test item name. The load test result file can be in formats such as JSON, CSV, or log text. The system first obtains the path to the load test root directory, and then recursively traverses this directory and all its subdirectories, scanning all files that match the target file extension and adding them one by one to the list of files to be parsed. For each file to be parsed, the system reads the file content and, according to preset field mapping rules, locates and extracts the hardware configuration information. The hardware configuration information refers to the hardware environment parameters of the inference server during the execution of this load test, including but not limited to GPU model and quantity, CPU model and number of cores, total memory capacity, etc. The hardware configuration information is usually automatically collected by the load testing tool during the test startup phase and written to the header or metadata area of ​​the result file. At the same time, the system also extracts model metadata from the same file. Model metadata refers to information describing the identity and version attributes of the tested model, including but not limited to model name, model version number, quantization accuracy, inference framework type, etc. By analyzing this information, the system can accurately reconstruct the hardware and software conditions under which each load test was conducted, providing comparable attribute basis for subsequent judgments on whether multiple load tests belong to the same test item.

[0030] (2) Combine the hardware configuration information, the model metadata, and the test item identifier into multiple fields to generate a test item hash value;

[0031] Specifically, the test item identifier is obtained from the stress test result file; the hardware configuration information, model metadata, and test item identifier are concatenated into a combined string according to a predetermined field order; a hash operation is performed on the combined string to generate a fixed-length test item hash value.

[0032] Furthermore, test item identifiers refer to tags used to distinguish different test cases or test scenarios, such as request type, input token length range, output length limit, concurrency settings, etc. These identifiers are also recorded in the load test result file or reflected in the filename. After parsing, the hardware configuration information and model metadata corresponding to each load test record have been obtained. At this point, the test item identifiers are extracted from the parsing results. According to predefined field arrangement rules, the fields in the hardware configuration information, the fields in the model metadata, and the test item identifiers are concatenated sequentially. During concatenation, specific delimiters can be used to connect the fields, forming a complete long string that uniquely represents the test conditions of this load test. The composition of the string ensures that any change in hardware configuration, model version, or test parameters will result in a different final string. A mature hash algorithm is used to calculate this string, outputting a fixed-length hash value. The generation of hash values ​​is deterministic; the same input will inevitably produce the same output. Therefore, multiple load tests of the same test item will yield completely consistent hash values, while any difference in conditions will lead to different hash values, thus achieving unique identification and reliable differentiation of test items.

[0033] (3) Group the load test data with the same hash value of the test item into the same load test dataset.

[0034] Specifically, all load test records are grouped by hash value of the test item as the grouping key; all load test records in each group are collected into a load test dataset; each load test dataset is associated with the corresponding test item hash value for use by the subsequent fluctuation filtering algorithm.

[0035] Furthermore, after parsing and calculating the hash values ​​of all load test result files, a set of load test record objects is held, each record accompanied by its corresponding test item hash value. A mapping table is established, using the hash value as the key and the list of load test records as the value. All load test records are traversed, and each record is appended to the list corresponding to its hash value. After the traversal is complete, each key-value pair in the mapping table represents an independent test item, where the key is the hash value and the value is the set of all load test records under that test item, i.e., the load test dataset. The records in the dataset are multiple independent load test results arranged in chronological or import order, completely consistent in hardware environment, model version, and test parameters, and have the prerequisite of being comparable. Each load test dataset, along with its test item hash value, is passed to the subsequent processing module as the input unit for first-letter delay extraction, coefficient of variation and relative range calculation, anomaly detection, and optimal data selection. By using hash value merging, even if the load test files are scattered in different subdirectories or imported in batches, the system can still accurately and completely aggregate all load test results under the same test conditions, avoiding omissions or incorrect associations, and ensuring the integrity of the dataset and the reliability of the fluctuation analysis conclusions.

[0036] S102. Calculate the first character delay of each load test data in the load test dataset, and calculate the coefficient of variation and relative range of the load test dataset based on the first character delay.

[0037] Specifically, the first-character delay index value is extracted from each record in the stress test dataset; the mean and standard deviation of the dataset are calculated based on all first-character delay values, and the ratio of the standard deviation to the mean is calculated as the coefficient of variation; the maximum and minimum values ​​in the dataset are determined based on all first-character delay values, the difference between the maximum and minimum values ​​is calculated, and the ratio of this difference to the mean is calculated as the relative range.

[0038] Furthermore, the initial latency is a key indicator for measuring the initial response speed of the inference service to user requests, directly reflecting the perceived latency. After extracting the initial latency of all load test records in the dataset, the arithmetic mean of these values ​​is calculated. The mean represents the average initial latency level of this test item across multiple load tests. The deviation of each initial latency value from the mean is calculated to obtain the standard deviation. A larger standard deviation indicates a more dispersed data distribution. Dividing the standard deviation by the mean yields the coefficient of variation, a dimensionless relative indicator that can eliminate the influence of the absolute value difference of initial latency in different test scenarios on the judgment of fluctuation. The maximum and minimum values ​​are found among all initial latency values, and the difference between the two is the absolute fluctuation range. Dividing this difference by the mean yields the relative range, which reflects the maximum deviation of extreme values ​​from the mean. The relative range complements the coefficient of variation, and the combined use of the two can comprehensively evaluate the data fluctuation from both the perspectives of overall dispersion and the impact of extreme values.

[0039] Furthermore, the step of calculating the coefficient of variation of the stress test dataset based on the initial character delay includes:

[0040] (1) Calculate the standard deviation and mean of the stress test dataset based on the first-word delay of each stress test data;

[0041] Specifically, the first character delay value is extracted from each record in the stress test dataset to form a list of first character delay values; the sum of all values ​​in the first character delay value list is divided by the number of values ​​in the list to obtain the mean; the square of the difference between each first character delay value and the mean is calculated, the sum of all squares is divided by the number of values, and the square root is taken to obtain the standard deviation.

[0042] (2) Calculate the coefficient of variation based on the ratio of the standard deviation to the mean.

[0043] Specifically, the coefficient of variation can be calculated using the following formula:

[0044] ;

[0045] in, The coefficient of variation;

[0046] Standard deviation;

[0047] This is the mean.

[0048] Furthermore, the implementation step of calculating the relative range of the stress test dataset based on the first-letter delay includes:

[0049] (1) Determine the maximum first-character delay and the minimum first-character delay in the stress test dataset, and calculate the difference between the maximum first-character delay and the minimum first-character delay;

[0050] Specifically, the process iterates through all load testing records in the dataset, extracting the first-letter latency value from each record. All values ​​are collected into a list, the length of which represents the number of load tests performed on the same test item. For example, if the same test item is tested three times, the list will contain three first-letter latency values. The maximum and minimum first-letter latency are obtained by iterating through the list. The difference between the maximum and minimum first-letter latency is calculated; this difference represents the absolute fluctuation range of the first-letter latency in the load testing data, also known as the range. The range directly reflects the absolute difference between the fastest and slowest responses in multiple load tests. A larger difference indicates more unstable first-letter latency performance across different rounds of load testing, and a more severe deviation from extreme values. This difference provides the numerator for the subsequent calculation of the relative range.

[0051] Furthermore, the difference can be calculated using the following formula:

[0052] ;

[0053] in, The difference;

[0054] Maximum first-letter delay;

[0055] This is the minimum first-letter delay.

[0056] (2) Calculate the mean of the stress test dataset based on the first character delay of each stress test data, and calculate the relative range based on the ratio of the difference to the mean.

[0057] Specifically, the relative range can be calculated using the following formula:

[0058] ;

[0059] in, The relative range;

[0060] The difference;

[0061] This is the mean.

[0062] S103. Based on the coefficient of variation and the relative range, perform anomaly judgment on each load test data in the load test dataset, remove abnormal data, obtain candidate datasets, and select the optimal load test data from the candidate datasets based on the throughput and latency of the load test data.

[0063] Specifically, the calculated coefficient of variation is compared with a first preset threshold, and the relative range is compared with a second preset threshold. If both indicators do not exceed their respective thresholds, the dataset is determined to be free of anomalies, and the current dataset is directly used as a candidate dataset. If either indicator exceeds the threshold, the dataset is determined to be anomaly, and the anomaly removal process begins. For datasets with anomalies, each load test data is sorted by its first-letter latency value from smallest to largest to determine the median. The difference between the first-letter latency of each record and the median is calculated, and the record with the largest and unique difference is identified as anomaly data and removed. The coefficient of variation and relative range are recalculated for the remaining records after removing the anomaly data. If both indicators do not exceed the threshold after recalculation, the remaining records are used as candidate datasets; otherwise, the test item is marked as data distortion and rejected. In the candidate dataset, the throughput and latency indicators of each load test record are compared, and the record with the highest throughput or lowest latency is selected as the optimal load test data for that test item.

[0064] Furthermore, the first and second preset thresholds are judgment criteria pre-set based on engineering experience and statistical principles. For example, the coefficient of variation threshold is set at 5%, and the relative range threshold is set at 10%. The anomaly judgment stage adopts a dual-indicator joint evaluation. When both indicators meet the threshold requirements simultaneously, the judgment is passed. If either indicator exceeds the standard, anomaly handling is triggered. After entering the anomaly removal process, the median is selected as the benchmark for deviation calculation to accurately reflect the central position of the normal data distribution and avoid misjudgment caused by the mean being skewed by outliers. The deviation of each record is obtained by subtracting the median from the first character delay of each record. The record with the largest and unique deviation is identified as the outlier causing the overall fluctuation. If the largest deviation is not unique, it indicates that there are multiple outliers in the dataset. In this case, the data quality is too low and should be directly marked as rejected. After removing the unique outlier, the remaining records are regarded as noise-reduced data and re-enter the calculation and judgment of the coefficient of variation and relative range. All records in the candidate dataset have been statistically verified to be stable and reliable. At this time, throughput and latency are introduced as performance optimization indicators. The one with the highest throughput is selected first. If the throughput is the same, the latency is further compared and the one with the lowest latency is selected. Thus, under the premise of ensuring data credibility, the most competitive performance test result is selected as the final representative data of this test item.

[0065] Furthermore, the implementation steps for anomaly detection of the stress test dataset based on the coefficient of variation and the relative range include:

[0066] (1) If the coefficient of variation is greater than the first preset threshold and the relative range is greater than the second preset threshold, it is determined that the stress test dataset is abnormal;

[0067] Specifically, the calculation results of the coefficient of variation and relative range of the load test dataset are obtained; the coefficient of variation is compared with a first preset threshold; the relative range is compared with a second preset threshold; when both conditions are met, the load test dataset is determined to be abnormal.

[0068] Furthermore, the first and second preset thresholds are pre-configured judgment criteria parameters based on engineering practice experience and the stringency of data stability requirements. For example, the first preset threshold can be set to 5%, and the second preset threshold can be set to 10%. The coefficient of variation is compared with the first preset threshold to determine whether the coefficient of variation exceeds the maximum allowable relative dispersion. The relative range is compared with the second preset threshold to determine whether the fluctuation range between extreme values ​​exceeds the allowable range relative to the mean. If both the coefficient of variation and the relative range exceed the limit, the stress test dataset is considered to be abnormal. The coefficient of variation reflects the overall dispersion, while the relative range reflects the degree of deviation of extreme values. They describe the data fluctuation characteristics from different dimensions. Only when the overall distribution is relatively dispersed and the difference between extreme values ​​is also large is it confirmed that the data does indeed have unacceptable fluctuations.

[0069] (2) If the coefficient of variation is less than or equal to the first preset threshold and the relative range is less than or equal to the second preset threshold, it is determined that the stress test dataset does not have any abnormalities.

[0070] Specifically, when both conditions are met simultaneously, such as the coefficient of variation being less than or equal to the first preset threshold and the relative range being less than or equal to the second preset threshold, the stress test dataset is determined to be free of anomalies and is directly used as a candidate dataset.

[0071] Furthermore, if neither of the two indicators exceeds its respective threshold upper limit (i.e., the coefficient of variation is less than or equal to the first preset threshold, and the relative range is less than or equal to the second preset threshold), then the load test dataset is deemed to have passed the fluctuation test. This indicates that the multiple load test results of the test item show good consistency in the first-letter delay indicator. From the perspective of overall dispersion, the fluctuation range of each load test data around the mean is within an acceptable range; from the perspective of extreme values, the difference between the best and worst performance is not significant relative to the mean. When both dimensions of the indicators meet the standards, it means that the repeatability and stability of the load test data for this test item meet the quality requirements, and the data reliability is high. At this point, the anomaly identification and removal process is skipped, and the current load test dataset is directly marked as a candidate dataset, entering the subsequent throughput and latency comparison stage. From this, the load test data with the best performance is selected as the final representative result of this test item. Compared with traditional methods that rely solely on a single indicator, this approach can more comprehensively evaluate data fluctuation, avoiding misjudgments of overall data quality due to accidental fluctuations in a single dimension of the indicator, thereby improving the robustness and accuracy of the filtering results.

[0072] Furthermore, the implementation step of selecting the optimal stress test data from the candidate dataset based on the throughput and latency of the stress test data includes:

[0073] (1) Obtain the throughput and latency of each load test data, and determine the maximum throughput and optimal latency of the candidate dataset based on the throughput and latency of each load test data;

[0074] Specifically, each load test record in the candidate dataset is traversed, and the throughput and latency metrics of each record are extracted. All extracted throughput values ​​are compared one by one, and the maximum value is determined as the maximum throughput of the candidate dataset. All extracted latency values ​​are compared one by one, and the minimum value is determined as the optimal latency of the candidate dataset.

[0075] Furthermore, the candidate dataset is a collection of load test data that has been verified as statistically stable and reliable after anomaly detection and removal. Each load test record in the candidate dataset is traversed, and throughput and latency are read from the performance metric fields of each record. Throughput refers to the number of requests successfully processed by the inference server per unit of time, usually measured in requests per second or tokens generated per second; a higher value indicates stronger server processing power. Latency refers to the total time from when a client sends a request to when it receives a complete response, usually measured in milliseconds; a lower value indicates a faster server response speed. All extracted throughput values ​​are placed in a list, and the maximum value is found through comparison. The maximum value represents the highest throughput level achievable by this test item in multiple stable load tests. All latency values ​​are placed in another list, and the minimum value is found through comparison. The minimum value represents the optimal latency level achievable by this test item in multiple stable load tests. By identifying the best-performing metric values ​​in the candidate dataset from both throughput and latency performance dimensions, a clear benchmark for selecting the optimal load test data is provided.

[0076] (2) The stress test data corresponding to the maximum throughput or the optimal delay is taken as the optimal stress test data.

[0077] Specifically, in the candidate dataset, determine the load test records that meet the condition that the throughput equals the maximum throughput; determine the load test records that meet the condition that the latency equals the optimal latency; and select one record from the above-mentioned records that meet the conditions as the optimal load test data for this test item according to the preset optimization rules.

[0078] Furthermore, if multiple load test records in the candidate dataset have throughput equal to the maximum throughput, then the latency of these records is compared, and the record with the lowest latency is selected. Alternatively, when the throughput difference is not significant, the record with the best latency is selected first, as lower latency directly translates to a better user experience. Regardless of the priority rule used, one load test record is selected from the candidate dataset as the optimal load test data for that test item. The record of the optimal load test data has undergone fluctuation testing in a statistical sense and represents the best performance level achievable by that test item in terms of performance metrics, and can be used as the final evaluation result output for that test item. By introducing a throughput and latency optimization mechanism while ensuring data stability, reliable and competitive representative data is selected from multiple load tests, significantly improving the credibility and practical reference value of the load test results.

[0079] Furthermore, the steps for removing abnormal data include:

[0080] (1) Determine the median of the stress test dataset and calculate the difference between the first word delay of each stress test dataset and the median;

[0081] Specifically, all load test data in the load test dataset are sorted in ascending order of their first-letter delay values ​​to form an ordered sequence; the median is determined based on the number and position of the values ​​in the ordered sequence; each load test record in the load test dataset is traversed, its first-letter delay value is extracted, and the absolute value of the difference between this value and the median is calculated as the deviation difference of that load test data.

[0082] Furthermore, if the number of values ​​in the sequence is odd, the median is the value in the middle of the sequence; if the number of values ​​is even, the median is the arithmetic mean of the two middle values. For example, when the first-word delays of three stress tests are 10ms, 11ms, and 100ms respectively, the sorted sequence is [10, 11, 100]. The middle value is the second value, 11ms, so the median is 11ms. The median, rather than the mean, is used as the benchmark for calculating deviation because the median is not sensitive to extreme values ​​and will not be skewed by individual abnormally large or small values, thus more accurately reflecting the central level that normal stress test data should be at. In the example above, the mean is approximately 40.3ms, which has been significantly increased by the abnormal value of 100ms. If the deviation is calculated based on the mean, the normal values ​​of 10ms and 11ms would show a larger deviation, leading to misjudgment.

[0083] Furthermore, the step of determining the median of the stress test dataset includes:

[0084] 1.1 Sort the load test data in the load test dataset in ascending order;

[0085] 1.2 Determine the median of the first character delay based on the sorting results.

[0086] Specifically, the first-character delay value of each load test record in the load test dataset is extracted to form a list of values ​​to be sorted. The first-character delay values ​​in the list are then sorted in ascending order to obtain an ordered sequence. The correspondence between each first-character delay value and its corresponding load test record is maintained, so that the original record can be located later based on the sorting position. The number of values ​​in the ordered sequence is determined. It is then determined whether the number of values ​​is odd or even, and the corresponding median determination rule is applied based on parity. If the number is odd, the value in the middle of the ordered sequence is taken as the median; if the number is even, the arithmetic mean of the values ​​in the two middle positions is taken as the median. Using the median as the central benchmark for calculating the deviation difference ensures the accuracy of anomaly identification and avoids misjudging normal data.

[0087] (2) Determine the pressure test data with the largest relative deviation based on the difference;

[0088] Specifically, the deviation difference of each load test data is obtained; the deviation difference of each load test data is divided by the median to obtain the relative deviation of each load test data; all relative deviation values ​​are compared one by one to determine the largest relative deviation and lock the load test data corresponding to the largest relative deviation. The relative deviation represents the relative magnitude of the first-letter delay of the load test data deviating from the normal level.

[0089] (3) If the pressure test data with the largest relative deviation is unique, the pressure test data shall be identified as abnormal pressure test data and removed.

[0090] (4) If the stress test data with the largest relative deviation is not unique, mark the stress test dataset as rejected or suggest retesting.

[0091] Specifically, it is determined whether the load test data corresponding to the maximum relative deviation is unique, that is, whether there exists one and only one load test record with a relative deviation equal to the maximum value; when it is determined to be unique, the load test record is marked as abnormal load test data; the abnormal load test data is removed from the load test dataset, and the remaining load test records constitute the noise-reduced load test dataset.

[0092] Furthermore, when the maximum relative deviation corresponds to two or more load test records, meaning there are at least two load test data sets with the same maximum relative deviation, it is determined that the maximum relative deviation is not unique. When it is determined that it is not unique, no individual record is removed from the dataset; instead, the entire load test dataset is marked as either rejected or recommended for retesting. When multiple outliers coexist, it indicates that the load test environment or server status of the test item is unstable, and the overall data quality is too low. Removing a single outlier record is insufficient to restore a stable and reliable dataset. In this case, no individual record removal or re-evaluation is performed; instead, the entire load test dataset is directly marked as either rejected or recommended for retesting. A rejection status indicates that the set of load test data cannot be adopted as a valid evaluation result, while a recommended for retesting status prompts testers to investigate the test environment and re-execute the load test for that test item. This prevents the risk of forcibly adopting unreliable data when the data quality has severely deteriorated, ensuring the rigor and credibility of the final output evaluation conclusions.

[0093] Furthermore, for large-scale load testing scenarios involving multiple data sets for the same test item, multiple inference servers for the same test item, and multiple test items, a decentralized distributed collaborative filtering mechanism can be adopted to improve overall filtering efficiency. The filtering task is pushed down to each computing node for local execution, and after the local filtering is completed, the key conclusions are broadcast to similar nodes.

[0094] Specifically, tasks are segmented according to test item identifiers. Each inference server's corresponding load testing data acquisition node is only responsible for processing the load testing datasets of one or more test items related to its own node, avoiding network transmission bottlenecks and single-point computing pressure caused by aggregating all raw data to a central node. Each node performs a complete fluctuation filtering process locally on its own load testing data, including first-letter delay extraction, coefficient of variation and relative range calculation, anomaly detection, and abnormal data removal. When a node completes the filtering of a test item dataset and reaches an acceptance or rejection conclusion, it encapsulates the conclusion into a broadcast message and sends it to other similar nodes in the cluster undertaking the same test item load testing task through a message bus or multicast mechanism. After receiving the broadcast, similar nodes use the filtering conclusions of other nodes as a reference. If the majority of nodes consistently reject the test item, the node can directly adopt the conclusion without recalculating, thereby reducing redundant computing overhead. If the conclusions of the nodes are inconsistent, further comprehensive analysis is triggered. Meanwhile, for scenarios with multiple test items, each node can process its assigned test items in parallel, and the nodes share the filtering progress and intermediate results of each test item through a broadcast mechanism, which shortens the overall filtering time of large-scale load test data.

[0095] Optionally, in one embodiment, after filtering is completed, the process includes:

[0096] After each node completes the hierarchical fluctuation filtering and outlier removal of its local load test data, it obtains a candidate dataset containing only stable load test records from its own node. Within this local candidate dataset, each node selects the load test record with the highest throughput or lowest latency, based on either a throughput-first or latency-first selection rule. This record is then used as the node's optimal load test data for that test item. Each node's decision is entirely based on its own directly collected raw load test data, ensuring a clear data source, a complete decision-making chain, and independence from the data quality of other nodes, thus avoiding cross-interference caused by data anomalies from other nodes. Furthermore, since each node can independently complete its final selection without waiting for filtering results from other nodes, decision latency is minimized. This approach is suitable for scenarios with high real-time requirements or where the operating environments of different inference servers vary significantly, making cross-node data merging unsuitable. The optimal load test data selected by each node can be reported to the central management node for aggregation and display. The central node only collects results and does not interfere with the independent decision-making process of each node.

[0097] Optionally, in another embodiment, after filtering is complete, the process further includes:

[0098] After each node completes local filtering and broadcasts its candidate datasets or filtering results, the central coordinating node, or through inter-node negotiation, aggregates the candidate data from all similar nodes into a global candidate dataset. The optimal load testing data is then determined from this global candidate dataset. Essentially, when a test item is load tested on multiple inference servers, each node first removes outlier data locally, generating a local candidate dataset. Each node then sends its local candidate dataset to the global coordinating node, or broadcasts it to all similar nodes to share their respective candidate datasets. The global coordinating node merges all candidate datasets from similar nodes to form a global candidate dataset containing all stable load testing records for that test item across all servers. Based on this, the global coordinating node re-executes the throughput and latency optimization logic, selecting the load testing record with the highest throughput or lowest latency from the global candidate dataset as the globally optimal load testing data for that test item across servers. This approach fully utilizes all valid load testing data from multiple inference servers, expanding the sample pool to improve the representativeness and competitiveness of the optimal data selection, and avoiding the bias of load testing results from a single server. Meanwhile, if a server has too few local candidate dataset samples due to occasional environmental fluctuations, the overall global data can compensate for the insufficient data volume of that server. This is suitable for homogeneous cluster scenarios where all inference servers have the same hardware configuration, consistent test environment, and mergeable stress test results. The final output of the globally optimal stress test data can serve as a unified performance benchmark for this test item in the entire cluster.

[0099] Furthermore, the method provided in this embodiment also includes:

[0100] (1) Write the acceptance or rejection status of each load test dataset to the first audit log table;

[0101] Specifically, after completing all the layered filtering and judgment processes for the load test dataset, the final processing status of the load test dataset is determined; an audit record corresponding to the load test dataset is generated, and the record content includes at least the test item identifier, the final status, and the processing timestamp; the generated audit record is inserted into the first audit log table for persistent storage.

[0102] Furthermore, after completing the entire filtering process for each load testing dataset, the final status of the dataset is determined. The final status includes an "accept" status and a "reject" status. An "accept" status indicates that the load testing dataset passed the volatility assessment, or although it initially failed, it successfully passed the assessment after anomaly removal and noise reduction. The system has successfully selected the optimal load testing data from this dataset as the representative result for this test item. A "reject" status indicates that the load testing dataset still fails the volatility assessment even after anomaly removal, or that multiple anomalies exist that cannot be uniquely removed. The system has marked the entire dataset as rejected or recommended a retest. An audit record is constructed for each processed load testing dataset. The core fields of the audit record include the test item hash value or test item identifier used to uniquely identify the test item, the final status field (values ​​are "accept" or "reject"), and the timestamp of record generation. The timestamp accurately records the time the audit information was written, providing a basis for subsequent tracing of filtering history by time. This audit record is then inserted into the first audit log table for persistent storage. The first audit log table serves as a summary record of the full filtering results, providing testers and managers with a global view. By querying this table, they can quickly understand which test items' load test data were accepted, which were rejected, and the processing completion time of each test item. This role is similar to the system's operational status ledger and is one of the fundamental components for achieving full lifecycle traceability of load test data.

[0103] (2) If the load test dataset triggers an anomaly judgment and removes abnormal load test data, the indicator information that triggers the fluctuation, the fluctuation calculation results of each layer, and the identifier of the removed abnormal load test data are written into the second audit log table.

[0104] Specifically, determine whether the current load testing dataset has undergone an anomaly judgment process during its filtering lifecycle, and whether this anomaly judgment process resulted in the actual removal of abnormal load testing data. If the above conditions are met, collect the key structured information generated during the filtering process, including the name and value of the indicator that triggered the fluctuation, the coefficient of variation and relative range calculated in the initial filtering stage, the coefficient of variation and relative range recalculated after the abnormal data was removed, and the record identifier of the removed abnormal load testing data in the original table. Construct the above structured information into an audit record and insert it into the second audit log table for persistent storage.

[0105] Furthermore, after completing the filtering process for each load testing dataset, it is determined whether the dataset has undergone the complete filtering path. Only when a dataset is deemed anomaly during the initial screening stage due to excessive coefficient of variation or relative range, thus triggering the anomaly identification and removal process, and successfully removing at least one anomalous load testing data point, is the trigger condition for writing to the second audit log table met. Datasets directly accepted after passing the initial screening do not need to be written to this table because they did not trigger anomaly handling. For datasets that triggered anomaly handling but had multiple anomalies and were ultimately rejected as a whole, selective recording is possible as needed. Once the trigger condition is met, process data generated by the dataset throughout the entire filtering chain is collected. The triggering metric information refers to the specific performance metric name that caused the initial screening to fail; in the current scheme, this is usually the first letter of the metric's delay. The specific exceeding value of this metric during the initial calculation is also recorded, such as a coefficient of variation of 8% or a relative range of 15%. The fluctuation calculation results at each layer include two layers of data: the first layer consists of the original values ​​of the coefficient of variation and relative range calculated during the initial screening stage, which are the direct basis for triggering anomaly judgment; the second layer consists of the results obtained by recalculating the coefficient of variation and relative range on the remaining load test data after removing abnormal data, which is the core basis for verifying the noise reduction effect and deciding whether to accept it in the end. The identifier of the removed abnormal load test data refers to the unique identifier in the original table of the load test record that was identified as abnormal by the system and removed from the dataset, such as the primary key ID or row number. Through this identifier, all field information of the record can be accurately located in the original data table. The information collected above is organized into a structured audit record according to the predefined table structure and inserted into the second audit log table for persistent storage. The second audit log table serves as a detailed operation log of the filtering process, recording the complete decision-making chain from anomaly discovery, anomaly location, anomaly removal to noise reduction verification. When subsequent review or questioning of the filtering results for a particular test item is required, auditors can query the second audit log to fully trace back the fluctuation indicators at that time, why the system judged it as abnormal, which specific data was removed, and whether the data quality improved after removal. This provides sufficient factual evidence for quality audits and dispute resolution. The two audit logs work together, with the first recording a summary of the final results and the second recording process details. Together, they form a complete audit traceability system, ensuring the traceability and transparency of the entire filtering process.

[0106] The filtering method for load testing data of model inference servers provided in this embodiment significantly improves the stability of load testing data and the credibility of evaluation results through multi-dimensional statistical analysis and precise anomaly removal. It constructs a dataset using multiple load tests of the same test item, calculates the coefficient of variation and relative range based on the initial delay, and evaluates data fluctuation from two dimensions: overall dispersion and the impact of extreme values. This overcomes the distortion problem of single-indicator judgment. After anomaly detection, it does not directly discard the entire dataset; instead, it locates and removes the unique anomaly sample based on the median, retaining valid data and significantly reducing retesting costs and time overhead. Simultaneously, it generates hash values ​​through hardware configuration, model metadata, and test item identifiers, accurately merging load testing data from the same scenario to ensure dataset comparability. In stable candidate datasets, it selects the optimal sample by combining throughput and latency, ensuring the results have both statistical reliability and performance representativeness. Furthermore, dual audit log tables fully record the filtering results and anomaly handling details, achieving full lifecycle traceability and verifiability of data, enhancing data governance transparency, effectively reducing occasional noise interference, improving the accuracy, efficiency, and standardization of model inference server load testing, and providing more reliable data support for performance evaluation.

[0107] Corresponding to the aforementioned embodiment of a filtering method for stress test data of a model inference server, this application also provides an embodiment of a filtering device for stress test data of a model inference server.

[0108] Figure 2 This is a schematic diagram of the second embodiment of the filtering device for model inference server load testing provided in this application. Please refer to... Figure 2 The apparatus provided in this embodiment includes a generation module 210 and a processing module 220;

[0109] The generation module 210 is used to obtain multiple load test data of the same test item in the model inference server to form a load test dataset;

[0110] The processing module 220 is used to calculate the first character delay of each load test data in the load test dataset, and calculate the coefficient of variation and relative range of the load test dataset based on the first character delay;

[0111] The processing module 220 is further configured to perform anomaly judgment on each load test data in the load test dataset according to the coefficient of variation and the relative range, remove abnormal data to obtain candidate datasets, and select the optimal load test data from the candidate datasets according to the throughput and latency of the load test data.

[0112] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0113] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0114] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0115] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A filtering method for model inference server stress testing data, characterized in that, The method includes: Obtain multiple load test data for the same test item from the model inference server to form a load test dataset; Calculate the first-word delay of each load test data in the load test dataset, and calculate the coefficient of variation and relative range of the load test dataset based on the first-word delay; Anomaly detection is performed on each load test data in the load test dataset based on the coefficient of variation and the relative range. Abnormal data is removed to obtain candidate datasets. The optimal load test data is then selected from the candidate datasets based on the throughput and latency of the load test data.

2. The method according to claim 1, characterized in that, The step of determining anomalies in the stress test dataset based on the coefficient of variation and the relative range includes: If the coefficient of variation is greater than a first preset threshold and the relative range is greater than a second preset threshold, it is determined that the stress test dataset is abnormal. If the coefficient of variation is less than or equal to the first preset threshold, and the relative range is less than or equal to the second preset threshold, it is determined that the stress test dataset does not have any abnormalities.

3. The method according to claim 1, characterized in that, The step of selecting the optimal load test data from the candidate dataset based on the throughput and latency of the load test data includes: Obtain the throughput and latency of each load test data point, and determine the maximum throughput and optimal latency of the candidate dataset based on the throughput and latency of each load test data point; The stress test data corresponding to the maximum throughput or the optimal latency is taken as the optimal stress test data.

4. The method according to claim 1, characterized in that, The calculation of the coefficient of variation and relative range of the stress test dataset based on the first character delay includes: The standard deviation and mean of the stress test dataset are calculated based on the first-word delay of each stress test data point; The coefficient of variation is calculated based on the ratio of the standard deviation to the mean.

5. The method according to claim 1, characterized in that, The step of calculating the coefficient of variation and relative range of the stress test dataset based on the first-letter delay also includes: Determine the maximum and minimum first-character delays in the stress test dataset, and calculate the difference between the maximum and minimum first-character delays; The mean of the load test dataset is calculated based on the first-word delay of each load test data point, and the relative range is calculated based on the ratio of the difference to the mean.

6. The method according to claim 1, characterized in that, The process of acquiring multiple load test data points for the same test item from the model inference server to form a load test dataset includes: Scan the load test directory structure and parse the hardware configuration information and model metadata in the load test files; The hardware configuration information, the model metadata, and the test item identifier are combined into multiple fields to generate a test item hash value. Load test data with the same hash value of the test item are grouped into the same load test dataset.

7. The method according to claim 1, characterized in that, The removal of abnormal data includes: Determine the median of the load test dataset, and calculate the difference between the first-word delay of each load test data and the median; The pressure test data with the largest relative deviation is determined based on the difference; If the stress test data with the largest relative deviation is unique, the stress test data is identified as abnormal stress test data and removed. If the load test data with the largest relative deviation is not unique, the load test dataset will be marked as rejected or recommended for retesting.

8. The method according to claim 1, characterized in that, The method further includes: Write the acceptance or rejection status of each load test dataset to the first audit log table; If the load test dataset triggers an anomaly detection and removes abnormal load test data, the indicator information that triggered the fluctuation, the fluctuation calculation results of each layer, and the identifier of the removed abnormal load test data will be written into the second audit log table.

9. The method according to claim 7, characterized in that, Determining the median of the stress test dataset includes: The load test data in the load test dataset are sorted in ascending order; Determine the median delay of the first character based on the sorting results.

10. A filtering device for stress test data of a model inference server, characterized in that, The device includes a generation module and a processing module; The generation module is used to obtain multiple load test data for the same test item in the model inference server to form a load test dataset; The processing module is used to calculate the first character delay of each load test data in the load test dataset, and calculate the coefficient of variation and relative range of the load test dataset based on the first character delay; The processing module is further configured to perform anomaly judgment on each load test data in the load test dataset based on the coefficient of variation and the relative range, remove abnormal data to obtain candidate datasets, and select the optimal load test data from the candidate datasets based on the throughput and latency of the load test data.

Images