An ultra-computing storage subsystem performance evaluation and adaptive configuration recommendation method

Abstract

The application discloses a kind of supercomputer storage subsystem performance evaluation and adaptive configuration recommendation method.The present application includes: obtaining platform and test target, concurrent process candidate set is generated based on platform core number, and complete test matrix is constructed in combination with access mode, read-write type combination.In the calculation node, serial read-write test is run, and performance index is collected to build IO performance baseline.Iterate test matrix, respectively execute two types of parallel read-write test of sharing, independent file, and collect different concurrent performance data to generate performance extension curve.Relying on curve, automatically identify peak value, saturation inflection point and performance degradation point, combined with optimization target, output optimal concurrency and access mode, compare the performance of two types of files to locate system bottleneck and output configuration optimization suggestion.The present application considers serial baseline calibration and multi-scene parallel stress test, automatically analyzes performance variation law, closed-loop output adaptive configuration scheme, and is convenient for quickly positioning storage read-write performance bottleneck.

Description

Technical Field

[0001] This invention relates to the field of high-performance computing and distributed storage evaluation technology, and in particular to a method for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem. Background Technology

[0002] As the core infrastructure for high-performance computing and large-scale AI training, the serial and parallel I / O performance of the storage subsystem of supercomputing directly determines end-to-end training efficiency and task throughput. Related technologies utilize the collaborative operation of parallel file systems, computing nodes, and storage media to construct a high-performance I / O system. Specifically, this system covers the entire process from data reading and writing to metadata management, including key aspects such as serial and parallel access, and shared and independent file modes. The choice of concurrency (number of processes / threads) has a significant impact on performance.

[0003] However, existing storage evaluation methods often rely on fixed concurrency levels or trial-and-error based on human experience, lacking adaptive selection and bottleneck diagnosis mechanisms. This can lead to test results that fail to reflect I / O behavior in real-world operating environments, or to significant differences in optimal concurrency parameters across different node sizes, thus impacting system tuning efficiency and resource utilization. Furthermore, existing solutions lack a systematic distinction between shared and independent file parallel access modes, making it difficult to automatically pinpoint the root causes of performance degradation such as lock contention, controller saturation, or physical bandwidth limitations, and consequently, failing to provide actionable configuration recommendations. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a method for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem.

[0006] Another objective of this invention is to provide a device for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem.

[0007] The third objective of this invention is to provide a computer device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a method for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem, comprising:

[0010] S10: Obtain target platform information and test objectives; automatically generate a candidate set of concurrent process counts based on the number of computing cores on the platform; and combine the candidate set of concurrent process counts, access mode set, and read / write type set into a test matrix. S20 directly performs serial read / write tests on the computing node, collects serial read / write performance indicators, and establishes the platform's serial input / output baseline capability. S30, based on the test matrix, perform parallel read and write tests on the computing nodes in shared file access mode and independent file access mode respectively, collect parallel read and write performance indicators under each concurrency level, and obtain the expansion curves of concurrency and performance indicators. S40 automatically identifies performance peaks, saturation inflection points, and degradation initiation points based on the expansion curve. It outputs recommended concurrent process counts and access modes according to optimization goals, and diagnoses bottleneck types based on performance comparisons of shared and independent files, generating configuration suggestions.

[0011] In one embodiment of the present invention, S10 includes: The candidate number of processes is incremented from 1 by powers of 2 until it reaches twice the number of cores, thus generating a set. , where C is the number of computational cores; The step of combining the candidate set of concurrent processes, the set of access patterns, and the set of read / write types into a test matrix includes: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.] With Read / Write Type Set The test matrix is ​​obtained by performing a Cartesian product operation on the candidate set P of concurrent processes. Each row of the test matrix corresponds to a set of test configurations to be executed.

[0012] In one embodiment of the present invention, S20 includes: Collect read bandwidth, write bandwidth, read IOPS, write IOPS, average read latency, and average write latency to establish the platform's serial input / output baseline capability.

[0013] In one embodiment of the present invention, S30 includes: In shared file access mode, multiple processes access the same file concurrently; in independent file access mode, each process accesses a different file. Read bandwidth, write bandwidth, read IOPS, write IOPS, read latency, and write latency were collected separately, and expansion curves were obtained under different concurrency levels.

[0014] In one embodiment of the present invention, it further includes: Only a preset subset of anchor points in the test matrix T is tested. The subset of anchor points includes the number of concurrent processes {1, 8, 32} and one serial baseline test. The performance metrics of the remaining concurrent points are predicted and completed by the performance prediction and diagnostic model module to obtain the complete expansion curve.

[0015] In one embodiment of the present invention, S40 includes: Construct the bandwidth curve BW(p) and calculate the relative growth rate of adjacent concurrent points. ; The number of processes at the point of maximum bandwidth is taken as the peak point. ; The number of processes whose relative growth rate first falls below a preset threshold α is taken as the saturation inflection point: ; The minimum number of processes whose bandwidth drops more than a preset threshold β after the peak is used as the degradation starting point: .

[0016] In one embodiment of the present invention, it further includes: When the optimization objective is to maximize bandwidth, the recommended number of processes is [value missing]. ; When efficiency is the primary optimization objective, the recommended number of processes is [number to be filled in]. ; When the optimization objective is constrained optimization, the delay threshold must be met. Maximize bandwidth under the premise that: ; and the interval To avoid outputting intervals together, the following is marked: The maximum number of processes in the candidate set of concurrent processes.

[0017] To achieve the above objectives, a second aspect of the present invention provides a device for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem, comprising: The test matrix generation module is used to obtain target platform information and test objectives, automatically generate a candidate set of concurrent process counts based on the number of cores on the platform, and combine the candidate set of concurrent process counts, access mode set, and read / write type set into a test matrix. The serial baseline test module is used to directly perform serial read and write tests on computing nodes, collect serial read and write performance indicators, and establish the platform's serial input and output baseline capability. The parallel read / write stress test module is used to perform parallel read / write tests on computing nodes in shared file access mode and independent file access mode according to the test matrix, collect parallel read / write performance indicators under various concurrency levels, and obtain the expansion curves of concurrency and performance indicators. The performance analysis and tuning module is used to automatically identify performance peaks, saturation inflection points, and degradation initiation points based on the expansion curve. It outputs recommended concurrent process counts and access modes according to the optimization goals, and diagnoses bottleneck types based on the performance comparison results of shared files and independent files, and generates configuration suggestions.

[0018] The present invention discloses a method and apparatus for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem, which can automatically select the optimal number of concurrent processes and identify performance degradation intervals, reducing the cost of manual parameter tuning; at the same time, it distinguishes between shared file and independent file access modes, automatically diagnoses bottlenecks and outputs actionable suggestions, thereby improving storage evaluation efficiency and cross-platform adaptability.

[0019] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing the method described in the first aspect embodiment.

[0020] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem according to an embodiment of the present invention; Figure 2 This is an overall block diagram of a storage subsystem performance evaluation and adaptive concurrent configuration recommendation system according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the storage subsystem performance evaluation and adaptive configuration recommendation method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a parallel file access mode according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the storage concurrency expansion bandwidth curve according to an embodiment of the present invention; Figure 6 This is a multi-task neural network structure and information flow graph according to an embodiment of the present invention; Figure 7 This is a flowchart of neural network training according to an embodiment of the present invention; Figure 8This is a structural diagram of a supercomputing storage subsystem performance evaluation and adaptive configuration recommendation device according to an embodiment of the present invention; Figure 9 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] The following description, with reference to the accompanying drawings, describes a method and apparatus for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem according to an embodiment of the present invention.

[0026] Figure 1 This is a flowchart of a method for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S10: Obtain target platform information and test objectives; automatically generate a candidate set of concurrent process counts based on the number of computing cores on the platform; and combine the candidate set of concurrent process counts, access mode set, and read / write type set into a test matrix. S20 directly performs serial read / write tests on the computing node, collects serial read / write performance indicators, and establishes the platform's serial input / output baseline capability. S30, based on the test matrix, perform parallel read and write tests on the computing nodes in shared file access mode and independent file access mode respectively, collect parallel read and write performance indicators under each concurrency level, and obtain the expansion curves of concurrency and performance indicators. S40 automatically identifies performance peaks, saturation inflection points, and degradation initiation points based on the expansion curve. It outputs recommended concurrent process counts and access modes according to optimization goals, and diagnoses bottleneck types based on performance comparisons of shared and independent files, generating configuration suggestions.

[0027] Furthermore, this invention proposes another method for performance evaluation and adaptive configuration recommendation of supercomputing storage subsystems, the overall process of which is as follows: Figure 3As shown, the process includes the following steps: Step S1: Obtain evaluation input and generate a concurrency parameter space. Obtain at least one of the following inputs: Target platform and file system information: parallel file system type, network topology, storage medium type, etc.; Test objectives: serial read / write, parallel read / write; shared file mode, independent file mode; Metrics and constraints: bandwidth, IOPS, average latency (or quantiles), test duration, file size, block size; Concurrency parameter space: set of process numbers (and optional set of thread numbers), set of node numbers.

[0028] The process of generating the concurrent parameter space: This step is based on the number of computing node cores detected by the platform. The system automatically generates a set of concurrent process counts according to preset rules. Specifically, the candidate process count values ​​start from 1 and increase in powers of 2 until they reach twice the number of cores, i.e.:

[0029] At the same time, the access mode set With Read / Write Type Set and the Cartesian product of the three The test matrix is ​​assembled, with each row of the test matrix corresponding to a set of test configurations to be executed.

[0030] Step S2: Serial I / O evaluation by direct testing on compute nodes. Serial read and write tests are performed directly on the compute nodes, and the data collected includes at least: read / write bandwidth; read / write IOPS; and average read / write latency. This data is used to establish the platform's serial I / O baseline capability and can be used for comparison and bottleneck identification in subsequent parallel tests.

[0031] Step S3: Parallel I / O evaluation. Two access modes are executed in a parallel scenario. The differences between the two modes are as follows: Figure 4 As shown: Shared file access mode: multiple processes concurrently access the same file; Independent file access mode: each process accesses a different file. Read / write bandwidth, IOPS, and latency were collected separately, and expansion curves were obtained under different concurrency levels.

[0032] Step S3+: Optional prediction mode, which can be executed using one of two modes: Full mode: All concurrent points in the test matrix T are tested one by one to obtain the complete expansion curve; Prediction mode: Only a subset of anchor points (such as the number of concurrent processes {1, 8, 32} and a single serial baseline) are tested, and the performance of the remaining concurrent points is predicted and completed by the performance prediction and diagnostic model module, thus obtaining the complete expansion curve while significantly reducing the number of tests. Curves obtained from both modes can be entered into steps S4 and S5 without distinction, meaning that the subsequent parameter selection, recommendation, and diagnostic processes do not differentiate between curves derived from actual tests and predictions.

[0033] Step S4: Adaptively select the optimal concurrency and output the recommended configuration. Repeat steps S2 or S3, or use the prediction mode of step S3+, under different process count configurations to obtain curves, forming a "process count - metric" curve, and automatically select the recommended configuration. An example of the resulting "concurrency - bandwidth" expansion curve and its peak point, degradation start point, and recommended interval is shown below. Figure 5 As shown. The recommended configuration generation process is as follows: Constructing bandwidth curves And calculate the relative growth rate of adjacent concurrent points:

[0034] The peak number of processes is the point where the bandwidth is at its maximum.

[0035] The saturation inflection point is when the relative growth rate first falls below a threshold. (e.g., 5%) of the process count:

[0036] The degradation begins at the peak value, after which the bandwidth decreases relative to the peak value beyond a threshold. Minimum number of processes (e.g., 10%):

[0037] It is understandable that the recommended number of processes is output based on the optimization objective: maximizing the objective takes... Efficiency First (Inflection Point) Target The constraint optimization objective is to satisfy the delay threshold. Maximize bandwidth under the premise that .Will The intervals marked as "Avoid Intervals" are also output.

[0038] Step S5: Bottleneck diagnosis and suggestion generation. Diagnostic conclusions are automatically generated based on the comparison results and expansion curves of shared files and independent files. The process for generating the diagnostic conclusions is as follows: First, the bandwidth ratio of shared files to independent files under the same read / write type is calculated. Then, determine the bottleneck type according to preset rules: if the write bandwidth of shared files is significantly lower than that of independent files under high concurrency ( ,like If the concurrency does not increase, it is determined to be "write serialization overhead caused by file system locking mechanism", and it is recommended to prioritize the use of independent file mode or limit the concurrency for write-intensive tasks; if the read bandwidth of shared files is significantly higher than that of independent files ( ,like If the read performance is "advantageous to read by sharing page cache", it is determined that read-intensive tasks should prioritize sharing files or using caching strategies. If the read and write performance of independent files decreases synchronously after a certain level of concurrency, it is determined that the concurrency is "limited by the physical limits of the storage subsystem or due to resource contention", and suggestions for limiting concurrency are output.

[0039] Furthermore, the overall system structure of the present invention is as follows: Figure 2 As shown, the system includes a test plan generation module, a benchmark execution module, a performance acquisition module, a scalability analysis module, a recommendation decision module, a diagnostic report module, and a performance prediction and diagnostic model module (preferred). The internal structure, processing procedures, and data flow relationships of each module are as follows: Understandably, the test plan generation module includes a platform detection subunit and a parameter space generation subunit. The platform detection subunit automatically obtains the number of cores, NUMA nodes, file system type, etc.; the parameter space generation subunit generates a process count set according to the rules described in step S1. And assemble the test matrix The output is then sent to the benchmark execution module. The benchmark execution module includes an execution environment adaptation subunit and a test execution subunit. The former completes process binding, mount path setting, and pre-test cache clearing on the compute node; the latter executes serial and parallel (shared / independent) read / write tests according to the test matrix, with each configuration group being executed repeatedly. Next. Performance acquisition module: includes a raw metric acquisition subunit, an outlier filtering subunit, and a normalization subunit. It sequentially completes the acquisition of bandwidth / IOPS / latency, and... Outlier filtering for repeated results and normalization to a structured data table. The extended analysis module includes a curve construction subunit, a difference analysis subunit, and a feature point detection subunit. It is based on a data table. Construct a concurrency-metric curve and calculate the relative growth rate. And identify the peak point according to the formula in step S4. saturation inflection point and the starting point of degradation In a preferred embodiment, the curves used in this module do not necessarily need to be constructed from full-scale experimental data. Instead, they can be predicted by the performance prediction and diagnostic model module below with a small number of anchor points, and then used to identify curves with the same definition. The recommendation decision module includes a single-objective recommendation subunit, a constraint optimization subunit, and an avoidance interval labeling subunit. Following step S4, it solves and outputs the number of recommended processes, recommended access modes, and avoidance intervals, providing separate recommendations for read and write scenarios. The diagnosis and reporting module includes a bottleneck classification subunit, a suggestion subunit, and a report output subunit. Following step S5, it calculates the bandwidth ratio. The process involves matching diagnostic rules, providing actionable suggestions, and organizing the results with charts into an evaluation report. In a preferred embodiment, the bottleneck classification sub-unit can be implemented by the multi-label diagnostic head of the following model module, with the threshold manually tuned in step S5. This is learned by the model.

[0040] Understandably, the storage performance prediction and bottleneck diagnosis module based on a multi-task neural network aims to automatically predict performance curves and diagnose bottleneck types under a limited number of actual tests. This invention further designs a multi-task neural network module, whose network structure and information flow are as follows: Figure 6 As shown. Its core idea is: by performing only a small number of inexpensive field tests on the target platform (such as serial baselines and individual anchor points), it is possible to predict all measurable parameters for a large number of untested concurrent configurations, and based on this, diagnose bottlenecks / advantage factors and output recommended configurations.

[0041] Furthermore, this invention designs a multi-task neural network for storage evaluation, denoted as the Storage Adaptive Evaluation Network (SAEN). It employs an integrated structure of a shared feature extraction backbone, a performance prediction head, and a multi-label diagnostic head. The prediction head output is cascaded to the diagnostic head via cross-configuration features, mutually constraining and reinforcing each other during training. The network as a whole consists of the following parts: an input encoding layer that encodes the target configuration features and platform context features into a normalized input vector; a shared feature extraction backbone composed of multiple fully connected layers, batch normalization layers, non-linear activation layers, and Dropout layers, outputting a shared feature representation independent of the specific task. Performance prediction head (regression branch): by Predict the observations for this configuration This includes bandwidth, IOPS, latency; and cross-configuration features. : Processing the output of the prediction head across the entire configuration grid yields cross-mode / cross-concurrency contrast features; Multi-label diagnostic head: with As input, it independently outputs the existence probabilities of multiple bottleneck factors and advantage factors, supporting the simultaneous existence of multiple factors.

[0042] Furthermore, regarding the input encoding layer and feature construction, the network input consists of two parts: target configuration features and platform context features, both of which can be obtained before the formal scanning test.

[0043] Target configuration features include: Target number of concurrent processes; Access mode; Reading and writing; , Block size vs. file size.

[0044] Furthermore, platform context features include: , Number of cores and number of NUMA nodes; File system types can be learned and embedded; Storage medium; This serves as the platform's serial I / O baseline.

[0045] Furthermore, the dimension of the concatenated input features is denoted as... (This embodiment) ), standardize continuous components according to training set statistics:

[0046] in For the training set Mean and standard deviation It is a small constant. After standardization As a shared backbone input.

[0047] Furthermore, a shared feature extraction backbone is constructed, consisting of three fully connected layers, each followed by batch normalization (BN), ReLU, and Dropout.

[0048]

[0049]

[0050] in , , These are learnable parameters. This represents shared features.

[0051] Furthermore, the performance prediction head, consisting of two fully connected layers, outputs the raw logits corresponding to seven measurable terms, and then applies appropriate transformations according to the value range of each measure:

[0052]

[0053] Will Divided into three groups and transformed separately: performance indicators (3-dimensional, spanning orders of magnitude, exponential restoration after logarithmic domain regression): Time decomposition percentage (3-dimensional, non-negative, sum to 1, using Softmax): Discrete coefficients of completion time for each process (1-dimensional, non-negative, using Softplus): Let the total predictable measurable value of this configuration be denoted as . .in Supports metadata bottleneck identification. Support for load imbalance determination.

[0054] Furthermore, cross-configuration features The structure of the configuration means that some bottleneck / advantage factors (lock contention, physical limits, cache advantages) cannot be determined by a single configuration and must be compared across modes and concurrency levels. Therefore, a performance prediction head is first used across the entire configuration grid (the set of concurrent processes). Predicted bandwidth Then calculate the features:

[0055]

[0056]

[0057] Cross-configuration feature vector .in Low indicator lock contention, Indicate the advantages of caching, Large and Indicates the physical limits. Here... That is, the bandwidth ratio in step S5 According to the subdivision of reading and writing, It shares the same name and meaning as step S4, thus aligning perfectly with the rule-based approach. All results are calculated from predictions, requiring no actual measurements.

[0058] Furthermore, the multi-label diagnostic head takes as input a concatenation of "shared features + predictable measurables + cross-configuration features", and outputs... Each independent factor probability (in this embodiment) ):

[0059]

[0060]

[0061] Understandable, The factors are as follows: Lock contention, Physical bandwidth / controller limit Metadata bottleneck Uneven load (the first four are bottleneck factors). Advantages of page cache sharing. The three types of conclusions (lock contention, physical limits, and cache advantages) are given by the rule method in step S5. This implementation method introduces metadata bottlenecks and load imbalances. Each factor is determined independently and can be true simultaneously.

[0062] Furthermore, the recommended configuration generation process, given the trained network and inexpensive test data on the target platform, involves the following steps: full-grid prediction. The prediction head is used to... Bandwidth curves were obtained from various concurrency levels, two modes, and read / write predictions. And the rest that can be measured. Peak point. .

[0063] Saturation inflection point. (like Degradation initiation point and avoidance interval. (like ), Mark the interval as an avoidable region. Constraint optimization. When there are no constraints, efficiency takes precedence. Performance priority Diagnostic-driven recommendations. For The diagnostic header is invoked based on the neighborhood configuration, and actionable suggestions for read and write scenarios are given according to the preset mapping based on the triggered factor set.

[0064] Furthermore, the network training process is as follows: Figure 7 As shown, the process includes the following steps: Training dataset construction: The data source is the full set of actual test records of the evaluation system on the test matrix, with each sample being... :enter Configure target features + platform context features (including the platform's serial baseline) ); Regression target Seven observables were measured for this configuration. ); Multi-label diagnostic goals As a multi-heat vector, each component is governed by independent expert rules ( Degradation range , , (Etc.) Weakly labeled and manually verified, multiple values ​​can be set to 1 simultaneously. Number of concurrent processes for sample coverage. Two access modes, read / write, multiple prototype machines, and combinations of various block sizes. Data augmentation: Add small Gaussian perturbations to continuous features and augment intermediate concurrency samples by logarithmic interpolation between adjacent concurrency points. Cascaded consistency: Calculate cross-configuration features. During training and inference, the prediction head output is computed on the grid. Training employs a two-stage process: first, the prediction head is trained, then its predictions to the grid are frozen to train the diagnostic head. Joint loss function: The regression loss is the sum of seven measurable terms (performance terms are in the logarithmic domain, with percentages and coefficients of variation in their respective domains).

[0065] Multi-label diagnostic loss:

[0066] Total loss ,in (e.g., 0.5) Balance the two tasks. (like The weight decay is represented by Adam (initial learning rate). Optimization and hyperparameters: Adam (initial learning rate) is used. ), Batch size 32, maximum 200 rounds, early stop, training / validation ratio 8:2.

[0067] The embodiments of this invention also have the following technical effects: Adaptive parameter selection: Automatically selects the optimal number of concurrent processes, reducing trial and error costs and improving evaluation efficiency; Diagnostic patterns: Distinguishes the performance differences and causes between shared file and independent file access patterns, and outputs strategic suggestions; Quantifiable scalability: Identifies the benefit and degradation ranges of concurrency improvement, providing a basis for system optimization decisions. Predicting the complete curve with a small number of measured points: As an implementation method of the above rule-based approach, a multi-task neural network can predict the complete concurrency expansion curve and output recommended configurations and bottleneck diagnoses with only a small number of measured anchor points. This significantly reduces the number of measurements while reusing the same feature points and diagnostic criteria, and replaces manual thresholds with data-driven approaches, enhancing cross-platform adaptability.

[0068] To achieve the above embodiments, such as Figure 8 As shown, this embodiment also provides a supercomputing storage subsystem performance evaluation and adaptive configuration recommendation device 10, including: The test matrix generation module 100 is used to obtain target platform information and test objectives, automatically generate a candidate set of concurrent process counts based on the number of cores on the platform, and combine the candidate set of concurrent process counts, access mode set, and read / write type set into a test matrix. The serial baseline test module 200 is used to directly perform serial read and write tests on the computing node, collect serial read and write performance indicators, and establish the platform's serial input and output baseline capability. The parallel read / write stress test module 300 is used to perform parallel read / write tests on computing nodes in shared file access mode and independent file access mode according to the test matrix, collect parallel read / write performance indicators under various concurrency levels, and obtain the expansion curves of concurrency and performance indicators. The performance analysis and tuning module 400 is used to automatically identify performance peaks, saturation inflection points, and degradation start points based on the expansion curve. It outputs recommended concurrent process counts and recommended access modes according to the optimization goals, and diagnoses bottleneck types based on the performance comparison results of shared files and independent files, and generates configuration suggestions.

[0069] An embodiment of the present invention provides a supercomputing storage subsystem performance evaluation and adaptive configuration recommendation device that can automatically select the optimal number of concurrent processes and identify performance degradation intervals, reducing the cost of manual parameter tuning; at the same time, it distinguishes between shared file and independent file access modes, automatically diagnoses bottlenecks and outputs actionable suggestions, improving storage evaluation efficiency and cross-platform adaptability.

[0070] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 9 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.

[0071] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0072] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0073] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem, characterized in that, include: S10: Obtain target platform information and test objectives; automatically generate a candidate set of concurrent processes based on the number of computing cores on the platform; and combine the candidate set of concurrent processes, access mode set, and read / write type set into a test matrix. S20 directly performs serial read / write tests on the computing node, collects serial read / write performance indicators, and establishes the platform's serial input / output baseline capability. S30, based on the test matrix, perform parallel read and write tests on the computing nodes in shared file access mode and independent file access mode respectively, collect parallel read and write performance indicators under each concurrency level, and obtain the expansion curves of concurrency and performance indicators. S40 automatically identifies performance peaks, saturation inflection points, and degradation initiation points based on the expansion curve. It outputs recommended concurrent process counts and access modes according to optimization goals, and diagnoses bottleneck types based on performance comparisons of shared files and independent files, generating configuration suggestions.

2. The method as described in claim 1, characterized in that, S10 includes: The candidate number of processes is incremented from 1 by powers of 2 until it reaches twice the number of cores, thus generating a set. , where C is the number of computational cores; The step of combining the candidate set of concurrent processes, the set of access patterns, and the set of read / write types into a test matrix includes: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.] With Read / Write Type Set The test matrix is ​​obtained by performing a Cartesian product operation on the candidate set P of concurrent processes. Each row of the test matrix corresponds to a set of test configurations to be executed.

3. The method as described in claim 1, characterized in that, S20 includes: Collect read bandwidth, write bandwidth, read IOPS, write IOPS, average read latency, and average write latency to establish the platform's serial input / output baseline capability.

4. The method as described in claim 1, characterized in that, S30 includes: In shared file access mode, multiple processes access the same file concurrently; in independent file access mode, each process accesses a different file. Read bandwidth, write bandwidth, read IOPS, write IOPS, read latency, and write latency were collected separately, and expansion curves were obtained under different concurrency levels.

5. The method as described in claim 4, characterized in that, The method further includes: Only a preset subset of anchor points in the test matrix T is tested. The subset of anchor points includes the number of concurrent processes {1, 8, 32} and one serial baseline test. The performance metrics of the remaining concurrent points are predicted and completed by the performance prediction and diagnostic model module to obtain the complete expansion curve.

6. The method as described in claim 1, characterized in that, S40 includes: Construct the bandwidth curve BW(p) and calculate the relative growth rate of adjacent concurrent points. ; The number of processes at the point of maximum bandwidth is taken as the peak point. ; The number of processes whose relative growth rate first falls below a preset threshold α is taken as the saturation inflection point: ； The minimum number of processes whose bandwidth drops more than a preset threshold β after the peak is used as the degradation starting point: 。 7. The method as described in claim 6, characterized in that, The method further includes: When the optimization objective is to maximize bandwidth, the recommended number of processes is [value missing]. ; When efficiency is the primary optimization objective, the recommended number of processes is [number to be filled in]. ; When the optimization objective is constrained optimization, the delay threshold must be met. Maximize bandwidth under the premise that: ； and the interval To avoid outputting intervals together, the following is marked: The maximum number of processes in the candidate set of concurrent processes.

8. A device for performance evaluation and adaptive configuration recommendation of a supercomputing storage subsystem, characterized in that, include: The test matrix generation module is used to obtain target platform information and test objectives, automatically generate a candidate set of concurrent process counts based on the number of cores on the platform, and combine the candidate set of concurrent process counts, access mode set, and read / write type set into a test matrix. The serial baseline test module is used to directly perform serial read and write tests on computing nodes, collect serial read and write performance indicators, and establish the platform's serial input and output baseline capability. The parallel read / write stress test module is used to perform parallel read / write tests on computing nodes in shared file access mode and independent file access mode according to the test matrix, collect parallel read / write performance indicators under various concurrency levels, and obtain the expansion curves of concurrency and performance indicators. The performance analysis and tuning module is used to automatically identify performance peaks, saturation inflection points, and degradation initiation points based on the expansion curve. It outputs recommended concurrent process counts and access modes according to the optimization goals, and diagnoses bottleneck types based on the performance comparison results of shared files and independent files, and generates configuration suggestions.

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images