Multi-dimensional benchmark test data analysis method and device based on modular framework

Abstract

The application discloses a kind of multi-dimension benchmark test data analysis method and device based on modular framework, it is related to test data analysis technical field, including: establishing unified connection obtains original data set and standardizes conversion formation structured data;Recognition tool type calls analysis plug-in interface conversion into format uniform comparable data;Construct data matrix, select benchmark system using adaptive algorithm and execute statistical fluctuation detection based on statistical rules, generate analysis result data;According to semantic analysis, optimal chart is matched and visual rendering is generated to generate report.It realizes benchmark test whole process automation closed loop, solves the tool island problem caused by tool data isomerism;Through plug-in architecture, the compatibility and efficiency of heterogeneous data processing are improved;Adaptive calibration and statistical detection are introduced, subjective error of manual selection benchmark is eliminated, the objectivity and scientificity of performance analysis are significantly enhanced, and intelligent generation of performance evaluation report is realized.

Description

Technical Field

[0001] This invention relates to the field of test data analysis technology, and in particular to a multi-dimensional benchmark test data analysis method and apparatus based on a modular framework. Background Technology

[0002] In the process of evaluating and selecting computer system performance, engineers need to use multiple benchmarking tools simultaneously, such as UnixBench, Stream, and SPEC CPU, to comprehensively evaluate system performance from multiple perspectives. However, while these tools offer comprehensive testing angles, they suffer from tool silos. Each benchmarking tool produces data in different formats, lacking a unified analysis platform. In particular, existing tools lack the ability to make cross-system comparisons when analyzing performance differences across multiple systems. Switching between different tools to process data leads to inefficiency, and the analysis results are limited to the raw data, lacking the ability to analyze data by system, CPU architecture, configuration parameters, and other dimensions. This results in a high degree of manual operation, high error rates, and time consumption, and a lack of unified comparison standards and analytical dimensions. Summary of the Invention

[0003] This invention provides a method and apparatus for multi-dimensional benchmark data analysis based on a modular framework, in order to solve the technical problem of low efficiency and unreliable results caused by differences between multi-dimensional benchmark data tools.

[0004] In a first aspect, embodiments of the present invention provide a multi-dimensional benchmark data analysis method based on a modular framework, including: S101 responds to test data analysis requests initiated by user terminals, establishes a unified connection with various benchmark testing tools and receives raw performance datasets, performs standardized conversion and validity checks on heterogeneous data formats in the raw performance datasets, and forms structured performance data with a unified format. S102: Identify the benchmark tool type based on the structured performance data, and call the corresponding type's preset data analysis plugin interface to convert the structured performance data into comparable data with a unified format; S103: Extract the fields corresponding to the preset analysis dimensions based on the comparable data, construct a data matrix for multi-dimensional comparative analysis, automatically select the corresponding benchmark system for performance comparison through an adaptive benchmark selection algorithm, and perform performance fluctuation detection based on statistical rules to form analysis result data; S104 performs semantic parsing on the analysis results data, performs visualization rendering based on the preset mapping rule library, and generates a visual standardized analysis report.

[0005] Secondly, embodiments of the present invention provide a multi-dimensional benchmark data analysis device based on a modular framework, comprising: The data access unit is used to respond to test data analysis requests, establish a unified connection with various benchmark testing tools, receive raw performance datasets, perform standardization transformation and validity checks, and form structured performance data with a unified format. The data conversion unit is used to identify the type of benchmark testing tool based on the structured performance data, and call the corresponding type of preset data analysis plugin interface to convert the structured performance data into comparable data in a uniform format. The data analysis unit is used to extract fields corresponding to preset analysis dimensions based on comparable data, construct a data matrix for multi-dimensional comparative analysis, automatically select the appropriate benchmark system for performance comparison through an adaptive benchmark selection algorithm, and perform performance fluctuation detection based on statistical rules to form analysis result data. The visualization report unit is used to perform semantic parsing on the analysis results data, and to perform visualization rendering based on a preset mapping rule library to form a visual and standardized analysis report.

[0006] Thirdly, embodiments of the present invention provide an electronic device, including: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the above-described multi-dimensional benchmark data analysis method based on a modular framework.

[0007] Fourthly, embodiments of the present invention provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the aforementioned multi-dimensional benchmark data analysis method based on a modular framework.

[0008] This invention provides a method and apparatus for multi-dimensional benchmark data analysis based on a modular framework. The method, in response to user terminal requests, establishes a unified connection with various benchmark tools and receives raw performance datasets. It then standardizes and transforms heterogeneous data formats to form structured performance data. Next, based on the tool type, it calls a preset data analysis plugin interface to convert the data into comparable data, thereby constructing a data matrix to perform multi-dimensional comparative analysis. An adaptive benchmark selection algorithm is used to automatically determine the benchmark system, and fluctuation detection is performed based on statistical rules to generate analysis result data. Finally, the analysis result data undergoes semantic parsing and is visualized and rendered according to a mapping rule base to generate a standardized analysis report. By establishing a unified data connection and standardized transformation, the entire process of benchmark data processing—from raw acquisition and in-depth analysis to result display—is automated and closed-loop, solving the problem of tool silos caused by the heterogeneity of data formats in various benchmarking tools. Standardized transformation and automated plugin routing significantly improve the processing efficiency and compatibility of heterogeneous performance data, reducing manual costs and error rates in the data preprocessing stage. The introduction of adaptive benchmarking algorithms and statistical fluctuation detection eliminates subjective errors in manually selecting benchmark systems, enhancing the objectivity, scientific rigor, and accuracy of performance analysis results, overcoming the shortcomings of traditional methods such as strong subjectivity and inconsistent comparison standards caused by manual benchmark system selection. By semantically parsing the analysis results data and automatically matching visualization templates, intelligent matching of analysis results and visualization charts is achieved, generating standardized analysis reports without manual intervention, significantly improving the efficiency and readability of performance evaluation report generation. Attached Figure Description

[0009] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a multi-dimensional benchmark data analysis method based on a modular framework, as described in Embodiment 1 of the present invention. Figure 2 This is a flowchart of a multi-dimensional benchmark data analysis method based on a modular framework, as described in Embodiment 2 of the present invention. Figure 3 This is a flowchart of a multi-dimensional benchmark data analysis method based on a modular framework, as described in Embodiment 3 of the present invention. Figure 4 This is a schematic diagram of the structure of a multi-dimensional benchmark test data analysis device based on a modular framework as described in Embodiment 4 of the present invention; Figure 5 This is a structural diagram of the electronic device described in Embodiment 5 of the present invention; Figure 6This is a flowchart illustrating one embodiment of the present invention. Detailed Implementation

[0010] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0011] Example 1 Figure 1 The flowchart of a multi-dimensional benchmark data analysis method based on a modular framework as described in Embodiment 1 of the present invention specifically includes the following steps: S101 responds to test data analysis requests initiated by user terminals, establishes a unified connection with various benchmark testing tools, receives raw performance datasets, performs standardized conversion and validity checks on heterogeneous data formats in the raw performance datasets, and forms structured performance data with a unified format.

[0012] When a test engineer initiates a test data analysis request through a user terminal, a unified connection is first established with various benchmarking tools (such as Unixbench, FIO, Stream, etc.) or the database or file system storing the results of these tools to receive the raw performance datasets generated by these tools. Since the output formats (such as JSON, XML, text logs) and units (such as KB / s, MB / s, ops / s) of different benchmarking tools vary, they need to be standardized into an internally unified format using preset conversion rules. Validity checks are then performed to verify the legality of each parameter, removing negative values ​​or data that significantly exceeds physical limits. Finally, it is determined whether the benchmarking tool type specified in the request is supported, resulting in structured performance data with a clear structure and unified fields.

[0013] S102 identifies the benchmark tool type based on the structured performance data and calls the corresponding type's preset data analysis plugin interface to convert the structured performance data into comparable data with a unified format.

[0014] By parsing the tool identifier field (such as benchmark_tool=FIO / Unixbench / Unknown, etc.) in the structured performance data, the type of benchmark testing tool to which it belongs can be identified. Then, it can be routed to the corresponding preset data analysis plugin interface, utilizing the specific tool's business logic encapsulated within the plugin (such as extracting single-core / multi-core scores for Unixbench, and classifying FIO by read / write mode). A unified interface is provided to further convert the tool-specific performance indicators in the structured performance data into unified, structured, and comparable data that includes environmental context (such as kernel version, CPU architecture) and key performance indicators. This eliminates the data format differences between different tools and provides a data structure that can be directly associated, queried, and compared horizontally for subsequent multi-dimensional comparative analysis.

[0015] S103: Extract the fields corresponding to the preset analysis dimensions based on the comparable data, construct a data matrix for multi-dimensional comparative analysis, automatically select the corresponding benchmark system for performance comparison through an adaptive benchmark selection algorithm, and perform performance fluctuation detection based on statistical rules to form analysis result data.

[0016] Based on the dimension fields specified in the request (such as kernel version, CPU architecture), the corresponding fields are extracted from the comparable data, and a multi-dimensional data matrix is ​​constructed using the extracted data. Then, an adaptive benchmark selection algorithm is used to automatically select the most suitable benchmark system for performance comparison from multiple candidate systems. Furthermore, based on statistical rules, the latest test data is compared with all other historical data to identify any unexpected performance fluctuations in the current test, ultimately generating analytical results data that includes performance comparison differences and fluctuations.

[0017] S104 performs semantic parsing on the analysis results data, performs visualization rendering based on the preset mapping rule library, and generates a visual standardized analysis report.

[0018] By performing deep semantic analysis on the results, the system identifies the characteristics of the current data, such as whether the data represents a trend over time or is a comparative data set across multiple systems. Based on a pre-defined mapping rule library, it automatically matches the optimal chart template (such as a line chart, a bar chart with error bars, or a radar chart) for rendering. Finally, the rendered charts and analysis results are assembled into a standardized analysis report for users to download or view.

[0019] This embodiment establishes a unified connection with various benchmarking tools and receives raw performance datasets in response to user terminal requests. It then standardizes and transforms heterogeneous data formats to form structured performance data. Subsequently, it calls preset data analysis plugin interfaces according to tool type to convert the data into comparable data, and then constructs a data matrix to perform multi-dimensional comparative analysis. It also uses an adaptive benchmark selection algorithm to automatically determine the benchmark system and performs fluctuation detection based on statistical rules to form analysis result data. Finally, it performs semantic parsing on the analysis result data and performs visualization rendering according to the mapping rule base to generate a standardized analysis report. By establishing a unified data connection and standardized transformation, the entire process of benchmark data processing—from raw acquisition and in-depth analysis to result display—is automated and closed-loop, solving the problem of tool silos caused by the heterogeneity of data formats in various benchmarking tools. Standardized transformation and automated plugin routing significantly improve the processing efficiency and compatibility of heterogeneous performance data, reducing manual costs and error rates in the data preprocessing stage. The introduction of adaptive benchmarking algorithms and statistical fluctuation detection eliminates subjective errors in manually selecting benchmark systems, enhancing the objectivity, scientific rigor, and accuracy of performance analysis results, overcoming the shortcomings of traditional methods such as strong subjectivity and inconsistent comparison standards caused by manual benchmark system selection. By semantically parsing the analysis results data and automatically matching visualization templates, intelligent matching of analysis results and visualization charts is achieved, generating standardized analysis reports without manual intervention, significantly improving the efficiency and readability of performance evaluation report generation.

[0020] Example 2 Figure 2 This is a flowchart of a multi-dimensional benchmark data analysis method based on a modular framework as described in Embodiment 2 of the present invention. This embodiment is an optimization based on the above embodiment. In this embodiment, S101 is specifically optimized as follows: Establish a unified data connection with multiple benchmarking tools and obtain the corresponding raw performance data; The raw performance data is parsed and performance indicators are extracted. The units of the performance indicators output by different benchmark testing tools are standardized and converted into a unified format and then filled into the structured data model to form structured performance data. Based on the pre-defined list of required fields and reasonable numerical ranges, the structured performance data is subjected to integrity checks and validity checks respectively.

[0021] Accordingly, the multi-dimensional benchmark data analysis method based on a modular framework provided in this embodiment specifically includes: S201 establishes a unified data connection with multiple benchmarking tools and obtains the corresponding raw performance data.

[0022] A unified data access layer supports multiple protocol connections, including but not limited to querying MySQL databases, reading files from the file system, or retrieving data via APIs. For example, a unified connection interface can be used to connect to various benchmarking tools. This interface defines three core methods: `connect(config)` to establish a connection based on configuration parameters; `fetch_data(query_params)` to retrieve raw data based on query conditions; and `disconnect()` to close the connection and release resources. Different connector classes can be implemented based on different types of data sources (benchmarking tools). For `MySQLConnector`, the connection logic is implemented for MySQL databases; for `FileSystemConnector`, the connection logic is implemented for file systems; and for `APIConnector`, the connection logic is implemented for REST API interfaces. Upon receiving an analysis request, the corresponding connector object is instantiated based on the data source configuration information (such as data source type, connection address, and authentication information), the `connect` method is called to establish a connection, and then the `fetch_data` method is called to perform the data retrieval operation.

[0023] S202 involves parsing the raw performance data and extracting performance metrics. The units of the performance metrics output by different benchmark testing tools are standardized and converted to a unified format and then filled into the structured data model to form structured performance data.

[0024] When parsing raw performance data and extracting performance metrics, specialized parsers can be implemented for the raw data formats of different benchmark tools to perform unit normalization. For example, the kilobytes per second (KB / s) output by tool A and the bytes per second (B / s) output by tool B can be uniformly converted to the internal standard unit MB / s; timestamps in different formats, such as Unix timestamps and ISO 8601 formats, can be uniformly converted to the standard date format. For instance, the raw output of the FIO tool typically contains multiple lines of text, including "read: io=19908KB, bw=403456000 B / s, iops=98500". Its specialized parser can use regular expressions to extract the bandwidth value 403456000 and the IOPS value 98500. For the UnixBench tool, its specialized parser can use regular expressions to extract the Dhrystone score and total score. The extracted metrics are then standardized using unit conversion. For example, a unit conversion mapping table can be maintained, defining the conversion relationship between the original units and internal standard units of each tool. This allows mapping the original unit of FIO bandwidth (bytes / second, B / s) and the original unit of the UnixBench File Copy metric (kilobytes / second, KBps) to megabytes / second (MB / s). The converted data is then populated into a structured data model, which can be designed using a relational data table structure. This model includes fields such as record_id (auto-incrementing primary key), system_id (foreign key, associated with the system information table), tool_type (enumeration type, values ​​unixbench / fio / stream), test_scenario (string, describing the test scenario), metric_name (string, metric name), metric_value (floating-point number, numerical value), metric_unit (string, unit), test_time (datetime, test time), and dimensions (JSON type, storage environment dimension information).

[0025] S203, based on the preset list of required fields and reasonable numerical ranges, performs integrity checks and validity checks on the structured performance data respectively.

[0026] After data population is complete, integrity and validity checks need to be performed. In the integrity check, a pre-defined list of required fields is used. This list can be manually defined for different tool types. If fields such as system kernel version or core performance metric values ​​are missing, the data is marked as invalid. For example, for the fio tool, the list requires records with metric_name as iops and bandwidth, and each record must include system_id, tool_type, metric_value, and test_time fields. All records are iterated to check for any missing required fields, and missing records are excluded. In the validity check, based on pre-defined validity thresholds (e.g., network latency cannot be negative, iops should be between 1000-2000000, bandwidth between 1-10000 MB / s, and unixbench score between 1-10000), the corresponding fields of each record are checked to ensure they fall within the specified ranges. Invalid data outside these ranges is filtered out to ensure the data quality for subsequent analysis.

[0027] S204 identifies the benchmark tool type based on the structured performance data and calls the corresponding type's preset data analysis plugin interface to convert the structured performance data into comparable data with a unified format.

[0028] Optionally, based on the various benchmarking tools established with the connection, a preset data analysis plugin interface is configured for the benchmarking tools to determine whether they can process the current data, perform analysis, and return analysis results in a unified format.

[0029] To achieve high scalability, a plug-in-based tool adaptation architecture is adopted to support rapid integration of new benchmarking tools. A unified plug-in interface is defined, and each benchmark's dedicated analyzer is encapsulated as an independent plug-in module. First, an abstract benchmark analyzer interface (IBenchmarkAnalyzer) is defined. This interface stipulates that all plug-ins encapsulating specialized analyzers must implement two methods: one for plug-in self-checking whether it can handle the currently identified tool type, and the other for executing specific business analysis logic.

[0030] An optional implementation of this embodiment is to maintain a tool processing routing table to build a mapping relationship between benchmark testing tools and corresponding dedicated analyzers, encapsulate each dedicated analyzer as an independent plugin, and implement a unified interface to form a preset data analysis plugin interface.

[0031] To achieve high scalability, each dedicated analyzer is encapsulated as an independent plugin. These plugins collectively implement a standard interface, which defines two standard methods: `can_analyze(data)`: receives structured performance data, determines whether the analyzer can process the data based on data characteristics (such as the `tool_type` field and data content characteristics), and returns a boolean value (true for processing, false for not processing); `analyze(data)`: executes the specific analysis logic, receives structured performance data as input, analyzes and processes it, and returns analysis results in a unified format. A dynamically updated tool processing routing table is maintained; this can be a mapping dictionary where the key is the benchmark tool type identifier and the value is the corresponding analyzer plugin instance, recording the mapping relationship between benchmark tools and dedicated analyzers. When a new plugin for a new benchmark tool is developed, it is simply placed in the plugin directory and the service is restarted; the plugin will automatically register its supported tool identifiers with the tool routing table. Subsequently, when the tool type of the benchmark tool is identified, the tool routing table will automatically assign the task to the plugin containing the corresponding dedicated analyzer to execute the targeted analysis algorithm and encapsulate the results into a uniformly formatted comparable data packet.

[0032] A plugin directory is maintained based on the independent plugins packaged for each dedicated analyzer, which is used to register the preset data analysis plugin interfaces of the corresponding benchmark tools.

[0033] Each benchmarking tool has its own independent plugin module, and each module contains an analyzer class that implements the IBenchmarkAnalyzer interface. For example, the UnixBenchAnalyzer plugin implements the `can_analyze` method to check if the input data's `tool_type` is `unixbench`, or if the data content contains keywords such as Dhrystone or System Benchmarks Index Score; it also implements the `analyze` method to extract 12 core metrics from UnixBench and, combined with environmental information, generate a performance data context package. Similarly, the FIOAnalyzer plugin implements the `can_analyze` method to check if the `tool_type` is `fio`, or if the data content contains keywords such as IOPS or BW; and it also implements the `analyze` method to parse IOPS, bandwidth, and latency metrics, categorize and aggregate them according to test mode (random read, sequential write, mixed random read / write, etc.), and generate a performance data context package. For the StreamAnalyzer plugin: Implement the `can_analyze` method to check if `tool_type` is `stream`, or check if the data content contains keywords such as `Copy`, `Scale`, `Add`, and `Triad`; implement the `analyze` method to extract four memory bandwidth values ​​and generate a performance data context package. Each plugin maintains a plugin directory. During startup, scanning this directory retrieves the compiled class files of all plugin modules stored within, and all modules in the directory are dynamically loaded. For each loaded plugin, check if it contains a class that implements the `IBenchmarkAnalyzer` interface. If so, instantiate the analyzer object and call its `can_analyze` method with a test data sample to confirm the tool types the analyzer can handle. After confirmation, register the analyzer object in the routing table, with the key being the tool type identifier that the analyzer can handle (e.g., unixbench, fio, stream).

[0034] Based on the identified benchmark tool type, the routing table is processed by querying the tool, and the corresponding preset data analysis plugin interface is called to analyze the structured performance data, forming comparable data in a uniform format.

[0035] Based on the `tool_type` field value in the structured performance data, the benchmark tool type can be identified, and the corresponding dedicated analyzer instance can be retrieved by querying the tool routing table. For example, when `tool_type=fio`, an `FIOAnalyzer` instance is retrieved from the routing table, its `analyze` method is called, and the structured data is passed in for analysis. After analysis, `FIOAnalyzer` returns a uniformly formatted, comparable dataset containing the performance data context. When support for new benchmark tools is needed, test engineers only need to write a new plugin module, implement the `IBenchmarkAnalyzer` interface, and place the plugin module's code file in the plugin directory. The plugin will be automatically scanned, loaded, and registered upon restart or hot reload, achieving plug-and-play extensibility. For example, comparable data includes: system identifier fields system_id and hostname; hardware dimension fields cpu_arch (e.g., x86_64, aarch64), cpu_model (e.g., Kunpeng 920), and disk_type (e.g., NVMe SSD, SATA HDD); software dimension fields os_name, os_version (e.g., CentOS 7.9, Kylin V10), and kernel_version; and business / project dimension fields project_name (e.g., “New version kernel performance verification”, “Server selection”). These reflect environmental context information, describing the system on which the test is performed, and are typically derived from system environment information at test startup or information manually specified by the user. It also includes tool information fields such as benchmark_tool (e.g., “fio”), benchmark_version, and test_scenario (e.g., “4k_random_read”); and performance metric fields, which are a list or dictionary of multiple testing tools. Each metric contains name, value, unit, category, etc. Metrics are encapsulations of various values ​​extracted from the output of benchmark testing tools, such as Dhrystone in Unixbench, bw and iops in FIO, etc. They reflect various performance metrics and are used to describe what kind of results were measured. They are specific performance data extracted, transformed, and standardized by the benchmark tool layer.

[0036] S205: Extract fields corresponding to preset analysis dimensions based on comparable data, construct a data matrix for multi-dimensional comparative analysis, automatically select the appropriate benchmark system for performance comparison through an adaptive benchmark selection algorithm, and perform performance fluctuation detection based on statistical rules to form analysis result data.

[0037] S206 performs semantic parsing on the analysis results data, performs visualization rendering based on the preset mapping rule library, and generates a visual standardized analysis report.

[0038] This embodiment establishes a unified data connection with multiple benchmarking tools, extracts performance metrics, and performs unified unit conversion. It also performs dual checks on data integrity and validity based on a list of required fields and reasonable numerical ranges. Each tool is configured with an independent analysis plugin and a unified standard interface is implemented. The maintenance tool manages the routing table and plugin directory, enabling automatic identification, registration, and routing of different tool types. Through a unified data access and verification mechanism for multiple benchmarking tools, differences in data formats and units of measurement between different benchmarking tools are eliminated, ensuring data quality from the source. A pluggable architecture, encapsulating each dedicated analyzer as an independent plugin module, achieves modular decoupling of tool analysis capabilities, providing high scalability. The maintenance tool manages the routing table and plugin directory, enabling dynamic loading and automatic registration of plugin modules. When new benchmarking tools need support, they can be used plug-and-play without modifying the core code, reducing system maintenance costs and upgrade difficulty.

[0039] Example 3 Figure 3 This is a flowchart of a multi-dimensional benchmark data analysis method based on a modular framework as described in Embodiment 3 of the present invention. This embodiment is an optimization based on the above embodiment. In this embodiment, S103 is specifically optimized as follows: Based on the preset analysis dimensions, extract the field content of the corresponding analysis dimensions from the comparable data and construct a dimension comparison matrix; An adaptive benchmark system selection is performed using a multidimensional weighted scoring algorithm, and the performance is compared with the benchmark system with the highest weighted score using a dimensional comparison matrix to obtain the performance comparison results. By using historical test results that exclude the most recent test result from the comparable data, the historical performance mean is calculated and the fluctuation control limit is determined based on statistical process control rules. The result is then compared with the most recent test result to obtain the performance fluctuation result.

[0040] Accordingly, the multi-dimensional benchmark data analysis method based on a modular framework provided in this embodiment specifically includes: S301 responds to test data analysis requests initiated by user terminals, establishes a unified connection with various benchmark testing tools, receives raw performance datasets, performs standardized conversion and validity checks on heterogeneous data formats in the raw performance datasets, and forms structured performance data with a unified format.

[0041] S302 identifies the benchmark tool type based on the structured performance data and calls the corresponding type's preset data analysis plugin interface to convert the structured performance data into comparable data with a unified format.

[0042] S303: Based on the preset analysis dimensions, extract the field content of the corresponding analysis dimensions from the comparable data and construct a dimension comparison matrix.

[0043] Based on the performance analysis dimensions included in the request, the corresponding row and column fields are extracted from the comparable data, and a dimension comparison matrix is ​​constructed. For example, with data from 100 servers, if the preset analysis dimensions included in the request are cpu_arch (CPU architecture) and os_version (operating system version), and the benchmark analysis tools used include FIO and Unixbench, the constructed dimension comparison matrix is ​​shown in Table 1. Table 1 Dimensional Comparison Matrix Using comparable data, the following parameters can be extracted from the context packets: os_version=CentOS7.9, cpu_arch=x86_64 for server A; os_version=Kylin V10, cpu_arch=x86_64 for server B; and os_version=Kylin V10, cpu_arch=aarch64 for server C. When constructing the dimensional comparison matrix, a two-dimensional matrix is ​​built using operating system version and CPU architecture as row indices and test items as column indices. The mean and standard deviation are calculated based on the results of multiple tests. For example, the dimensions include hardware dimensions cpu_arch, cpu_model, and memory_size; software dimensions os_name, os_version, and kernel_version; configuration dimensions disk_type and filesystem; and business dimension project_name. After obtaining the analysis dimension information, these field names are used as candidate analysis dimensions. When the user selects or determines a set of dimensions based on the analysis objective, a temporary data cube is built in memory to prepare for the next step of analysis.

[0044] S304 uses a multidimensional weighted scoring algorithm to adaptively select a benchmark system, and uses a dimensional comparison matrix to compare the performance with the benchmark system with the highest weighted score to obtain the performance comparison results.

[0045] To select the most suitable benchmark system for comparison, a multi-dimensional weighted scoring algorithm is used to score candidate systems from multiple different dimensions. The optimal benchmark system is then selected and its performance is compared with a dimensional comparison matrix. When selecting a benchmark system, different weights can be assigned to different dimensions according to actual performance requirements. The benchmark system selected after weighted scoring can then be used for performance comparisons with a focus on specific user needs. The benchmark system should possess the best representativeness and stability to ensure the fairness and reliability of the comparison results.

[0046] Optionally, for multiple pre-set candidate systems, weighted calculations are performed on the dimensions of data completeness, test sample sufficiency, performance stability, and environmental typicality, and the system with the highest score is determined as the benchmark system.

[0047] When performing multi-dimensional weighted scoring, relevant information from all candidate systems can be collected based on user-defined and selected candidate systems. Data completeness score, test sample sufficiency score, performance stability score, and environment typicality score are calculated separately. These four scores are then weighted and summed to obtain a total score for each candidate system. The system with the highest score is selected as the benchmark system for performance comparison. For example, the data completeness score checks whether the system has completed all specified benchmark test suites, such as running both Unixbench and Stream and FIO tests. A higher completion rate results in a higher score, ensuring the benchmark system provides comprehensive comparison dimensions. The test sample sufficiency score checks whether each test item has sufficient repeated samples, such as whether the FIO test has been run more than three times. A full score is awarded if the required number of samples is met, while points are deducted proportionally for insufficient samples, indicating that more samples mean lower randomness in individual test results and more reliable mean values. The performance stability score calculates the coefficient of variation (COP) based on multiple sampling results for each test item. The coefficient of performance (CV) is calculated as the standard deviation divided by the mean. A smaller CV indicates less performance fluctuation and higher stability. A lower average CV across all test items results in a higher score for this item. Selecting a system with stable performance as the benchmark makes the comparison more meaningful and avoids misjudgments due to drastic fluctuations in the benchmark itself. The environmental typicality score checks historical data for mainstream configuration information, such as the most frequently purchased CPU models and OS versions in the past six months. It calculates the matching degree between the candidate system configuration and the mainstream configuration; a higher matching degree results in a higher score. The most mainstream and common configuration is selected as the benchmark, ensuring that the performance improvement rate conclusions are relevant to most real-world scenarios. The weighted sum score weights and sums the above four items separately. For example, data completeness score 0.2, test sample sufficiency score 0.2, performance stability score 0.4, and environmental typicality score 0.2. The total score for each candidate system is calculated, and the system with the highest score is selected as the benchmark. Additional validation can be performed: if the difference between the highest and second-highest scores is extremely small (e.g., <5%), the user can be prompted that "there are two candidate benchmarks; manual selection is recommended" to avoid arbitrary algorithmic decisions.

[0048] S305: Using historical test results that exclude the latest test result from the comparable data, calculate the historical performance mean and determine the fluctuation control limit based on statistical process control rules, and compare it with the latest test result to obtain the performance fluctuation result.

[0049] To identify abnormal increases or decreases in performance metrics within time-series performance data, volatility detection can be performed by combining threshold triggering rules and statistical process control (SPC) rules. This involves selecting historical test results excluding the most recent test result, determining control limits based on SPC rules, and then comparing these results with the most recent test result to determine if performance fluctuations exist. Integrating the performance comparison results with the performance volatility results yields the analysis results.

[0050] One optional implementation of this embodiment is to compare the mean change of the latest test result with the historical test results in the comparable data, and determine whether there is a suspected anomaly based on whether the difference in the comparison exceeds a preset fluctuation threshold.

[0051] A simple threshold trigger rule can be used to identify suspected anomalies, such as performance changes exceeding ±10%. Based on the requirements for performance stability or fluctuation range, a fluctuation threshold can be preset. The average of historical data excluding the most recent test result is calculated and compared with the most recent test result. Whether the difference between the two exceeds the preset fluctuation threshold determines whether a suspected anomaly exists. For example, the fluctuation threshold can be set to 20%. If the difference between the last test result and the historical data average exceeds 20%, it is considered a suspected performance fluctuation anomaly.

[0052] Based on statistical process control rules, control lines for the center line, upper control limit, and lower control limit are determined by excluding the latest test result from historical test results. These control lines are then compared with the latest test result to determine whether there are statistical anomalies, thus obtaining the performance fluctuation results.

[0053] Drawing on statistical process control rules from industrial quality control and statistical theories, this approach provides a more rigorous statistical analysis of fluctuations based on historical operating patterns. Using historical test results excluding the most recent test, the mean and standard deviation are calculated to determine the center line, upper control limit, and lower control limit. Then, based on the most recent test result, if it falls outside this range, even if the percentage fluctuation does not appear abnormal, it can be considered a departure from historical operating patterns and thus classified as a statistical anomaly. This, combined with any suspected anomalies, forms the performance fluctuation result. For example, based on historical data [P1, ..., Pn-1] (excluding the most recent point to be tested), the mean (μ) and standard deviation (σ) are calculated. Then, the center line (CL) is set to μ, the upper control limit (UCL) to μ + 3σ, and the lower control limit (LCL) to μ - 3σ (3σ is a common choice, covering 99.7% of normally distributed data). Finally, the most recent data point Pn is checked; if Pn > UCL or Pn < LCL, it is classified as a statistical anomaly.

[0054] S306 performs semantic parsing on the analysis results data, performs visualization rendering based on the preset mapping rule base, and generates a visual standardized analysis report.

[0055] Specifically, the analysis results data is parsed and the analysis type field, data structure field, number of indicators field, and number of dimensions field are extracted.

[0056] The analysis results are in JSON format, containing performance comparison results and performance fluctuation results. Key fields extracted include: `analysis_type` (analysis type), with values ​​such as "multi_dimension_comparison", "time_series_trend", and "correlation_analysis"; `data_structure` (data structure), with values ​​such as "category_vs_values" and "series_over_time"; `metric_count` (number of metrics to display); and `dimension_count` (number of dimensions involved).

[0057] Based on the extracted fields, the best chart template is matched in the preset chart data feature mapping rule library.

[0058] Based on the extracted fields, the best chart template is matched in the preset chart type-data feature mapping rule base. This rule base is a configurable mapping table that defines the chart templates corresponding to different combinations of fields.

[0059] For example, if the analysis_type field is "multi_dimension_comparison" and the data_structure field is "category_vs_values" (category vs. value), the metric_count field has 1 value, and the number of dimensions is greater than 5, this indicates a scenario comparing a single continuous numerical indicator (score) across categorical dimensions (OS version). This primarily involves comparing categories versus values. Based on the mapping rule library, the optimal chart representation is a bar chart to visually represent the numerical differences between categories. Furthermore, since the data also includes standard deviation, a clustered bar chart template with error bars can be matched to visually display the fluctuation range of each data set. If the data structure field includes "time series," the optimal chart representation is a line chart with trend prediction lines. If the analysis_type field includes "abnormal fluctuations," the optimal chart representation is a box plot or control chart to highlight performance anomalies.

[0060] The analysis results data and optimal chart templates are used to render and generate a standardized analysis report.

[0061] The dynamically generated charts and analysis conclusions are automatically populated into a standardized analysis report template. A chart rendering engine is then invoked for visualization. Based on the input matching template identifier and analysis results data, the rendering engine generates the basic structure of the charts (coordinate system, axes, legend), and then maps the analysis results data onto the visual elements of the charts, forming an analysis report to present to test engineers for intuitive review and analysis of the benchmark test results. For example, candidate system names can be mapped to the horizontal axis category, the performance mean to the height of the bars, and the standard deviation to the length of the error bars at the top of the bars. The rendering engine also automatically generates chart titles (e.g., "Comparison of UnixBench Total Scores under Different Operating System Versions and CPU Architectures"), axis labels (e.g., "Tested System," "UnixBench Total Score"), and legend descriptions based on the dimensional information in the analysis results data. After the charts are rendered, they can be assembled into a standardized analysis report in HTML / CSS format. Text information from the analysis results is filled into text placeholders in the report template, and the rendered charts are embedded into these placeholders as Base64-encoded images or SVG vector graphics, forming a complete report file that is returned to the user's terminal. The user's terminal can then view the complete performance analysis report in their browser. For example... Figure 6 The diagram shown is a flowchart of one implementation of this method.

[0062] This embodiment extracts dimensional fields from comparable data and constructs a dimensional comparison matrix. It then adaptively selects a benchmark system using a multi-dimensional weighted scoring algorithm that incorporates data completeness, sample sufficiency, performance stability, and environmental typicality. Combining the variation range of the latest test points and historical averages, along with the centerline and upper and lower control limits calculated based on statistical process control rules, it performs dual performance fluctuation judgment. Finally, it analyzes the semantic features of the analysis results and matches the best chart template from the rule base to complete the rendering of a standardized report. Through multi-dimensional weighted scoring calculations, the determination process of the benchmark system has a quantitative basis, ensuring the fairness and rationality of performance comparisons. The introduction of statistical process control (SPC) theory can keenly capture abnormal performance fluctuations that exceed statistical regularities, effectively distinguishing between normal system jitter and true performance degradation, improving the sensitivity and reliability of anomaly detection. Through semantically driven visualization mapping, it ensures that complex multi-dimensional analysis data can be presented in the most intuitive and logical chart form, providing users with an easy-to-understand analysis report.

[0063] Example 4 Figure 4 This is a schematic diagram of a multi-dimensional benchmark data analysis device based on a modular framework according to Embodiment 4 of the present invention. In this embodiment, the multi-dimensional benchmark data analysis device based on a modular framework includes: The data access unit 810 is used to respond to test data analysis requests, establish a unified connection with various benchmark testing tools, receive raw performance datasets, perform standardization conversion and validity checks, and form structured performance data with a unified format. The data conversion unit 820 is used to identify the type of benchmark testing tool based on the structured performance data and call the corresponding type of preset data analysis plugin interface to convert the structured performance data into comparable data with a unified format. The data analysis unit 830 is used to extract fields corresponding to preset analysis dimensions based on comparable data, construct a data matrix for multi-dimensional comparative analysis, automatically select the corresponding benchmark system for performance comparison through an adaptive benchmark selection algorithm, and perform performance fluctuation detection based on statistical rules to form analysis result data. The visualization report unit 840 is used to perform semantic parsing on the analysis results data, perform visualization rendering according to the preset mapping rule library, and form a visual standardized analysis report.

[0064] This embodiment establishes a unified connection through a data access unit and performs data standardization conversion to form structured performance data. The data conversion unit identifies the type of benchmark testing tool and calls the corresponding type of preset data analysis plugin interface to generate comparable data. The data analysis unit constructs a data matrix and adaptively selects a benchmark system for comparison. Based on statistical rules, performance fluctuation detection is performed to form analysis result data. The visualization report unit parses the data and performs visualization rendering to form a standardized analysis report. By establishing a unified data connection and standardized transformation, the entire process of benchmark data processing—from raw acquisition and in-depth analysis to result display—is automated and closed-loop, solving the problem of tool silos caused by the heterogeneity of data formats in various benchmarking tools. Standardized transformation and automated plugin routing significantly improve the processing efficiency and compatibility of heterogeneous performance data, reducing manual costs and error rates in the data preprocessing stage. The introduction of adaptive benchmarking algorithms and statistical fluctuation detection eliminates subjective errors in manually selecting benchmark systems, enhancing the objectivity, scientific rigor, and accuracy of performance analysis results, overcoming the shortcomings of traditional methods such as strong subjectivity and inconsistent comparison standards caused by manual benchmark system selection. By semantically parsing the analysis results data and automatically matching visualization templates, intelligent matching of analysis results and visualization charts is achieved, generating standardized analysis reports without manual intervention, significantly improving the efficiency and readability of performance evaluation report generation.

[0065] The multi-dimensional benchmark data analysis device based on a modular framework provided in this embodiment of the invention can execute the multi-dimensional benchmark data analysis method based on a modular framework provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0066] Example 5 Figure 5 This is a structural diagram of an electronic device according to Embodiment 5 of the present invention. Figure 5 A block diagram is shown of an exemplary electronic device 12 suitable for implementing embodiments of the present invention. Figure 5 The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0067] like Figure 5 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0068] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0069] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0070] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0071] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0072] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the electronic device 12 / server / computer, and / or with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. Figure 5 As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although... Figure 5 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0073] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the multi-dimensional benchmark data analysis method based on a modular framework provided in the embodiments of the present invention.

[0074] Example 6 Embodiment 6 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the multi-dimensional benchmark data analysis method based on a modular framework as provided in the above embodiments.

[0075] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0076] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0077] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0078] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A multi-dimensional benchmark data analysis method based on a modular framework, characterized in that, include: S101 responds to test data analysis requests initiated by user terminals, establishes a unified connection with various benchmark testing tools and receives raw performance datasets, performs standardized conversion and validity checks on heterogeneous data formats in the raw performance datasets, and forms structured performance data with a unified format. S102: Identify the benchmark tool type based on the structured performance data, and call the corresponding type's preset data analysis plugin interface to convert the structured performance data into comparable data with a unified format; S103: Extract the fields corresponding to the preset analysis dimensions based on the comparable data, construct a data matrix for multi-dimensional comparative analysis, automatically select the corresponding benchmark system for performance comparison through an adaptive benchmark selection algorithm, and perform performance fluctuation detection based on statistical rules to form analysis result data; S104 performs semantic parsing on the analysis results data, performs visualization rendering based on the preset mapping rule library, and generates a visual standardized analysis report.

2. The method according to claim 1, characterized in that, S101 includes: Establish a unified data connection with multiple benchmarking tools and obtain the corresponding raw performance data; The raw performance data is parsed and performance indicators are extracted. The units of the performance indicators output by different benchmark testing tools are standardized and converted into a unified format and then filled into the structured data model to form structured performance data. Based on the pre-defined list of required fields and reasonable numerical ranges, the structured performance data is subjected to integrity checks and validity checks respectively.

3. The method according to claim 1, characterized in that, S102 further includes: Based on the various benchmarking tools that have established connections, a preset data analysis plugin interface is configured for the benchmarking tools to determine whether they can process the current data, perform analysis, and return analysis results in a unified format.

4. The method according to claim 3, characterized in that, The configuration of preset data analysis plugin interfaces for benchmark testing tools includes: By maintaining a tool processing routing table to build a mapping relationship between benchmark testing tools and corresponding dedicated analyzers, each dedicated analyzer is encapsulated as an independent plugin and a unified interface is implemented to form a preset data analysis plugin interface; Based on the independent plugins packaged for each dedicated analyzer, maintain a plugin directory to register the preset data analysis plugin interfaces of the corresponding benchmark tools; Based on the identified benchmark tool type, the routing table is processed by querying the tool, and the corresponding preset data analysis plugin interface is called to analyze the structured performance data, forming comparable data in a uniform format.

5. The method according to claim 1, characterized in that, S103 includes: Based on the preset analysis dimensions, extract the field content of the corresponding analysis dimensions from the comparable data and construct a dimension comparison matrix; An adaptive benchmark system selection is performed using a multidimensional weighted scoring algorithm, and the performance is compared with the benchmark system with the highest weighted score using a dimensional comparison matrix to obtain the performance comparison results. By using historical test results that exclude the most recent test result from the comparable data, the historical performance mean is calculated and the fluctuation control limit is determined based on statistical process control rules. The result is then compared with the most recent test result to obtain the performance fluctuation result.

6. The method according to claim 5, characterized in that, S103 further includes: For multiple candidate systems, weighted calculations are performed based on the dimensions of data completeness, test sample sufficiency, performance stability, and environmental typicality, and the system with the highest score is determined as the benchmark system. The mean change of the latest test result and the historical test results in the comparable data are compared, and the presence of suspected anomalies is determined based on whether the difference exceeds the preset fluctuation threshold. Based on statistical process control rules, control lines for the center line, upper control limit, and lower control limit are determined by excluding the latest test result from historical test results. These control lines are then compared with the latest test result to determine whether there are statistical anomalies, thus obtaining the performance fluctuation results.

7. The method according to claim 1, characterized in that, S104 includes: Parse and analyze the results data, and extract the analysis type field, data structure field, number of indicators field, and number of dimensions field; Based on the extracted fields, the best chart template is matched in the preset chart data feature mapping rule library; The analysis results data and optimal chart templates are used to render and generate a standardized analysis report.

8. A multi-dimensional benchmark data analysis device based on a modular framework, used to implement the multi-dimensional benchmark data analysis method based on a modular framework as described in any one of claims 1-7, characterized in that, include: The data access unit is used to respond to test data analysis requests, establish a unified connection with various benchmark testing tools, receive raw performance datasets, perform standardization transformation and validity checks, and form structured performance data with a unified format. The data conversion unit is used to identify the type of benchmark testing tool based on the structured performance data, and call the corresponding type of preset data analysis plugin interface to convert the structured performance data into comparable data in a uniform format. The data analysis unit is used to extract fields corresponding to preset analysis dimensions based on comparable data, construct a data matrix for multi-dimensional comparative analysis, automatically select the appropriate benchmark system for performance comparison through an adaptive benchmark selection algorithm, and perform performance fluctuation detection based on statistical rules to form analysis result data. The visualization report unit is used to perform semantic parsing on the analysis results data, and to perform visualization rendering based on a preset mapping rule library to form a visual and standardized analysis report.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-dimensional benchmark data analysis method based on a modular framework as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the multi-dimensional benchmark data analysis method based on a modular framework as described in any one of claims 1-7.

Images