Distributed storage test acceleration method and device, electronic equipment and storage medium

Abstract

The application discloses a distributed storage test acceleration method and device, electronic equipment and a storage medium, relates to the technical field of distributed storage, and through operation semantics and dependency relationship identification of a test case of a to-be-tested distributed storage system and decomposition into independently executable test slices, a multilayer parallel pipeline including a test environment preparation layer, a test data management layer, a test operation execution layer and a test result verification layer is constructed based on the dependency relationship to realize parallel processing of the test slices, and system resources and test slice execution states are monitored in real time during execution and are dynamically scheduled and distributed according to the monitoring results and the dependency relationship, and meanwhile, parallel execution results are distributed collected, consistency verified and test reports are generated, so that the problems of long test period, serious resource waste and insufficient test reliability caused by linear execution of test cases, insufficient resource utilization and lack of systematicness of result verification in the prior art can be solved.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a distributed storage testing acceleration method and apparatus, electronic device and storage medium. Background Technology

[0002] Distributed storage is a core component of modern IT infrastructure, widely used in cloud storage, big data platforms, and other fields. With the acceleration of digital transformation, its system scale continues to expand, data distribution, consistency mechanisms, and network topology are becoming increasingly complex, and failure scenarios are becoming more diverse, placing higher demands on the comprehensiveness and efficiency of testing.

[0003] Current automated testing of distributed storage mainly relies on traditional linear execution models, which have significant bottlenecks: First, it depends on management and manual definition, resulting in a high blocking rate and easily causing test execution to be blocked; second, test data preparation lacks intelligent prefetching and caching mechanisms, leading to repeated generation and distribution that is time-consuming; third, resource utilization is low, with insufficient average utilization of computing, storage, and network resources, and test nodes, bandwidth, and storage I / O often remaining idle; fourth, the fault recovery mechanism is imperfect, requiring the entire test process to be rerun when nodes fail or the network is interrupted, resulting in wasted time. These problems significantly lengthen the testing cycle, increase manpower and resource costs, and severely restrict the development iteration speed and quality assurance capabilities of distributed storage systems. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for accelerating distributed storage testing, in order to at least solve the problem of long testing cycles in automated distributed storage testing in related technologies.

[0005] This application provides a method for accelerating distributed storage testing, including:

[0006] The test cases of the distributed storage system under test are analyzed to identify the operational semantics and dependencies, and the test cases are decomposed into multiple independently executable test slices.

[0007] Based on the aforementioned dependencies, a multi-layer parallel pipeline is constructed, enabling the test slices to be processed in parallel within the corresponding pipeline layers. The multi-layer parallel pipeline includes a test environment preparation layer, a test data management layer, a test operation execution layer, and a test result verification layer.

[0008] During execution, the system resource status and test slice execution status are monitored in real time, and the test slices are dynamically scheduled and allocated based on the monitoring results and the aforementioned dependencies.

[0009] The intermediate and final results generated by parallel execution are collected and verified in a distributed manner, and a test report is generated.

[0010] This application also provides a distributed storage testing acceleration device, including:

[0011] The identification unit is used to analyze the test cases of the distributed storage system under test, identify the operational semantics and dependencies, and decompose the test cases into multiple independently executable test slices.

[0012] The building unit is used to build a multi-layer parallel pipeline based on the dependency relationship, so that the test slice is processed in parallel in the corresponding pipeline layer. The multi-layer parallel pipeline includes a test environment preparation layer, a test data management layer, a test operation execution layer and a test result verification layer.

[0013] The monitoring unit is used to monitor the system resource status and test slice execution status in real time during the execution process, and to dynamically schedule and allocate test slices based on the monitoring results and the dependencies.

[0014] The generation unit is used to perform distributed collection and consistency verification of intermediate and final results generated by parallel execution, and to generate test reports.

[0015] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above methods.

[0016] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above methods.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0018] This application provides a distributed storage testing acceleration method, apparatus, electronic device, and storage medium. It identifies and decomposes test cases of the distributed storage system under test into independently executable test slices based on operational semantics and dependencies. A multi-layered parallel pipeline, comprising a test environment preparation layer, a test data management layer, a test operation execution layer, and a test result verification layer, is constructed based on these dependencies to achieve parallel processing of the test slices. During execution, system resources and the execution status of the test slices are monitored in real time, and dynamic scheduling and allocation are performed based on monitoring results and dependencies. Simultaneously, the parallel execution results are collected in a distributed manner, consistency verification is performed, and a test report is generated. Therefore, it can solve the problems of long test cycles, serious resource waste, and insufficient test reliability in existing technologies caused by linear execution of test cases, inefficient dependency management, insufficient resource utilization, and lack of systematic result verification. This achieves the technical effects of improving the efficiency of automated distributed storage testing, optimizing resource utilization, ensuring the consistency and accuracy of test results, and reducing test costs.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 A flowchart illustrating a distributed storage testing acceleration method provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a distributed storage test acceleration device provided in an embodiment of this application. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0025] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0026] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] The specific application environment architecture or specific hardware architecture on which the distributed storage test acceleration method depends is described here.

[0028] Embodiments of this application provide a method for accelerating distributed storage testing. Figure 1 This is a flowchart illustrating a distributed storage testing acceleration method provided in an embodiment of this application.

[0029] like Figure 1 As shown, the method includes the following steps:

[0030] Step 101: Analyze the test cases of the distributed storage system under test, identify the operational semantics and dependencies, and decompose the test cases into multiple independently executable test slices.

[0031] In the embodiments of this disclosure, step 101 aims to systematically analyze the test cases of the distributed storage system under test, accurately identify the semantic features of each operation in the test cases and the dependencies between operations, and then reasonably decompose the originally holistic test cases into multiple test slices with independent execution capabilities. The identification of operation semantics covers the essential definition of core behaviors such as various functional operations, resource interactions, and parameter configurations during the testing process, while the identification of dependencies covers the relationships between operations in dimensions such as data flow, control logic, and resource consumption. The decomposition of test slices is based on the core principle of "independent execution," ensuring that each slice can still complete specific test objectives even without other unrelated operations, providing a foundation for subsequent parallel processing. As one implementation method, a combination of static analysis and dynamic monitoring can be used to identify operation semantics and dependencies. Dependencies are analyzed by constructing a relationship graph, and the slice granularity is adjusted according to test requirements and resource conditions, ultimately outputting a set of test slices that meet the requirements.

[0032] By accurately identifying operational semantics and dependencies and splitting test slices, the limitations of the traditional linear execution of test cases are broken, maximizing the parallel execution potential of test tasks. At the same time, it reduces the blocking problems caused by dependency confusion in subsequent execution, providing a key prerequisite for the construction and dynamic scheduling of multi-layer parallel pipelines, and effectively improving the flexibility and execution efficiency of distributed storage automated testing.

[0033] Step 102: Construct a multi-layer parallel pipeline based on the dependency relationship, so that the test slices are processed in parallel in the corresponding pipeline layers. The multi-layer parallel pipeline includes a test environment preparation layer, a test data management layer, a test operation execution layer, and a test result verification layer.

[0034] In the embodiments of this disclosure, based on the aforementioned identified test case dependencies, a multi-layered parallel pipeline is constructed, encompassing a test environment preparation layer, a test data management layer, a test operation execution layer, and a test result verification layer. This allows each independently executable test slice to achieve efficient parallel processing within its corresponding pipeline layer, according to its functional attributes and dependency requirements. Each pipeline layer carries the core tasks of different stages of distributed storage testing. The test environment preparation layer is responsible for building a runtime environment adapted to the test requirements; the test data management layer focuses on the generation, distribution, and integrity verification of test data; the test operation execution layer enables concurrent execution of test slices; and the test result verification layer performs preliminary collection and verification of results. Each layer performs its specific function while also coordinating to ensure the orderly parallel advancement of the testing process. As one implementation method, the test environment preparation layer can use configuration management tools to perform multi-node environment initialization and software deployment in parallel; the test data management layer can use distributed tools to generate test data and distribute it through message middleware; the test operation execution layer can use a concurrency framework to support concurrent execution of test operations; and the test result verification layer can use a stream processing framework to achieve parallel collection of results, thus adapting to the testing requirements of large-scale distributed storage.

[0035] By constructing a multi-layered parallel pipeline with clearly defined functions based on dependencies, the testing process is decomposed into independent stages that can be advanced in parallel. This breaks the parallelism limitations of traditional linear testing processes and effectively solves the problems of resource idleness and low efficiency caused by poor connection of test tasks and insufficient parallel processing in existing technologies. It achieves efficient collaboration and optimized resource allocation in each stage of testing, and significantly improves the overall processing efficiency and process regularity of distributed storage automated testing.

[0036] Step 103: During execution, the system resource status and test slice execution status are monitored in real time, and the test slices are dynamically scheduled and allocated based on the monitoring results and the aforementioned dependencies.

[0037] In the embodiments of this disclosure, during the execution of test slices in a multi-layer parallel pipeline, the system resource status (including the occupancy and available capacity of various resources such as computing, storage, and network) and the execution status of the test slices (including execution progress, whether they are running normally, and whether any anomalies have occurred) are continuously monitored in real time. Based on the monitoring results and the aforementioned identified dependencies, the execution priority, allocation nodes, and execution sequence of each test slice are dynamically scheduled and rationally allocated to ensure that the execution of the test slices both conforms to dependency constraints and adapts to real-time resource conditions. As one implementation method, dynamic allocation can be achieved through a multi-dimensional scheduling mechanism. By using monitoring tools to collect resource data and execution status information in real time, suitable nodes are allocated for different types of test slices, such as compute-intensive and I / O-intensive ones. At the same time, when node failures, uneven loads, or dependency conflicts are detected, the scheduling strategy is adjusted in a timely manner to ensure the progress of the test.

[0038] By combining real-time monitoring and dynamic scheduling, the system effectively solves the problems of resource waste and test blockage caused by rigid resource allocation, delayed handling of dependency conflicts, and insufficient fault recovery capabilities in existing technologies. It achieves optimized allocation of system resources, ensures the continuity and stability of test execution, and further improves the overall efficiency and reliability of distributed storage automated testing.

[0039] Step 104: Distributed collection and consistency verification of intermediate and final results generated by parallel execution, and generation of test report.

[0040] In the embodiments of this disclosure, intermediate and final results generated by the parallel execution of test slices in a multi-layer parallel pipeline are comprehensively collected in a distributed manner to ensure that test results across nodes and pipeline layers are aggregated without omission. Simultaneously, systematic consistency verification is performed on the collected results to verify the logical correctness and consistency of the data during parallel execution. Finally, a test report containing key test information is generated based on the verified valid results. Distributed collection focuses on achieving efficient aggregation and low-latency transmission of results, consistency verification checks for potential deviations in results caused by parallel execution, and the test report clearly presents the test execution status, core indicators, and verification conclusions, providing a complete basis for test evaluation. As one implementation method, a stream processing framework can be used to achieve streaming collection and preprocessing of results, employing methods such as hash comparison and logical verification to complete consistency verification, ultimately generating a test report covering multiple dimensions including execution summary, performance indicators, and fault analysis.

[0041] By collecting distributed results, verifying systematic consistency, and generating standardized test reports, this technology effectively solves the problems of scattered parallel execution results, difficulty in ensuring consistency, and lack of comprehensive support for test conclusions in existing technologies. It ensures the accuracy and completeness of test results, provides a reliable basis for the test evaluation of distributed storage systems, and further improves the closed-loop quality of automated testing.

[0042] This application provides a method for accelerating distributed storage testing. It identifies and decomposes test cases of the distributed storage system under test into independently executable test slices based on operational semantics and dependencies. A multi-layered parallel pipeline, comprising a test environment preparation layer, a test data management layer, a test operation execution layer, and a test result verification layer, is constructed based on these dependencies to achieve parallel processing of the test slices. During execution, system resources and the execution status of the test slices are monitored in real time, and dynamic scheduling and allocation are performed based on monitoring results and dependencies. Simultaneously, the parallel execution results are collected in a distributed manner, consistency verification is performed, and test reports are generated. Therefore, this method can solve the problems of long test cycles, serious resource waste, and insufficient test reliability caused by linear execution of test cases, inefficient dependency management, insufficient resource utilization, and lack of systematic result verification in existing technologies. It achieves the technical effects of improving the efficiency of automated testing of distributed storage, optimizing resource utilization, ensuring the consistency and accuracy of test results, and reducing test costs.

[0043] As a specific implementation of this disclosure, based on the basic scheme, the analysis of test cases for the distributed storage system under test is further defined to identify the operational semantics and dependencies therein, and the test cases are decomposed into multiple independently executable test slices, including: performing static semantic analysis and dynamic behavior capture on the test cases to obtain an operation sequence with semantic tags; constructing a directed acyclic graph representing the dependencies based on the data transfer relationships, execution order constraints, and shared resource occupancy among the operations in the operation sequence; and performing topological partitioning of the operations based on the directed acyclic graph to form the test slices.

[0044] Specifically, when analyzing test cases of the distributed storage system under test, static analysis tools such as abstract syntax tree parsing are first used to parse static information such as function calls and parameter configurations in the test scripts. At the same time, dynamic instrumentation frameworks such as Pin are used to capture dynamic behaviors such as I / O operation sequences during test case execution. Through the collaboration of these two methods, static semantic analysis and dynamic behavior capture are completed, and each operation is labeled with semantic tags such as "data write" and "configuration change", forming an operation sequence with semantic tags. Then, based on this operation sequence, data transit dependencies are determined by analyzing the transit relationships of input and output variables between operations, execution order constraints are clarified by parsing conditional branches and loop structures, and resource dependencies are defined by identifying the occupancy of shared resources such as file locks. Combining the above three types of dependency relationships, a directed acyclic graph (DAG) representing all dependency logic is constructed using graph algorithms. Finally, a topological sorting algorithm is used to sort the operations in the DAG in order, and the topology is divided according to the principle of "operations without dependencies or whose dependencies are satisfied are grouped together" to ensure that each group of operations can be executed independently, ultimately forming multiple test slices that meet the requirements.

[0045] By combining static and dynamic methods to accurately capture operational semantics, constructing a standardized directed acyclic graph based on multi-dimensional dependencies, and then generating test slices through topology partitioning, the independence and rationality of test slices are ensured. This effectively avoids parallel execution conflicts caused by incomplete dependency identification or improper slice partitioning, providing a reliable foundation for the efficient operation of subsequent multi-layer parallel pipelines.

[0046] As a specific implementation of this disclosure, based on the basic scheme, the static semantic analysis and dynamic behavior capture of the test cases to obtain operation sequences with semantic tags are further defined, including: extracting operation call logic by parsing the syntax structure of the test script; tracking system calls and input / output behaviors during test execution, and assigning functional semantic tags to the captured operations.

[0047] Specifically, when performing static semantic analysis on test cases, an abstract syntax tree parsing tool is used to deconstruct the syntactic structure of the test script layer by layer. This involves traversing syntactic elements such as function definitions, parameter passing, and nested statements to accurately extract the operation call logic, clarifying the call order, associated parameters, and triggering conditions of each operation, thus forming a static operation logic list. During test execution, dynamic instrumentation frameworks such as Pin are used to track system call behavior in real time (including I / O operations, resource allocation and release, configuration modification, and other related system calls). Simultaneously, the input data format, output result form, and interaction process with external components for each operation are recorded, achieving comprehensive capture of dynamic execution behavior. Based on the statically extracted operation call logic and dynamically captured execution behavior, combined with the functional scenario requirements of distributed storage testing, each operation is assigned a corresponding functional semantic label. For example, data writing operations are labeled "data writing," system parameter adjustment operations are labeled "configuration change," and resource allocation operations are labeled "resource occupation." Finally, these are integrated to form a clearly structured and semantically explicit sequence of operations with semantic labels.

[0048] By combining static syntax parsing with dynamic behavior tracking, the integrity of operation call logic extraction is ensured, and the authenticity of execution behavior capture is achieved. The assigned functional semantic tags accurately reflect the core attributes of the operation, providing accurate and reliable basic data for subsequent dependency identification and test slice division, effectively avoiding subsequent process deviations caused by semantic ambiguity.

[0049] As a specific implementation of this disclosure, based on the basic scheme, the step of performing topological partitioning of operations according to the directed acyclic graph to form the test slice includes: predicting the execution time and resource consumption of each operation or operation group based on historical execution data; and adjusting the granularity or resource allocation strategy of the test slice based on the prediction results.

[0050] Specifically, when topologically partitioning operations based on a Directed Acyclic Graph (DAG) to form test slices, historical execution data is first extracted from the database. This data includes key indicators such as the actual execution time, CPU utilization, memory consumption, storage I / O throughput, and network bandwidth usage of each operation and operation group in similar past test cases. Subsequently, machine learning models such as regression models are used to train and analyze this historical data to construct execution time prediction models and resource consumption prediction models. These models predict the expected execution time of the current operation or operation group to be partitioned, as well as the scale of demand for various resources such as computing, storage, and network. Based on the above prediction results, if an operation group is predicted to have a short execution time and low resource consumption, multiple such operations are combined into one test slice to reduce scheduling overhead. If an operation is predicted to have a long execution time or high resource consumption, it is split into multiple sub-operation groups to form independent test slices to avoid single-point blocking. At the same time, test slices with high resource demand are preferentially allocated to high-performance nodes or nodes with high resource idle rates, while slices with low resource demand are flexibly allocated to ordinary nodes. This completes the dynamic adjustment of test slice granularity and resource allocation strategy, ultimately forming an optimized set of test slices.

[0051] By using historical data-driven prediction to accurately match slice granularity with resource requirements, we can avoid scheduling redundancy caused by overly fine slices and solve the parallel blocking problem caused by overly coarse slices. At the same time, we can achieve precise resource allocation, significantly improve the parallel execution efficiency of test slices and the system resource utilization, and ensure the smoothness of subsequent pipeline execution.

[0052] As a specific implementation of this disclosure, based on the basic scheme, the construction of a multi-layer parallel pipeline based on the dependency relationship is further defined, so that the test slices are processed in parallel in the corresponding pipeline layers, including: in the test environment preparation layer, the environment initialization, software deployment and parameter configuration of the test nodes are completed in parallel; in the test data management layer, test data is generated in parallel, the data is distributed to each test node and integrity is verified; in the test operation execution layer, the test actions contained in the test slice are executed concurrently and cross-node cross-validation is performed; in the test result verification layer, the output of each node is collected in parallel and the results are compared and analyzed.

[0053] Specifically, when building a multi-layered parallel pipeline based on dependencies, the test environment preparation layer uses configuration management tools such as Ansible to perform environment initialization operations on multiple test nodes, including basic system configuration, installation of dependent components, and parallel deployment of distributed storage software such as Ceph. Key parameters such as network buffer size and storage I / O scheduling strategies are uniformly optimized according to test requirements to ensure consistency across nodes. The test data management layer uses distributed data generation tools such as Fio to generate large-scale test data (e.g., 1TB of data) in parallel. Message middleware such as Kafka efficiently distributes the data to each test node, and a checksum verification mechanism is used to verify the integrity of the distributed data, preventing data loss or corruption during transmission. The test operation execution layer uses multi-threading or concurrent frameworks such as MPI to drive each test node to synchronously execute test actions in the test slice. During execution, cross-node cross-validation is achieved by comparing the execution results of different nodes, and core metrics such as data read / write consistency are verified in real time. The test result verification layer uses Apache... Streaming frameworks such as Flink collect test results from each node in parallel, and use difference detection algorithms such as text diff or numerical comparison to perform batch comparison and analysis of the results to accurately identify abnormal data. Then, based on template engines such as Jinja2, the analysis results are integrated to form a standardized preliminary report.

[0054] By precisely implementing parallel tasks across each pipeline layer, the entire process of test environment preparation, data processing, operation execution, and result verification is carried out in parallel. This not only avoids the waiting time in traditional linear processes, but also ensures the accuracy of the testing process through cross-node verification and integrity checks, significantly improving the overall throughput and reliability of distributed storage testing.

[0055] As a specific implementation of this disclosure, based on the basic solution, it is further limited to the test environment preparation layer, where a configuration management tool is used to perform batch and parallel environment settings for multiple test nodes.

[0056] Specifically, in the test environment preparation layer, Ansible is selected as the configuration management tool. Standardized playbook scripts are written to define the basic system configurations required for the test environment (such as operating system version adaptation, firewall rule configuration, user permission settings), the installation list of dependent components (such as drivers, middleware, and test toolsets), and the deployment parameters and runtime environment requirements of distributed storage software (such as Ceph). Leveraging the batch parallel execution capability of this configuration management tool, environment configuration instructions are sent to multiple target test nodes simultaneously, achieving synchronous initialization of multiple nodes and avoiding the time accumulation caused by traditional serial configuration. In the software deployment phase, the tool ensures that the storage software versions of all test nodes are consistent by remotely executing deployment scripts in batches, and automatically completes the unified optimization of key parameters such as network buffer size, storage IO scheduling strategy, and cluster communication protocol according to preset parameters. The entire process does not require manual operation on each node. The tool's status verification function provides real-time feedback on the configuration progress and results of each node, ensuring that the runtime environment of all test nodes is completely consistent and meets the test requirements.

[0057] By using configuration management tools to enable batch parallel environment setup for multiple test nodes, the test environment preparation cycle is significantly shortened, errors and inconsistencies caused by manual configuration are avoided, and the manpower cost of environment setup is reduced. This provides a standardized and highly consistent basic environment support for the parallel execution of subsequent test slices.

[0058] As a specific implementation of this disclosure, based on the basic scheme, it is further limited to the test data management layer, where a distributed data generation tool is used to concurrently generate test data, and asynchronous data distribution is performed through a message middleware.

[0059] Specifically, in the test data management layer, Fio is selected as the distributed data generation tool. Parameters such as data capacity, data format, and read / write mode are configured according to the test scenario requirements. Concurrent generation of test data is achieved through multi-node distributed deployment, for example, simultaneously generating 1TB of diverse test data to meet the large-scale testing needs of distributed storage. Simultaneously, Kafka is introduced as a message middleware to build an asynchronous data distribution architecture. The concurrently generated test data is written to a message queue according to a preset partitioning strategy. Each test node, acting as a consumer, autonomously pulls the corresponding data shards from the queue, initiating the distribution process without waiting for all data to be generated, thus achieving parallel progress in data generation and distribution. During data distribution, a checksum verification mechanism is used to verify the integrity of the data pulled by each node, comparing the checksum value with the checksum result of the original generated data to ensure that no data is lost or corrupted during asynchronous transmission, ultimately completing the deployment of test data for each test node.

[0060] The concurrent capabilities of distributed data generation tools significantly improve the efficiency of test data generation. By leveraging the asynchronous distribution characteristics of message middleware, the time-series dependency of data generation and distribution is broken, reducing waiting time. At the same time, integrity verification ensures data reliability, effectively solving the problem of long preparation time for large-scale test data and providing efficient and reliable data support for the parallel execution of subsequent test operations.

[0061] As a specific implementation of this disclosure, based on the basic scheme, the resource status of the real-time monitoring system and the execution status of the test slice are further defined. The test slice is dynamically scheduled and allocated according to the monitoring results and the dependencies, including: continuously collecting the resource utilization indicators and network status of each computing node; dynamically determining the execution node and startup sequence of the test slice according to resource availability, the dependencies and task queue status; and automatically triggering task retry, migration or skipping mechanisms when node failure, network interruption or task execution timeout is detected.

[0062] Specifically, during real-time monitoring of system resource status and test slice execution status, collection components are deployed on each computing node through monitoring agents such as Prometheus to continuously collect resource utilization indicators such as CPU utilization, remaining memory, storage I / O throughput, network bandwidth utilization, and network latency. Simultaneously, network status data such as network connection stability and data transmission success rate are captured. All monitoring data is uploaded in real-time to the central scheduling node for aggregation and analysis. The central scheduler combines the collected resource availability data, a pre-built dependency graph (DAG), and task queue status such as task queue length and priority of pending slices on each node, using a multi-objective optimization algorithm to dynamically decide the execution node for the test slice—placing the test slice on the central scheduler. Computation-intensive slices are assigned to nodes with high CPU idle rates, while I / O-intensive slices are assigned to nodes with sufficient storage resources. The slice startup sequence is strictly determined by dependencies, ensuring that subsequent slices are started only after the preceding dependent slices are completed. When a node failure is detected (such as abnormal CPU or memory crashes), unfinished test slices on that node are automatically migrated to healthy nodes with sufficient resources for continued execution. When a network interruption is detected, a network reconnection mechanism is triggered first; if reconnection fails, unfinished slices are retried. When a test slice execution time exceeds a preset threshold (set based on historical execution data) and is deemed timed out, if the slice does not affect the overall test logic, it is skipped directly; if it is a critical slice, it is reassigned to another node for execution.

[0063] By comprehensively monitoring resource and network status, and combining dependency relationships and dynamic scheduling of task queues, we ensure that the execution nodes of test slices are in line with the real-time status of the timing adaptation system. At the same time, we prevent test process interruptions through targeted fault handling mechanisms, effectively improving resource utilization and test execution continuity, and reducing time waste caused by abnormal situations.

[0064] As a specific implementation of this disclosure, based on the basic scheme, the dynamic scheduling and allocation process is further defined to adopt a multi-objective optimization strategy, and the optimization objectives include at least one of the following: shortest overall test time, highest resource utilization, and highest system reliability.

[0065] Specifically, the multi-objective optimization strategy adopted in the dynamic scheduling and allocation process first clarifies the quantitative evaluation indicators of each optimization objective: the shortest overall test time is measured by the total time from the start of the test task to its completion; the highest resource utilization rate is measured by the comprehensive average of CPU utilization, memory usage, storage I / O throughput and network bandwidth utilization; and the highest system reliability is measured by the test task's fault-free completion rate and anomaly recovery success rate. Based on the above indicators, a scheduling optimization model is constructed using multi-objective optimization algorithms such as NSGA-II. The real-time resource status of each computing node, the dependencies between test slices, task priorities, and historical execution data are used as input parameters. The algorithm solves for the Pareto optimal scheduling solution set. The central scheduler, considering the requirements of the current test scenario, selects a suitable scheduling scheme from the solution set. If the test scenario prioritizes iterative efficiency, the scheme with the shortest overall test time as the core objective is selected, and high-priority test slices are allocated to nodes with idle resources to shorten the parallel execution cycle. If resource conservation is prioritized, the highest resource utilization is ensured, and load balancing algorithms are used to avoid node idleness or overload. If test stability is prioritized, the highest system reliability is the primary objective, and critical test slices are allocated to nodes with low failure rates, with redundant resources reserved to handle anomalies. Simultaneously, the achievement status of each objective is fed back in real time during the scheduling process, and optimization weights are dynamically adjusted to ensure a coordinated balance among multiple objectives.

[0066] By using a multi-objective optimization strategy to coordinate test time, resource utilization, and system reliability, the problem of neglecting one aspect for another caused by a single optimization objective is avoided. This approach can maximize the compression of the test cycle, make full use of hardware resources to reduce waste, and improve the test process's ability to withstand anomalies. It significantly enhances the overall performance and scenario adaptability of distributed storage automated testing.

[0067] As a specific implementation of this disclosure, based on the basic scheme, the distributed collection and consistency verification of intermediate and final results generated by parallel execution is further defined, including: real-time aggregation of result data reported by each test node in a streaming processing manner; and consistency comparison of results from different nodes for the same test target to determine whether they meet preset semantic consistency or data consistency constraints.

[0068] Specifically, when performing distributed collection and consistency verification of intermediate and final results generated by parallel execution, a streaming collection channel is constructed using a stream processing framework such as Apache Flink. During the execution of test slices, each test node encapsulates intermediate results, final outputs, and execution logs in a preset format and reports them to the stream processing channel in real time. The framework achieves low-latency aggregation of result data through distributed node deployment, and performs real-time filtering, deduplication, and key indicator extraction preprocessing on the reported data to ensure efficient and complete data transmission. In the consistency verification phase, for the same test target results from different nodes, the preset constraint standards are first defined—semantic consistency constraints focus on the correctness of operational logic, such as verifying whether the functional feedback after each node performs the "data write" operation conforms to the preset semantic rules. Data consistency constraints focus on the uniformity of the data itself. The check value of the output data of each node is calculated by hash comparison algorithm and compared with the benchmark value, or version vectors are used to record the data update sequence to verify the cross-node data synchronization status. Then, the system judges whether the results of all nodes simultaneously meet the semantic and data consistency constraints, and accurately identifies result deviations.

[0069] Real-time aggregation of results is achieved through streaming processing, avoiding the delays and data omissions caused by traditional batch collection. Combined with targeted consistency constraint comparison, the logical correctness and data consistency of parallel execution results are ensured, effectively solving the problems of scattered results and difficulty in verifying consistency in distributed testing, and providing accurate and reliable core data support for test reports.

[0070] As a specific implementation of this disclosure, based on the basic scheme, it is further defined that when the consistency verification finds inconsistent results, the abnormal results are automatically identified and a targeted retesting process is triggered.

[0071] Specifically, when verifying the consistency of parallel execution results, the consistency verification engine uses methods such as hash comparison and version vector to compare the intermediate and final results of different test nodes for the same test target dimension by dimension. When a result is detected to be inconsistent with the preset semantic consistency constraints or data consistency constraints, an anomaly identification mechanism is automatically activated—recording the test node ID, test slice number, inconsistent data fields and verification difference corresponding to the abnormal result. At the same time, the abnormal result is tagged with "to be reviewed" and stored in the abnormal result database to form a traceable anomaly record. Subsequently, a targeted retesting process is triggered. Based on the pre-built dependency graph (DAG), the scope of the anomaly association is located, and only the test slice directly related to the abnormal result (rather than the entire test case) is selected and reassigned to healthy nodes with idle resources for execution. During the re-execution process, the data transmission and execution status are monitored in real time to ensure that the test conditions are consistent with the original execution scenario. After the retest is completed, the results are verified for consistency again to form a closed-loop process.

[0072] By automatically identifying abnormal results, problems can be accurately traced, and targeted retesting can be triggered instead of a full rerun. This effectively avoids the time wasted by restarting the entire testing process when results are inconsistent, as is the case with existing technologies. This significantly improves the efficiency of exception handling, ensuring the accuracy of test results and further compressing the overall testing cycle.

[0073] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0074] Corresponding to the aforementioned distributed storage testing acceleration method, this disclosure also proposes a distributed storage testing acceleration device. Since the device embodiments of this disclosure correspond to the aforementioned method embodiments, details not disclosed in the device embodiments can be referred to the aforementioned method embodiments, and will not be repeated here.

[0075] Figure 2 This is a schematic diagram of the structure of a distributed storage test acceleration device provided in an embodiment of this disclosure, as shown below. Figure 2 As shown, it includes:

[0076] The identification unit 21 is used to analyze the test cases of the distributed storage system under test, identify the operational semantics and dependencies therein, and decompose the test cases into multiple independently executable test slices.

[0077] Construction unit 22 is used to construct a multi-layer parallel pipeline based on the dependency relationship, so that the test slice is processed in parallel in the corresponding pipeline layer. The multi-layer parallel pipeline includes a test environment preparation layer, a test data management layer, a test operation execution layer and a test result verification layer.

[0078] Monitoring unit 23 is used to monitor the system resource status and test slice execution status in real time during the execution process, and to dynamically schedule and allocate test slices based on the monitoring results and the dependencies.

[0079] The generation unit 24 is used to perform distributed collection and consistency verification of intermediate and final results generated by parallel execution, and to generate a test report.

[0080] This application provides a distributed storage testing acceleration device that identifies and decomposes test cases of the distributed storage system under test into independently executable test slices based on their operational semantics and dependencies. A multi-layered parallel pipeline, comprising a test environment preparation layer, a test data management layer, a test operation execution layer, and a test result verification layer, is constructed based on these dependencies to achieve parallel processing of the test slices. During execution, system resources and the execution status of the test slices are monitored in real time, and dynamic scheduling and allocation are performed based on the monitoring results and dependencies. Simultaneously, the parallel execution results are collected in a distributed manner, consistency verification is performed, and test reports are generated. Therefore, this device can solve the problems of long test cycles, serious resource waste, and insufficient test reliability in existing technologies caused by linear execution of test cases, inefficient dependency management, insufficient resource utilization, and lack of systematic result verification. It achieves the technical effects of improving the efficiency of automated testing of distributed storage, optimizing resource utilization, ensuring the consistency and accuracy of test results, and reducing test costs.

[0081] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0082] For a description of the features in the embodiment corresponding to the distributed storage test acceleration device, please refer to the relevant description of the embodiment corresponding to the distributed storage test acceleration method, which will not be repeated here.

[0083] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the distributed storage test acceleration method.

[0084] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the distributed storage test acceleration method.

[0085] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0086] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the distributed storage test acceleration method.

[0087] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the distributed storage test acceleration method.

[0088] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0089] The above provides a detailed description of a distributed storage testing acceleration method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for accelerating distributed storage testing, characterized in that, include: The test cases of the distributed storage system under test are analyzed to identify the operational semantics and dependencies, and the test cases are decomposed into multiple independently executable test slices. This process includes: performing static semantic analysis and dynamic behavior capture on the test cases to obtain operation sequences with semantic tags; constructing a directed acyclic graph (DAG) representing the dependencies based on the data transfer relationships, execution order constraints, and shared resource occupancy among the operations in the operation sequences; and performing topological partitioning of the operations based on the DAG to form the test slices. A multi-layered parallel pipeline is constructed based on the aforementioned dependencies, enabling the test slices to be processed in parallel within their respective pipeline layers. This multi-layered parallel pipeline includes a test environment preparation layer, a test data management layer, a test operation execution layer, and a test result verification layer. Specifically, constructing the multi-layered parallel pipeline based on the aforementioned dependencies, enabling the test slices to be processed in parallel within their respective pipeline layers, includes: parallelly performing environment initialization, software deployment, and parameter configuration for test nodes in the test environment preparation layer; parallelly generating test data, distributing the data to each test node, and performing integrity verification in the test data management layer; concurrently executing the test actions contained in the test slice and performing cross-node cross-validation in the test operation execution layer; and parallelly collecting the outputs of each node in the test result verification layer, performing result comparison and analysis. During execution, the system resource status and test slice execution status are monitored in real time, and the test slices are dynamically scheduled and allocated based on the monitoring results and the aforementioned dependencies. The intermediate and final results generated by parallel execution are collected and verified in a distributed manner, and a test report is generated.

2. The distributed storage testing acceleration method according to claim 1, characterized in that, The static semantic analysis and dynamic behavior capture of the test cases yield an operation sequence with semantic tags, including: Extract the operation call logic by parsing the syntax structure of the test script; During the test run, system calls and input / output behaviors are tracked, and the captured operations are assigned functional semantic labels.

3. The distributed storage testing acceleration method according to claim 1, characterized in that, The step of performing topological partitioning of the operations based on the directed acyclic graph to form the test slice includes: Based on historical execution data, predict the execution time and resource consumption of each operation or operation group; Adjust the granularity or resource allocation strategy of the test slice based on the prediction results.

4. The distributed storage testing acceleration method according to claim 1, characterized in that, In the test environment preparation layer, a configuration management tool is used to perform batch and parallel environment settings for multiple test nodes.

5. The distributed storage testing acceleration method according to claim 1, characterized in that, In the test data management layer, a distributed data generation tool is used to generate test data concurrently, and the data is distributed asynchronously through a message middleware.

6. The distributed storage testing acceleration method according to claim 1, characterized in that, The real-time monitoring system resource status and test slice execution status, based on the monitoring results and the dependencies, dynamically schedules and allocates test slices, including: Continuously collect resource utilization metrics and network status of each computing node; The execution node and startup sequence of the test slice are dynamically determined based on resource availability, the aforementioned dependencies, and the task queue status. When a node failure, network interruption, or task execution timeout is detected, a task retry, migration, or skip mechanism is automatically triggered.

7. The distributed storage testing acceleration method according to claim 6, characterized in that, The dynamic scheduling and allocation process adopts a multi-objective optimization strategy, with optimization objectives including at least one of the following: shortest overall test time, highest resource utilization, and highest system reliability.

8. The distributed storage testing acceleration method according to claim 1, characterized in that, The distributed collection and consistency verification of intermediate and final results generated by parallel execution includes: The results data reported by each test node are aggregated in real time using a streaming processing method; Consistency comparison is performed on results from different nodes targeting the same test objective to determine whether they meet preset semantic consistency or data consistency constraints.

9. The distributed storage testing acceleration method according to claim 8, characterized in that, When consistency verification finds inconsistent results, the abnormal results are automatically identified and a targeted retesting process is triggered.

10. A distributed storage testing acceleration device, characterized in that, include: The identification unit is used to analyze test cases of the distributed storage system under test, identify the operational semantics and dependencies, and decompose the test cases into multiple independently executable test slices. The analysis of test cases of the distributed storage system under test, identifying the operational semantics and dependencies, and decomposing the test cases into multiple independently executable test slices includes: performing static semantic analysis and dynamic behavior capture on the test cases to obtain operation sequences with semantic tags; constructing a directed acyclic graph (DAG) representing dependencies based on the data transfer relationships, execution order constraints, and shared resource occupancy among the operations in the operation sequence; and performing topological partitioning of the operations based on the DAG to form the test slices. A construction unit is used to build a multi-layer parallel pipeline based on the dependencies, enabling the test slices to be processed in parallel within the corresponding pipeline layers. The multi-layer parallel pipeline includes a test environment preparation layer, a test data management layer, a test operation execution layer, and a test result verification layer. The process of building the multi-layer parallel pipeline based on the dependencies, enabling the test slices to be processed in parallel within the corresponding pipeline layers, includes: in the test environment preparation layer, performing environment initialization, software deployment, and parameter configuration of the test nodes in parallel; in the test data management layer, generating test data in parallel, distributing the data to each test node, and performing integrity verification; in the test operation execution layer, concurrently executing the test actions contained in the test slice and performing cross-node cross-validation; and in the test result verification layer, collecting the outputs of each node in parallel and performing result comparison and analysis. The monitoring unit is used to monitor the system resource status and test slice execution status in real time during the execution process, and to dynamically schedule and allocate test slices based on the monitoring results and the dependencies. The generation unit is used to perform distributed collection and consistency verification of intermediate and final results generated by parallel execution, and to generate test reports.

11. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the distributed storage test acceleration method according to any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the distributed storage test acceleration method according to any one of claims 1-9.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the distributed storage test acceleration method according to any one of claims 1-9.

Images