Performance test method, device and equipment of distributed file system and medium

Abstract

This invention discloses a performance testing method, apparatus, device, and medium for distributed file systems, applicable to the field of data processing. The method includes: acquiring at least two tasks to be tested using big data methods; generating a scheduling strategy for the tasks to be tested through the Airflow platform; performing load testing on each task to be tested sequentially based on the scheduling strategy to obtain processing results; and summarizing each processing result to obtain the load testing result. This invention improves the efficiency of performance testing for distributed file systems.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and particularly relates to a performance testing method and device of a distributed file system, equipment and medium. BACKGROUND

[0002] With the development of computer technology, data is growing explosively, and various new distributed systems for big data computing have emerged in succession. The data processed by these systems is stored in the underlying distributed file system. The distributed file system is used to provide file storage services for upper-layer applications in a distributed environment. Compared with a single computer file system, the distributed file system has good scalability. It is usually built on multiple computer nodes, so its storage capacity can usually be expanded with the increase in the number of storage nodes. In addition, multiple storage nodes usually provide concurrent access mechanisms, which makes the read-write throughput of the distributed file system also scalable to meet the needs of upper-layer applications.

[0003] For a distributed file system, performance testing can well reflect the support capability of the system for upper-layer applications. With more and more distributed file systems being widely used in various distributed environments, it is particularly important to perform unified performance testing on the distributed file system. First, the performance testing results can guide users to select the most suitable distributed file system for the application. A unified performance testing method can compare the performance of the same application on different distributed file systems, thereby giving users guidance for selection. Second, performance testing can continuously improve the distributed file system itself. For developers, a unified performance testing method can use different applications to test the same distributed file system, discover problems in a timely manner, reflect the bottlenecks and deficiencies of the system, and thus can be optimized and improved in a targeted manner.

[0004] Existing performance testing methods for distributed file systems include HadoopBenchmark and HiBench for HDFS, which utilize the Hadoop MapReduce computing platform for distributed testing of HDFS; and IOR, a testing tool based on the MPI platform, which uses the MPI computing platform to test the distributed file systems used during MPI computation. However, these performance testing solutions do not adequately support unified performance testing for different distributed file systems. First, these solutions are designed to be tightly coupled with a single distributed file system, making it difficult to directly compare test results from different distributed file systems. Second, these solutions select specific computing platforms to adapt to the distributed environment, incurring significant overhead during performance testing and resulting in inaccurate results. Furthermore, the test cases used in these solutions are typically simple file operations, which differ considerably from actual application access patterns, making it difficult to simulate complex and diverse real-world applications. Consequently, the performance test results cannot provide guidance for users to select the most suitable distributed file system, failing to achieve the desired testing effect. Summary of the Invention

[0005] This invention provides a performance testing method, apparatus, computer device, and storage medium for distributed file systems, in order to improve the accuracy of performance testing for distributed file systems.

[0006] A performance testing method for a distributed file system includes:

[0007] Use big data methods to obtain at least two tasks to be tested;

[0008] Using the Airflow platform, each of the tasks to be tested is parsed to obtain the task elements, and based on the task elements, a scheduling strategy for the task to be tested is generated using a directed acyclic graph.

[0009] Based on the scheduling strategy, each of the tasks to be tested is subjected to load testing in sequence to obtain the processing results;

[0010] The results of each processing step are summarized to obtain the stress test results.

[0011] Optionally, based on the scheduling strategy, each of the tasks to be tested is subjected to load testing in sequence to obtain the processing results, including:

[0012] The task to be tested is split into pieces to obtain N basic piece data.

[0013] According to a preset load balancing distribution mode, at least one target node is selected from the node set, and the basic shard data is distributed to the target node to perform stress testing processing;

[0014] Results of the stress testing processing of each target node are aggregated to obtain the processing result.

[0015] Optionally, the sharding processing of the to-be-tested task to obtain N basic shard data comprises:

[0016] The number of data records of the basic task data is obtained, and each data record is sequentially numbered;

[0017] For each data record, the number of the data record is used to perform a modulo operation on the number N of shards to obtain a number modulo of the data record.

[0018] Data records with the same number modulo are divided into the same shard set, and data records in each shard set are taken as a basic shard data to obtain N basic shard data.

[0019] Optionally, the distribution of the basic shard data to the target node to perform stress testing processing comprises:

[0020] Based on the configuration and current resource usage data of the node, the effective data amount of the node currently performing data processing is determined.

[0021] If the data amount in the basic shard data is greater than the effective data amount, the basic shard data is segmented according to a preset dimension to obtain K target shard data.

[0022] The K target shard data is executed using a thread pool to obtain the stress testing processing result.

[0023] Optionally, the selection of at least one target node from the node set according to the preset load balancing distribution mode comprises:

[0024] Through cluster communication, the activity of each node in the cluster is detected to obtain a node state.

[0025] The node with the active node state is taken as an effective node.

[0026] For each effective node, a health check is performed on the effective node to obtain a check result.

[0027] acquire actual physical addresses of applications corresponding to the basic shard data, and read actual physical addresses of each of the effective nodes, determine physical distances between the basic shard data and each of the effective nodes based on the actual physical addresses corresponding to the basic shard data and the actual physical addresses of each of the effective nodes;

[0028] determine a target node based on the inspection result of each of the effective nodes and each of the physical distances.

[0029] Optionally, the determining the target node based on the inspection result of each of the effective nodes and each of the physical distances comprises:

[0030] for each of the physical distances, performing normalization processing on the physical distance, and taking a result obtained as a distance weight of an effective node corresponding to the physical distance;

[0031] based on the inspection result of each of the effective nodes, obtain a health score of the effective node;

[0032] for each of the effective nodes, according to the distance weight and the health score, calculate a matching score of the effective node in a preset manner;

[0033] acquire an effective node with a maximum score of the matching score as the target node.

[0034] A performance testing device of a distributed file system comprises:

[0035] a task acquisition module configured to acquire at least two to-be-tested tasks in a big data manner;

[0036] a strategy generation module configured to perform task analysis on each of the to-be-tested tasks through an Airflow platform to obtain a task element composition, and generate a scheduling strategy of the to-be-tested task in a directed acyclic graph manner based on the task element composition;

[0037] a stress testing processing module configured to perform stress testing processing on each of the to-be-tested tasks in sequence based on the scheduling strategy to obtain a processing result;

[0038] a result determination module configured to aggregate each of the processing results to obtain a stress testing result.

[0039] Optionally, the stress testing processing module comprises:

[0040] a shard unit configured to perform shard processing on the to-be-tested tasks to obtain N basic shard data;

[0041] The stress testing unit is configured to select at least one target node from the node set according to a preset load balancing distribution manner, and distribute the basic shard data to the target node to perform stress testing processing.

[0042] The aggregation unit is configured to aggregate the stress testing processing results of each target node to obtain the processing result.

[0043] Optionally, the sharding unit comprises:

[0044] The numbering subunit is configured to obtain the number of data records of the basic task data, and sequentially number each data record.

[0045] The modulo subunit is configured to perform modulo operation on the number of each data record and the number N of shards to obtain the number modulo of the data record.

[0046] The data sharding subunit is configured to divide data records with the same number modulo into the same shard set, and take the data records in each shard set as a basic shard data to obtain N basic shard data.

[0047] Optionally, the stress testing processing module comprises:

[0048] The data filtering unit is configured to determine the effective data amount of the node for data processing based on the configuration and current resource usage data of the node.

[0049] The data segmentation unit is configured to perform data segmentation on the basic shard data according to a preset dimension to obtain K target shard data if the data amount in the basic shard data is greater than the effective data amount.

[0050] The multi-thread processing unit is configured to execute the processing instruction on the K target shard data in the manner of thread pool to obtain the stress testing processing result.

[0051] Optionally, the stress testing unit comprises:

[0052] The state detection subunit is configured to perform activity detection on each node in the cluster by cluster communication to obtain the node state.

[0053] The effective node determination subunit is configured to take the node with the activated state as an effective node.

[0054] The health check subunit is configured to perform health check on each effective node to obtain a check result.

[0055] The physical distance determination subunit is configured to obtain actual physical addresses of applications corresponding to the basic shard data, read actual physical addresses of each of the effective nodes, and determine physical distances between the basic shard data and each of the effective nodes based on the actual physical addresses of the applications corresponding to the basic shard data and the actual physical addresses of the effective nodes.

[0056] The target node determination subunit is configured to determine a target node based on the inspection result of each of the effective nodes and each of the physical distances.

[0057] Optionally, the target node determination subunit comprises:

[0058] The weight determination element is configured to normalize each of the physical distances, and take a result as a distance weight of an effective node corresponding to the physical distance.

[0059] The health score element is configured to obtain a health score of each of the effective nodes based on the inspection result of the effective node.

[0060] The matching score calculation element is configured to calculate a matching score of each of the effective nodes according to the distance weight and the health score of the effective node in a preset manner.

[0061] The target node selection element is configured to obtain an effective node with a maximum matching score as the target node.

[0062] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the performance test method of the distributed file system when executing the computer program.

[0063] A computer readable storage medium stores a computer program, and the computer program implements the steps of the performance test method of the distributed file system when executed by a processor.

[0064] The performance test method, device, computer device, and storage medium of the distributed file system provided by the application obtain at least two to-be-tested tasks in a big data manner, perform task analysis on each to-be-tested task through an Airflow platform to obtain a task element composition, generate a scheduling strategy of the to-be-tested task in a directed acyclic graph manner based on the task element composition, perform stress test processing on each to-be-tested task in sequence based on the scheduling strategy to obtain a processing result, and aggregate each processing result to obtain a stress test result. The application realizes the production scheduling strategy manner, performs rapid performance test on diversified test tasks of the distributed file system, and improves the performance test efficiency of the distributed file system. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor under the premise of the drawings.

[0066] Figure 1 is a schematic diagram of an application environment of the performance test method of the distributed file system in an embodiment of the present application;

[0067] Figure 2 is a flowchart of the performance test method of the distributed file system in an embodiment of the present application;

[0068] Figure 3 is a structural schematic diagram of the performance test device of the distributed file system in an embodiment of the present application;

[0069] Figure 4 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0071] The performance test method of the distributed file system provided in the present application can be applied in an application environment such as Figure 1 , wherein the terminal device communicates with the server through the network. The terminal device can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0072] The system framework 100 can include a terminal device, a network and a server. The network is used as a medium to provide a communication link between the terminal device and the server. The network can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0073] The user can use the terminal device to interact with the server through the network to receive or send messages, etc.

[0074] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III), MP4 players (Moving Picture Eperts Group Audio Layer IV), laptops, and desktop computers, etc.

[0075] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0076] It should be noted that the performance testing method for the distributed file system provided in this embodiment of the invention is executed by the server, and correspondingly, the performance testing device for the distributed file system is set in the server.

[0077] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown in this embodiment is merely illustrative. Depending on the implementation needs, there can be any number of terminal devices, networks, and servers. The terminal devices in this embodiment can specifically correspond to application systems in actual production.

[0078] In one embodiment, such as Figure 2 As shown, a performance testing method for a distributed file system is provided, and this method is applied to... Figure 1 The following steps, S201 to S204, are used as an example of the server in the example.

[0079] S201: Use big data to obtain at least two tasks to be tested.

[0080] Specifically, this embodiment is geared towards distributed file systems and uses a big data approach to obtain at least two tasks to be tested from different file systems.

[0081] S202: Using the Airflow platform, each task to be tested is parsed to obtain the task elements, and based on the task elements, a scheduling strategy for the task to be tested is generated using a directed acyclic graph.

[0082] Specifically, Airflow is a programmable, scheduled, and monitored workflow platform based on a directed acyclic graph (DAG). Airflow can define a set of dependent tasks and execute them sequentially according to their dependencies. Airflow provides a rich set of command-line tools for system management, while its web management interface can also easily manage and schedule tasks and monitor task execution status in real time, facilitating system operation and maintenance.

[0083] In this context, task element composition refers to the combination of various test elements and test processes included in the task. It should be noted that, in this embodiment, the specific task element composition of each task to be tested, obtained from the distributed file system, is determined by the source distributed file system. The task element composition of each task to be tested is not entirely the same; therefore, the test processes for each task to be tested may be the same or different.

[0084] Airflow is compatible with various components, and users can choose from several preset schemes based on their needs. Among these, the most critical actuators in the preset schemes include the following four types:

[0085] SequentialExecutor: A single-process sequential execution of tasks. It is the default executor and is typically used only for testing.

[0086] LocalExecutor: Executes tasks locally in a multi-process environment;

[0087] CeleryExecutor: Distributed scheduling, commonly used in production environments;

[0088] DaskExecutor: Dynamic task scheduling, mainly used for data analysis;

[0089] Preferably, this embodiment uses CeleryExecutor as the executor.

[0090] Before step S202, the method further includes:

[0091] Using big data tools such as Hadoop and Spark, we designed parallel programs that run on distributed systems.

[0092] S203: Based on the scheduling strategy, perform load testing on each task to be tested in sequence to obtain the processing results.

[0093] Optionally, based on the scheduling strategy, each task to be tested is subjected to load testing in sequence, and the processing results include:

[0094] The test task is split into pieces to obtain N basic piece data.

[0095] According to the preset load balancing distribution method, at least one target node is selected from the node set, and the basic shard data is allocated to the target node for load testing.

[0096] The results of the stress test for each target node are summarized to obtain the processing result.

[0097] Furthermore, the task to be tested is split into pieces, resulting in N basic piece data, including:

[0098] Obtain the number of data records for the basic task data and assign a sequential number to each data record;

[0099] For each data record, the number of data records is used to perform a modulo operation on the number of fragments N to obtain the modulo of the data record number;

[0100] Data records with the same number modulo are grouped into the same shard set, and the data records in each shard set are used as a basic shard data, resulting in N basic shard data.

[0101] Specifically, the number of data records of the basic task data is obtained, a shard list is generated in the cache records, the data records of the basic task data are written into the shard list, and each data record is numbered in turn. Then, for each data record, the number of the data record is used to perform a modulo operation on the number of shards N to obtain the modulo of the number of the data record.

[0102] The modulo operation is a mathematical operation whose basic form is a%b, or a mod b, which represents the remainder when a is divided by b.

[0103] For example, in one specific implementation, the number of data records of the acquired basic task data is 50. Each data record is numbered, resulting in 50 numbers from 1 to 50. The preset data segment N is 4. Modulo operations are performed on 4 using numbers 1 to 50 respectively. The modulo of 4 is 1 for numbers 1, 5, 9, 13, 17, 21, 25, 29, 33, 37, 41, 45, and 49; modulo 1 for numbers 2, 6, 10, 14, 18, 22, 26, 30, 34, 38, 42, 46, and 50; and modulo 2 for numbers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, and 47. The modulo of numbers 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, and 48, taken modulo 4, results in a modulo of 0. After obtaining the modulo of each data entry, the data records with a modulo of 1 are used as the first basic partition, i.e., the data records corresponding to numbers 1, 5, 9, 13, 17, 21, 25, 29, 33, 37, 41, 45, and 49 are placed into the first basic partition. The data records with a modulo of 2 are used as the second basic partition, i.e., the data records corresponding to numbers 2, 6, 10, 48, 49, 40, 41, 45, and 49 are placed into the second basic partition. The data records corresponding to numbers 14, 18, 22, 26, 30, 34, 38, 42, 46, and 50 are placed into the second basic shard data. Data records with numbers modulo 3 are used as the third basic shard data, specifically the data records corresponding to numbers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, and 47. Data records with numbers modulo 0 are used as the fourth basic shard data, specifically the data records corresponding to numbers 4, 8, 12, 16, 20, 24, 28, and 50. The data records corresponding to 32, 36, 40, 44, and 48 are placed into the fourth basic data shard. In this way, the 50 data records are divided into four basic data shards according to the characteristics of the number modulus. Adjacent data records are assigned to different data shards and will be subsequently assigned to different node servers. This ensures that the time difference between the execution of processing instructions for adjacent data records is not too large during subsequent data processing, and that the time difference between obtaining the execution results is not too large, which helps to improve the efficiency of the execution result aggregation.

[0104] It is worth noting that the modulo operation and shard aggregation described in this embodiment are both processed in the cache, and the final basic shard data is also stored in the shard list. This processing method can effectively save the system resources of the central server, save sharding time, and improve sharding efficiency.

[0105] Furthermore, allocating the basic sharded data to the target node for load testing includes:

[0106] Based on the node's configuration and current resource usage data, determine the effective amount of data that the node is currently processing;

[0107] If the amount of data in the basic shard data is greater than the amount of effective data, then the basic shard data is split according to the preset dimensions to obtain K target shard data.

[0108] Using a thread pool, processing instructions are executed on K target data shards to obtain the results of the load test.

[0109] The preset dimension can be determined based on the actual data attributes, and there are no restrictions here.

[0110] The effective data volume can be flexibly set according to the node configuration. The effective data volume of different nodes may be different, and no specific limit is made here.

[0111] Furthermore, selecting at least one target node from the node set according to the preset load balancing distribution method includes:

[0112] The activity of each node in the cluster is detected through cluster communication to obtain the node status;

[0113] Nodes that are active are considered valid nodes.

[0114] For each valid node, a health check is performed to obtain the check results;

[0115] Obtain the actual physical address of the application corresponding to the basic shard data, and read the actual physical address of each valid node. Based on the actual physical address of the basic shard data and the actual physical address of the valid node, determine the physical distance between the basic shard data and each valid node.

[0116] The target node is determined based on the inspection results of each valid node and each physical distance.

[0117] Health checks are used to determine the service availability of node servers (ECS instances). The health check mechanism improves the overall availability of front-end services and avoids the impact of back-end ECS anomalies on the overall service. Existing main methods for conducting health checks include Layer 7 checks, Layer 4 checks, and load balancing.

[0118] Optionally, the categories of health checks include node throughput, number of packets, number of requests, response speed, latency, and round-trip time (RTT).

[0119] It should be understood that the nodes in the cluster are distributed, meaning different nodes can have the same physical address or different physical addresses (depending on the actual situation, they usually have different physical addresses, for example, one node in province A and one node in province B). The closer the physical address corresponding to the client IP address of the data access request is to the physical address of the node, the shorter the time spent on information propagation (data sending and receiving). In this embodiment, determining the target node through health checks and physical distance is beneficial for obtaining efficient target nodes and improving data transmission efficiency.

[0120] Preferably, the target node is determined based on the inspection results of each valid node and each physical distance, including:

[0121] For each physical distance, the physical distance is normalized, and the result is used as the distance weight of the corresponding valid node.

[0122] Based on the inspection results of each valid node, a health score for each valid node is obtained.

[0123] For each valid node, a matching score is calculated based on distance weight and health score using a preset method.

[0124] The node with the highest matching score is selected as the target node.

[0125] The preset method can be a pre-defined weighting calculation formula.

[0126] S204: Summarize each processing result to obtain the stress test result.

[0127] In this embodiment, at least two tasks to be tested are obtained using big data; a scheduling strategy for the tasks to be tested is generated through the Airflow platform; based on the scheduling strategy, each task to be tested is subjected to load testing in sequence to obtain the processing results; and each processing result is summarized to obtain the load testing result. This achieves rapid performance testing for diverse test tasks of distributed file systems using a production scheduling strategy, thereby improving the performance testing efficiency of distributed file systems.

[0128] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0129] In one embodiment, a performance testing apparatus for a distributed file system is provided, which corresponds one-to-one with the performance testing methods for distributed file systems described in the above embodiments. For example... Figure 3As shown, the performance testing device for this distributed file system includes a task acquisition module 31, a policy generation module 32, a load testing processing module 33, and a result determination module 34. Details of each module are as follows:

[0130] Task acquisition module 31 is used to acquire at least two tasks to be tested using big data methods;

[0131] The strategy generation module 32 is used to generate the scheduling strategy of the task to be tested through the Airflow platform;

[0132] The load testing processing module 33 is used to perform load testing on each of the tasks to be tested in sequence based on the scheduling strategy, and obtain the processing results.

[0133] The result determination module 34 is used to summarize each of the processing results to obtain the stress test result.

[0134] Optionally, the pressure testing processing module 33 includes:

[0135] The sharding unit is used to shard the task to be tested to obtain N basic shard data.

[0136] The load testing unit is used to select at least one target node from the node set according to a preset load balancing distribution method, and allocate the basic shard data to the target node to perform load testing processing;

[0137] The summarization unit is used to summarize the stress test results of each target node to obtain the processing result.

[0138] Optionally, the fragmentation unit includes:

[0139] The numbering subunit is used to obtain the number of data records of the basic task data and to sequentially number each data record.

[0140] The modulus subunit is used to perform a modulo operation on the number of segments N using the number of the data record for each data record, so as to obtain the modulo of the number of the data record.

[0141] The data sharding subunit is used to group data records with the same number modulus into the same shard set, and to use the data records in each shard set as a basic shard data to obtain N basic shard data.

[0142] Optionally, the pressure testing processing module 33 includes:

[0143] The data filtering unit is used to determine the effective amount of data that a node is currently processing, based on the node's configuration and current resource usage data.

[0144] The data segmentation unit is used to segment the basic data according to a preset dimension to obtain K target data segments if the amount of data in the basic data segments is greater than the amount of effective data.

[0145] A multi-threaded processing unit is used to execute the processing instructions on K target data shards using a thread pool to obtain the results of the load test.

[0146] Optionally, the pressure testing unit includes:

[0147] The state detection subunit is used to detect the activity of each node in the cluster and obtain the node status through cluster communication.

[0148] The valid node determination subunit is used to identify nodes whose node status is active as valid nodes.

[0149] The health check subunit is used to perform a health check on each of the valid nodes and obtain the check results.

[0150] The physical distance determination subunit is used to obtain the actual physical address of the application corresponding to the basic shard data, and read the actual physical address of each of the effective nodes. Based on the actual physical address corresponding to the basic shard data and the actual physical address of the effective node, the physical distance between the basic shard data and each of the effective nodes is determined.

[0151] The target node determination subunit is used to determine the target node based on the inspection results of each of the valid nodes and each of the physical distances.

[0152] Optionally, the target node determining subunit includes:

[0153] The weight determination element is used to normalize the physical distance for each physical distance and use the result as the distance weight of the valid node corresponding to the physical distance.

[0154] A health scoring element is used to obtain a health score for each valid node based on the inspection results of each valid node.

[0155] A matching score calculation element is used to calculate the matching score of each valid node according to the distance weight and the health score, using a preset method.

[0156] The target node selection element is used to obtain the valid node with the highest matching score, which is then used as the target node.

[0157] The terms "first" and "second" in the above-mentioned modules / units are only used to distinguish different modules / units and are not intended to specify which module / unit has a higher priority or any other limiting meaning. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The module divisions appearing in this application are merely logical divisions; in actual applications, different division methods may be used.

[0158] Specific limitations regarding the performance testing device for distributed file systems can be found in the limitations of the performance testing methods for distributed file systems described above, and will not be repeated here. Each module in the aforementioned performance testing device for distributed file systems can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0159] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data involved in performance testing methods for distributed file systems. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a performance testing method for a distributed file system.

[0160] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the performance testing method for the distributed file system described in the above embodiment, for example... Figure 2 The steps S201 to S204 shown, as well as other extensions and related steps of the method, are examples. Alternatively, when the processor executes a computer program, it implements the functions of each module / unit of the performance testing device for the distributed file system in the above embodiments, for example... Figure 3The functions of modules 31 to 34 are shown. To avoid repetition, they will not be described again here.

[0161] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0162] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, video data, etc.).

[0163] The memory can be integrated into the processor or it can be set up separately from the processor.

[0164] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the steps of the performance testing method for the distributed file system described in the above embodiments, for example... Figure 2 The steps S201 to S204 shown, as well as other extensions and related steps of the method, are examples. Alternatively, when a computer program is executed by a processor, it implements the functions of each module / unit of the performance testing device for the distributed file system in the above embodiments, for example... Figure 3 The functions of modules 31 to 34 are shown. To avoid repetition, they will not be described again here.

[0165] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0167] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A performance testing method for a distributed file system, characterized in that, The method includes: Use big data methods to obtain at least two tasks to be tested; The Airflow platform parses each task to be tested to obtain task element composition. Based on the task element composition, a scheduling strategy for the task to be tested is generated using a directed acyclic graph. The task element composition refers to the combination of various test elements and test processes contained in the task. The specific task element composition is determined by the source distributed file system. The Airflow platform uses CeleryExecutor for distributed scheduling. Based on the scheduling strategy, the task to be tested is sharded according to the number of the data record in the task to be tested. Data records with adjacent numbers are assigned to different shards to obtain N basic shards. The basic shards are then assigned to the target node to perform load testing to obtain the processing result. The scheduling strategy is based on the task elements and is generated using a directed acyclic graph. The results of each processing step are summarized to obtain the stress test results.

2. The performance testing method for a distributed file system according to claim 1, characterized in that, Based on the scheduling strategy, each of the tasks to be tested is subjected to load testing in sequence, and the processing results include: According to the preset load balancing distribution method, at least one target node is selected from the node set, and the basic shard data is allocated to the target node for load testing. The results of the stress test for each target node are summarized to obtain the processing result.

3. The performance testing method for a distributed file system according to claim 2, characterized in that, The step of segmenting the test task based on the data record number in the test task to obtain N basic segment data includes: Obtain the number of data records of the basic task data, and sequentially number each data record; For each of the data records, the number of fragments N is moduloed using the data record number to obtain the modulo of the data record number; Data records with the same number modulus are grouped into the same shard set, and each data record in the shard set is used as a basic shard data, resulting in N basic shard data.

4. The performance testing method for a distributed file system according to claim 2, characterized in that, The step of allocating the basic shard data to the target node for load testing includes: Based on the node's configuration and current resource usage data, determine the effective amount of data that the node is currently processing; If the amount of data in the basic shard data is greater than the amount of effective data, then the basic shard data is divided according to a preset dimension to obtain K target shard data. Using a thread pool, the processing instructions are executed on K target data shards to obtain the results of the load test.

5. The performance testing method for a distributed file system according to claim 2, characterized in that, The step of selecting at least one target node from the node set according to a preset load balancing distribution method includes: The activity of each node in the cluster is detected through cluster communication to obtain the node status; Nodes that are active are considered valid nodes. For each of the valid nodes, a health check is performed on the valid node to obtain the check results; Obtain the actual physical address of the application corresponding to the basic shard data, and read the actual physical address of each of the effective nodes. Based on the actual physical address of the basic shard data and the actual physical address of the effective node, determine the physical distance between the basic shard data and each of the effective nodes. The target node is determined based on the inspection results of each of the valid nodes and each of the physical distances.

6. The performance testing method for a distributed file system according to claim 5, characterized in that, The determination of the target node based on the inspection results of each of the valid nodes and each of the physical distances includes: For each physical distance, the physical distance is normalized, and the result is used as the distance weight of the effective node corresponding to the physical distance. Based on the inspection results of each valid node, a health score is obtained for each valid node; For each valid node, a matching score is calculated using a preset method based on the distance weight and the health score. The node with the highest matching score is selected as the target node.

7. A performance testing device for a distributed file system, characterized in that, The device includes: The task acquisition module is used to acquire at least two tasks to be tested using big data methods. The strategy generation module is used to parse each of the tasks to be tested through the Airflow platform to obtain the task element composition, and generate the scheduling strategy of the task to be tested based on the task element composition using a directed acyclic graph. The task element composition refers to the combination of each test element and each test process contained in the task. The specific task element composition is determined by the source distributed file system. The Airflow platform uses the CeleryExecutor executor for distributed scheduling. The load testing processing module is used to perform load testing processing on the task under test based on the scheduling strategy and the number of the data record in the task under test. The data records with adjacent numbers are assigned to different data shards to obtain N basic data shards. The basic data shards are then assigned to the target node to perform load testing processing to obtain the processing result. The scheduling strategy is based on the task elements and is generated in the form of a directed acyclic graph. The result determination module is used to summarize each of the processing results to obtain the stress test results.

8. The performance testing device for a distributed file system according to claim 7, characterized in that, The stress testing processing module includes: The load testing unit is used to select at least one target node from the node set according to a preset load balancing distribution method, and allocate the basic shard data to the target node to perform load testing processing; The summarization unit is used to summarize the stress test results of each target node to obtain the processing result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the performance testing method for the distributed file system as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the performance testing method for the distributed file system as described in any one of claims 1 to 6.

Images