Concurrency capability testing method and device of system, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a concurrent capability testing method and device of a system, a storage medium and electronic device.The concurrent capability testing method of the system comprises the steps that a to-be-tested task in a network file system is split to generate a task pool; determining a plurality of threads configured for each client corresponding to the network file system, and constructing a thread pool according to the plurality of threads corresponding to the plurality of clients; performing task scheduling on each thread in the thread pool so as to enable the plurality of threads to execute a plurality of subtasks in the task pool; and testing the concurrency capability of the network file system according to the execution information of executing the plurality of sub-tasks by the plurality of threads. According to the embodiment of the invention, the problem that the network file system runs unstably in the multi-thread multi-client scene due to the lack of a method for testing the concurrency capability of the network file system in the multi-thread multi-client scene in the related technology can be solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communications, and in particular, to a method and device for testing the concurrency capability of a system, a storage medium, and an electronic device. Background Art

[0002] The Network File System (NFS) is one of the current mainstream heterogeneous platform sharing file systems. It is mainly used in the UNIX (an operating system) environment and can support file sharing between different types of systems through the network. NFS allows a system to share directories and files with others on the network. By using NFS, a large number of users and different programs on the same client can access files on the remote system as if they were local files, allowing each computer node to use network shared directories as easily as using local resources. NFS-Ganesha (a software that implements the NFS protocol) is a user-mode network file system. Compared with kernel-mode NFS, user-mode NFS-Ganesha is easier to manage and maintain, and is widely used in big data and massive distributed file storage scenarios.

[0003] After the NFS service shared directory is shared to the network through NFS-Ganesha, the client can mount the shared directory to achieve data transmission between the client and the server. Currently, a single NFS service node is sufficient to support thousands of clients to mount and access at the same time, and each client can start hundreds of threads to access the NFS shared directory mounted by the current client. The access block sizes of these different clients and different threads on the client are diverse and random. These real-time updates and multi-faceted interactions of data pose a severe test to the multi-concurrency capabilities of the data storage system in random block size scenarios.

[0004] Therefore, in the related art, there is a lack of testing methods for the concurrency capabilities of the network file system in a multi-threaded multi-client scenario, which has led to the problem that the network file system operates unstably in a multi-threaded multi-client scenario and has not yet been effectively solved. Summary of the invention

[0005] The embodiments of the present application provide a method and device for testing the concurrency capability of a system, a storage medium, and an electronic device, so as to at least solve the problem in the related art that there is a lack of a method for testing the concurrency capability of a network file system in a multi-threaded multi-client scenario, which causes the network file system to operate unstably in a multi-threaded multi-client scenario.

[0006] According to an embodiment of the present application, a method for testing the concurrency capability of a system is provided, which specifically includes: splitting the tasks to be tested in a network file system to generate a task pool; determining multiple threads configured for each client corresponding to the network file system, and building a thread pool according to the multiple threads corresponding to multiple clients; scheduling tasks for each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool; and testing the concurrency capability of the network file system according to execution information of the multiple threads executing the multiple subtasks.

[0007] In an exemplary embodiment, a task to be tested in a network file system is split to generate multiple subtasks, including: determining a first task size of the task to be tested, and determining a data block size of each preset data block and a proportional relationship between multiple preset data blocks, wherein the proportional relationship is used to indicate a proportion corresponding to each preset data block, and the data block size is used to indicate a second task size of each subtask to be split; and splitting the task to be tested according to the proportional relationship, the first task size, and the data block size of each preset data block to generate multiple subtasks.

[0008] In an exemplary embodiment, the task to be tested is split according to the proportional relationship, the first task size and the data block size of each preset data block to generate multiple subtasks, including: determining a first product of the ratio of each preset data block and the data block size of each preset data block, and determining a sum of multiple first products; calculating a quotient of the first task size and the sum, and determining whether the quotient is an integer; when the quotient is determined to be an integer, determining a first number of subtasks of each data block size to be split according to the quotient and the proportional relationship, and splitting the task to be tested into the multiple subtasks according to the first number and the data block size of each preset data block; when the quotient is determined to be a non-integer, rounding down the quotient to determine a target value; and determining a target value according to the target value. The method comprises the following steps: determining a second number of subtasks of each data block size to be split according to the value and the proportional relationship, and splitting the task to be tested into multiple first subtasks according to the second number and the data block size of each preset data block; and determining a third task size of the task to be split in the task to be tested according to the difference between the quotient value and the target value; sorting the multiple preset data blocks according to the data block size of each preset data block to determine the size relationship between the multiple preset data blocks; determining a second number of subtasks of each data block size to split the task to be split into according to the size relationship; splitting the task to be split according to the second number and the data block size of each preset data block to generate multiple second subtasks; and determining the first subtask and the second subtask as multiple subtasks in the task pool.

[0009] In an exemplary embodiment, task scheduling is performed on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool, including: establishing a communication relationship between the multiple clients, and synchronizing the execution progress of each subtask executed by each thread according to the communication relationship; and scheduling each thread according to the execution progress.

[0010] In an exemplary embodiment, task scheduling is performed on each thread according to the execution progress, including: when the execution progress indicates that any thread among the multiple threads has completed the third subtask in the task pool, determining the number of first tasks allowed to be executed by any thread and the number of second tasks that the any thread has executed; determining whether to allow the any thread to execute other tasks in the task pool according to the first number of tasks and the second number of tasks; when it is determined that the any thread is allowed to execute other tasks in the task pool, obtaining a fourth subtask in the task pool and sending the fourth subtask to the any thread so that the any thread executes the fourth subtask, wherein the fourth subtask is any task in the task pool.

[0011] In an exemplary embodiment, before testing the concurrency capability of the network file system based on the execution information of the multiple threads executing the multiple subtasks, the method also includes: obtaining the task time consumed by each thread to execute each subtask in the task pool, the number of input and output operations per second corresponding to each thread, and the bandwidth information corresponding to each thread; and determining the task time consumed, the number of input and output operations per second, and the bandwidth information as the execution information.

[0012] In an exemplary embodiment, obtaining the number of input and output operations per second corresponding to each thread includes: monitoring a third number of subtasks distributed by the task pool to the multiple threads per second according to a timer corresponding to the task pool; and monitoring a fourth number of execution requests corresponding to each thread received by the task pool per second according to the timer, wherein the execution request is used to instruct each thread to execute a request for each subtask in the task pool; and determining the number of input and output operations per second corresponding to each thread according to the third number and the fourth number.

[0013] According to another embodiment of the present application, a system concurrency capability testing device is provided, including: a splitting module, used to split the tasks to be tested in the network file system to generate a task pool; a determination module, used to determine the multiple threads configured for each client corresponding to the network file system, and build a thread pool according to the multiple threads corresponding to the multiple clients; a scheduling module, used to schedule tasks for each thread in the thread pool, so that the multiple threads execute the multiple subtasks in the task pool; and a testing module, used to test the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks.

[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps in the above method embodiment when running.

[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0017] Through the concurrency testing method of the system of the present application, the tasks to be tested in the network file system are split to generate a task pool; multiple threads configured for each client corresponding to the network file system are determined, and a thread pool is constructed according to the multiple threads corresponding to the multiple clients; each thread in the thread pool is task-scheduled so that multiple threads execute multiple subtasks in the task pool; the concurrency of the network file system is tested according to the execution information of multiple threads executing multiple subtasks. That is to say, the embodiment of the present application defines a task execution method of a network file system in a multi-client multi-threaded scenario, multiple threads configured by multiple clients constitute a thread pool, and the task to be tested is split into multiple subtasks to generate a task pool, so that each thread in the thread pool randomly executes each task in the task pool, and then the concurrency of the network file system is tested by the execution information of the tasks in the task pool executed by the network file system in the multi-threaded multi-client scenario. Through the optional embodiment of the present application, the problem of the unstable operation of the network file system in the multi-threaded multi-client scenario caused by the lack of a testing method for the concurrency of the network file system in the multi-threaded multi-client scenario in the related art is solved. Then, the concurrency capability of the network file system in a multi-threaded and multi-client scenario is tested to ensure that the network file system can run stably in a multi-threaded and multi-client scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal of a system concurrent capability testing method according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a method for testing the concurrency capability of a system according to an embodiment of the present application;

[0020] Figure 3 It is a flow chart of a method for testing the concurrent capability of multi-client multi-threaded random block size of a file system according to an optional embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of the working mode of the task pool and the thread pool according to an optional embodiment of the present application;

[0022] Figure 5 It is a structural block diagram of a concurrent capability testing device of a system according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0025] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal of a method for testing concurrent capabilities of a system in an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0026] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining the interactive state in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0028] Figure 2 is a flowchart of a method for testing the concurrent capability of a system according to an embodiment of the present application, which can be applied to Figure 1 In a computer terminal, such as Figure 2 As shown, the process includes the following steps:

[0029] Step S202, splitting the tasks to be tested in the network file system to generate a task pool;

[0030] Among them, the task pool includes: multiple subtasks.

[0031] Step S204, determining multiple threads configured for each client corresponding to the network file system, and constructing a thread pool according to the multiple threads corresponding to the multiple clients;

[0032] For example, the network file system corresponds to 5 clients, and each client is configured with 10 threads. Then the thread pool includes 50 threads.

[0033] Step S206, performing task scheduling on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool;

[0034] Step S208: testing the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks.

[0035] Through the concurrency testing method of the system of the present application, the tasks to be tested in the network file system are split to generate a task pool; multiple threads configured for each client corresponding to the network file system are determined, and a thread pool is constructed according to the multiple threads corresponding to the multiple clients; each thread in the thread pool is task-scheduled so that multiple threads execute multiple subtasks in the task pool; the concurrency of the network file system is tested according to the execution information of multiple threads executing multiple subtasks. That is to say, the embodiment of the present application defines a task execution method of a network file system in a multi-client multi-threaded scenario, multiple threads configured by multiple clients constitute a thread pool, and the task to be tested is split into multiple subtasks to generate a task pool, so that each thread in the thread pool randomly executes each task in the task pool, and then the concurrency of the network file system is tested by the execution information of the tasks in the task pool executed by the network file system in the multi-threaded multi-client scenario. Through the optional embodiment of the present application, the problem of the unstable operation of the network file system in the multi-threaded multi-client scenario caused by the lack of a testing method for the concurrency of the network file system in the multi-threaded multi-client scenario in the related art is solved. Then, the concurrency capability of the network file system in a multi-threaded and multi-client scenario is tested to ensure that the network file system can run stably in a multi-threaded and multi-client scenario.

[0036] Optionally, in the above step S202, the task to be tested in the network file system is split to generate multiple subtasks, including: determining a first task size of the task to be tested, and determining a data block size of each preset data block and a proportional relationship between multiple preset data blocks, wherein the proportional relationship is used to indicate a proportion corresponding to each preset data block, and the data block size is used to indicate a second task size of each subtask to be split; and splitting the task to be tested according to the proportional relationship, the first task size and the data block size of each preset data block to generate multiple subtasks.

[0037] Wherein, the task to be tested is split according to the proportional relationship, the first task size and the data block size of each preset data block to generate multiple subtasks, including: determining a first product of the proportion of each preset data block and the data block size of each preset data block, and determining a sum of multiple first products; calculating a quotient of the first task size and the sum, and determining whether the quotient is an integer; when the quotient is determined to be an integer, determining a first number of subtasks of each data block size to be split according to the quotient and the proportional relationship, and splitting the task to be tested into the multiple subtasks according to the first number and the data block size of each preset data block; when the quotient is determined to be a non-integer, rounding down the quotient to determine a target value; calculating a quotient of the target value and the The method comprises the following steps: determining a second number of subtasks of each data block size to be split according to a proportional relationship, and splitting the task to be tested into multiple first subtasks according to the second number and the data block size of each preset data block; and determining a third task size of the task to be split in the task to be tested according to the difference between the quotient and the target value; sorting the multiple preset data blocks according to the data block size of each preset data block to determine the size relationship between the multiple preset data blocks; determining the second number of subtasks of each data block size to split the task to be split into according to the size relationship; splitting the task to be split according to the second number and the data block size of each preset data block to generate multiple second subtasks; and determining the first subtask and the second subtask as multiple subtasks in the task pool.

[0038] It is understandable that the above technical solution defines how to finely split a large task to be tested (for example, writing or reading a large file) to generate multiple subtasks when testing the performance of a network file system (for example, NFS), thereby simulating the scenario of multi-client and multi-threaded random access to data of different block sizes in actual applications. Specifically:

[0039] 1) Determine the first task size: The total size of the task to be tested needs to be determined, for example, testing a 1GB file write operation. This 1GB is the "first task size", which is the basis for task splitting.

[0040] 2) Determine the data block size and ratio of each preset data block: define multiple preset data block sizes, such as 4KB, 64KB, 128KB, 256KB and 1MB, and the ratio of these data blocks in the total task, for example, 4KB data blocks account for 20% of the total number of blocks, 64KB data blocks also account for 20%, and so on. In actual applications, the size of data blocks is random and diverse, so it is necessary to set the ratio to simulate the performance of the network file system when processing data blocks of different sizes.

[0041] 3) Split tasks according to the proportional relationship, the first task size, and the data block size: Based on the above information, the number of subtasks for each data block size and the specific size of each subtask can be calculated. For example, if a 1GB file is to be split into data block subtasks that meet the above ratios, then it is necessary to calculate the total number of 4KB, 64KB, 128KB, 256KB, and 1MB blocks, as well as the specific locations (offsets) of these blocks in the file. By calculating the relationship between the number of blocks and the total task size, the number of occurrences of each block can be determined, thereby generating a specific subtask list.

[0042] 4) Subtask generation: The task management module will generate multiple subtasks in a specific task pool based on these calculation results. Each subtask contains information such as data block size, file offset, etc. These subtasks will be placed in a task pool, waiting to be assigned to different clients and threads for execution.

[0043] The task splitting and subtask generation process defined by the above technical solution can ensure that when testing the network file system, the complex scenario of multi-client multi-threaded random access to data blocks of different sizes can be fully and accurately simulated, thereby evaluating the system's concurrent processing capabilities and performance in this scenario.

[0044] Optionally, the above step S206 performs task scheduling on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool, including: establishing a communication relationship between the multiple clients, and synchronizing the execution progress of each thread to execute each subtask according to the communication relationship; and scheduling each thread according to the execution progress.

[0045] It is understandable that the above technical solution involves thread pool management, task scheduling and communication synchronization between multiple clients, specifically:

[0046] 1) Establishing a communication relationship between the multiple clients: In the test environment, there may be multiple clients accessing the file system at the same time, and multiple threads are running on each client. In order to ensure the smooth progress of the test, a communication mechanism needs to be established between these clients. This communication relationship can be established through a network protocol or bus to ensure that each client can exchange information with other clients, such as: thread status, task completion status, etc.

[0047] 2) Synchronize the execution progress of each subtask of each thread according to the communication relationship: Once the communication relationship is established, the thread distribution module can obtain the execution status and progress of all threads in real time through this channel. This synchronization can help each client understand which tasks have been completed, which threads are idle, and the progress of the entire test. For example, when a thread completes a subtask, a progress report will be generated to determine whether the thread is ready to accept new tasks.

[0048] 3) Scheduling tasks for each thread according to the execution progress: Based on the above-mentioned execution progress synchronization, the thread distribution module can make more effective task scheduling decisions. It will ensure that all threads are working efficiently and that some threads will not wait for new tasks for a long time. Specifically, when the module detects that a thread has completed a subtask and reported the status of this subtask, it can select the next appropriate subtask from the task pool and assign it to the thread, ensuring the continuity of the thread and the randomness of the task. This scheduling strategy not only takes into account the current state of the thread, but also the characteristics of the tasks in the task pool and the load of the file system to achieve the best concurrent testing effect.

[0049] In summary, the above technical solution aims to enable multiple threads on multiple clients to efficiently and orderly execute concurrent read and write test tasks of random block sizes under a unified framework through an effective communication and scheduling mechanism, thereby accurately evaluating the performance of the file system in high-concurrency random access scenarios.

[0050] Among them, task scheduling is performed on each thread according to the execution progress, including: when the execution progress indicates that any thread among the multiple threads has completed the third subtask in the task pool, determining the number of first tasks allowed to be executed by any thread and the number of second tasks that have been executed by any thread; determining whether to allow any thread to execute other tasks in the task pool according to the first number of tasks and the second number of tasks; when it is determined that any thread is allowed to execute other tasks in the task pool, obtaining a fourth subtask in the task pool, and sending the fourth subtask to any thread, so that any thread executes the fourth subtask, wherein the fourth subtask is any task in the task pool.

[0051] It is understandable that the above technical solution defines the process of dynamically scheduling tasks based on the execution progress of threads. Specifically:

[0052] 1) Monitor execution progress: Continuously monitor the execution progress of all threads in the task pool. The task pool contains a series of tasks to be executed, each of which may have different block sizes and positions.

[0053] 2) Completion status detection: When it is detected that a thread (any thread in the thread pool) has completed the third subtask currently being executed by the thread, the thread dispatch module will make the next scheduling decision.

[0054] 3) Determine the number of tasks: Calculate the number of first tasks that the thread is allowed to execute since the start of the test (i.e., the total number of tasks assigned to the thread) and the number of second tasks it has actually executed, and then evaluate whether the thread has reached or exceeded the expected number of tasks executed.

[0055] 4) Scheduling decision: Based on the number of first tasks and the number of second tasks, the thread dispatch module determines whether to allow the thread to continue to execute other tasks in the task pool. If the number of first tasks is less than the number of second tasks, the thread is allowed to continue to execute other tasks; if the number of first tasks is greater than or equal to the number of second tasks, the thread is not allowed to continue to execute other tasks.

[0056] 5) Task allocation: While allowing the thread to continue executing other tasks, a new task (the fourth subtask) is selected from the task pool and sent to the thread for execution. The key here is that the selected tasks are random and non-repetitive to ensure that the test covers all preset block sizes and positions.

[0057] The above technical solution ensures fair and efficient task scheduling in a multi-threaded environment, and also meets the requirements of random concurrency testing, that is, each thread has the opportunity to execute tasks of different block sizes, so as to comprehensively evaluate the multi-concurrency performance of the file system.

[0058] Optionally, before testing the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks in the above step S208, the method further includes: obtaining the task time consumed by each thread to execute each subtask in the task pool, the number of input and output operations per second corresponding to each thread, and the bandwidth information corresponding to each thread; and determining the task time consumed, the number of input and output operations per second, and the bandwidth information as the execution information.

[0059] Among them, obtaining the number of input and output operations per second corresponding to each thread includes: monitoring a third number of subtasks distributed by the task pool to the multiple threads per second according to a timer corresponding to the task pool; and monitoring a fourth number of execution requests corresponding to each thread received by the task pool per second according to the timer, wherein the execution request is used to instruct each thread to execute a request for each subtask in the task pool; and determining the number of input and output operations per second corresponding to each thread according to the third number and the fourth number.

[0060] It is understandable that before testing the concurrency capability of a network file system, it is necessary to collect and determine the key performance indicators of thread execution tasks, specifically how to obtain the time consumed by the thread to execute tasks, the number of input and output operations per second (IOPS) and bandwidth information, and use this information as execution information for subsequent test analysis. Specifically:

[0061] 1) Task duration: records the time required for each thread to execute each subtask in the task pool, which is an important indicator for measuring the response speed and efficiency of the network file system.

[0062] 2) Input and output operations per second (IOPS): used to evaluate the number of read and write operations that the network file system can handle per unit time. It can be calculated by monitoring the number of subtasks distributed by the task pool to multiple threads per second (the third number) and the number of requests received per second for each thread to execute subtasks (the fourth number). The third number reflects the system's ability to assign tasks, while the fourth number reflects the thread's ability to execute tasks and give feedback to the system. The combination of the two can yield the IOPS of the network file system in a multi-threaded environment.

[0063] Specific:

[0064] Determine the third quantity: Use a timer to monitor the number of subtasks distributed by the task pool to threads per second. This data reflects the task allocation rate of the system in a multi-threaded concurrent environment.

[0065] Determine the fourth quantity: Use a timer to monitor the number of requests received per second for each thread to execute subtasks. This part of the data reflects the rate at which the thread executes operations and feeds back to the system.

[0066] Determine IOPS: By comparing and analyzing the third and fourth quantities, you can calculate the number of read and write operations that each thread can complete per second, that is, IOPS.

[0067] 3) Bandwidth information: measures the amount of data that can be transmitted per unit time, which is directly related to the data throughput capacity of the network file system.

[0068] The above technical solution aims to comprehensively collect key performance data of the network file system in a multi-threaded concurrent environment, and provide a basis for subsequent system concurrency capability testing. By monitoring and recording the above information, the performance of the network file system in processing multi-client, multi-threaded random concurrent requests can be more accurately evaluated, including response speed, IOPS and bandwidth.

[0069] In order to better understand the process of the concurrent capability testing method of the above-mentioned system, the implementation process of the concurrent capability testing method of the above-mentioned system is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0070] The optional embodiment of the present application proposes a method for testing the concurrent capability of a file system with multiple clients and multiple threads and random block sizes. Through multi-client multi-thread transmission, each thread carries a random block size to test the data transmission capability of the file system in complex scenarios with multiple clients, multiple threads on the client, and random read and write block sizes in the threads. Specifically, the method tests the concurrent capability of the file system based on the time taken to complete the test task, the number of input / output operations per second (IOPS) during the test, and the bandwidth as the measurement criteria. The method of the optional embodiment of the present application is mainly composed of four modules:

[0071] Task management module, thread distribution module, communication module, data transmission module. Among them:

[0072] Figure 3 : is a flowchart of a method for testing the concurrent capability of a file system with multiple clients and multiple threads and random block sizes according to an optional embodiment of the present application. Figure 3 As shown:

[0073] Preparation steps:

[0074] This step corresponds to Figure 3 Mounting shared directories and setting task parameters: You need to extract tasks from the file system to mount shared directories and set task parameters for each task. Then send the tasks to the task management module for task management.

[0075] 1) Task management module (execution Figure 3 Task management steps in ):

[0076] In the task management module, the tasks to be tested are split according to the configured file size to be tested, the number of clients, the number of client thread configurations, the test block size and the corresponding ratio. According to the file size to be tested, the test block size and the corresponding ratio, the number of tasks that need to be executed by the thread is split, the block size allocated to each task is determined, and the tasks are placed in the task pool.

[0077] Specific description: Build a task management module on a client participating in the test to complete the splitting of the tasks to be tested. According to the size of the file to be tested, the test block size and the corresponding ratio in the test parameters, split the total number of thread tasks and the offset and block size corresponding to each thread task, abstract the concept of task pool, and put all tasks into the task pool. For example, if the test file to be written is 1G, the test block size and ratio are (4k, 20, 64k, 20, 128k, 20, 256k, 20, 1m, 20), which means that the number of input / output (IO) of the block size of 4k, 64k, 128k, 256k, and 1m accounts for 20% of the total IO number, then the file size 1024*1024k=(4k+64k+128k+256k+1m)*×, where × is the total IO number. × is 710.42, which is rounded to 710. At this time, the remaining file size is 1024*1024k-(4k+64k+128k+256k+1m)*710=616k. Then the remaining 616k is allocated in descending order according to the block size, that is, 256*2+64*1+4k*10=616. Finally, the IO block size and IO times are determined to be (4k, 720, 64k, 711, 128k, 710, 256k, 712, 1m, 710), and the total IO times is 3563 times. The offset is calculated in a circular manner according to the block size sequence (4k, 64k, 128k, 256k, 1m), that is, the offset of the first task is 0, and the block size is 4k; the offset of the second task is 4k, and the block size is 64k; the offset of the third task is 68k, and the block size is 128k; the offset of the fourth task is 196k, and the block size is 256k; the offset of the fifth task is 452k, and the block size is 1m; the offset of the sixth task is 1476k, and the block size is 4k. That is, there are 3563 tasks in the task pool, of which there are 720 tasks with a block size of 4k, 711 tasks with a block size of 64k, 710 tasks with a block size of 128k, 712 tasks with a block size of 256k, and 710 tasks with a block size of 1m.

[0078] 2)Thread distribution module:

[0079] Can execute Figure 3 The thread in performs the operation.

[0080] In the thread distribution module, concurrent scheduling and management of threads provided by each client are implemented, and the threads provided by each client are abstracted into the test thread pool for unified management. Data transmission tasks are assigned to each thread, and the single task execution time, IOPS, bandwidth information, etc. of each thread are recorded.

[0081] Specifically: Build a thread distribution module on a client participating in the test, and abstract a thread pool based on the number of configured clients and the number of threads provided on each client. The thread pool contains all available threads for this test, and the thread distribution module coordinates the task scheduling of all threads in the thread pool. Each time a task thread is distributed, the time taken to execute the task of the current thread is recorded and output to the log. By starting a timer task, the size of the test block and the number of read and write requests distributed per second are recorded to form test process data such as IOPS and bandwidth. The test process is quantified and output to the log, and the cluster's ability to cope with multi-client segment multi-threaded random concurrency scenarios is observed in real time.

[0082] For example: Figure 4 Schematic diagram of the task pool and thread pool working mode according to an optional embodiment of the present application. Figure 4 As shown in the figure: a thread distribution module is built on a client participating in the test. According to the number of configured clients and the number of threads provided on each client, a thread pool is abstracted. The thread pool contains all available threads for this test. The thread distribution module coordinates the task scheduling of all threads in the thread pool. For example, there are 5 clients in total, and 100 threads are provided on each client for testing, so there are 500 threads in the thread pool. The concurrency management module schedules these 500 threads, randomly taking a task from the task pool without putting it back and executing it in turn. After a thread is executed, it takes a task from the thread pool again. Weaken the existence of the client, integrate the threads of each client, build a thread pool, and the threads in the thread pool randomly take tasks from the task pool in turn until all tasks in the task pool are executed. Record the client or thread ID (ID) from which each thread comes, the block size of the task, the offset, the time it takes to complete the task, the total time it takes to complete the task, and other data and output them to the log. By using the timer module and combining global variables to record the number of read and write task distributions and the total block size, the data is updated every second to form two quantitative indicators, IOPS and bandwidth, and output to the log. By analyzing data such as IOPS, bandwidth, and task duration, we can draw conclusions on the ability of the storage environment to test and evaluate the multi-client multi-threaded random concurrent scenarios.

[0083] 3) Communication module:

[0084] The communication relationship can be used to determine whether the thread task is completed. If the thread task is completed, determine whether the test task is completed. If the thread task is not completed, continue with the thread execution step. If the test task is completed, perform the task distribution step. If the test task is not completed, continue to perform the task management step. Specifically:

[0085] The communication module is used to realize the communication between multiple clients. This module is used to synchronize the task transmission status of different threads on each client, report the thread status to the thread distribution module, and assist in completing concurrent scheduling.

[0086] Specifically, a communication bus is built between the various clients to synchronize the task completion status of each thread on each client, so that the thread distribution module can schedule all threads in the thread pool to go to the task management module to receive tasks.

[0087] That is, a communication bus is built between each client, and each client monitors the task thread status on the client through thread communication and other methods. When the task thread completes the data transmission task, it reports the thread task completion to the thread distribution module through the communication module and receives a new transmission task. The task contains the read and write offset, block size and other information of the thread.

[0088] 4) Data transmission module:

[0089] The actual operation of the thread task is completed in the data transmission module, and the data of the specified block size of the specified file is read or written.

[0090] That is, the data transmission module implements interaction with the underlying layer, and completes the reading or writing of data for the specified file through interaction with the data disk.

[0091] The optional embodiment of the present application proposes a method for testing the concurrent capability of a file system with multi-client multi-threaded random block size. The method is mainly composed of a task management module, a thread distribution module, a communication module and a data transmission module. In the task management module, the concept of a task pool is abstracted, the task to be tested is split, and the task is split according to the configured file size to be tested, the test block size and the corresponding ratio, and all thread tasks are placed in the task pool. In the thread distribution module, concurrent scheduling and management of threads provided by each client are implemented, and the threads provided by each client are abstracted to the test thread pool for unified management, and data transmission tasks are assigned to each thread. The tasks received by each thread in turn are random, carrying different block sizes, and forming task delay, IOPS, bandwidth and other measurement indicators. In the communication module, communication between multiple clients is implemented, and this module is used to synchronize the task transmission of different threads on each client, and the thread status is reported to the thread distribution module to assist in completing concurrent scheduling. In the data transmission module, the actual operation of the thread task is implemented to complete the reading or writing of data. By comparing the task delay, IOPS, bandwidth and other indicators in a certain model test process, the ability of the storage environment to cope with the multi-client multi-threaded random concurrent scenario is reflected.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0093] In the present embodiment, a concurrent capability testing device of a system is also provided, and the device is used to implement the above-mentioned embodiment and preferred implementation mode, and the descriptions which have been made are omitted. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0094] Figure 5 is a structural block diagram of a concurrent capability testing device of a system according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0095] A splitting module 52 is used to split the tasks to be tested in the network file system to generate a task pool;

[0096] A determination module 54 is used to determine multiple threads configured for each client corresponding to the network file system, and to construct a thread pool according to the multiple threads corresponding to the multiple clients;

[0097] A scheduling module 56, configured to perform task scheduling on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool;

[0098] The testing module 58 is used to test the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks.

[0099] Through the concurrency testing device of the system of the present application, the tasks to be tested in the network file system are split to generate a task pool; multiple threads configured for each client corresponding to the network file system are determined, and a thread pool is constructed according to the multiple threads corresponding to the multiple clients; each thread in the thread pool is task-scheduled so that multiple threads execute multiple subtasks in the task pool; the concurrency of the network file system is tested according to the execution information of multiple threads executing multiple subtasks. That is to say, the embodiment of the present application defines a task execution method of a network file system in a multi-client multi-threaded scenario, multiple threads configured by multiple clients constitute a thread pool, and the task to be tested is split into multiple subtasks to generate a task pool, so that each thread in the thread pool randomly executes each task in the task pool, and then the concurrency of the network file system is tested by the execution information of the tasks in the task pool executed by the network file system in the multi-threaded multi-client scenario. Through the optional embodiment of the present application, the problem of the unstable operation of the network file system in the multi-threaded multi-client scenario caused by the lack of a testing method for the concurrency of the network file system in the multi-threaded multi-client scenario in the related art is solved. Then, the concurrency capability of the network file system in a multi-threaded and multi-client scenario is tested to ensure that the network file system can run stably in a multi-threaded and multi-client scenario.

[0100] In an exemplary embodiment, the splitting module 52 is also used to determine the first task size of the task to be tested, and to determine the data block size of each preset data block and the proportional relationship between multiple preset data blocks, wherein the proportional relationship is used to indicate the proportion corresponding to each preset data block, and the data block size is used to indicate the second task size of each subtask to be split; the task to be tested is split according to the proportional relationship, the first task size and the data block size of each preset data block to generate multiple subtasks.

[0101] In an exemplary embodiment, the splitting module 52 is further used to determine a first product of the ratio of each preset data block and the data block size of each preset data block, and determine the sum of multiple first products; calculate the quotient of the first task size and the sum, and determine whether the quotient is an integer; when the quotient is determined to be an integer, determine the first number of subtasks of each data block size to be split according to the quotient and the ratio, and split the task to be tested into the multiple subtasks according to the first number and the data block size of each preset data block; when the quotient is determined to be a non-integer, round the quotient down to determine a target value; determine the number of subtasks of each data block size to be split according to the target value and the ratio. A second number of tasks, and splitting the task to be tested into multiple first subtasks according to the second number and the data block size of each preset data block; and determining the third task size of the task to be split in the task to be tested according to the difference between the quotient and the target value; sorting the multiple preset data blocks according to the data block size of each preset data block to determine the size relationship between the multiple preset data blocks; determining the second number of subtasks of each data block size into which the task to be split is split according to the size relationship; splitting the task to be split according to the second number and the data block size of each preset data block to generate multiple second subtasks; determining the first subtask and the second subtask as multiple subtasks in the task pool.

[0102] In an exemplary embodiment, the scheduling module 56 is further used to establish a communication relationship between the multiple clients, and synchronize the execution progress of each subtask of each thread according to the communication relationship; and schedule tasks for each thread according to the execution progress.

[0103] In an exemplary embodiment, the scheduling module 56 is also used to determine the number of first tasks allowed to be executed by any thread and the number of second tasks that the any thread has executed when the execution progress indicates that any thread among the multiple threads has completed the third subtask in the task pool; determine whether to allow the any thread to execute other tasks in the task pool based on the first number of tasks and the second number of tasks; and when it is determined that the any thread is allowed to execute other tasks in the task pool, obtain a fourth subtask from the task pool and send the fourth subtask to the any thread so that the any thread executes the fourth subtask, wherein the fourth subtask is any task in the task pool.

[0104] In an exemplary embodiment, the test module 58 is also used to obtain the task time consumed by each thread to execute each subtask in the task pool, the number of input and output operations per second corresponding to each thread, and the bandwidth information corresponding to each thread; and determine the task time consumed, the number of input and output operations per second, and the bandwidth information as the execution information.

[0105] In an exemplary embodiment, the test module 58 is also used to monitor a third number of subtasks distributed by the task pool to the multiple threads per second according to a timer corresponding to the task pool; and to monitor a fourth number of execution requests corresponding to each thread received by the task pool per second according to the timer, wherein the execution request is used to instruct each thread to execute a request for each subtask in the task pool; and determine the number of input and output operations per second corresponding to each thread based on the third number and the fourth number.

[0106] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps in the above method embodiment when running.

[0107] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0108] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0109] S1, split the tasks to be tested in the network file system to generate a task pool;

[0110] S2, determining multiple threads configured for each client corresponding to the network file system, and building a thread pool according to the multiple threads corresponding to the multiple clients;

[0111] S3, performing task scheduling on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool;

[0112] S4, testing the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks.

[0113] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in the above method embodiment.

[0114] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0115] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0116] S1, split the tasks to be tested in the network file system to generate a task pool;

[0117] S2, determining multiple threads configured for each client corresponding to the network file system, and building a thread pool according to the multiple threads corresponding to the multiple clients;

[0118] S3, performing task scheduling on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool;

[0119] S4, testing the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks.

[0120] An embodiment of the present application further provides a computer program product, including a computer program, which is implemented to perform the steps in the above method embodiment when executed by a processor.

[0121] Optionally, in this embodiment, the computer program product may be executed by a processor to perform the following steps:

[0122] S1, split the tasks to be tested in the network file system to generate a task pool;

[0123] S2, determining multiple threads configured for each client corresponding to the network file system, and building a thread pool according to the multiple threads corresponding to the multiple clients;

[0124] S3, performing task scheduling on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool;

[0125] S4, testing the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks.

[0126] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0127] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0128] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for testing the concurrent capability of a system, characterized in that: include: Split the tasks to be tested in the network file system to generate a task pool; Determine multiple threads configured for each client corresponding to the network file system, and build a thread pool according to the multiple threads corresponding to the multiple clients; Performing task scheduling on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool; The concurrency capability of the network file system is tested according to the execution information of the multiple threads executing the multiple subtasks.

2. The method according to claim 1, characterized in that Split the task to be tested in the network file system into multiple subtasks, including: Determine a first task size of the task to be tested, and determine a data block size of each preset data block and a proportional relationship between a plurality of preset data blocks, wherein the proportional relationship is used to indicate a proportion corresponding to each preset data block, and the data block size is used to indicate a second task size of each subtask to be split; The task to be tested is split according to the proportional relationship, the first task size and the data block size of each preset data block to generate a plurality of subtasks.

3. The method according to claim 2, characterized in that The task to be tested is split according to the proportional relationship, the first task size, and the data block size of each preset data block to generate a plurality of subtasks, including: Determine a first product of the ratio of each preset data block and the data block size of each preset data block, and determine a sum of a plurality of first products; Calculating a quotient of the first task size and the sum value, and determining whether the quotient is an integer; In the case where the quotient is determined to be an integer, determining a first number of subtasks of each data block size to be split according to the quotient and the proportional relationship, and splitting the task to be tested into the plurality of subtasks according to the first number and the data block size of each preset data block; When it is determined that the quotient is a non-integer, rounding the quotient down to determine the target value; Determine a second number of subtasks of each data block size to be split according to the target value and the proportional relationship, and split the task to be tested into a plurality of first subtasks according to the second number and the data block size of each preset data block; and Determining a third task size of the task to be split in the task to be tested according to a difference between the quotient and the target value; sorting the plurality of preset data blocks according to the data block size of each preset data block to determine the size relationship between the plurality of preset data blocks; Determine, according to the size relationship, a second number of subtasks of the size of each data block into which the task to be split is to be split; Splitting the task to be split according to the second number and the data block size of each preset data block to generate a plurality of second subtasks; The first subtask and the second subtask are determined as a plurality of subtasks in the task pool.

4. The method according to claim 1, characterized in that: Performing task scheduling on each thread in the thread pool so that the multiple threads execute multiple subtasks in the task pool includes: Establishing a communication relationship between the multiple clients, and synchronizing the execution progress of each subtask by each thread according to the communication relationship; Task scheduling is performed on each thread according to the execution progress.

5. The method according to claim 4, characterized in that Performing task scheduling on each thread according to the execution progress includes: In a case where the execution progress indicates that any thread among the multiple threads has completed the third subtask in the task pool, determining the number of first tasks allowed to be executed by the any thread and the number of second tasks that the any thread has executed; Determine whether to allow any thread to execute other tasks in the task pool according to the first task quantity and the second task quantity; When it is determined that any one of the threads is allowed to execute other tasks in the task pool, a fourth subtask is obtained from the task pool and the fourth subtask is sent to any one of the threads so that the any one of the threads executes the fourth subtask, wherein the fourth subtask is any one of the tasks in the task pool.

6. The method according to claim 1, characterized in that Before testing the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks, the method further includes: Obtaining the task time consumed by each thread to execute each subtask in the task pool, the number of input and output operations per second corresponding to each thread, and the bandwidth information corresponding to each thread; The task duration, the number of input and output operations per second, and the bandwidth information are determined as the execution information.

7. The method according to claim 6, characterized in that Obtain the number of input and output operations per second corresponding to each thread, including: monitoring a third number of subtasks distributed by the task pool to the multiple threads per second according to a timer corresponding to the task pool; and Monitoring, according to the timer, a fourth number of execution requests corresponding to each thread received by the task pool per second, wherein the execution request is used to instruct each thread to execute a request for each subtask in the task pool; The number of input and output operations per second corresponding to each thread is determined according to the third number and the fourth number.

8. A concurrent capability testing device for a system, characterized in that: include: The splitting module is used to split the tasks to be tested in the network file system to generate a task pool; A determination module is used to determine multiple threads configured for each client corresponding to the network file system, and to construct a thread pool according to the multiple threads corresponding to the multiple clients; A scheduling module, used for performing task scheduling on each thread in the thread pool so that the multiple threads execute the multiple subtasks in the task pool; The testing module is used to test the concurrency capability of the network file system according to the execution information of the multiple threads executing the multiple subtasks.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images