Block I / O playback method, system and device and computer storage medium

Abstract

The invention discloses a block I / O playback method, system and device and a computer storage medium, and relates to the technical field of computers, original I / O data recorded in the process that an operating system tests a block device are obtained; analyzing block I / O data in the original I / O data to obtain initial block I / O data; extracting target block I / O data belonging to the same block I / O recording process from the initial block I / O data; determining a recording timestamp of the I / O data of the target block; adjusting the record timestamp according to an adjustment rule for keeping the time interval between the I / Os of the blocks unchanged to obtain a playback timestamp; and submitting the target block I / O data to a kernel block layer according to the playback timestamp. According to the method, the problem of block I / O timestamp conflict caused by different block I / O records can be avoided; in addition, the time interval between the block I / Os after playback can be kept unchanged, and the playback accuracy of the block I / Os in the kernel layer is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a block I / O playback method, system, device, and computer storage medium. Background Art

[0002] During device development, when a problem occurs during device testing, it is necessary to locate, analyze, and resolve the issue. For example, in the Linux operating system (OS), most block devices managed by Linux are storage devices, and data transmission is the core functional area of storage devices. Therefore, when performing system-level testing on the data transmission function of block devices, it is necessary to test whether the Block I / O (block input and output) sent to the device by the OS is completed normally and correctly by the device. During this process, if the problem encountered during the test can be reproduced with a high probability, such as by attempting to reproduce it using I / O record and playback, it will help the problem-solving process.

[0003] However, the Linux runtime is divided into the user layer and the kernel layer. I / O playback is performed in the user layer. Even if the user layer sends completely consistent I / O sequences to the kernel layer multiple times, it cannot guarantee that the kernel layer will generate a consistent block I / O sequence each time, resulting in poor block I / O playback accuracy.

[0004] In summary, how to accurately replay block I / O is a problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a block I / O playback method that can, to a certain extent, solve the technical problem of how to accurately replay block I / O. This application also provides a block I / O playback system, an electronic device, and a computer-readable storage medium.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A block I / O playback method, comprising:

[0008] Get the raw I / O data recorded during the operating system's testing of the block device;

[0009] Parsing the block I / O data in the original I / O data to obtain initial block I / O data;

[0010] Extracting target block I / O data belonging to the same block I / O recording process from the initial block I / O data;

[0011] Determining a recording timestamp of the target block I / O data;

[0012] Adjusting the recording timestamp according to an adjustment rule that keeps the time interval between block I / Os unchanged to obtain a playback timestamp;

[0013] Submit the target block I / O data to the kernel block layer according to the playback timestamp.

[0014] In an exemplary embodiment, obtaining raw I / O data recorded during the operating system's testing of the block device includes:

[0015] During the operating system's block device testing process, the block device data transfer test tool is used to collect block I / O operation events that occur in the kernel block layer of the block device to obtain the original file;

[0016] Converting the original file to obtain a text file;

[0017] Parsing the text file to obtain original I / O data corresponding to the block I / O operation;

[0018] The raw I / O data includes target device information, number of CPU cores, read / write direction, occurrence time, block I / O action, access start position and access size.

[0019] In an exemplary embodiment, parsing the block I / O data in the original I / O data to obtain the initial block I / O data includes:

[0020] Extracting block I / O data whose block I / O action is completed and submitted from the original I / O data as block I / O data to be processed;

[0021] The I / O data of the block to be processed that occurs earliest is used as the benchmark block I / O data;

[0022] Set the record timestamp of the reference block I / O data to zero;

[0023] Determine a time difference between the generation time of the other block I / O data to be processed and the generation time of the reference block I / O data, and use the time difference as a recording timestamp of the other block I / O data to be processed;

[0024] The read / write direction, access start position, access size and timestamp of the block I / O data to be processed are used as the initial block I / O data.

[0025] In an exemplary embodiment, after taking the read / write direction, access start position, access size and timestamp of the block I / O data to be processed as the initial block I / O data, the method further includes:

[0026] Create the block I / O data file to be replayed;

[0027] Determining the generation time of the block I / O data file to be replayed;

[0028] Generate a file name of the block I / O data file to be replayed according to the generation time, the target device information and the number of cores of the central processing unit;

[0029] Record the initial block I / O data to the corresponding block I / O data file to be replayed.

[0030] In an exemplary embodiment, extracting target block I / O data belonging to the same block I / O recording process from the initial block I / O data includes:

[0031] Receive selection information input by the user;

[0032] Determining the block I / O data file to be replayed selected by the user according to the selection information;

[0033] Using the block I / O data file to be replayed selected by the user as the target data file to be replayed;

[0034] Detecting whether the generation time in the file name of the target data file to be replayed is consistent;

[0035] If the generation time in the file name of the target data file to be replayed is consistent, the initial block I / O data recorded in the target data file to be replayed is used as the target block I / O data.

[0036] In an exemplary embodiment, submitting the target block I / O data to the kernel block layer according to the playback timestamp includes:

[0037] Using the target device information and the number of CPU cores as name difference comparison targets, determining the number of file name types of the target data file to be replayed;

[0038] Using the file name type number as the node type number of the device node;

[0039] Determine the total amount of the target block I / O data;

[0040] Determining the total number of the device nodes according to the total amount of data;

[0041] According to the number of node types and the total number of nodes, create corresponding device nodes in the kernel layer;

[0042] The target block I / O data is concurrently transmitted to the corresponding device node, so that the device node submits the target block I / O data to the kernel block layer according to the playback timestamp.

[0043] In an exemplary embodiment, after submitting the target block I / O data to the kernel block layer according to the playback timestamp, the method further includes:

[0044] Collecting statistics on the execution information of the target block I / O data;

[0045] Problem analysis, location, and debugging are performed on data transmission performance of the operating system transmitting data to the block device based on the execution information.

[0046] A block I / O playback system, comprising:

[0047] The original I / O data acquisition module is used to obtain the original I / O data recorded during the operating system's test of the block device;

[0048] A parsing module, configured to parse the block I / O data in the original I / O data to obtain initial block I / O data;

[0049] an extraction module, configured to extract target block I / O data belonging to the same block I / O recording process from the initial block I / O data;

[0050] A timestamp determination module, configured to determine a recording timestamp of the target block I / O data;

[0051] A timestamp adjustment module, configured to adjust the recording timestamp according to an adjustment rule that keeps the time interval between block I / Os unchanged, to obtain a playback timestamp;

[0052] A submission module is used to submit the target block I / O data to the kernel block layer according to the playback timestamp.

[0053] An electronic device, comprising:

[0054] memory for storing computer programs;

[0055] A processor is configured to implement the steps of any of the above block I / O playback methods when executing the computer program.

[0056] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above block I / O playback methods.

[0057] The present application provides a block I / O playback method, which obtains raw I / O data recorded during an operating system's testing of a block device; parses the block I / O data in the raw I / O data to obtain initial block I / O data; extracts target block I / O data belonging to the same block I / O recording process from the initial block I / O data; determines a recording timestamp for the target block I / O data; adjusts the recording timestamp according to an adjustment rule that maintains a constant time interval between block I / Os to obtain a playback timestamp; and submits the target block I / O data to a kernel block layer according to the playback timestamp. This application parses the block I / O data in the original I / O data to obtain the initial block I / O data for playback at the kernel layer; and it is necessary to extract the target block I / O data belonging to the same block I / O recording process from the initial block I / O data for playback, which can avoid the block I / O timestamp conflict problem caused by non-same block I / O records and ensure the accurate playback of block I / O; in addition, it is necessary to submit the target block I / O data to the kernel block layer according to the adjustment rule that keeps the time interval between block I / Os unchanged, which can keep the time interval between block I / Os unchanged after playback, further ensuring the playback accuracy of block I / O in the kernel layer. A block I / O playback system, electronic device and computer-readable storage medium provided by this application also solve corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0059] Figure 1 A flowchart of a block I / O playback method provided in an embodiment of the present application;

[0060] Figure 2 It is a meaning diagram related to block I / O playback;

[0061] Figure 3 This is a schematic diagram of the content structure of the block I / O data file to be replayed;

[0062] Figure 4 Schematic diagram for generating playback timestamps;

[0063] Figure 5 This is a structural diagram for Linux system testing;

[0064] Figure 6 is a schematic diagram of recording I / O data in an exemplary embodiment;

[0065] Figure 7 is a schematic diagram of playback of block I / O data in an exemplary embodiment;

[0066] Figure 8 A schematic diagram of the structure of a block I / O playback system provided in an embodiment of the present application;

[0067] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0068] Figure 10 Another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0070] In order to facilitate the understanding of the present application, the technical terms involved in this application are first explained. User layer: In order to improve its own stability, security and performance optimization, the OS divides itself into the user layer and the kernel layer during operation. The user layer is relatively restricted and cannot directly access hardware resources and kernel data at will. Kernel layer: The kernel layer has great authority and almost unlimited functions. It is located between the user layer and the hardware resources. In most cases, when the user layer wants to transmit data to the device, it must first pass through the kernel layer to perform some processing on the user layer data before transferring it to the device. Kernel module: also known as a driver module or kernel driver module, is a software module that can be loaded into the OS kernel and become a part of the OS kernel. It works in the OS kernel layer, can interact with devices or other kernel modules, and can provide an interface to the user layer to allow the user layer to use specific functions of the kernel layer or transfer data with the kernel layer. Block devices are the most common type of storage device management used by Linux. The smallest unit of data storage access for these devices is a block. Their characteristic is that data can be directly accessed from any block on the device, without having to sequentially read the previous block before the next. Currently, most data storage devices, such as hard drives and USB flash drives, are classified as block devices by Linux. I / O (Input / Output) specifically refers to the input and output of data, i.e., data transfer, particularly data transfer initiated by the user layer. Block I / O, often abbreviated as BIO or bio, is a kernel-layer data structure created after user-layer I / O is processed and converted by some internal OS mechanisms. The kernel layer then uses and further processes the BIO and ultimately sends it to the target device. The smallest unit of block I / O transfer is a block. System-level testing, as used in this document, specifically refers to all testing methods and tools that test the device under test based on the OS, such as launching a test program on the OS or loading a kernel module to assist in testing. This is different from testing the bare metal hardware device directly, bypassing the OS.

[0071] See also Figure 1 , Figure 1 A flowchart of a block I / O playback method provided in an embodiment of the present application.

[0072] An embodiment of the present application provides a block I / O playback method, which may include the following steps:

[0073] Step S101: obtaining original I / O data recorded during the process of the operating system testing the block device.

[0074] In practical applications, when the operating system of a server or other computer device tests a block device, it uses a block device data transfer test tool to record the I / O data during the test for subsequent I / O playback. The I / O data recorded by the block device data transfer test tool includes both user-layer I / O data and kernel-layer I / O data. Therefore, it is possible to first obtain the raw I / O data recorded during the operating system's block device testing process for subsequent processing. The operating system can be an operating system such as Linux that has a user-layer and kernel-layer structure.

[0075] In an exemplary embodiment, the block device data transmission test tool can be a blktrace toolset or a fio tool or other tools, which is not specifically limited in this application. Among them, the blktrace tool can monitor one or more target devices specified by the user at the kernel layer, collect BIO operation event details and generate one or more original trace files, and then use the btrecord tool to parse the original trace file to generate a data file that can be used for playback, and then use the btreplay tool to read the data file, construct an I / O sequence at the user layer and submit it to the kernel layer. The fio tool can initiate batch I / O for a certain device, and can also record its own I / O sequence. The parameter -write_iolog can record important information of all I / O initiated by fio at the user layer. After the fio execution is completed, the recorded information will be saved to a file. When you want to replay the I / O later, start fio again and use the -read_iolog parameter to specify the recording information file. The I / O sequence will be constructed at the user layer and submitted to the kernel layer.

[0076] It should be noted that the number of original trace files generated by one blktrace execution is related to the number of target devices specified by the user for monitoring during the blktrace execution and the specific CPU cores involved in the BIO process. For example, if the user specifies monitoring of three target devices, and during the monitoring process, 5, 2, and 6 CPU cores respectively participate in BIO operation events of different target devices, then 5+2+6=13 original trace files will be generated. In the simplest extreme case, if the user specifies monitoring of one target device and only one CPU core participates in the BIO operation events of the device during the monitoring period, only one original trace file will be generated. It should also be noted that the blktrace toolset implements kernel-layer BIO recording and user-layer I / O playback; fio's -write_iolog and -read_iolog parameters implement user-layer I / O recording and user-layer I / O playback. The recorded I / O can only come from the fio program itself. In addition, fio's -read_iolog parameter also accepts the specified BIO operation event sequence file recorded and processed by the blktrace toolset, and then replays I / O at the user layer. This allows BIO generated by any program to be replayed by fio, but in general, fio performs I / O playback at the user layer. In other words, both the blktrace toolset and the fio tool perform I / O playback at the user level. However, the Linux runtime is divided into the user level and the kernel level. The I / O of the user level to the block device may need to go through a series of levels and processes such as page cache and kernel level processing and sorting according to its specific attributes before it can form a BIO. This causes differences between I / O and BIO in specific moments and corresponding relationships. Even if the user level sends a completely consistent I / O sequence to the kernel level multiple times, it is impossible to guarantee that the kernel level will generate a consistent BIO sequence each time. In other words, the means of I / O construction and playback at the user level is difficult to provide sufficiently accurate and stable support for block device system-level data transmission testing. The block I / O playback solution provided in this application can accurately replay kernel-level block I / O, thereby providing sufficiently accurate and stable support for block device system-level data transmission testing, making it easy to reproduce problems that occur during the block device system-level data transmission testing process with a high probability, and providing assistance in solving the problem.

[0077] Step S102: parsing the block I / O data in the original I / O data to obtain initial block I / O data.

[0078] In actual applications, the original I / O data includes I / O data collected by the user layer and block I / O data collected by the kernel layer. This application focuses on replaying block I / O data, so the block I / O data in the original I / O data can be parsed to obtain the initial block I / O data for playback.

[0079] It should be noted that the user-layer I / O data in the original I / O data can be directly replayed at the user layer with the help of the blktrace toolset or the fio tool, and this application does not make any specific restrictions here.

[0080] In an exemplary embodiment, the BIO information required by this application should at least include: target device information, occurrence time, access start location, access size, etc. Linux commonly uses major and minor device numbers to represent a specific device, so target device information can be represented by major and minor. The read / write direction refers to reading data from / writing data to the target device, the access start location refers to the storage location of the target device from which access begins, and the access size refers to how much storage space on the target device is accessed. Furthermore, the data recorded by the block device data transfer test tool must be converted before it can be used. Therefore, in the process of obtaining the raw I / O data recorded during the operating system's block device testing, the block device data transfer test tool can be used to collect block I / O operation events occurring at the kernel block layer of the block device during the operating system's block device testing to obtain a raw file; convert the raw file to obtain a text file; and parse the text file to obtain the raw I / O data corresponding to the block I / O operation. The raw I / O data includes target device information, number of CPU cores, read / write direction, occurrence time, block I / O action, access start location, and access size.

[0081] For ease of understanding, taking the block device data transmission test tool blktrace as an example, first use the blktrace tool to capture all BIO operation events that occur in the kernel block layer within a period of time for one or more specified block devices. After blktrace is executed, one or more original trace files will be automatically generated in the execution directory, which is a form of BIO sequence record; the original trace file is large in size and contains content that is not related to block I / O playback. In order to facilitate storage and simplify the processing during subsequent playback, it is necessary to extract information related to block I / O playback from it. Another tool in the blktrace tool set, blkparse, can be introduced. This tool can convert the specified original trace file into a text trace file that is easy for users to read. The content of the text trace file generated by blkparse is roughly as follows:

[0082] 259,2 5 1 0.000000000 3919 QR 231024 + 8 [app1];

[0083] 259,2 5 1 0.000016893 3923 GR 231032 + 512 [app2];

[0084] 259,2 5 1 0.000058902 3917 PW 231040 + 8 [app3].

[0085] Each line of record here corresponds to the detailed information of a BIO operation event, among which the meanings of several sections related to block I / O playback are as follows Figure 2 As shown. A BIO will go through multiple operation events in the kernel, which can be Figure 2 The "BIO Action" field in the record represents different stages of a BIO's life cycle. This application focuses on the case where the value is "Q". This BIO action indicates that a BIO is created and submitted using the kernel block layer interface submit_bio(). It can be considered the starting point of a BIO's life cycle, and the main properties of the BIO are determined at this moment.

[0086] Accordingly, in the process of parsing the block I / O data in the original I / O data to obtain the initial block I / O data, the block I / O data with the block I / O action of completed submission can be extracted from the original I / O data as the block I / O data to be processed; the block I / O data to be processed with the earliest occurrence time is used as the benchmark block I / O data; the recording timestamp of the benchmark block I / O data is set to zero; the time difference between the occurrence time of other block I / O data to be processed and the occurrence time of the benchmark block I / O data is determined, and the time difference is used as the recording timestamp of other block I / O data to be processed; the read / write direction, access starting position, access size and timestamp of the block I / O data to be processed are used as the initial block I / O data.

[0087] In an exemplary embodiment, after the read / write direction, access start position, access size and timestamp of the block I / O data to be processed are used as the initial block I / O data, a block I / O data file to be replayed can also be created; the generation time of the block I / O data file to be replayed is determined; the file name of the block I / O data file to be replayed is generated according to the generation time, target device information and the number of CPU cores, so that the target device information in the BIO important information is reflected in the file name, which helps to reduce the size of the BIO data file to be replayed and improve the block I / O playback efficiency; the initial block I / O data is recorded to the corresponding block I / O data file to be replayed.

[0088] In a specific application scenario, a possible format of the block I / O data file to be replayed is: multiple pieces of recorded block I / O important information are stored in sequence in the file, and each piece of block I / O important information includes: read / write direction, timestamp, access start position, and access size. The timestamp can be the precise time point of the block I / O event in the format of "seconds.nanoseconds". The time of the first block I / O event recorded by blktrace is used as the base moment and recorded as 0.000000000. The time when each subsequent block I / O event occurs is recorded as the difference from the base moment. For example, the 0.000058902 in the text trace file content example above indicates that the BIO event occurred 58902 nanoseconds after the first block I / O event captured by blktrace. Overall, the internal structure of the block I / O data file to be replayed is similar to the C language structure array format. The content structure of the block I / O data file to be replayed can be as follows. Figure 3 As shown, the block I / O data file to be replayed can be named record_{%major%}_{%minor%}_{%core%}_{%date%}_{%time%}, where {%major%} and {%minor%} are replaced with the major and minor values of the target device, and {%core%} is replaced with the CPU The core value, {%date%}, and {%time%} must be replaced with the current date and time when the file is generated. For example, the file name generated at 15:21:53 on January 8, 2025 might be: record_259_2_5_20250108_152153. One or more block I / O data files to be replayed are parsed and refined one by one to generate one or more corresponding block I / O data files to be replayed. Multiple block I / O data files to be replayed generated in the same batch must have exactly the same {%date%} and {%time%} in their file names, indicating that they come from the same block I / O recording process. Furthermore, the block I / O data files to be replayed generated in the same batch can be copied or moved to a suitable directory for subsequent playback.

[0089] Step S103: extracting target block I / O data belonging to the same block I / O recording process from the initial block I / O data.

[0090] In actual applications, during the block device system-level data transmission test, multiple tests and I / O records may be performed. If all initial block I / O data are directly replayed, unreasonable BIO timestamp conflicts may occur in multiple BIO data files to be replayed generated by different BIO recording processes. For example, multiple BIO events for the same device occurring at the same time on the same CPU core will cause a conflict, affecting the playback of block I / O at the kernel layer. To avoid this situation, the target block I / O data belonging to the same block I / O recording process must be extracted from the initial block I / O data. In this way, when the target block I / O data is subsequently replayed at the kernel layer, there will be no timestamp conflict problem for block I / O data recorded at different times.

[0091] In an exemplary embodiment, in the process of extracting target block I / O data belonging to the same block I / O recording process from the initial block I / O data, selection information input by the user can be received; based on the selection information, the block I / O data file to be replayed selected by the user is determined; the block I / O data file to be replayed selected by the user is used as the target data file to be replayed; it is detected whether the generation time in the file name of the target data file to be replayed is consistent, for example, by detecting whether the generation time in the file name of the target data file to be replayed is consistent through the {% date%} and {% time%} fields of the file name; if the generation time in the file name of the target data file to be replayed is consistent, the initial block I / O data recorded in the target data file to be replayed is used as the target block I / O data.

[0092] Step S104: Determine the recording timestamp of the target block I / O data.

[0093] Step S105: adjusting the recording timestamp according to the adjustment rule of keeping the time interval between block I / Os unchanged to obtain a playback timestamp.

[0094] Step S106: Submit the target block I / O data to the kernel block layer according to the playback timestamp.

[0095] In actual applications, each target block I / O data has its own recording timestamp, which is also the timestamp of the block I / O data recorded during the test. This recording timestamp will affect the playback moment of the block I / O data. However, considering that the recording timestamp has already occurred, the recording timestamp needs to be converted into a future timestamp so that the target block I / O data can be played back according to the converted timestamp. That is, the recording timestamp of the target block I / O data needs to be determined, and the recording timestamp is adjusted according to the adjustment rule that keeps the time interval between block I / Os unchanged to obtain the playback timestamp. In this way, the time interval between the playback timestamps of each target block I / O data is consistent with the time interval between the recording timestamps. Subsequently, if the target block I / O data is submitted to the kernel block layer according to the playback timestamp, it can be guaranteed that the target block I / O data will be played back in sequence and on time in the kernel layer.

[0096] In an exemplary embodiment, the timestamp of each BIO in the block I / O data file to be replayed can be temporarily adjusted to a specific time in the future. The calculation method can be: future = app_start_time + delay, new_timestamp = future + old_timestamp. App_start_time is the exact time when playback begins, and delay is an estimated fixed duration, such as 5 seconds. Delay must be greater than the total time from the start of playback to the completion of sending the block I / O data file content to be replayed. This ensures that by the time future arrives, all device nodes in the block I / O playback module have completed receiving the adjusted block I / O data file content to be replayed. Old_timestamp is the original timestamp in the block I / O's important information before the adjustment, and new_timestamp is the timestamp in the block I / O's important information after the adjustment. New_timestamp is no longer the difference from the base time of the first block I / O, but a specific future time. Therefore, new_timestamp indicates the expected future time point for playback of the corresponding block I / O. Figure 4 This indicates that the adjustment to the playback block I / O data file content is limited to the timestamp member of each BIO in the file content. Combined with the above calculation method, it can be seen that the time interval between each BIO before the modification (that is, the difference between the original timestamps) is equal to the time interval between each BIO after the modification. Therefore, this modification does not affect the accuracy of the time interval between each BIO during playback.

[0097] In an exemplary embodiment, in the process of submitting the target block I / O data to the kernel block layer according to the playback timestamp, the target device information and the number of CPU cores can be used as the name difference comparison targets to determine the number of file name types of the target data file to be replayed; the number of file name types is used as the number of node types of the device node; the total amount of data of the target block I / O data is determined; based on the total amount of data, the total number of nodes of the device nodes is determined, for example, the total number of nodes of the device nodes can be determined based on the total amount of data and the processing capacity of the device nodes; corresponding device nodes are created in the kernel layer according to the number of node types and the total number of nodes; the target block I / O data is concurrently transmitted to the corresponding device nodes, so that the device nodes submit the target block I / O data to the kernel block layer according to the playback timestamp. The advantage of doing this is that when the user layer concurrently accesses multiple different device nodes, the kernel will run multiple device instances concurrently and run on multiple CPU cores. They run independently and in parallel without affecting each other, thus avoiding the situation where the user layer concurrently accesses the same device node (device instance) and requires it to replay block I / Os with too dense timestamps. Because a single device instance only runs on one CPU core, it cannot guarantee the accurate replay of each block I / O at the time corresponding to the high-density block I / O timestamp, resulting in block I / O accumulation or omission.

[0098] In a specific application scenario, during the process of generating device nodes, a kernel layer module for block I / O playback can be loaded, recorded as a BIO playback module, and the module is required to allocate and generate one or more device nodes according to the block I / O data files to be replayed selected by the user. For example, the BIO playback module can allocate and generate one or more / dev / replay_{%major%}_{%minor%}_{%core%} device nodes according to the number and file names of the block I / O data files to be replayed in the previous step, where {%major%}, {%minor%}, and {%core%} are all from For example, if three I / O data files from the same batch (record_259_0_3_20250108_152153, record_259_1_0_20250108_152153, and record_259_2_5_20250108_152153) are selected during execution, the BIO replay module will allocate and generate three corresponding device nodes: / dev / replay_259_0_3, / dev / replay_259_1_0, and / dev / replay_259_2_5. The I / O data file information can be transferred between the user layer and the kernel module through various methods, such as kernel module parameters, ioctl, and sysfs. Furthermore, after the BIO playback module creates the above-mentioned device node, it can start a corresponding number of multiple processes or multiple threads according to the number of block I / O data files to be replayed selected by the user. Each process / thread temporarily adjusts the content of a block I / O data file to be replayed and sends it to the corresponding device node. In other words, based on the correspondence between the {%major%}, {%minor%}, and {%core%} in the block I / O data file name to be replayed and the device node name, the content of the block I / O data file to be replayed can be temporarily adjusted and sent to the corresponding device node in a concurrent manner.

[0099] Correspondingly, the BIO replay module runs in the kernel layer in the form of one or more device instances. Each device instance obtains its own first BIO important information from the adjusted block I / O data file content to be replayed, allocates and fills a preliminary BIO associated with the target device major and minor according to the information, and calls the kernel submit_bio() interface at the new_timestamp time of the BIO to submit it to the kernel block layer. Then, it obtains the second BIO important information, constructs a preliminary BIO associated with the target device major and minor, and submit_bio() at the new_timestamp time of the BIO... until all BIOs to be replayed of this device node are submitted by submit_bio(). After completing the replay of all BIO sequences, the aforementioned kernel layer replay module is uninstalled, and one or more / dev / replay_{%major%}_{%minor%}_{%core%} device nodes are destroyed.

[0100] In an exemplary embodiment, after the target block I / O data is submitted to the kernel block layer according to the playback timestamp, the execution information of the target block I / O data can also be counted; based on the execution information, the data transmission performance of the operating system transmitting data to the block device is analyzed, located and debugged.

[0101] In an exemplary embodiment, the status information of the user layer and the kernel layer can also be recorded when the operating system starts testing the block device. Accordingly, before extracting the target block I / O data belonging to the same block I / O recording process from the initial block I / O data, the user layer and the kernel layer can be restored to the corresponding state according to the recorded status information of the user layer and the kernel layer, the user layer I / O data in the original I / O data can be extracted, the user layer I / O data can be submitted to the user layer for playback, and the block I / O data generated by the kernel layer can be recorded. It is detected whether the block I / O data generated by the kernel layer is consistent with the initial block I / O data. If not, the recorded status information of the user layer and the kernel layer is deleted. If consistent, the recorded status information of the user layer and the kernel layer is saved, so that when the I / O playback is performed later, the user layer and the kernel layer can be restored to the corresponding state according to the recorded status information of the user layer and the kernel layer, and the target block I / O data belonging to the same block I / O recording process from the initial block I / O data and subsequent steps are executed. In this way, the present application can detect whether the recorded status information of the user layer and kernel layer will affect the generation of block I / O through the recorded initial block I / O data, and when the recorded status information of the user layer and kernel layer does not affect the generation of block I / O, the status of the user layer and kernel layer can be restored to the status during the test during I / O playback, and the test status of the operating system can be further replayed, thereby realizing the playback of the I / O test environment, improving the integrity of the I / O playback, and facilitating more accurate restoration of the process of the operating system testing block devices.

[0102] The present application provides a block I / O playback method, which obtains raw I / O data recorded during an operating system's testing of a block device; parses the block I / O data in the raw I / O data to obtain initial block I / O data; extracts target block I / O data belonging to the same block I / O recording process from the initial block I / O data; determines a recording timestamp for the target block I / O data; adjusts the recording timestamp according to an adjustment rule that maintains a constant time interval between block I / Os to obtain a playback timestamp; and submits the target block I / O data to a kernel block layer according to the playback timestamp. This application parses the block I / O data in the original I / O data to obtain the initial block I / O data for playback at the kernel layer; and extracts the target block I / O data belonging to the same block I / O recording process from the initial block I / O data for playback, which can avoid the block I / O timestamp conflict problem caused by non-same block I / O records and ensure accurate playback of block I / O; in addition, it is necessary to submit the target block I / O data to the kernel block layer according to the adjustment rule that keeps the time interval between block I / Os unchanged, which can keep the time interval between block I / Os unchanged after playback, further ensuring the playback accuracy of block I / O in the kernel layer.

[0103] In order to facilitate the understanding of the block I / O playback solution provided by this application, the test process of block devices in Linux is now explained as an example. Among them, most of the block devices managed by Linux are storage devices, and data transmission belongs to the core functional category of storage devices. Therefore, when performing system-level testing on the data transmission function of block devices, it is necessary to strictly test whether the Block I / O sent to the device by the OS is completed normally and correctly by the device. The test tools and methods used in system-level testing are mainly located in the user layer and kernel layer, such as Figure 5 As shown, there are two types of I / O batch testing methods: sequential I / O and random I / O. Sequential I / O involves performing block-by-block I / O on the entire storage area of a block device or a specific storage area. For example, the I / O sequence is: block 10 -> block 11 -> block 12, etc. This is primarily used to test the sequential read and write functionality and performance of a block device. Random I / O involves performing I / O on randomly selected blocks within the entire storage area of a block device or a specific storage area. For example, the I / O sequence is: block 12 -> block 3 -> block 65, etc. This is primarily used to test the random read and write functionality and performance of a block device.

[0104] Assuming that the block device data transmission test tool is blktrace, write the user layer application app_record and the kernel layer BIO playback module, and the two work together to complete the recording and playback of I / O data. The recording process is as follows: Figure 6 As shown, the following process is included:

[0105] app_record starts executing; specify the target device;

[0106] Call blktrace to start monitoring the target device;

[0107] Start testing the system-level data transmission function of the target device;

[0108] End the test, the test passed or there was a problem; stop the blktrace program;

[0109] Call blkparse to convert the original trace file into a text trace file;

[0110] Filter the record lines with BIO action "Q" from the text trace file;

[0111] Extract important BIO information from the record line and save it as a BIO data file to be played back;

[0112] app_record has completed execution.

[0113] Accordingly, the playback process can be as follows Figure 7 As shown, the following process is included:

[0114] The app_replay main process specifies the BIO data file to be replayed; determines whether it is from the same BIO recording process. If so, it analyzes the file name and number of the BIO data to be replayed, loads the BIO replay module, and passes the number and file name of the BIO data to be replayed to the BIO replay module; if not, it ends;

[0115] The BIO playback module receives information such as the number and file name of the BIO data files to be played back, and allocates and generates device nodes according to certain rules based on the information;

[0116] The app_replay main process starts multiple processes / threads according to the number of BIO data files to be replayed;

[0117] The app_replay subprocess / subthread modifies the timestamp of each important BIO information in the BIO data file to be replayed; the modified content of the BIO data file to be replayed is sent to the corresponding device node;

[0118] The device node / device instance receives the modified content of the BIO data file to be replayed; determines whether all BIOs to be replayed have been traversed. If so, the app_replay subprocess / subthread ends; if not, it retrieves the next BIO's important information, allocates and fills the BIO, associates it with the target device, waits for the appropriate time to submit the BIO, and returns to the step of determining whether all BIOs to be replayed have been traversed.

[0119] After the app_replay main process waits for all child processes / child threads to end, it unloads the BIO replay module and ends.

[0120] This embodiment combines user-level applications and kernel-level modules to implement a Linux BIO sequence recording and playback method. Focusing on Linux kernel block-level I / O rather than user-level I / O, this method more closely resembles the actual data transfer process within the scope of storage device system-level testing, improving the accuracy and stability of data transfer recording and playback. This embodiment implements recording and playback functionality by simply executing the corresponding user-level applications (app_record, app_replay) and loading an independent kernel module (the BIO playback module), without modifying the OS kernel code. This avoids contamination of existing kernel functionality or performance degradation. The independent kernel module can be uninstalled after playback completes, leaving no impact on other programs or data during application execution. After execution, no remaining data remains except for the BIO data file to be played back, minimizing disruption to the normal operation of the entire OS. The recorded BIO sequence information is persistently stored as a file with a concise file name, facilitating management and future access when playback is required. In general, this embodiment aims to reduce the difficulty of BIO record-playback operations and reduce the interference of such operations on the OS, while maximizing the probability of recurrence of data transmission problems.

[0121] See also Figure 8 , Figure 8 A schematic diagram of the structure of a block I / O playback system provided in an embodiment of the present application.

[0122] An embodiment of the present application provides a block I / O playback system, which may include:

[0123] The original I / O data acquisition module 101 is used to acquire the original I / O data recorded during the operating system's test of the block device;

[0124] The parsing module 102 is used to parse the block I / O data in the original I / O data to obtain initial block I / O data;

[0125] An extraction module 103 is configured to extract target block I / O data belonging to the same block I / O recording process from the initial block I / O data;

[0126] A timestamp determination module 104 is used to determine a recording timestamp of the target block I / O data;

[0127] The timestamp adjustment module 105 is used to adjust the recording timestamp according to the adjustment rule of keeping the time interval between block I / Os unchanged to obtain a playback timestamp;

[0128] The submission module 106 is used to submit the target block I / O data to the kernel block layer according to the playback timestamp.

[0129] In an embodiment of the present application, a block I / O playback system is provided, wherein the raw I / O data acquisition module may include:

[0130] The original file acquisition unit is used to collect block I / O operation events of the block device at the kernel block layer through the block device data transmission test tool during the process of the operating system testing the block device to obtain the original file;

[0131] A file conversion unit, used for converting the original file to obtain a text file;

[0132] A file parsing unit, used for parsing text files to obtain original I / O data corresponding to block I / O operations;

[0133] The raw I / O data includes target device information, number of CPU cores, read / write direction, occurrence time, block I / O action, access start position, and access size.

[0134] An embodiment of the present application provides a block I / O playback system, wherein the parsing module may include:

[0135] A data extraction unit is used to extract block I / O data for which block I / O action is completed and submitted from the original I / O data as block I / O data to be processed;

[0136] A setting unit, configured to use the to-be-processed block I / O data with the earliest occurrence time as the reference block I / O data;

[0137] A zeroing unit, used for setting the recording timestamp of the reference block I / O data to zero;

[0138] a difference determination unit, configured to determine a time difference between a time when other block I / O data to be processed occurs and a time difference between the time when the reference block I / O data occurs, and use the time difference as a recording timestamp for the other block I / O data to be processed;

[0139] The data processing unit is used to take the read / write direction, access start position, access size and timestamp of the block I / O data to be processed as the initial block I / O data.

[0140] The block I / O playback system provided in the embodiment of the present application may further include:

[0141] A file creation unit is configured to create a file of the block I / O data to be replayed after the data processing unit uses the read / write direction, access start position, access size and timestamp of the block I / O data to be processed as the initial block I / O data;

[0142] A time determination unit, used to determine the generation time of the block I / O data file to be replayed;

[0143] A name generation unit, used to generate a file name for the block I / O data file to be replayed based on the generation time, target device information, and the number of CPU cores;

[0144] The recording unit is used to record the initial block I / O data to the corresponding block I / O data file to be played back.

[0145] An embodiment of the present application provides a block I / O playback system, wherein the extraction module may include:

[0146] A selection receiving unit, configured to receive selection information input by a user;

[0147] A selection response unit is used to determine the block I / O data file to be replayed selected by the user according to the selection information;

[0148] A file processing unit, configured to use the block I / O data file to be replayed selected by the user as a target data file to be replayed;

[0149] The file detection unit is used to detect whether the generation time in the file name of the target data file to be replayed is consistent; if the generation time in the file name of the target data file to be replayed is consistent, the initial block I / O data recorded in the target data file to be replayed is used as the target block I / O data.

[0150] In an embodiment of the present application, a block I / O playback system is provided, wherein a submission module may include:

[0151] a type number determination unit, configured to determine the number of file name types of a target data file to be replayed, using target device information and the number of CPU cores as name difference comparison targets;

[0152] A type number setting unit, used for taking the file name type number as the node type number of the device node;

[0153] A total amount determination unit, used for determining the total amount of target block I / O data;

[0154] A quantity determination unit, used for determining the total number of device nodes according to the total amount of data;

[0155] The node creation unit is used to create corresponding device nodes in the kernel layer according to the number of node types and the total number of nodes;

[0156] The data sending unit is used to concurrently transmit the target block I / O data to the corresponding device node, so that the device node submits the target block I / O data to the kernel block layer according to the playback timestamp.

[0157] The block I / O playback system provided in the embodiment of the present application may further include:

[0158] The data response module is used to submit the target block I / O data to the kernel block layer according to the playback timestamp, and then count the execution information of the target block I / O data; based on the execution information, the data transmission performance of the operating system to the block device is analyzed, located and debugged.

[0159] The present application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the block I / O playback method provided in the embodiments of the present application. Figure 9 , Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0160] An electronic device provided by an embodiment of the present application includes a memory 201 and a processor 202. The memory 201 stores a computer program, and when the processor 202 executes the computer program, the steps of the block I / O playback method described in any of the above embodiments are implemented.

[0161] See also Figure 10 Another electronic device provided in an embodiment of the present application may further include: an input port 203 connected to the processor 202 for transmitting commands inputted from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside world; and a communication module 205 connected to the processor 202 for enabling communication between the electronic device and the outside world. The display unit 204 may be a display panel, a laser scanning display, etc. The communication method adopted by the communication module 205 includes but is not limited to Mobile High-Definition Link (MHL), Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), wireless connection: Wireless Fidelity (WiFi), Bluetooth communication technology, Bluetooth low energy communication technology, and communication technology based on IEEE802.11s.

[0162] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the block I / O playback method described in any of the above embodiments are implemented.

[0163] The computer-readable storage medium involved in this application includes random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.

[0164] An embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the block I / O playback method described in any of the above embodiments.

[0165] For descriptions of the relevant portions of the block I / O playback system, electronic device, and computer-readable storage medium provided in the embodiments of the present application, please refer to the detailed description of the corresponding portions in the block I / O playback method provided in the embodiments of the present application, and will not be repeated here. In addition, portions of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0166] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0167] The above description of the disclosed embodiments will enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A block I / O playback method, characterized in that: include: Get the raw I / O data recorded during the operating system's testing of the block device; Parsing the block I / O data in the original I / O data to obtain initial block I / O data; Extracting target block I / O data belonging to the same block I / O recording process from the initial block I / O data; Determining a recording timestamp of the target block I / O data; Adjusting the recording timestamp according to an adjustment rule that keeps the time interval between block I / Os unchanged to obtain a playback timestamp; Submit the target block I / O data to the kernel block layer according to the playback timestamp.

2. The method according to claim 1, characterized in that The obtaining of raw I / O data recorded during the operating system's testing of the block device includes: During the operating system's block device testing process, the block device data transfer test tool is used to collect block I / O operation events that occur in the kernel block layer of the block device to obtain the original file; Converting the original file to obtain a text file; Parsing the text file to obtain original I / O data corresponding to the block I / O operation; The raw I / O data includes target device information, number of CPU cores, read / write direction, occurrence time, block I / O action, access start position and access size.

3. The method according to claim 2, characterized in that The parsing of the block I / O data in the original I / O data to obtain the initial block I / O data includes: Extracting block I / O data whose block I / O action is completed and submitted from the original I / O data as block I / O data to be processed; The I / O data of the block to be processed that occurs earliest is used as the benchmark block I / O data; Set the record timestamp of the reference block I / O data to zero; Determine a time difference between the generation time of the other block I / O data to be processed and the generation time of the reference block I / O data, and use the time difference as a recording timestamp of the other block I / O data to be processed; The read / write direction, access start position, access size and timestamp of the block I / O data to be processed are used as the initial block I / O data.

4. The method according to claim 3, characterized in that After the read / write direction, access start position, access size and timestamp of the block I / O data to be processed are used as the initial block I / O data, the method further includes: Create the block I / O data file to be replayed; Determining the generation time of the block I / O data file to be replayed; Generate a file name of the block I / O data file to be replayed according to the generation time, the target device information and the number of cores of the central processing unit; Record the initial block I / O data to the corresponding block I / O data file to be replayed.

5. The method according to claim 4, characterized in that The step of extracting target block I / O data belonging to the same block I / O recording process from the initial block I / O data includes: Receive selection information input by the user; Determining the block I / O data file to be replayed selected by the user according to the selection information; Using the block I / O data file to be replayed selected by the user as the target data file to be replayed; Detecting whether the generation time in the file name of the target data file to be replayed is consistent; If the generation time in the file name of the target data file to be replayed is consistent, the initial block I / O data recorded in the target data file to be replayed is used as the target block I / O data.

6. The method according to claim 5, characterized in that Submitting the target block I / O data to the kernel block layer according to the playback timestamp includes: Using the target device information and the number of CPU cores as name difference comparison targets, determining the number of file name types of the target data file to be replayed; Using the file name type number as the node type number of the device node; Determine the total amount of the target block I / O data; Determining the total number of the device nodes according to the total amount of data; According to the number of node types and the total number of nodes, create corresponding device nodes in the kernel layer; The target block I / O data is concurrently transmitted to the corresponding device node, so that the device node submits the target block I / O data to the kernel block layer according to the playback timestamp.

7. The method according to claim 1, characterized in that After submitting the target block I / O data to the kernel block layer according to the playback timestamp, the method further includes: Collecting statistics on the execution information of the target block I / O data; Problem analysis, location, and debugging are performed on data transmission performance of the operating system transmitting data to the block device based on the execution information.

8. A block I / O playback system, characterized in that: include: The original I / O data acquisition module is used to obtain the original I / O data recorded during the operating system's test of the block device; A parsing module, configured to parse the block I / O data in the original I / O data to obtain initial block I / O data; an extraction module, configured to extract target block I / O data belonging to the same block I / O recording process from the initial block I / O data; A timestamp determination module, configured to determine a recording timestamp of the target block I / O data; A timestamp adjustment module, configured to adjust the recording timestamp according to an adjustment rule that keeps the time interval between block I / Os unchanged, to obtain a playback timestamp; A submission module is used to submit the target block I / O data to the kernel block layer according to the playback timestamp.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the block I / O playback method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the block I / O playback method according to any one of claims 1 to 7.

Images