Virtual generation method and system for file system test data, equipment and medium

Abstract

The invention discloses a virtual generation method, system and device for file system test data and a medium, and relates to the technical field of virtual generation, and the method comprises the following steps: generating a memory file system through a user space memory file system FUSE; reading directory information in the memory file system, and generating a filename list and a directory list through a periodic specified directory number generation rule and a filename rule; and opening any file in the current directory list, judging whether the file exists or not through the file name list, successfully opening the file when the current file exists, reading the opened file, and generating test data with a specified length based on the content generation factor of the file, the read data offset and the requested byte number. The memory file system is a read-only file system, reading is carried out only when the current file exists, the virtual test data is generated in real time when the current file is read, the virtual test data is not used for a resident memory and is not stored in a disk, and dependence on disk storage is avoided.

Description

Technical Field

[0001] This invention relates to the field of virtual generation technology, and in particular to a method, system, device and medium for virtually generating file system test data. Background Technology

[0002] Enterprise-level storage systems, as a crucial component of computer systems, provide persistent data storage, efficient data management, and access capabilities, making them an indispensable part of modern information technology infrastructure. With the continuous growth of data volume and the diversification of application scenarios, the design and implementation of storage systems are constantly evolving and optimizing. During the development of various storage systems, long-term testing of their functionality, stability, and other indicators is necessary. One of the most important and fundamental tests is verifying the file data storage function and its stability—the essential function of a storage system.

[0003] Before testing the storage system, a certain number of files need to be generated and stored on disk for subsequent testing. Test cases are written to write these pre-generated test files into the storage system under test to test its storage functionality. During read tests, the contents of the files written to the storage system are read and compared with the contents of the original generated files to verify the correctness of the stored data.

[0004] When the number and types of test files are small, it is not possible to comprehensively test the storage system. However, due to disk capacity limitations, it is not possible to generate a large number of test files. This is especially true when the amount of data to be tested is massive, or when many storage nodes need to be tested. For example, when testing 100 storage nodes to write test data with a total size of 1PB, a total of 1 billion files, or a single 10TB file, it is usually impossible to generate such a massive amount of test data, and it also requires high equipment costs. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a method, system, device, and medium for virtually generating file system test data, thereby solving the problem that the prior art is usually unable to generate such a massive amount of test data, and even if it is possible, it requires a considerable amount of time and equipment costs.

[0006] The present invention specifically provides the following technical solution: A method for virtually generating file system test data includes the following steps: A memory file system is generated in user space using the user-space memory file system FUSE; the memory file system is a read-only file system. When the memory file system is mounted, the directory information in the memory file system is read, and a list of filenames and a list of directories are generated by periodically specifying the number of directories and the filename rules. Open any file in the current directory list, check if the file exists by checking the filename list, open the file successfully if it exists, read the opened file, and generate test data of a specified length based on the file content generation factor, the data offset read, and the number of bytes requested.

[0007] Preferably, the step of generating the filename list and directory list by periodically specifying the directory number generation rule and filename rule specifically involves: Based on periodic filename rules and specified directory number generation rules, obtain the directory generation factor, root directory generation factor, and parent directory content summary factor in the current directory; among them, the directory generation factor is the factor that determines the file name, directory name, and number of generated files in the current directory, the root directory generation factor is the factor that determines the data for generating the root directory, and the parent directory content summary factor is the summary factor of the list of all generated files and directories in the parent directory. Generate a fixed number and fixed-size list of filenames and directories using directory generation factors, root directory generation factors, and parent directory content summary factors.

[0008] Preferably, the filename rules and the rules for generating the specified number of directories are as follows: When the directory depth exceeds the maximum directory level, no subdirectory information is generated. The root directory parent directory generation factor is set to a fixed value, and rules are set for the maximum number of directories, the maximum number of files, the maximum file name length, and the maximum directory name length. The maximum number of directories is the maximum number of subdirectories contained in the specified directory, the maximum number of files is the maximum number of files contained in the specified directory, the maximum filename length specifies the maximum length of filenames, and the maximum directory name length specifies the maximum length of directory names.

[0009] Preferably, the step of generating a fixed number and fixed size list of filenames and a list of directories using a directory generation factor, a root directory generation factor, and a parent directory content summary factor specifically involves: Calculate the product of the directory generation factor and the parent directory content summary factor, and then take the remainder based on the maximum number of files to determine the number of files generated. Multiply the product of the directory generation factor and the parent directory content summary factor by the root directory generation factor, and based on the multiplication result, take the remainder according to the maximum number of files to generate the number of directories; Generate a first string consisting of the maximum filename length multiplied by the number of files, and divide the first string equally according to the number of files to use as file extensions, thereby generating a list of filenames; Generate a second string consisting of the maximum filename length multiplied by the number of directories. Divide the second string equally according to the number of files, and use it as the name of each directory to generate a directory list.

[0010] Preferably, test data of a specified length is generated based on the content generation factor of the file, the data offset read, and the number of bytes requested, specifically as follows: The test data is generated by taking the content generation factor, the data offset to be read, and the number of bytes to be requested as input data. The content generation factor is used to specify the variable factor for generating the file content. The data offset to be read is obtained by reading the index. The number of bytes to be requested includes the content length and the data range. The content length is used to determine the file size, and the data range is used to specify the value range of a single byte of data.

[0011] Preferably, the range of values ​​specified as a single byte data content is specifically as follows: Obtain the product of the data offset information and the variable factor, and obtain the result of shifting the data offset information bit left by 2 bits. Perform a bitwise AND operation on the sum of the product and the result of shifting left by 2 bits according to the value range of a single byte data content to obtain the requested number of bytes.

[0012] This invention provides a virtual generation system for file system test data, comprising: The file system generation module is used to generate a memory file system in user space through the user space memory file system FUSE; the memory file system is a read-only file system. The directory and file name generation module is used to read the directory information in the memory file system when the memory file system is mounted, and generate a file name list and a directory list by periodically specifying the directory number generation rules and file name rules. The data generation module is used to open any file in the current directory list, determine whether the file exists by checking the file name list, open the file successfully if it exists, read the opened file, and generate test data of a specified length based on the content generation factor of the file, the data offset read, and the number of bytes requested.

[0013] The present invention provides a computer device, including a memory and a processor. The memory stores a program, and when the program is executed by the processor, the processor performs the steps of the above-described method for virtually generating file system test data.

[0014] The present invention provides a storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the above-described method for virtually generating test data of a file system are implemented.

[0015] Compared with the prior art, the present invention has the following significant advantages: This invention constructs a memory file system. When a directory content reading request is made based on the memory file system, a list of filenames and a list of directories are generated. Any file in the current directory list is opened, and the existence of the file is determined by the filename list. If the file exists, it is successfully opened and read. Test data of a specified length is generated based on the file's content generation factor, the data offset read, and the number of bytes requested. Since the memory file system is a read-only file system, it only reads the file if it exists, and virtual test data is generated in real time only during reading. It is not used to reside in memory or stored on disk, avoiding dependence on disk storage, improving the efficiency and repeatability of data generation, and reducing generation time and equipment costs. It is very suitable for the needs of large-scale memory file system testing. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of reading file content provided by the present invention; Figure 2 This is a schematic diagram of the directory reading method provided by the present invention; Figure 3 This is a flowchart of a method for virtually generating file system test data provided by the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] The purpose of this invention is to provide an efficient, repeatable, and storage-space-free method for virtually generating test data for a memory file system. This method generates a memory file system using a user-space memory file system (FUSE). The file content is generated as fixed data using a pre-defined generation method, thereby achieving the virtual generation of test data for the memory file system.

[0019] FUSE (Filesystem in Userspace) is a framework that allows developers to create in-memory file systems in user space. It does not provide specific commands or parameters, but rather provides an interface library that allows developers to implement custom in-memory file systems using various programming languages ​​(such as C, C++, Python, etc.).

[0020] The working principle of USE can be summarized in the following steps: Kernel Module: The FUSE core module is a module loaded into the Linux kernel that is responsible for communicating with the FUSE library in user space.

[0021] User-space library: The FUSE library provides a set of APIs that allow user-space programs to perform various operations on the memory file system. User-space memory file system programs respond to memory file system operation requests through these APIs.

[0022] Communication mechanism: Kernel modules and user-space libraries communicate through communication pipes, which are usually implemented through Unix sockets or similar mechanisms.

[0023] Memory file system operation mapping: When a user-space program receives a request for operation from the memory file system, it translates these requests into actual file or data operations (such as read, write, create, delete, etc.) and returns the results to the caller.

[0024] like Figure 1 , Figure 2 and Figure 3 As shown, the present invention provides a method for virtually generating file system test data, which specifically includes the following steps: Step S1: Generate a memory file system in user space using the user space memory file system FUSE; the memory file system is a read-only file system.

[0025] Step S2: When the memory file system is mounted, read the directory information in the memory file system, and generate a list of filenames and a list of directories by periodically specifying the number of directories and the filename rules.

[0026] Based on periodic filename rules and a specified number of directory generation rules, this algorithm obtains the directory generation factor, root directory generation factor, and parent directory content summary factor for the current directory. The algorithm description includes the following key input information: the directory generation factor, which determines the names and number of files and directories generated in the current directory; and the parent directory content summary factor, which is a summary factor of the list of all generated files and directories in the parent directory.

[0027] Generate a fixed number and fixed-size list of filenames and directories using directory generation factors, root directory generation factors, and parent directory content summary factors.

[0028] Specifically, the filename rules and the rules for generating the specified number of directories are as follows: The maximum directory level specifies the maximum number of subdirectories that can be included in the generated memory file system. When the directory depth exceeds the maximum directory level, no subdirectory information is generated. The root directory parent directory generation factor is set to a fixed value; in one embodiment, the fixed value is an initial value of 1. Rules are also set for the maximum number of directories, the maximum number of files, the maximum filename length, and the maximum directory name length. The maximum number of directories refers to the maximum number of subdirectories contained in a specified directory; the maximum number of files refers to the maximum number of files contained in a specified directory; the maximum filename length specifies the maximum length of filenames; and the maximum directory name length specifies the maximum length of directory names.

[0029] A fixed number and fixed-size list of filenames and directories are generated using a directory generation factor, a root directory generation factor, and a parent directory content summary factor, specifically as follows: Calculate the product of the directory generation factor and the parent directory content summary factor, and then take the remainder based on the maximum number of files to determine the number of generated files.

[0030] Multiply the product of the directory generation factor and the parent directory content summary factor by the root directory generation factor, and based on the result of the multiplication, take the remainder according to the maximum number of files to generate the number of directories.

[0031] Generate a first string consisting of the maximum filename length multiplied by the number of files. Divide the first string into equal parts according to the number of files, and use each part as the file extension (the last 3 characters of each filename string) to generate a list of filenames.

[0032] Generate a second string consisting of the maximum filename length multiplied by the number of directories. Divide the second string equally according to the number of files, and use it as the name of each directory to generate a directory list.

[0033] The string constructed from the filename is subjected to string factorization to generate the first factor result. The first factor result is then moduloed by the maximum filename length to obtain the file size.

[0034] The string constructed from the list of filenames and the list of directories is subjected to string factorization to generate a second factor result. The second factor result is then used to obtain the content summary factor of this directory by taking the remainder of the maximum number of directories plus the maximum number of files; where the maximum number of directories is the maximum number of subdirectories contained in the specified directory.

[0035] Step S3: Open any file in the current directory list, determine whether the file exists by checking the file name list, open the file successfully if it exists, and read the opened file. Generate test data of a specified length based on the file content generation factor, the data offset read, and the number of bytes requested.

[0036] A periodic, fixed-data generation method is used to generate repeatedly readable virtual file content, which can be in the form of binary or string data. Repeated reading of the same file will generate the same content.

[0037] Get the variable factors of the specified generated file content, and obtain the data offset information by reading the current file's Offset.

[0038] Obtain the product of the data offset information and the variable factor, and obtain the result of shifting the data offset information bit left by 2 bits. Perform a bitwise AND operation on the sum of the product and the result of shifting left by 2 bits according to the value range of the single byte data content to generate the content of the specified length and obtain the requested number of bytes.

[0039] Specifically, this involves using a content generation factor, the read data offset, and the requested number of bytes as input data to generate test data containing binary or string data. The invention includes the following key input information: [Content Generation Factor], a variable factor specifying the generated file content; this factor determines the periodicity of the generated data, thus affecting whether the file content is identical; [Read Index], the current file offset, which affects the value of the currently generated byte data; [Content Length], affecting the length of readable content, i.e., the file size; and [Data Range], specifying the range of values ​​for a single byte of data.

[0040] Unit byte generation algorithm: *Unit Byte = [Content generation factor] multiplied by [Read index], the result of the operation is left-shifted by 2 bits by [Read index], the result is added to [Read index], and then bitwise ANDed with [Data range].

[0041] By inputting different [read indexes] using the above algorithm, content of a specified length can be generated.

[0042] It also includes the following steps: String factor conversion algorithm: used to convert a specified string into a fixed factor value.

[0043] Algorithm description: The factor is obtained by converting each character in the specified string of the specified length into an integer value and then multiplying them in sequence to obtain a fixed factor value.

[0044] This invention virtually generates test data based on filename and directory list information, and also includes the following steps: FUSE memory file system API operation module: The init method generates a list of root directories by specifying the number of directories and filename rules when called by the memory file system.

[0045] The getattr method retrieves file or directory attributes based on virtual file attributes in the current directory.

[0046] The `readdir` method generates a list of the current directory based on the obtained file or directory attributes when reading directory contents. During directory reading, a read request is sent, and a list is generated in real-time using the directory and file generation algorithm module. Specifically, the directory name string is converted using string factoring to obtain a directory generation factor. The root directory generation factor, parent directory content summary factor, and directory generation factor are used to generate the number of files and directories (including the maximum number of directories and files). The number of files, directories, maximum filename length, and maximum directory name length are used to obtain a directory list and a filename list. Finally, the directory list and filename list are used to generate the current directory content summary factor and a list, which are then provided for reading.

[0047] The `open` method: When opening a file, you pass in the file path and the opening mode. It checks if the file exists in the current list of filenames. If it exists, the file is opened successfully; otherwise, it fails.

[0048] The `read` method: When reading file content, it generates the file content based on the input file path, the read offset, and the requested number of bytes. The read request is processed by a generation algorithm to generate the file content in real time based on the requested file information. Specifically: the filename string is converted to a string factor to obtain a content generation factor; the content generation factor, file size information, read index, content length, and data range are input into the generation algorithm; and the generated data is sent to the file to be read.

[0049] The `release` method: Releases the opened virtual file resources.

[0050] The `write` method: This memory file system is a read-only memory file system. This API does not support it.

[0051] The `mkdir` method: This memory file system is a read-only memory file system. This API does not support it.

[0052] The `unlink` method: This memory file system is a read-only memory file system. This API does not support it.

[0053] The `rename` method: This memory file system is a read-only memory file system. This API does not support it.

[0054] Simulated memory file system registration module: Initializes the memory file system based on key input information and registers it with the operating system to bind operation functions with FUSE's API, so that the FUSE core module can call and process memory file system requests.

[0055] Based on the above method, the present invention provides a virtual generation system for file system test data, including: a file system generation module, a directory and file name generation module, and a data generation module.

[0056] The file system generation module is used to generate a memory file system in user space using the user space memory file system FUSE; the memory file system is a read-only file system; the directory and file name generation module is used to read the directory information in the memory file system when the memory file system is mounted, and generate a file name list and a directory list by periodically specifying the number of directories and file name rules; the data generation module is used to open any file in the current directory list, determine whether the file exists by checking the file name list, open the file successfully if it exists, read the opened file, and generate test data of a specified length based on the content generation factor of the file, the data offset read, and the number of bytes requested.

[0057] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a program, and when the program is executed by the processor, the processor performs the steps of a method for virtually generating file system test data.

[0058] According to the disclosed embodiments, the computer device can communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth communication, etc.) or with any device that enables the computing device to communicate with one or more other computing devices (e.g., router, demodulator, etc.).

[0059] The present invention also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of a method for virtually generating file system test data.

[0060] According to the disclosed embodiments, the storage medium can be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, the storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0061] The above description, in conjunction with specific preferred embodiments, provides a more detailed explanation of the present invention. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for virtually generating test data for a file system, characterized in that, Includes the following steps: A memory file system is generated in user space using the user-space memory file system FUSE; the memory file system is a read-only file system. When the memory file system is mounted, the directory information in the memory file system is read, and a list of filenames and a list of directories are generated by periodically specifying the number of directories and the filename rules. Open any file in the current directory list, check if the file exists by checking the filename list, open the file successfully if it exists, read the opened file, and generate test data of a specified length based on the file content generation factor, the data offset read, and the number of bytes requested.

2. The method for virtually generating file system test data as described in claim 1, characterized in that, The process of generating a list of filenames and a list of directories by periodically specifying a number of directories and filename rules is as follows: Based on periodic filename rules and specified directory number generation rules, obtain the directory generation factor, root directory generation factor, and parent directory content summary factor in the current directory; among them, the directory generation factor is the factor that determines the file name, directory name, and number of generated files in the current directory, the root directory generation factor is the factor that determines the data for generating the root directory, and the parent directory content summary factor is the summary factor of the list of all generated files and directories in the parent directory. Generate a fixed number and fixed-size list of filenames and directories using directory generation factors, root directory generation factors, and parent directory content summary factors.

3. The method for virtually generating file system test data as described in claim 2, characterized in that, The filename rules and the rules for generating the specified number of directories are as follows: When the directory depth exceeds the maximum directory level, no subdirectory information is generated. The root directory parent directory generation factor is set to a fixed value, and rules are set for the maximum number of directories, the maximum number of files, the maximum file name length, and the maximum directory name length. The maximum number of directories is the maximum number of subdirectories contained in the specified directory, the maximum number of files is the maximum number of files contained in the specified directory, the maximum filename length specifies the maximum length of filenames, and the maximum directory name length specifies the maximum length of directory names.

4. The method for virtually generating file system test data as described in claim 3, characterized in that, The process of generating a fixed number and fixed-size list of filenames and directories using a directory generation factor, a root directory generation factor, and a parent directory content summary factor specifically involves: Calculate the product of the directory generation factor and the parent directory content summary factor, and then take the remainder based on the maximum number of files to determine the number of files generated. Multiply the product of the directory generation factor and the parent directory content summary factor by the root directory generation factor, and based on the multiplication result, take the remainder according to the maximum number of files to generate the number of directories; Generate a first string consisting of the maximum filename length multiplied by the number of files, and divide the first string equally according to the number of files to use as file extensions, thereby generating a list of filenames; Generate a second string consisting of the maximum filename length multiplied by the number of directories. Divide the second string equally according to the number of files, and use it as the name of each directory to generate a directory list.

5. The method for virtually generating file system test data as described in claim 1, characterized in that, Test data of a specified length is generated based on the content generation factor of the file, the data offset read, and the number of bytes requested, specifically as follows: The test data is generated by taking the content generation factor, the data offset to be read, and the number of bytes to be requested as input data. The content generation factor is used to specify the variable factor for generating the file content. The data offset to be read is obtained by reading the index. The number of bytes to be requested includes the content length and the data range. The content length is used to determine the file size, and the data range is used to specify the value range of a single byte of data.

6. The method for virtually generating file system test data as described in claim 5, characterized in that, The specified range of values ​​for a single byte of data content is as follows: Obtain the product of the data offset information and the variable factor, and obtain the result of shifting the data offset information bit left by 2 bits. Perform a bitwise AND operation on the sum of the product and the result of shifting left by 2 bits according to the value range of a single byte data content to obtain the requested number of bytes.

7. A virtual generation system for file system test data, characterized in that, include: The file system generation module is used to generate a memory file system in user space using the user space memory file system FUSE. The memory file system is a read-only file system; The directory and file name generation module is used to read the directory information in the memory file system when the memory file system is mounted, and generate a file name list and a directory list by periodically specifying the directory number generation rules and file name rules. The data generation module is used to open any file in the current directory list, determine whether the file exists by checking the file name list, open the file successfully if it exists, read the opened file, and generate test data of a specified length based on the content generation factor of the file, the data offset read, and the number of bytes requested.

8. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a program that, when executed by the processor, causes the processor to perform the steps of a method for virtually generating file system test data as described in any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the virtual generation method for file system test data according to any one of claims 1 to 6.

Images