Small file dynamic aggregation method, device and readable medium based on virtual file system

Abstract

The present invention discloses a method, device and readable medium for dynamically aggregating small files based on a virtual file system. In response to determining that a storage carrier exists, the storage carrier is opened, an operation handle of the file is obtained, a magic number is read from a handle table of the operation handle, and in response to determining that the magic number is legal, initialization information of the storage carrier is read and deserialized to obtain an initialization information structure; a physical file system instance is created based on the initialization information structure, a mapping relationship from a first set to a second set is constructed, and elements of the first set and the second set both include reading, writing and updating; a mount point is constructed, and a directory structure stored in the storage carrier is constructed according to the characteristics of the host file system, and file operations of the operating system on the mount point are obtained and intercepted; the element corresponding to the file operation in the second set is searched for and called through the mapping relationship, and the object of the corresponding element operation is redirected to the storage carrier, thereby realizing real-time dynamic reading, writing and aggregation of massive small files.

Description

Technical Field

[0001] The present invention relates to the field of computer storage, and in particular to a method, device and readable medium for dynamically aggregating small files based on a virtual file system. Background Art

[0002] Files under 1MB are generally considered small files. With the rapid development of the internet (especially mobile internet), the Internet of Things, cloud computing, and big data, data is growing explosively. The use of massive amounts of small files is becoming increasingly commonplace in everyday life. Various applications generate small files, such as emojis sent via instant messaging, office documents, and short videos. Because small files contain relatively little data, metadata access performance significantly impacts small file access performance. In disk file systems, directory entries (dentries), index nodes (inodes), and data are stored in different locations on the storage medium. Therefore, accessing a single file requires at least three independent accesses. This translates to a large number of random accesses, which is highly inefficient for disks. Disk file systems organize disk data using blocks and use multi-level pointers or hash trees in inodes to index file data blocks. For large files, data blocks are allocated as contiguously as possible, ensuring good spatial locality. For small files, data blocks may be scattered in different locations on the disk, resulting in a large amount of disk fragmentation (including internal fragmentation and external fragmentation), which not only degrades access performance but also leads to a large amount of disk space waste.

[0003] Small file consolidation is currently the most successful strategy for optimizing the small file problem and has been adopted by multiple distributed storage systems, including Facebook Haystack and Taobao File System (TFS). This strategy allows multiple logical files to share the same physical file, consolidating and storing multiple small files into a single large file for efficient small file storage. This mechanism is well-suited for distributed storage systems using WORM (Write Once Read Many) mode, but is unsuitable for storage systems that allow frequent overwrites and deletions. File overwrites and deletions can cause internal fragmentation in large files. Implementing space management and timely defragmentation is complex and incurs overhead. Failure to address fragmentation and instead employing append-only writes wastes storage capacity and destroys data locality, increasing the randomness of data distribution and degrading read performance. Furthermore, supporting random reads and writes raises practical challenges in handling large and small files uniformly, allowing small files to grow into large files and large files to degenerate into small files. Summary of the Invention

[0004] In response to the above-mentioned technical problems, the embodiment of the present application aims to propose a method, device and readable medium for dynamically aggregating small files based on a virtual file system to solve the technical problems mentioned in the above background technology section.

[0005] In a first aspect, the present invention provides a method for dynamically aggregating small files based on a virtual file system, comprising the following steps:

[0006] In response to determining that a storage carrier exists, opening the storage carrier, obtaining an operation handle of the file, reading a magic number from a handle table of the operation handle, and in response to determining that the magic number is valid, reading initialization information of the storage carrier and deserializing it to obtain an initialization information structure;

[0007] A physical file system instance is created based on the initialization information structure, and a mapping relationship is constructed from a first set to a second set. The first set is a set of access services provided by the virtual file system to the filter driver, and the second set is a set of access services provided by the physical file system to the virtual file system. Elements of both the first set and the second set include read, write, and update.

[0008] Build a mount point, construct a directory structure stored in the storage medium based on the host file system characteristics, obtain the operating system's file operations on the mount point and intercept them;

[0009] The element corresponding to the file operation in the second set is searched through the mapping relationship and called, and the object of the corresponding element operation is redirected to a storage carrier.

[0010] Preferably, obtaining and intercepting file operations on the mount point by the operating system specifically includes:

[0011] Register a filter driver to obtain the file operations of the operating system on the mount point, and intercept the file operations of the operating system on the mount point through the filter driver.

[0012] As an option, it also includes:

[0013] In response to determining that the exit signal is received, the mount point is removed, the filter driver is uninstalled, the cache is flushed, the mapping relationship, the first set, and the second combination are released, and the process ends.

[0014] As an option, it also includes:

[0015] In response to determining that no exit signal is received, repeating the steps of: searching for an element corresponding to the file operation in the second set through a mapping relationship and calling the element, and redirecting the object of the corresponding element operation to a storage carrier.

[0016] As an option, it also includes:

[0017] In response to determining that the storage carrier does not exist, a storage carrier is created, the specified physical file system type and related parameters are serialized according to the initialization information structure to obtain initialization information, and the initialization information is written into the storage carrier.

[0018] As an option, it also includes:

[0019] In response to determining that the magic number is illegal, the process ends.

[0020] Preferably, the initialization information structure includes a magic number Magic Number, the physical file system type FSType used, the size ExtendInfoSize occupied by the extended information, the extended information ExtendInfo and its corresponding size, type and offset. When reading the initialization information, it is read according to the type and offset.

[0021] In a second aspect, the present invention provides a small file dynamic aggregation device based on a virtual file system, comprising the following steps:

[0022] a deserialization module configured to, in response to determining that a storage carrier exists, open the storage carrier, obtain an operation handle of the file, read a magic number from a handle table of the operation handle, and in response to determining that the magic number is legal, read initialization information of the storage carrier and deserialize it to obtain an initialization information structure;

[0023] a mapping relationship establishment module configured to create a physical file system instance based on the initialization information structure and construct a mapping relationship from a first set to a second set, where the first set is a set of access services provided by the virtual file system to the filter driver, and the second set is a set of access services provided by the physical file system to the virtual file system, and elements of both the first set and the second set include read, write, and update;

[0024] The interception module is configured to construct a mount point, construct a directory structure stored in the storage medium according to the characteristics of the host file system, obtain the file operation of the operating system on the mount point and intercept it;

[0025] The calling module is configured to search for the element corresponding to the file operation in the second set through a mapping relationship and call it, and redirect the object of the corresponding element operation to a storage carrier.

[0026] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The method for dynamic aggregation of small files based on a virtual file system proposed in the present invention intercepts the file operations of the operating system on the mount point obtained by the filter driver, and maps the actual creation, reading and writing operations to reading and writing of a specific storage medium through the virtual file system, so as to realize the dynamic aggregation, reading, writing, addition and deletion of a large number of small files stored in a specific folder (mount drive letter).

[0030] (2) The small file dynamic aggregation method based on the virtual file system proposed in the present invention can realize the real-time dynamic reading and writing of massive small files, and aggregate them into a large storage medium. After unloading, they can be quickly migrated to any location, solving the problem of slow migration of massive small files. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 is a diagram of an exemplary device architecture to which an embodiment of the present application may be applied;

[0033] Figure 2 This is a flow chart of a method for dynamically aggregating small files based on a virtual file system according to an embodiment of the present application;

[0034] Figure 3 This is a flowchart of a method for dynamically aggregating small files based on a virtual file system according to an embodiment of the present application;

[0035] Figure 4 This is a result diagram before processing a large number of small files using the small file dynamic aggregation method based on the virtual file system according to an embodiment of the present application;

[0036] Figure 5 This is a diagram showing the effect of processing a large number of small files using the small file dynamic aggregation method based on the virtual file system according to an embodiment of the present application;

[0037] Figure 6Schematic diagram of a small file dynamic aggregation device based on a virtual file system according to an embodiment of the present application;

[0038] Figure 7 It is a structural diagram of a computer device suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0040] Figure 1 An exemplary device architecture 100 is shown to which a method for dynamically aggregating small files based on a virtual file system or a device for dynamically aggregating small files based on a virtual file system according to an embodiment of the present application can be applied.

[0041] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0042] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, 103.

[0043] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitations are given here.

[0044] The server 105 may be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server may process the acquired files or data and generate processing results.

[0045] It should be noted that the small file dynamic aggregation method based on the virtual file system provided in the embodiment of the present application can be executed by the server 105, or by the terminal devices 101, 102, and 103. Accordingly, the small file dynamic aggregation device based on the virtual file system can be set in the server 105, or in the terminal devices 101, 102, and 103.

[0046] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. If the processed data does not need to be acquired remotely, the above-described apparatus architecture may not include a network, but only require servers or terminal devices.

[0047] Figure 2 A method for dynamically aggregating small files based on a virtual file system provided by an embodiment of the present application is shown, comprising the following steps:

[0048] S1, in response to determining that a storage carrier exists, opening the storage carrier, obtaining an operation handle of the file, reading a magic number from a handle table of the operation handle, in response to determining that the magic number is legal, reading initialization information of the storage carrier and deserializing it to obtain an initialization information structure.

[0049] In a specific embodiment, it also includes:

[0050] In response to determining that the storage carrier does not exist, a storage carrier is created, the specified physical file system type and related parameters are serialized according to the initialization information structure to obtain initialization information, and the initialization information is written into the storage carrier.

[0051] Specifically, the process of the embodiment of the present application is as follows Figure 3 As shown, it can be used for any file, including large files and small files. The transmission of large files can fully utilize the disk performance without using the process of the embodiment of the present application. The dynamic aggregation effect for small files is more obvious. Generally, files with a size of less than 1MB are considered small files. First, determine whether the storage carrier exists. If it does not exist, create it. When creating, it is necessary to determine whether the remaining parameters are legal. If the storage carrier is specified as a file, sparse file, etc. that can be created, then create it, otherwise end the process. During the creation process, the physical file system type and related parameters specified by the parameters, such as sector size, total number of sectors, cluster size and encryption parameters, are serialized according to the initialization information structure and written to the storage carrier. After the storage carrier is created, that is, the storage carrier exists, the storage carrier can be opened to obtain the operation handle of the small file.

[0052] In a specific embodiment, it also includes:

[0053] In response to determining that the magic number is illegal, the process ends.

[0054] Specifically, read the magic number MagicNumber of size 8 bytes at offset 0 of the operation handle (ie, the handle table), and determine whether the magic number MagicNumber is legal. If it is legal, proceed to step S2, otherwise terminate the process.

[0055] S2, create a physical file system instance based on the initialization information structure, and build a mapping relationship from the first set to the second set. The first set is the access service set provided by the virtual file system to the filter driver, and the second set is the access service set provided by the physical file system to the virtual file system. The elements of the first set and the second set both include read, write, and update.

[0056] In a specific embodiment, the initialization information structure includes a magic number Magic Number, the physical file system type FSType used, the size ExtendInfoSize occupied by the extended information, the extended information ExtendInfo and its corresponding size, type and offset. When reading the initialization information, it is read according to the type and offset.

[0057] Specifically, the initialization information structure is shown in Table 1:

[0058] Table 1 Initialization information structure of storage carrier

[0059] name size type offset Magic Number 8 Raw data 0 FSType 4 unsigned int 8 ExtendInfoSize 4 unsigned int 12 ExtendInfo TLV Structure 16

[0060] Among them, Magic Number is a specific identifier that can be used to distinguish files. Its type is raw data, which means it is unprocessed or simplified data. FSType is used to identify the type of physical file system used. ExtendInfoSize indicates the size (in bytes) occupied by the extended information, and ExtendInfo indicates the extended information. When reading the initialization information, it is necessary to read according to the type and offset described here. TLV is the abbreviation of Tag (type), Length (length), Value (value), and is a data encoding format. Among them, Value can be a raw data type or a TLV structure.

[0061] Assume that there exists a first set, a second set, and a mapping relationship between the first set and the second set. The first set is the set of access services provided by the virtual file system VFS to the filter driver, whose elements include but are not limited to read, write, and update. The second set is the set of access services provided by the physical file system to the virtual file system VFS, whose elements include but are not limited to read, write, and update. The mapping relationship is to map one element in the first set to a specific element in the second set. The operation refers to an element in the first set.

[0062] The file system, primarily used in an operating system, organizes and allocates space on storage devices and is responsible for file storage, protection, and retrieval. Specifically, it is responsible for creating files for users, storing, reading, modifying, and retrieving files, controlling file access, and revoking files when they are no longer in use. File systems include logical file systems and physical file systems. The logical file system supports system calls, while a physical file system instance manages the permanent storage of data and, after creation, can dump data to a storage medium. A storage medium can be a regular file on the host file system, a sparse file, a sparse file virtualized by another driver, or other media with storage capabilities and a read / write interface. The virtual file system (VFS) is an interface layer between the physical and logical file systems. It abstracts the implementation details of the file system, making different file systems appear identical to the operating system and other processes running in the system, allowing the logical file system to be connected to multiple different physical file systems. The host file system is the external file system environment in which this method runs, such as the Windows NTFS file system.

[0063] S3, constructing a mount point, constructing a directory structure stored in the storage medium according to the host file system characteristics, obtaining the file operation of the operating system on the mount point and intercepting it.

[0064] In a specific embodiment, obtaining and intercepting the file operation of the operating system on the mount point specifically includes:

[0065] Register a filter driver to intercept the operating system's file operations on the mount point through the filter driver.

[0066] Specifically, the created mount point is empty by default, and a directory structure needs to be constructed in order to represent normal stored data. The file operations of the operating system on the mount point are intercepted. This file covers both large and small files without making any special distinction. All file (folder) operations in the operating system can be captured by the driver. The driver is a software component that allows the operating system and the device to communicate with each other. After the driver obtains data from the device, it returns the data to the operating system, and the operating system returns the data to the application. All file (folder) operations within a specific folder (mount drive letter) are filtered through the filter driver, and the actual creation, read, and write operations are mapped to read and write operations on a specific storage medium through the VFS. The filter driver can filter the I / O operations of one or more file systems or file system volumes. This method can achieve dynamic aggregation, reading, writing, addition, and deletion of a large number of small files stored in a specific folder (mount drive letter).

[0067] S4, searching for the element corresponding to the operation in the second set through the mapping relationship and calling it, redirecting the object of the corresponding element operation to a storage carrier.

[0068] Specifically, since the mapping relationship identifies the elements (operations) in the first set with the elements (operations) in the second set, the mapping relationship can be used to find the corresponding elements (operations) in the second set for the file operation, call the corresponding elements (operations) to perform specific file storage behavior, and redirect the object of the element (operation) to a storage medium. The storage medium can be migrated to any location after being unmounted.

[0069] In a specific embodiment, it also includes:

[0070] In response to determining that the exit signal is received, the mount point is removed, the filter driver is uninstalled, the cache is flushed, the mapping relationship, the first set, and the second combination are released, and the process ends.

[0071] In a specific embodiment, it also includes:

[0072] In response to determining that the exit signal has not been received, step S4 is repeated.

[0073] Specifically, determine whether an exit signal is received. If an exit signal is received, remove the mount point, uninstall the filter driver, flush the cache, release the mapping relationship, the first set and the second set, and end the process. Otherwise, repeat step S4.

[0074] The above steps S1-S4 do not represent the order of the steps, but are only step symbols.

[0075] refer to Figure 4Before the small file dynamic aggregation method based on the virtual file system proposed in the embodiment of the present application is adopted, there are 1938505 files and 103497 folders in the folder. After the small file dynamic aggregation method based on the virtual file system proposed in the embodiment of the present application is adopted, the effect diagram is as follows: Figure 5 As shown, it can not only realize the real-time dynamic reading and writing of massive small files, but also aggregate them into a large storage medium. After writing, they can be quickly migrated to any location, solving the problem of slow migration of massive small files.

[0076] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a small file dynamic aggregation device based on a virtual file system. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0077] The embodiment of the present application provides a device for dynamically aggregating small files based on a virtual file system, comprising the following steps:

[0078] Deserialization module 1 is configured to, in response to determining that a storage carrier exists, open the storage carrier, obtain an operation handle of the file, read a magic number from a handle table of the operation handle, and in response to determining that the magic number is legal, read initialization information of the storage carrier and deserialize it to obtain an initialization information structure;

[0079] a mapping relationship establishment module 2 configured to create a physical file system instance based on the initialization information structure and construct a mapping relationship from a first set to a second set, where the first set is a set of access services provided by the virtual file system to the filter driver, and the second set is a set of access services provided by the physical file system to the virtual file system, and elements of both the first set and the second set include read, write, and update;

[0080] Interception module 3 is configured to construct a mount point, construct a directory structure stored in the storage medium according to the characteristics of the host file system, obtain the file operations of the operating system on the mount point and intercept them;

[0081] The calling module 4 is configured to search for the element corresponding to the operation in the second set through the mapping relationship and perform the call, and redirect the object of the corresponding element operation to a storage carrier.

[0082] Reference below Figure 7 , which shows an electronic device (eg Figure 1 A structural diagram of a computer device 700 (server or terminal device shown). Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0083] like Figure 7 As shown, the computer device 700 includes a central processing unit (CPU) 701 and a graphics processing unit (GPU) 702, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 703 or the program loaded from the storage part 709 to the random access memory (RAM) 704. Various programs and data required for the operation of the device 700 are also stored in the RAM 704. The CPU 701, GPU 702, ROM 703 and RAM 704 are connected to each other via a bus 705. An input / output (I / O) interface 706 is also connected to the bus 705.

[0084] The following components are connected to the I / O interface 706: an input section 707 including a keyboard, a mouse, and the like; an output section 708 including a display such as a liquid crystal display (LCD), a speaker, and the like; a storage section 709 including a hard disk and the like; and a communication section 710 including a network interface card such as a LAN card or a modem. The communication section 710 performs communication processing via a network such as the Internet. A drive 711 may also be connected to the I / O interface 706 as needed. A removable medium 712, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 711 as needed, so that a computer program read therefrom can be installed into the storage section 709 as needed.

[0085] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication part 710, and / or installed from a removable medium 712. When the computer program is executed by the central processing unit (CPU) 701 and the graphics processing unit (GPU) 702, the above-mentioned functions defined in the method of the present application are executed.

[0086] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable medium, or any combination thereof. Computer-readable media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, apparatuses, or components, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or component. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution apparatus, device, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0087] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0089] The modules involved in the embodiments described in this application may be implemented in software or hardware, and may also be set in a processor.

[0090] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: in response to determining that a storage carrier exists, opens the storage carrier, obtains an operation handle of the file, reads a magic number in the handle table of the operation handle, and in response to determining that the magic number is legal, reads the initialization information of the storage carrier and deserializes it to obtain an initialization information structure; creates a physical file system instance based on the initialization information structure, constructs a mapping relationship from a first set to a second set, the first set is an access service set provided by the virtual file system to the filter driver, and the second set is an access service set provided by the physical file system to the virtual file system, and the elements of the first set and the second set both include read, write, and update; constructs a mount point, constructs a directory structure stored in the storage carrier according to the characteristics of the host file system, obtains the file operation of the operating system on the mount point and intercepts it; searches for the element corresponding to the file operation in the second set through the mapping relationship and calls it, and redirects the object of the corresponding element operation to the storage carrier.

[0091] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for dynamically aggregating small files based on a virtual file system, characterized in that: The following steps are involved: In response to determining that a storage carrier exists, opening the storage carrier, obtaining an operation handle of the file, reading a magic number from a handle table of the operation handle, and in response to determining that the magic number is valid, reading initialization information of the storage carrier and deserializing the information to obtain an initialization information structure; Creating a physical file system instance based on the initialization information structure, and constructing a mapping relationship from a first set to a second set, where the first set is a set of access services provided by the virtual file system to the filter driver, and the second set is a set of access services provided by the physical file system to the virtual file system, wherein elements of both the first set and the second set include read, write, and update; Constructing a mount point, constructing a directory structure stored in the storage medium according to the characteristics of the host file system, obtaining and intercepting file operations on the mount point by the operating system; The element corresponding to the file operation in the second set is searched through the mapping relationship and called, and the object of the corresponding element operation is redirected to the storage carrier.

2. The method for dynamically aggregating small files based on a virtual file system according to claim 1, characterized in that: The obtaining and intercepting the file operation of the operating system on the mount point specifically includes: Register a filter driver, obtain the file operation of the operating system on the mount point, and intercept the file operation of the operating system on the mount point through the filter driver.

3. The method for dynamically aggregating small files based on a virtual file system according to claim 2, characterized in that: Also includes: In response to determining that the exit signal is received, the mount point is removed, the filter driver is uninstalled, the cache is flushed, the mapping relationship, the first set and the second combination are released, and the process ends.

4. The method for dynamically aggregating small files based on a virtual file system according to claim 1, characterized in that: Also includes: In response to determining that no exit signal is received, repeating the steps: searching the mapping relationship for an element corresponding to the file operation in the second set and calling it, and redirecting the object of the corresponding element operation to the storage carrier.

5. The method for dynamically aggregating small files based on a virtual file system according to claim 1, characterized in that: Also includes: In response to determining that the storage carrier does not exist, a storage carrier is created, the specified physical file system type and related parameters are serialized according to the initialization information structure to obtain initialization information, and the initialization information is written into the storage carrier.

6. The method for dynamically aggregating small files based on a virtual file system according to claim 1, characterized in that: Also includes: In response to determining that the magic number is illegal, the process ends.

7. The method for dynamically aggregating small files based on a virtual file system according to claim 1, characterized in that: The initialization information structure includes a magic number Magic Number, the physical file system type FSType used, the size ExtendInfoSize occupied by the extended information, the extended information ExtendInfo and their corresponding sizes, types and offsets. When reading the initialization information, it is read according to the type and offset.

8. A small file dynamic aggregation device based on a virtual file system, characterized in that: The following steps are involved: a deserialization module configured to, in response to determining that a storage carrier exists, open the storage carrier, obtain an operation handle for the file, read a magic number from a handle table of the operation handle, and, in response to determining that the magic number is valid, read initialization information of the storage carrier and deserialize the information to obtain an initialization information structure; a mapping relationship establishment module configured to create a physical file system instance based on the initialization information structure and construct a mapping relationship from a first set to a second set, wherein the first set is a set of access services provided by the virtual file system to the filter driver, and the second set is a set of access services provided by the physical file system to the virtual file system, wherein elements of the first set and the second set both include read, write, and update; an interception module configured to construct a mount point, construct a directory structure stored in the storage medium according to characteristics of the host file system, obtain file operations on the mount point by the operating system, and intercept the operations; The calling module is configured to search for the element corresponding to the file operation in the second set through the mapping relationship and call it, and redirect the object of the corresponding element operation to the storage carrier.

9. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images