Data processing method, computer device and storage medium based on stream directory

Abstract

The present application is applicable to the field of computer technology, and provides a data processing method based on a stream directory, a computer device, and a storage medium, including: obtaining multiple source files in a source file system and a file access order for the multiple source files; running a first process, reading the first metadata of each source file according to the file access order, and generating the second metadata of an initial stream directory based on each first metadata, wherein the initial stream directory belongs to a subdirectory of a specified parent directory of a target file system; running a second process, reading the file content of each source file according to the file access order, and filling at least one data block associated with the initial stream directory based on the content of each file, wherein the first process and the second process are executed in parallel; updating the initial stream directory based on the second metadata and at least one data block to obtain a target stream directory. The present application can reduce I / O operations when frequently accessing source files and improve data processing efficiency.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a data processing method, a computer device, and a storage medium based on a stream directory. Background Art

[0002] With the explosive growth of data generated by artificial intelligence, application scenarios for operations on massive small files (relatively small file sizes, such as 1KB, 4KB, etc.) usually require processing millions of small files. For example, image processing and computer vision applications usually require processing a large number of image files, and the size of a single image file is relatively small. For genomics data analysis scenarios, the amount of data generated in genomics research is huge, including sequencing data, expression data, etc. These data are usually stored in the form of small files.

[0003] In the above application scenarios, the presence of a large number of small files will increase the metadata overhead of the source file system (the file system that stores small files), because each file needs to be allocated an index node inode. The processing of a large number of small files may cause the input / output (I / O) (such as disk I / O or memory I / O) bottleneck of the file system, resulting in poor system performance. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a data processing method, a computer device, and a storage medium based on a stream directory, which can reduce I / O operations when frequently accessing source files and improve data processing efficiency.

[0005] A first aspect of an embodiment of the present application provides a data processing method based on a stream directory, including:

[0006] Acquire multiple source files in a source file system and a file access order for the multiple source files;

[0007] Running a first process, reading first metadata of each of the source files according to the file access order, and generating second metadata of an initial stream directory based on each of the first metadata, wherein the initial stream directory belongs to a subdirectory of a specified parent directory of the target file system;

[0008] Running a second process, reading the file content of each of the source files according to the file access order, and filling at least one data block associated with the initial stream directory based on the content of each of the files, wherein the first process and the second process are executed in parallel;

[0009] Based on the second metadata and the at least one data block, the initial stream directory is updated to obtain a target stream directory.

[0010] In an implementation of the first aspect, generating second metadata of the initial stream directory based on each of the first metadata includes:

[0011] Obtaining a slice structure for metadata in the target file system;

[0012] For each attribute in the slice structure, determining a target value of the attribute based on an initial value adapted to the attribute in each of the first metadata;

[0013] A target slice structure including each of the target values ​​is determined as second metadata of the initial stream directory.

[0014] In an implementation of the first aspect, filling at least one data block associated with the initial stream directory based on the content of each file includes:

[0015] Determine the total data volume of the contents of the plurality of files;

[0016] Determining the required number of data blocks based on the block capacity of the data blocks of the target file system and the total data volume;

[0017] Allocating the required number of target data blocks to the initial stream directory;

[0018] According to the file access sequence, the contents of each file are sequentially filled into each target data block.

[0019] In an implementation manner of the first aspect, updating the initial stream directory based on the second metadata and the at least one data block to obtain a target stream directory includes:

[0020] Obtaining the index node of the initial stream directory, where the index node and the stream directory have a one-to-one correspondence;

[0021] updating a metadata attribute of the index node based on the second metadata;

[0022] Based on the at least one data block, updating a data block attribute of the index node;

[0023] The initial stream directory after the index node is updated is used as the target stream directory.

[0024] In an implementation of the first aspect, the method further includes:

[0025] If an access request for the target stream directory is received, an access mode for the target stream directory is obtained, wherein the access mode is a first access mode based on a portable operating system interface or a second access mode based on a memory mapping interface;

[0026] Based on the access mode, the target stream directory is accessed to obtain an access result adapted to the access request.

[0027] In an implementation of the first aspect, accessing the target stream directory based on the access mode to obtain an access result adapted to the access request includes:

[0028] If the access mode is the second access mode based on the memory mapping interface, directly mapping the target stream directory to the virtual address space of the file access process;

[0029] Access the file content of the target stream directory in the virtual address space to obtain an access result adapted to the access request.

[0030] In an implementation of the first aspect, accessing the target stream directory based on the access mode to obtain an access result adapted to the access request includes:

[0031] If the access mode is a first access mode based on a portable operating system interface, selecting a target access interface adapted to the access request from at least one access interface of the target file system for a stream directory, the access interface being created based on the portable operating system interface;

[0032] Parsing the access request to obtain interface parameters adapted to the target access interface;

[0033] Based on the interface parameters, the target access interface is called to access the target stream directory to obtain an access result that matches the access request.

[0034] In an implementation of the first aspect, the storage unit of the source file system is a data block of a first capacity, and the storage unit of the target file system is a data block of a second capacity, where the second capacity is greater than the first capacity.

[0035] A second aspect of an embodiment of the present application provides a data processing device based on a stream directory, including:

[0036] An acquisition module, used for acquiring a plurality of source files in a source file system and a file access sequence for the plurality of source files;

[0037] A first reading module is used to run a first process, read the first metadata of each of the source files according to the file access order, and generate the second metadata of an initial stream directory based on each of the first metadata, wherein the initial stream directory belongs to a subdirectory of a specified parent directory of the target file system;

[0038] A second reading module is used to run a second process, read the file content of each of the source files according to the file access order, and fill at least one data block associated with the initial stream directory based on the content of each of the files, wherein the first process and the second process are executed in parallel;

[0039] An updating module is used to update the initial stream directory based on the second metadata and the at least one data block to obtain a target stream directory.

[0040] A third aspect of an embodiment of the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0041] A fourth aspect of an embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0042] In a first aspect of an embodiment of the present application, source files in a source file system and an access order for the source files are obtained, second metadata of an initial stream directory is generated through a first process according to the access order, and data blocks associated with the initial stream directory are filled synchronously through a second process running in parallel, and finally the initial stream directory is efficiently updated to generate a target stream directory, and independent processing processes for multiple source files are converted into an overall processing process for a target stream directory, thereby reducing the number of I / O operations of the target file system and improving data processing efficiency.

[0043] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a schematic diagram of the system structure of the target file system provided in an embodiment of the present application;

[0046] Figure 2 is a schematic diagram of a slice structure of an index node provided in an embodiment of the present application;

[0047] Figure 3 It is a flow chart of a data processing method based on a stream directory provided in an embodiment of the present application;

[0048] Figure 4 is a flowchart of a method for generating second metadata provided by an embodiment of the present application;

[0049] Figure 5 It is a flowchart of a method for filling a target data block provided in an embodiment of the present application;

[0050] Figure 6 It is a flowchart of a method for updating an index node of a stream directory provided in an embodiment of the present application;

[0051] Figure 7 It is a flowchart of a method for accessing a stream directory provided in an embodiment of the present application;

[0052] Figure 8 is a flowchart of an access method based on a second access mode provided in an embodiment of the present application;

[0053] Fig. 9 is a flowchart of an access method based on a first access mode provided in an embodiment of the present application;

[0054] Fig.10 is an example diagram of an application scenario of a target file system provided in an embodiment of the present application;

[0055] Fig.11 is a schematic diagram of a data processing device based on a stream directory provided in an embodiment of the present application;

[0056] Fig.12 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0058] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0059] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0060] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0061] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0062] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0063] like Figure 1 As shown, it is a structural diagram of the target file system provided by an embodiment of the present application. In the application, the disk space can be divided into multiple partitions, each partition can be formatted as a target file system, and the boot block is the first block of the disk partition, which records some information of the target file system partition. The program and data for booting and loading the current partition are stored in this block. The target file system can be divided into block groups. Each target file system contains multiple block groups, and each block group contains multiple metadata areas and data areas. The metadata area is the area for storing block bitmaps (block map), inode tables (inode table), inode bitmaps (inode map), etc.; the data area is the area for storing file data, and some block groups may also include super blocks (Super block) and block group descriptors.

[0064] In applications, the super block is used to record the overall information of the target file system, including the total number, usage, remaining amount, size of the index node inode and data block, as well as the format and related information of the target file system.

[0065] In the application, the group descriptor describes the block numbers where each block group starts and ends, and describes the block numbers between which each block group (inodemap, blockmap, inode table) is located.

[0066] In the application, a block can only be used by one file, and the block bitmap records all used and unused block numbers. When adding a new file, query the block bitmap to determine the unused blocks to record the file data. When deleting a file, first find the corresponding block number from the block bitmap, then update the flag to unused, and finally release the block.

[0067] In applications, the inode map has the same design concept as the block bitmap and is used to record the used and unused index node (inode) numbers.

[0068] In the application, the inode table is used to record the collection of all inodes in the block group. The location of the inode can be found by using the inode number and the starting position of the inode table as an offset. The data block, as described in the previous section, is where the file or directory data is stored.

[0069] Regarding the inode, each stream directory (stream directory is also a file) in the target file system has only one inode. The inode stores the inode number, file metadata (such as file type, permissions, file owner, size, timestamp, link count, block usage, etc.), and the addressing pointer for the data block (that is, the pointer to the block occupied by the file content).

[0070] In applications, such as Figure 2As shown in the figure, the data structure of the index node inode is a public structure called a slice (islice). A slice is a key / value pair of metadata, a possible data pointer, a hash chain, an iterator chain, and the position of the entity relative to the beginning of the metadata. All pointers are zero-relative, not absolute. In the figure, the hash header is the starting part of the metadata chain, which contains a pointer to the next islice. An islice is a fragment in the metadata chain, which contains some important information, such as key (inode number), value (statistical data, real-time data), data block number, index node block number, inext (pointer to the next slice islice), etc.

[0071] Among them, the key is the index node inode number, which is a number that uniquely identifies a file in the target file system and corresponds to the file's inode number. The value is statistical data and real-time data, which stores some statistical information about the file, such as file size, creation time, modification time, etc. The data block information includes the data block number and data offset, which indicates the physical location of the file data and the offset of the data block relative to the beginning of the file. The index node information includes the index node block number and index node offset, indicating the location and offset of the index node. Extended attributes contain some additional attributes or information, such as file permissions, ownership, etc. The data block is where the file content is actually stored. The stream directory root node is a specific node in the target file system, which is used to manage the streaming data or other related data of the file.

[0072] In the application, the smallest storage unit of the target file system is a fixed-size block, which is much larger than the block size of existing file systems such as Ext. Everything in the target file system is a file, and the stream directory in the target file system is also a file. Each block can only store the data of one stream directory. The stream directory can be associated with multiple blocks, and the blocks of a file in the file system are not necessarily continuous.

[0073] In applications, different computer architectures may have different endianness and alignment requirements. For example, some architectures use little-endian mode, while others use big-endian mode. In order to ensure data consistency and cross-platform compatibility, the target file system uses the uint8_t data type, which is a fixed-size data type that represents an exact one-byte (8-bit) storage space. That is, for any file system using the data type uint8_t, its size is always one byte, and the bytes are independent, not affected by the endianness, and no additional padding is required to ensure alignment. Using uint8_t can avoid alignment differences between different architectures and ensure that data can be correctly parsed on any platform. Data types such as uint32_t and uint64_t often require additional padding in some architectures to meet the alignment requirements in disk space and memory space. Therefore, when uint32_t or uint64_t is part of a data structure, the size of the entire structure may be larger than the storage space required for the actual data. Therefore, in the target file system, when data is written to the actual data block of the disk, the data of various different fields are usually converted into byte sequences (i.e., uint8_t type sequences). The uint32_t type data in the memory can be converted into uint8_t[4] on the disk, the uint16_t type data in the memory can be converted into uint8_t[2] on the disk, and the char type data in the memory can be converted into uint8_t on the disk.

[0074] In the application, the basic data structure (byte sequence of type unit8) of the stream directory based on the target file system is published in the form of a binary description header, indicating that the definition of the data structure is published in binary form, including description information on how to parse and operate these data. The binary description header provides metadata of the data structure, making it easier for applications to understand and operate these data. By providing a binary description header, applications can create different data structure variants as needed without having to design the entire data structure from scratch, so applications can easily create custom variants. The data structure of the stream directory is not only applicable to general file systems, but can also be used to process compound files, that is, files containing multiple different types of data, such as the BAM (Binary Alignment Map) format of genomic data, because it has unique streaming, management, and random search properties. The target file system provides an application programming interface (API) for creating a library. The various interfaces provided in the specific creation library API are shown in Table 1:

[0075]

[0076] Table 1

[0077] Among them, the inode number included in the sdirNew method for creating a blank stream directory has a unique standard style. The stream directory root inode is an ID requested by the library, plus the standard file system in user space (FUSE) root inode number 1. On a distributed file system (DFS) that supports stream directories, a large number of stream directories can be created with a given ID and stored as independent file collections. Each stream directory is assigned a unique path in the DFS and mounted in the DFS that supports stream directories. The creation API returns an abstract sdir_t structure, which contains the location information of the root inode. This location information refers to the offset relative to the start of the metadata or actual data, that is, it points to the specific location of the slice where the data element is located.

[0078] In the application, in order to facilitate the application's access to the stream directory, the target file system will also provide a stream directory library API. The stream directory library is read-only, and the stream index is a single file that can be stored on any medium, which can be a file or a storage device. This library is the smallest library and only performs sequential streaming processing. There will also be a more traditional random access library. Using this library as a small file storage repository will produce theoretical performance close to that of media streaming, even if the average file size is 10KB. The various interfaces provided in the specific stream directory library API are shown in Table 2:

[0079]

[0080] Table 2

[0081] In the application, the target file system also provides a random access API for the stream directory. The random API provides access to extended attributes and support for Portable Operating System Interface (POSIX) compatible file systems. The provided distributed file system includes support for stream directories. Specifically, the random API includes the access interfaces shown in Table 3:

[0082]

[0083] Table 3

[0084] Based on the above description of the target file system, the data processing method based on the stream directory provided by the target file system provided by the embodiment of the present application is further described. In the application scenarios of small file read and write operations (such as Web computer device log processing scenarios, Internet of Things device data collection scenarios, genomics data analysis scenarios or image processing and recognition, etc.), you can use Figure 1 The target file system reduces various I / O operations of the file system. The storage unit of the source file system for storing small files is a data block of the first capacity (for the sake of distinction, it can be called the first data block), and the storage unit of the target file system is a data block of the second capacity (for the sake of distinction, it can be called the second data block). The second capacity is much larger than the first capacity. In the data preprocessing stage of the specific mass small file reading and writing application, the computer device merges the M small files in the source file system into Figure 1 The target file system shown has N stream directories (file forms provided by the target file system, directories are also files), where M and N are both positive integers, and M is much larger than N. When performing a merge operation on M small files, the computer device first creates an initial stream directory in the target file system, allocates an index node inode to the initial stream directory, and can also pre-allocate one or more empty continuous second data blocks to the initial stream directory through a block allocator. On this basis, the computer device implements the data processing method based on the stream directory provided in this embodiment.

[0085] In one embodiment, Figure 3 As shown, a data processing method based on a stream directory is provided, and the method is applied to a computer device as an example for explanation, including the following steps S101 to S104. Among them:

[0086] Step S101: Acquire multiple source files in a source file system and a file access sequence for the multiple source files.

[0087] In the application, the source file system is another file system different from the target file system, the source file is a relatively small file stored in the source file system, such as a source file of 1KB, 4KB, etc., there are many types of source file systems, such as ext, xfs, ntfs, etc., the storage unit of the source file system is the first data block of the first capacity, the first capacity is generally 1024 bytes (1K), 2048 bytes (2K) or 4096 bytes (4K), etc., and the first capacity of the first data block is set when the source file system is created. In the application, there is a pre-set file access order for multiple source files.

[0088] Step S102, running the first process, reading the first metadata of each of the source files according to the file access order, and generating the second metadata of the initial stream directory based on each of the first metadata, wherein the initial stream directory belongs to a subdirectory of a specified parent directory of the target file system.

[0089] In applications, files in a file system usually contain metadata and file content. Metadata is information about the file, and file content is the actual data stored in the file. File metadata and file content are usually stored separately. Metadata is stored in the file's index node, while file content is stored in the file system's data block.

[0090] For the source file system, the first metadata of the source file and the file content of the source file are stored separately. The computer device can implement read and write processing for the source file by running two independent processes in parallel. The two independent processes include a first process for performing metadata processing of the file and a second process for performing actual content processing of the file.

[0091] For the target file system, the second metadata of the stream directory (also called stream file) and the file content of the stream directory are also stored separately. In the process of writing the stream directory, the computer device obtains the file access order for multiple source files, runs the first process, and generates the second metadata of the initial stream directory in the target file system based on the metadata of each source file. The second metadata is the metadata in the target file system used to describe the information of the stream directory.

[0092] In the application, the metadata block of a single stream directory can be set to host the first metadata of 5,000 to 10,000 source small files, and the optimal size of the metadata is 2 blocks. If the index node count in the stream directory is within 2 blocks, the probability of zero cache miss in random metadata access can be guaranteed to be 100%, that is, there is no cache miss at all during data access.

[0093] Step S103, running the second process, reading the file content of each of the source files according to the file access order, and filling at least one data block associated with the initial stream directory based on the content of each file, the first process and the second process are executed in parallel.

[0094] In the application, the computer device uses the first process to read the first metadata of the source file while running the second process in parallel, and reads the file content of each source file in sequence according to the file access order, and fills the file content of the source file into the plurality of second data blocks associated with the pre-created initial stream directory in sequence. The number of second data blocks is determined based on the capacity of the file content of all source files. When the computing power is sufficient, the first process and the second process can be executed in parallel to improve the data processing efficiency. Among them, the computer device can use the sdirNew interface for creating a blank stream directory shown in the aforementioned Table 1 to pre-create the initial stream directory.

[0095] Step S104: based on the second metadata and the at least one data block, update the initial stream directory to obtain a target stream directory.

[0096] In the application, the index node of the initial stream directory is used to store metadata information of the initial stream directory and block pointers pointing to multiple second data blocks (such as addressing pointers pointing to data blocks, etc.). The computer device updates the metadata information in the index node of the initial stream directory through the second metadata obtained from the first metadata of multiple source files, and updates the data block attributes in the index node of the initial stream directory based on the filled multiple second data blocks. After the initial stream directory is updated, the target stream directory is obtained.

[0097] Exemplarily, the file size of the source file is exactly the data block size of the source file system, which is 4KB, and the size of the stream directory of the target file system is exactly the data block size of the target file system, which is 16MB. Then 16MB is approximately 4096 times of 4KB, that is, 16MB=16777216 Bytes, 4 KB = 4096 Bytes, that is, one stream directory can store the file contents of approximately 4096 source files, that is, no less than 4096 open and close operations are required for the 4096 source files. After generating the stream directory for storing the 4096 source files, only one open and close operation is required for the stream directory.

[0098] In this embodiment, by obtaining the source files in the source file system and the access order for the source files, the second metadata of the initial stream directory is generated by the first process according to the access order, and the data blocks associated with the initial stream directory are filled synchronously by the second process running in parallel, and finally the initial stream directory is efficiently updated to generate the target stream directory, and the independent processing processes for multiple source files are converted into an overall processing process for the target stream directory, thereby reducing the I / O operations when the source files are frequently accessed and improving the data processing efficiency.

[0099] In one embodiment, Figure 4As shown, the implementation process of step S102 includes the following steps S201 to S203. Among them:

[0100] Step S201, obtaining a slice structure for metadata in the target file system.

[0101] In the application, the metadata in the target file system is stored in the index node. The data structure of the index node is Figure 2 As shown in the slice structure, the computer device reads the slice structure associated with the index node storing the metadata of the initial stream directory file, and the slice structure stores the initial value of the metadata of the initial stream directory.

[0102] Step S202: for each attribute in the slice structure, based on the initial value adapted to the attribute in each of the first metadata, determine a target value for the attribute.

[0103] In the application, for each attribute in the slice structure, the computer device reads the initial value associated with the attribute in each source file, and converts the target value of the attribute based on the preset conversion relationship and the initial value of the metadata of each source file. For example, the file size attribute in the slice structure can be the sum of the file sizes of multiple source files.

[0104] Step S203: determine the target slice structure including each of the target values ​​as the second metadata of the initial stream directory.

[0105] In the application, the obtained target slice structure including each target value is used as the second metadata of the initial stream directory.

[0106] In this embodiment, by obtaining the metadata slice structure of the target file system and determining the target value of each attribute based on the initial value in the first metadata, the second metadata of the initial stream directory is finally formed, thereby effectively managing and updating the metadata of the file system and improving the performance and data consistency of the file system.

[0107] In one embodiment, Figure 5 As shown, the implementation process of step S103 includes the following steps S301 to S303. Among them:

[0108] Step S301, determining the total data volume of the contents of the plurality of files.

[0109] In the application, the number of second data blocks allocated by the computer device for the initial stream directory in the target file system can be determined based on the total data volume of the file contents of all source files. The computer device sums the content sizes of the file contents of each source file as the total data volume of the multiple file contents.

[0110] Step S302: Determine the required number of data blocks based on the block capacity of the data blocks of the target file system and the total data volume.

[0111] In the application, the computer device determines the required number of the second data blocks according to the block capacity of the data block (i.e., the second data block) of the target file system (the block capacity of the second data block is the second capacity, such as the block capacity of the second data block is 16MB) and the total data volume of the plurality of source files. For example, the ratio of the total data volume of the source file to the block capacity of the second data block can be directly used as the required number of the second data blocks, or a preset number can be added to the ratio as the required number of the second data blocks.

[0112] Step S303: Allocate the required number of target data blocks to the initial stream directory.

[0113] In the application, the computer device allocates the required number of target data blocks for the initial stream directory in the target file system. The multiple target data blocks are continuous. In the application, data blocks can also be dynamically allocated, that is, allocated while filling.

[0114] Step S304: filling the contents of each file into each target data block in sequence according to the file access order.

[0115] In the application, according to the file access order for the source files, the second process is run to sequentially fill the file contents of each source file into the required number of target data blocks. It should be noted that the second data blocks can be allocated in real time during the filling process, and at least one second data block is pre-allocated when creating the initial stream directory. During the file data filling process, after the allocated second data blocks are used up, multiple continuous new second data blocks can be allocated in real time and filled until the file contents of all source files are filled.

[0116] In this embodiment, by accurately calculating the total data volume of the file content and the number of required data blocks, an appropriate number of data blocks are effectively allocated to the initial stream directory, and data is filled in according to the file access order, thereby improving the space utilization and data access efficiency of the file system.

[0117] In one embodiment, Figure 6 As shown, the implementation process of step S104 includes the following steps S401 to S404. Among them:

[0118] Step S401, obtaining the index node of the initial stream directory, where the index node and the stream directory have a one-to-one correspondence.

[0119] In the application, in the target file system, each stream directory corresponds to an inode, which contains metadata of the stream directory, such as file size, permissions, owner, creation time, etc.

[0120] Step S402: Update the metadata attribute of the index node based on the second metadata.

[0121] In the application, after multiple source files are merged into the target stream directory, the first metadata of the source files will no longer be valid. The second metadata is obtained by merging the first metadata of multiple source files in the order of file access. The first metadata can be merged according to custom merging rules, such as setting the file permissions in the second metadata to the maximum or minimum permissions in all source files, or setting the file size in the second metadata to the sum of the file sizes of each source file (or the sum of the actual contents of the source files).

[0122] Step S403: updating the data block attribute of the index node based on the at least one data block.

[0123] In the application, the index node of the stream directory includes a block pointer (also called an addressing pointer) for pointing to a data block, and the pointer points to the first data block of at least one second data block continuously allocated to the stream directory. The computer device reads the physical address of the first data block of the at least one data block and the physical address of the last data block, and updates the data block attribute in the index node.

[0124] Step S404: The initial stream directory whose index node has been updated is used as the target stream directory.

[0125] In the application, after the index node of the initial stream directory is updated according to the second metadata and at least one filled data block, the target stream directory is obtained.

[0126] In this embodiment, by updating the metadata attributes and data block attributes of the index node of the initial stream directory to obtain the target stream directory, effective management and updating of the initial stream directory is achieved, thereby improving the performance and data consistency of the file system.

[0127] In one embodiment, Figure 7 As shown, the access implementation process for the target stream directory includes the following steps S501 to S502. Among them:

[0128] Step S501: if an access request for the target stream directory is received, an access method for the target stream directory is obtained, wherein the access method is a first access method based on a portable operating system interface or a second access method based on a memory mapping interface.

[0129] In the application, the target stream directory includes the first metadata of multiple source files and the file contents of all source files. In the target file system, the access method for the target stream directory includes at least a first access method and a second access method, and the first access method is determined based on the portable operating system interface. In the application, if the target file system is directly mounted in the operating system, the target stream directory serves as the mount point of the target file system. At this time, the computer device can provide multiple stream directory access interfaces that support the portable operating system interface POSIX (such as the access interface list shown in Table 2 above) according to the target file system to respond to the application's access request for the target stream directory.

[0130] The second access method is based on the memory mapping interface. Memory mapping access can directly map the contents of the file into the virtual address space of the process, so that the file contents can be directly accessed and operated like memory. Memory mapping avoids multiple file I / O operations, reduces the number of CPU cache failures, and improves the speed of data processing. The file contents can be operated directly without explicit read and write calls. Multiple processes can map the same file at the same time, thereby sharing data between processes.

[0131] Step S502: Based on the access mode, access the target stream directory to obtain an access result that matches the access request.

[0132] In the application, the file content in the target stream directory is accessed through the designated access method for the target stream directory, and an access result adapted to the access request is obtained.

[0133] In the application, when accessing the file contents of the stream directory, the object hash can be used to ensure that the data has not been tampered with or damaged during transmission. In the target file system, specific search operations include path name, inode number, and extended attribute name. Each type of metadata has a hash function and a 128K hash header. Thousands of very small files can be efficiently accessed randomly.

[0134] In this embodiment, by providing a flexible and efficient access mechanism, the ability to process stream directory requests can be significantly improved while ensuring the stability and performance of the system.

[0135] In one embodiment, Figure 8 As shown, the implementation process of step S502 includes the following steps S601 to S602. Among them:

[0136] Step S601: If the access mode is the second access mode based on the memory mapping interface, the target stream directory is directly mapped to the virtual address space of the file access process.

[0137] In the application, when the access mode is the second access mode based on the memory mapping interface, the computer device directly maps the target stream directory (including the entire directory structure and file content) to the virtual address space of the file access process.

[0138] Step S602: access the file content of the target stream directory in the virtual address space to obtain an access result that matches the access request.

[0139] In the application, the file access process can directly access the file content of the target stream directory in the virtual address space, and finally obtain the access result adapted to the access request.

[0140] In this embodiment, the second access method based on the memory mapping interface can directly operate the file content in the memory, reducing the number of data copies and improving the speed of data processing.

[0141] In one embodiment, Fig. 9 As shown, the implementation process of step S502 includes the following steps S701 to S703. Among them:

[0142] Step S701, if the access mode is the first access mode based on the portable operating system interface, then select a target access interface adapted to the access request from at least one access interface of the target file system for the stream directory, and the access interface is created based on the portable operating system interface.

[0143] In the application, the target file system may provide multiple access interfaces for the stream directory (as shown in Table 1, Table 2, and Table 3). When the computer device receives an access request for the stream directory of the target file system, it selects a target access interface that is adapted to the access request from at least one directory access interface of the target file system for the stream directory.

[0144] Step S702: parse the access request to obtain interface parameters adapted to the target access interface.

[0145] In the application, the computer device parses the access request and obtains interface parameters that adapt the current access request to the target access interface.

[0146] Step S703: Based on the interface parameters, the target access interface is called to access the target stream directory to obtain an access result that matches the access request.

[0147] In the application, the target access interface is called through the interface parameters to access the target stream directory and obtain the access result adapted to the access request.

[0148] In this embodiment, the first access mode based on POSIX is applicable to general file operations and can ensure cross-platform compatibility and stability.

[0149] In one embodiment, for the processing scenario of a large number of small files, through the data processing method based on the stream directory provided in this embodiment, in the data preprocessing stage, the massive scattered small files in the source file system are merged into the stream directory storage in the target file system through the stream directory, thereby reducing the number of file closing times. In the data access stage, if the access mode is memory mapping access, the entire file (including its subdirectories) is mapped to the virtual memory space of the access process, further directly avoiding multiple file I / O operations.

[0150] In one embodiment, the above processing method for merging multiple source files to obtain a stream directory can be applied to erasure coding (EC) storage, such as Fig.10 The architecture example of erasure code storage shown in Figure 1 uses the target file system management in the erasure code block device. The size of a stream directory block is 16MB and is distributed on 8 nodes. Each node has 8 executors. The 8 executors in a node read in parallel in the storage controller, and each node also reads in parallel, so there are 64 executors reading in parallel.

[0151] The read size on each executor is 256K bytes, providing a throughput of over 200 Megabits per second (MBS), and possibly even up to 250MBS. A 24G SAS (Serial Attached SCSI) controller can be used, which can transfer 24 gigabits per second (24 Gbps), resulting in a streaming rate of 8*24G, or 192G for the entire stripe. This far exceeds the possible acceptance rate of the client's Network Interface Controller (NIC), which may be as low as 10G. Read failures only occur if the disk access node crashes or the network controller is damaged. If so, there are two redundant erasure code stores that can be read to repair the stripe. The specific configuration of the redundancy mode that provides erasure coding is called k=64, m=36, which means there are 64 second data blocks, 36 redundancy check blocks, can tolerate up to 6 concurrent disk failures and 2 concurrent complete node failures, and does not require any decoding when reading, so the failure-free reading loss of the erasure code is zero.

[0152] The present application also provides a data processing device based on a stream directory, which is used to execute the steps in the above-mentioned data processing method embodiment based on a stream directory. Fig.11 As shown, the data processing device 900 based on the stream directory provided in the embodiment of the present application includes:

[0153] The acquisition module 910 is used to acquire multiple source files in the source file system and a file access sequence for the multiple source files.

[0154] The first reading module 920 is used to run the first process, read the first metadata of each of the source files according to the file access order, and generate the second metadata of the initial stream directory based on each of the first metadata, wherein the initial stream directory belongs to a subdirectory of a specified parent directory of the target file system.

[0155] The second reading module 930 is used to run the second process, read the file content of each source file according to the file access order, and fill at least one data block associated with the initial stream directory based on the content of each file. The first process and the second process are executed in parallel.

[0156] The updating module 940 is configured to update the initial stream directory based on the second metadata and the at least one data block to obtain a target stream directory.

[0157] In one embodiment, the first reading module is used to obtain a slice structure for metadata in the target file system; for each attribute in the slice structure, based on an initial value adapted to the attribute in each of the first metadata, a target value for the attribute is determined; and the target slice structure including each of the target values ​​is determined as the second metadata of the initial stream directory.

[0158] In one embodiment, the second reading module is used to determine the total data volume of the contents of multiple files; determine the required number of data blocks based on the block capacity of the data blocks of the target file system and the total data volume; allocate the required number of target data blocks to the initial stream directory; and fill the contents of each file into each target data block in sequence according to the file access order.

[0159] In one embodiment, the update module is used to obtain the index node of the initial stream directory, and there is a one-to-one correspondence between the index node and the stream directory; update the metadata attributes of the index node based on the second metadata; update the data block attributes of the index node based on the at least one data block; and use the initial stream directory with the updated index node as the target stream directory.

[0160] In one embodiment, the acquisition module is also used to obtain an access method for the target stream directory if an access request for the target stream directory is received, and the access method is a first access method based on a portable operating system interface or a second access method based on a memory mapping interface; based on the access method, the target stream directory is accessed to obtain an access result that is adapted to the access request.

[0161] In one embodiment, the acquisition module is also used to directly map the target stream directory to the virtual address space of the file access process if the access method is a second access method based on a memory mapping interface; access the file content of the target stream directory in the virtual address space to obtain an access result that is adapted to the access request.

[0162] In one embodiment, the acquisition module is also used to select a target access interface adapted to the access request from at least one access interface of the target file system for the stream directory if the access mode is a first access mode based on a portable operating system interface, and the access interface is created based on the portable operating system interface; parse the access request to obtain interface parameters adapted to the target access interface; based on the interface parameters, call the target access interface to access the target stream directory to obtain an access result adapted to the access request.

[0163] In one embodiment, the storage unit of the source file system is a data block of a first capacity, and the storage unit of the target file system is a data block of a second capacity, and the second capacity is greater than the first capacity.

[0164] In applications, each module in the image processing device may be a software program module, or may be implemented by different logic circuits integrated in a processor, or may be implemented by multiple distributed processors.

[0165] Fig.11 1 is a schematic diagram of the structure of a computer device for implementing data processing based on a stream directory provided by an embodiment of the present application. Fig.11 As shown, the computer device 1000 of this embodiment includes: at least one processor 1010 ( Fig.11 Only one is shown in the figure) a processor, a memory 1020, and a computer program 1030 stored in the memory 1020 and executable on the at least one processor 1010, and when the processor 1010 executes the computer program 1030, the steps in any of the above-mentioned data transmission method embodiments are implemented.

[0166] The computer device 1000 may include, but is not limited to, a processor 1010 and a memory 1020. Those skilled in the art will appreciate that Fig.11The computer device 1000 is merely an example and does not constitute a limitation on the computer device 1000 . The computer device 1000 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device 1000 may also include input and output devices, network access devices, etc.

[0167] The processor 1010 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0168] In some embodiments, the memory 1020 may be an internal storage unit of the computer device 1000, such as a hard disk or memory of the computer device 1000. In other embodiments, the memory 1020 may also be an external storage device of the computer device 1000, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 1000. Further, the memory 1020 may also include both an internal storage unit of the computer device 1000 and an external storage device. The memory 1020 is used to store an operating system, an application, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 1020 may also be used to temporarily store data that has been output or is to be output. It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application, and their specific functions and technical effects can be specifically referred to the method embodiment part, which will not be repeated here.

[0169] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0170] An embodiment of the present application further provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0171] An embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, a mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium, and the readable storage medium can be a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data processing method based on a stream directory, characterized in that: include: Acquire multiple source files in a source file system and a file access order for the multiple source files; Running a first process to read first metadata of each of the source files according to the file access order; Performing data conversion on the first metadata of each source file to obtain first metadata of a single-byte type, and generating second metadata of an initial stream directory under a target file system based on each first metadata of a single-byte type; Running a second process to read the file content of each of the source files from a plurality of first data blocks of the source file system associated with the source files according to the file access order; Performing data conversion on the file content of each of the source files to obtain a single-byte file content, and filling at least one second data block of the target file system associated with the initial stream directory based on the single-byte file content, wherein the first process and the second process are executed in parallel, and a second capacity of the second data block is much larger than a first capacity of the first data block; Based on the second metadata and the at least one second data block, updating the initial stream directory to obtain a target stream directory; If an access request for the target stream directory is received and the access method for the target stream directory is the second access method based on the memory mapping interface, the target stream directory is directly mapped to the virtual address space of the file access process, and the file content of the target stream directory in the virtual address space is accessed to obtain an access result that is adapted to the access request.

2. The data processing method based on the stream directory according to claim 1, characterized in that: The step of generating second metadata of the initial stream directory under the target file system based on each of the first metadata of the single-byte type includes: Obtaining a slice structure for metadata in the target file system; For each attribute in the slice structure, determining a target value of the attribute based on an initial value adapted to the attribute in each of the first metadata; The target slice structure including each of the target values ​​is determined as the second metadata of the initial stream directory under the target file system.

3. The data processing method based on the stream directory according to claim 1, characterized in that: The filling of at least one second data block of the target file system associated with the initial stream directory based on the file contents of each single-byte type comprises: Determine the total data volume of the contents of the plurality of files; Determining the required number of data blocks based on the block capacity of the data blocks of the target file system and the total data volume; Allocating the required number of target data blocks to the initial stream directory; According to the file access sequence, the contents of each file are sequentially filled into each target data block.

4. The data processing method based on the stream directory according to claim 1, characterized in that: The updating the initial stream directory based on the second metadata and the at least one second data block to obtain a target stream directory includes: Obtaining the index node of the initial stream directory, where the index node and the stream directory have a one-to-one correspondence; updating a metadata attribute of the index node based on the second metadata; Based on the at least one second data block, updating a data block attribute of the index node; The initial stream directory after the index node is updated is used as the target stream directory.

5. The data processing method based on the stream directory according to claim 1, characterized in that: The method further comprises: If an access request for the target stream directory is received and the access method for the target stream directory is a first access method based on a portable operating system interface, the target stream directory is accessed based on the first access method to obtain an access result that matches the access request.

6. The data processing method based on the stream directory according to claim 5, characterized in that: The accessing the target stream directory based on the first access mode to obtain an access result adapted to the access request includes: Selecting a target access interface adapted to the access request from at least one access interface of the target file system for a stream directory, the access interface being created based on a portable operating system interface; Parsing the access request to obtain interface parameters adapted to the target access interface; Based on the interface parameters, the target access interface is called to access the target stream directory to obtain an access result that matches the access request.

7. The data processing method based on a stream directory according to any one of claims 1 to 6, characterized in that: The storage unit of the source file system is a first data block of a first capacity, and the storage unit of the target file system is a second data block of a second capacity.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images