Data processing method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a computer program product, and the method comprises the steps: dividing a storage space into a plurality of storage domains, and creating a corresponding file system instance for each storage domain; wherein each file system instance manages data operation in a corresponding storage domain; in response to a data operation request for a target file, determining a target file system instance of the target file; and sequentially writing data or metadata of the target file into a storage domain managed by the target file system instance. According to the method, lock competition and cache interference are reduced through a domain-isolated multi-instance architecture, and the concurrency performance is improved; the data and the metadata are sequentially written into the tail of the log, disk addressing and fragmentation are reduced to the maximum extent, and therefore the utilization rate and the overall performance of the storage medium are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a data processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, modern data centers place extremely high demands on the throughput and latency performance of file systems. Traditional file systems face severe challenges, especially when processing massive amounts of large files (such as video streams, database backups, and log archives).

[0003] In current technologies, mainstream file systems (such as Ext4 and XFS) typically run in the operating system kernel space. When applications read and write files, they need to frequently switch contexts between user space and kernel space, which brings significant CPU overhead. Furthermore, to ensure metadata consistency, traditional file systems in multi-core CPU environments typically use global locks or fine-grained locks to manage shared resources (such as global directory trees and free block bitmaps). With the widespread adoption of many-core processors and high-speed NVMe SSDs, this "shared memory"-based concurrency control mechanism often becomes a performance bottleneck, causing the CPU to spend a significant amount of time contending for lock resources, preventing it from fully utilizing hardware performance.

[0004] On the other hand, traditional file systems mostly employ an update-in-place write strategy. While this strategy was acceptable in the era of mechanical hard drives, on SSDs, random writes can lead to severe write amplification, shortening device lifespan and degrading performance. Although Log-based File Systems (LFS) solve the random write problem through sequential writes, traditional LFS is prone to severe file fragmentation when handling large files, and its garbage collection overhead is enormous, often resulting in unpredictable performance fluctuations. Summary of the Invention

[0005] The purpose of this invention is to provide a data processing method, apparatus, device, storage medium, and computer program product that can significantly improve concurrent throughput in a multi-core environment and optimize the writing and recycling efficiency of large files.

[0006] To achieve the above objectives, embodiments of the present invention provide a data processing method, including: The storage space is divided into multiple storage domains, and a corresponding file system instance is created for each storage domain; wherein each file system instance manages data operations within the corresponding storage domain; In response to a data operation request for a target file, determine the target file system instance of the target file; The data or metadata of the target file is sequentially written into the storage domain managed by the target file system instance.

[0007] As an improvement to the above scheme, the step of determining the target file system instance of the target file in response to a data operation request for the target file further includes: If the current file system instance receiving the data operation request is different from the target file system instance, the target storage domain to which the data operation request belongs is identified, and the data operation request is forwarded to the target file system instance that manages the target storage domain for processing.

[0008] As an improvement to the above solution, the method further includes: One of the multiple file system instances is configured as a master instance, and the other file system instances are configured as slave instances; wherein, the master instance is used to receive modification requests for the root directory, update the root directory information, and synchronize the updated root directory information to all slave instances; the multiple file system instances adopt a shared-nothing architecture, and each file system instance has its own private memory area.

[0009] As an improvement to the above solution, the method further includes: Each instance sends directory information within its managed storage domain to the master instance; The main instance aggregates all received directory information to build a global root directory view, and starts the file system service after the aggregation is completed.

[0010] As an improvement to the above scheme, the storage domain is logically divided into multiple fixed-size fragments; Different writing strategies are executed based on the size of the record to be written in the data operation request: If the size of the record to be written is less than the preset threshold, it will be appended to the end of the metadata log as the first record. If the size of the record to be written is between the preset threshold and the fragment size, it will be appended to the end of the data log as the second record. If the size of the record to be written is greater than or equal to the fragment size, it is treated as the third record, and a new free fragment is directly allocated for continuous writing.

[0011] As an improvement to the above scheme, the metadata log and the data log are stored in different shards of the storage domain, respectively; Specifically, the content of the metadata log is cached in the private memory area of ​​the file system instance; In response to the shard used to store metadata logs being full, a new shard is allocated, and a link record pointing to the new shard is added to the end of the original shard.

[0012] This invention also provides a data processing apparatus, comprising: The resource binding module is configured to: divide the storage space into multiple storage domains and create a corresponding file system instance for each storage domain; wherein each file system instance manages data operations within its corresponding storage domain; The request distribution module is configured to: in response to a data operation request for a target file, determine the target file system instance of the target file; The data writing module is configured to sequentially write the data or metadata of the target file into the storage domain managed by the target file system instance.

[0013] This invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the data processing method described in any of the preceding claims.

[0014] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute any of the data processing methods described above.

[0015] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the data processing method described above.

[0016] Compared to existing technologies, the beneficial effects of the data processing method, apparatus, device, storage medium, and computer program product provided by the embodiments of the present invention are as follows: By dividing the storage space into multiple storage domains and creating a corresponding file system instance for each storage domain; wherein each file system instance manages data operations within its corresponding storage domain; in response to a data operation request for a target file, a target file system instance for the target file is determined; and the data or metadata of the target file is sequentially written into the storage domain managed by the target file system instance. The embodiments of the present invention reduce lock contention and cache interference through a domain-isolated multi-instance architecture, improving concurrency performance; by sequentially writing data and metadata to the end of the log, disk addressing and fragmentation are minimized, thereby significantly improving the utilization rate of the storage medium and overall performance. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a preferred embodiment of a data processing method provided by the present invention; Figure 2 This is a schematic diagram of the system architecture in a data processing method provided by the present invention; Figure 3 This is a schematic diagram of a preferred embodiment of a data processing device provided by the present invention; Figure 4 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 , Figure 1 This is a flowchart illustrating a preferred embodiment of a data processing method provided by the present invention. The data processing method includes: Step S100: Divide the storage space into multiple storage domains and create a corresponding file system instance for each storage domain; wherein each file system instance manages data operations within the corresponding storage domain; Step S200: In response to a data operation request for the target file, determine the target file system instance of the target file; In step S300, the target file system instance sequentially writes the data or metadata of the target file into the storage domain managed by the target file system instance.

[0020] In practical implementation, this invention provides a user-space journaling file system optimized for large files. This system is deployed in the user space of the operating system (such as Linux) of the computing device, bypassing the traditional kernel virtual file system (VFS). Please refer to... Figure 2 , Figure 2 This is a schematic diagram of the system architecture of a data processing method provided by the present invention. The system mainly includes three modules: a file system API layer, a resource partitioning module composed of file system instances (shards) and hardware layer disk partitioning, and a journaling file system module. The file system API layer is the interface provided by the file system in this embodiment of the invention, designed to provide applications with access to file system functions. These functions include file system formatting, opening, closing, reading, and writing files, as well as creating and deleting directories. The API layer encapsulates the requests initiated by the application into specific parameters and sends them to the file system execution thread. Once the file system completes the corresponding operation and returns data, the API layer passes the result to the application. The file system API acts as a bridge between the application and the file system, enabling the application to easily utilize the file system's functions for file and directory management operations.

[0021] The underlying hardware of this system includes a multi-core processor (CPU) and high-performance storage media (such as NVMe SSD). To maximize the utilization of multi-core performance and improve the ability of multi-threaded concurrent read and write operations, this invention adopts a share-nothing architecture, dividing the file system into multiple independent instances, partitioning the CPU, disk, and memory. Each file system instance thread corresponds to a separate CPU core, and each CPU has a private memory area and a private disk space. Furthermore, two CPUs cannot access the same disk space. The specific configuration is as follows: 1. Resource partitioning: The physical storage medium is logically divided into multiple fixed-size storage domains. Each storage domain corresponds to an instance of the file system, meaning each file system instance (Shard) is responsible for its own corresponding storage domain. Simultaneously, the system's memory is also partitioned into multiple non-overlapping private memory regions.

[0022] 2. Instance Binding: Multiple file system instances (Shard 1, Shard 2... ShardN) are launched in user space, each bound to an independent CPU physical core. The primary task of a file system instance is to operate within its domain, thus avoiding frequent communication between different instances and improving parallelism. Furthermore, when an instance receives a request belonging to another domain, it forwards the request to other instances belonging to that domain.

[0023] 3. Exclusive Management: Each file system instance exclusively manages one storage domain. For example, Shard 1 can only read and write data in Domain 1, and Shard 2 can only read and write data in Domain 2. There are no shared locks between instances, and all cross-domain interactions are completed through message passing.

[0024] like Figure 1 As shown, this embodiment describes the data processing flow of the file system in detail.

[0025] In step S100, the storage space is divided into multiple storage domains, and a corresponding file system instance is created for each storage domain; wherein each file system instance manages data operations within the corresponding storage domain.

[0026] When the system starts, it reads the configuration file and starts the corresponding number of file system instances based on the number of CPU cores. Each instance initializes its private memory pool and mounts its corresponding disk storage domain. At this point, a physical resource isolation wall is established between the instances.

[0027] Specifically, the physical storage medium is divided into N disjoint storage domains, each managed by only one user-space file system instance. Domain partitioning can be equal or non-uniform based on workload, device characteristics, or tenant needs.

[0028] Each instance uses a private memory area to store runtime metadata structures (such as directory trees, index nodes, mapping relationships, sharding status, etc.) to avoid lock contention between instances reading and writing to the same memory structure.

[0029] Optionally, each instance can be bound to a different computing unit or processing core, further reducing the overhead of cross-core resource access and scheduling.

[0030] In one embodiment, the physical storage medium is logically divided into N storage domains. In user space, N corresponding file system instances run. Each file system instance is preferably bound to a dedicated CPU core to achieve better parallel performance. Each instance is responsible for managing all file data and metadata operations within its corresponding domain. A client application initiates a request through a standard file operation interface.

[0031] In step S200, in response to a data operation request for the target file, the target file system instance of the target file is determined.

[0032] Specifically, when a client application initiates a file operation request (such as a write request), the request may be received by any file system instance. The instance receiving the request first determines the storage domain to which the target file or data belongs.

[0033] If the target domain is one that the instance itself manages, then the instance will process the request directly. For write requests, the processing method is to append the metadata changes and file data sequentially to the log structure it manages.

[0034] If the target domain is managed by another instance, the receiving instance forwards the request to the file system instance that manages the target domain through an efficient inter-process communication (IPC) mechanism, such as shared memory or message queues, for processing by the target instance.

[0035] Once processing is complete, the instance responsible for the task returns the result to the client. It's worth noting that this "shared-nothing" architecture ensures that instances do not need to wait for each other when processing requests within their own domain, significantly improving parallelism.

[0036] In step S300, the data or metadata of the target file is sequentially written into the storage domain managed by the target file system instance.

[0037] Specifically, after receiving a write request, the target file system instance performs the core log write operation. It should be noted that existing file systems typically store metadata and data in blocks allocated on the device. When writing a file to such a file system, some metadata structures on the disk are updated, blocks are allocated for the data, and the data is eventually flushed to the disk. Under this disk management strategy, even writing a continuous stream of bytes can result in many random writes reaching the storage device. Obviously, this affects the read and write speed of upper-layer applications. To address this problem, this embodiment of the invention proposes a journaling file system, including two logs: a metadata log and a data log. This system effectively improves the processing speed of a large number of continuous write operations by appending data and metadata to the logs.

[0038] First, to facilitate log and disk space management, this embodiment of the invention divides the storage domain of each file system instance into multiple shards of the same size, for example, each shard is 64MB. Data and metadata logs are recorded on each shard. For the metadata log, if the metadata of a shard is full, a new shard is allocated, and the record pointing to the new shard is appended to the log as the last entry. For the data log, the distribution of data shards is recorded in the metadata.

[0039] Secondly, the metadata log records all file system operations on metadata. Since this embodiment uses a log structure, records describing the metadata of the same file may be scattered across different parts of the disk. Therefore, to avoid the high cost of reading multiple small blocks of data to retrieve metadata, all metadata is stored in memory. When starting the file system from scratch, the entire metadata log needs to be read, its records reconstructed, and loaded into memory.

[0040] For example, the specific startup process is as follows: 1. Create a boot record: Create a boot record at a specific offset position on the device. This record is initially created by an auxiliary program, similar to a disk formatting tool (such as mkfs). 2. System Startup: During system startup, an attempt is made to load the boot record from the specified offset location; 3. Verify the boot record: Verify the authenticity and consistency of the boot record. The boot record includes information such as file system metadata, metadata, and data log offsets; 4. Load Metadata: If the boot record verification passes, all metadata is loaded from the logs on the disk to complete the system startup.

[0041] It should be noted that the currently active shard has a write buffer in memory, and both data and metadata are treated as log records, which are sequentially appended to the end of the currently active shard.

[0042] Furthermore, in order to address the write amplification and large file transfer overhead issues faced by traditional log file systems during garbage collection, this embodiment of the invention introduces a data volume-based hierarchical strategy in step S300.

[0043] In this embodiment, it is assumed that the fragment size is 64MB and the system sets the preset threshold to 4KB (small record threshold).

[0044] Scenario 1: Writing a small record (i.e., the first record) (Data < 4KB) Processing method: It will be treated as a metadata log and directly appended to the shard currently used to store metadata (MetaLog Head). This is understandable, as metadata is typically updated frequently and is small in size; centralized storage is beneficial for fast indexing and caching.

[0045] Scenario 2: Writing a medium-sized record (i.e., the second record) (4KB ≤ Data < 64MB) Processing method: Append the current data to the Data Log Head. When the data to the Data Log Head is full, close it and write it to disk, then allocate a new data to the disk, thus taking advantage of the sequential write capability of the SSD.

[0046] Scenario 3: Writing a large record (i.e., the third record) (Data ≥ 64MB) Processing method: Instead of appending the data to the end of a fragment, the system directly allocates one or more complete, brand-new fragments from the free fragment pool. The data is then directly filled into these new fragments. In the log stream, only a link record or pointer record pointing to these new fragments is recorded. For example, the link record contains the physical address and checksum of the new fragment, using a 64-bit offset pointer for cross-fragment addressing. In essence, when the system performs garbage collection, for these large, complete fragments, if the file is deleted, the entire fragment is released; if the file is valid, there's no need to read 64MB of data and rewrite it to a new location—only the fragment index needs to be retained. This significantly reduces memory bandwidth usage and I / O latency during GC.

[0047] In this embodiment, a hierarchical writing strategy is adopted for data records of different sizes. In particular, large records are directly allocated complete fragments, which avoids a large amount of data transfer during log compression and significantly reduces system jitter.

[0048] In one embodiment, because the present invention employs a share-nothing architecture, the directory tree needs to be maintained consistently. Each file system instance not only maintains its own root directory but also needs to track and store aggregate information of the root directories of all other instances. For example, this embodiment of the present invention uses an asymmetric master-slave model to replicate data between instances, where one instance acts as a reliable data source, while the other instances synchronize with it. Exemplarily, a first file system instance is selected as the master instance, and the rest are slave instances. The metadata of the global root directory is maintained by the master. When a second file system instance needs to create a new file located in the root directory, the second file system instance sends a request to the first file system instance. The first file system instance updates the directory tree in memory, records the mapping of the new file to the second file system instance, and synchronizes the changes to all slave instances (or only queries when needed). It should be noted that once the file is created, the data reading and writing within the file is entirely handled independently by the slave (i.e., the second file system instance), without the participation of the master (i.e., the first file system instance), thus avoiding the master becoming a performance bottleneck.

[0049] In other words, when the file system starts, each file system instance sends its root directory information to the master instance. The entire system startup is only complete when the master instance receives the root directory information from all instances. The instance that starts the entire file system is the master instance, and this role remains unchanged during the file system's operation. Furthermore, to ensure data consistency across different cores, only the master instance can modify the root directory; other instances must synchronize data from the master instance. If a slave instance wants to modify the root directory, it needs to send a request to the master instance. Once the master instance agrees, it synchronizes the updated root directory to all slave instances.

[0050] In another embodiment, when the system crashes and restarts, parallel loading is first used, that is, all instances start in parallel, scan the log tail of the storage domain they manage, rebuild their respective local metadata mapping tables, and then converge. All slave instances send the rebuilt local directory information to the master instance, and the master instance merges all the information to build a complete global file system view, marking the completion of system startup and the start of providing services to the outside world.

[0051] It should be noted that in a log-structured file system, new data is appended to the end of the log, while old data is not immediately deleted. Over time, this old data may become outdated and invalid. If left unmanaged, this invalid data will occupy valuable disk space and may eventually exhaust available space. To address this issue, embodiments of the present invention employ a compaction mechanism to free up space by periodically cleaning up this invalid data, a process similar to garbage collection. At runtime, the file system can use various heuristics to determine when and which shards need garbage collection; for example, in embodiments of the present invention, compaction can be determined by judging the proportion of valid data in a shard. However, compaction introduces additional disk operations, which may lead to a decrease in the performance of user-initiated operations. To balance this, compaction should be performed as infrequently as possible, but still frequently enough not to cause space problems. For example, the specific steps of compaction can be divided into: Shard selection: To minimize the impact on performance, the most suitable shards need to be selected for compression. This can be done by calculating the ratio of valid data in the current shard to the total size of the shard. When this ratio is relatively small, it indicates that the shard contains a lot of invalid data, so it can be selected for compression.

[0052] Compression Implementation: First, all currently valid data in the specified shard is read into memory. Then, a new shard is allocated to rewrite this data, and the data layout within the new shard is determined. Next, these new shards are written to disk, and the metadata in memory is updated to reflect the new data items. Finally, the old shards containing invalid data are released.

[0053] In the above embodiments, this invention effectively solves the lock contention bottleneck and SSD write amplification problem in modern high-concurrency storage systems by constructing a shared-nothing architecture based on "core-instance-domain" binding in user space, combined with a log-level writing strategy optimized for large files. It is particularly suitable for application scenarios with extremely high requirements for large file throughput, such as video processing and big data analysis. Furthermore, the file system is implemented in user space, and its core feature is the adoption of a log-like structure, appending disk operations sequentially to the end of the log, thereby improving efficiency when handling large write operations. To ensure good scalability and conform to the design principle of one thread per core and shared resources, each thread only processes the part of the directory tree that does not intersect with other threads, thereby reducing the need for synchronization operations and better leveraging the advantages of higher bandwidth, lower latency, and lower power consumption of SSDs.

[0054] Accordingly, the present invention also provides a data processing apparatus capable of implementing all the processes of the data processing methods described in the above embodiments.

[0055] Please see Figure 3 , Figure 3 This is a schematic diagram of a preferred embodiment of a data processing apparatus provided by the present invention. The data processing apparatus includes: Resource binding module 301 is configured to: divide the storage space into multiple storage domains, create a corresponding file system instance for each storage domain, and manage data operations within the corresponding storage domain for each file system instance; The request distribution module 302 is configured to: determine the target file system instance of the target file in response to a data operation request for the target file; The data writing module 303 is configured to sequentially write the data or metadata of the target file into the storage domain managed by the target file system instance.

[0056] Preferably, the request distribution module is further configured to: If the current file system instance receiving the data operation request is different from the target file system instance, the target storage domain to which the data operation request belongs is identified, and the data operation request is forwarded to the target file system instance that manages the target storage domain for processing.

[0057] Preferably, the resource binding module is further configured as follows: One of the multiple file system instances is configured as a master instance, and the other file system instances are configured as slave instances; wherein, the master instance is used to receive modification requests for the root directory, update the root directory information, and synchronize the updated root directory information to all slave instances; the multiple file system instances adopt a shared-nothing architecture, and each file system instance has its own private memory area.

[0058] Preferably, the resource binding module is further configured as follows: Each instance sends directory information within its managed storage domain to the master instance; The main instance aggregates all received directory information to build a global root directory view, and starts the file system service after the aggregation is completed.

[0059] Preferably, the storage domain is logically divided into multiple fixed-size fragments; Different writing strategies are executed based on the size of the record to be written in the data operation request: If the size of the record to be written is less than the preset threshold, it will be appended to the end of the metadata log as the first record. If the size of the record to be written is between the preset threshold and the fragment size, it will be appended to the end of the data log as the second record. If the size of the record to be written is greater than or equal to the fragment size, it is treated as the third record, and a new free fragment is directly allocated for continuous writing.

[0060] Preferably, the metadata log and the data log are stored in different shards of the storage domain; Specifically, the content of the metadata log is cached in the private memory area of ​​the file system instance; In response to the shard used to store metadata logs being full, a new shard is allocated, and a link record pointing to the new shard is added to the end of the original shard.

[0061] In specific implementations, the working principle, control process, and technical effects of the data processing device provided in the embodiments of the present invention are the same as those of the data processing methods in the above embodiments, and will not be repeated here.

[0062] Please see Figure 4 , Figure 4 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 401, a memory 402, and a computer program stored in the memory 402 and configured to be executed by the processor 401. When the processor 401 executes the computer program, it implements the data processing method described in any of the above embodiments.

[0063] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 402 and executed by the processor 401 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0064] The processor 401 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 401 can be any conventional processor. The processor 401 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0065] The memory 402 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory 402 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD), and a flash card, or it can be other volatile solid-state storage devices.

[0066] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 4 The structural diagram is merely an example of the terminal device described above and does not constitute a limitation on the terminal device described above. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components.

[0067] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the data processing method described in any of the above embodiments.

[0068] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the data processing method described in any of the above embodiments.

[0069] In this invention, the file system creates multiple file system instances in user space and divides the file storage medium into multiple non-overlapping storage domains. Each instance only accesses its corresponding storage domain and uses a private memory area and private disk space. Instances do not share disk write paths and metadata structures, thereby avoiding cross-instance resource contention.

[0070] Within the storage domain, this invention employs a dual-log sequential append write path. The metadata log records directory entries, inodes, attribute changes, etc., and management is based on shards. When the current metadata shard is full, a jump record is appended to the end of the log, and the system switches to a new shard, maintaining log continuity and structural visibility. The data log records file data content. File data is categorized and stored according to its relationship to shard size: smaller data can be directly merged into the end of the metadata log to reduce fragmentation in the data log; medium-sized data is written to the end of the data log; large records are directly allocated to new shards, enabling large-block continuous writing and maintaining physical continuity and read performance.

[0071] During system startup, the boot record is read and log offset information is obtained. The metadata log is replayed to reconstruct the metadata and load it into memory. During runtime, all path resolution and write operations are performed based on memory-state metadata, reducing the overhead of random reads and small block accesses.

[0072] In this invention, global root directory consistency is achieved through a master-slave model. The master instance is responsible for unique modifications to the root directory, while slave instances receive updates synchronized from the master instance. All instances maintain an aggregated global root view. When an instance receives a request but the target object is located in another storage domain, it relies on the aggregated view to identify the target storage domain and delegates the request to the corresponding instance for execution.

[0073] To ensure the sustainability of space for sequential appending, the system selects shards for compression and reclamation based on the ratio of valid data within a shard to its total size. The compression process includes reading valid data, rewriting it to a new shard, updating metadata in memory, and releasing the old shard, thereby reducing space waste and maintaining a clean log structure.

[0074] This invention provides a data processing method, apparatus, device, storage medium, and computer program product capable of processing data in various scenarios, including large and small files. In particular, it offers higher bandwidth and lower latency, achieving optimal performance in the following specific scenarios: for example, reading and writing M / G / TB level files; the absence of hot files or directories; multiple users working on their respective files without sharing; any user able to access any file without conflict; and most operations being sequential read / write. It is understood that the above performance advantages are achieved precisely because this invention employs a share-nothing architecture and log appending. These embodiments enable continuous writing of large files to fill the entire filesystem, and reading can also be performed sequentially. For the aforementioned sequential read / write operations, the file system performance is even more superior.

[0075] Specifically, this invention optimizes the reading and writing of large files. For write-heavy, read-light scenarios, appending large files to data and metadata logs enables contiguous disk storage, significantly improving write and read performance. Based on Share-nothing, this invention better utilizes the multi-core capabilities of the CPU and leverages the advantages of high-performance SSDs, achieving highly parallel multi-threaded read and write capabilities. Implementing the file system in user space provides high flexibility, portability, and ease of development. A file system running in user space can respond more flexibly to application needs, while reducing system calls and simplifying I / O scheduling and management, thereby improving system efficiency and performance. In other words, running in user space avoids frequent switching between kernel and user modes; the use of log-based sequential writing perfectly adapts to SSD characteristics and reduces write amplification.

[0076] In summary, this invention provides a data processing method for a user-space log-based file system. Data and metadata are sequentially written to the end of the log, minimizing disk seeks and fragmentation, thereby improving storage media utilization and overall performance. Furthermore, this invention utilizes a share-nothing architecture to achieve a one-to-one binding between CPU cores, file system instances, and disk domains. There is no resource sharing between instances, completely eliminating lock contention overhead in multi-core environments and achieving linear performance scaling with the number of CPU cores. This parallel processing approach significantly improves system throughput and concurrency, making it particularly suitable for handling large-scale files and high-concurrency writes. In addition, this invention employs a master-slave model for jointly maintaining the data root directory, ensuring data synchronization and consistency among multiple instances. These mechanisms not only improve system reliability but also guarantee that each instance obtains the latest data status, thus preventing data inconsistencies.

[0077] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0078] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: The storage space is divided into multiple storage domains, and a corresponding file system instance is created for each storage domain; wherein each file system instance manages data operations within the corresponding storage domain; In response to a data operation request for a target file, determine the target file system instance of the target file; The data or metadata of the target file is sequentially written into the storage domain managed by the target file system instance.

2. The data processing method as described in claim 1, characterized in that, The step of determining the target file system instance of the target file in response to a data operation request for the target file further includes: If the current file system instance receiving the data operation request is different from the target file system instance, the target storage domain to which the data operation request belongs is identified, and the data operation request is forwarded to the target file system instance that manages the target storage domain for processing.

3. The data processing method as described in claim 1, characterized in that, The method further includes: One of the multiple file system instances is configured as a master instance, and the other file system instances are configured as slave instances; wherein, the master instance is used to receive modification requests for the root directory, update the root directory information, and synchronize the updated root directory information to all slave instances; the multiple file system instances adopt a shared-nothing architecture, and each file system instance has its own private memory area.

4. The data processing method as described in claim 3, characterized in that, The method further includes: Each instance sends directory information within its managed storage domain to the master instance; The main instance aggregates all received directory information to build a global root directory view, and starts the file system service after the aggregation is completed.

5. The data processing method as described in claim 1, characterized in that, The storage domain is logically divided into multiple fixed-size fragments; Different writing strategies are executed based on the size of the record to be written in the data operation request: If the size of the record to be written is less than the preset threshold, it will be appended to the end of the metadata log as the first record. If the size of the record to be written is between the preset threshold and the fragment size, it will be appended to the end of the data log as the second record. If the size of the record to be written is greater than or equal to the fragment size, it is treated as the third record, and a new free fragment is directly allocated for continuous writing.

6. The data processing method as described in claim 5, characterized in that, The metadata log and the data log are respectively stored in different shards of the storage domain; Specifically, the content of the metadata log is cached in the private memory area of ​​the file system instance; In response to the shard used to store metadata logs being full, a new shard is allocated, and a link record pointing to the new shard is added to the end of the original shard.

7. A data processing apparatus, characterized in that, include: The resource binding module is configured to: divide the storage space into multiple storage domains and create a corresponding file system instance for each storage domain; wherein each file system instance manages data operations within its corresponding storage domain; The request distribution module is configured to: in response to a data operation request for a target file, determine the target file system instance of the target file; The data writing module is configured to sequentially write the data or metadata of the target file into the storage domain managed by the target file system instance.

8. A terminal device, characterized in that, The device includes a processor and a memory, wherein the memory stores a computer program and the computer program is configured to be executed by the processor, wherein the processor, when executing the computer program, implements the data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the data processing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1 to 6.

Images