File management method, device and system and storage medium

Abstract

The invention relates to a file management method, device and system and a storage medium. The method comprises the steps that when an operation request of a client for a target file is received, metadata management operation matched with the operation request is executed in an embedded database of a file management system through a metadata manager of the file management system, and metadata of the target file is obtained; the metadata comprises a data reference identifier; analyzing the data reference identifier to obtain a data storage position; the data storage position is used for indicating a storage position of the target file in a target data storage rear end, and the target data storage rear end is one of a plurality of data storage rear ends butted with a storage abstraction layer of the file management system; and executing a file management operation matched with the operation request on the target file based on the data storage position in the target data storage rear end through a storage abstraction layer of the file management system. According to the scheme provided by the invention, the problems of function redundancy and hierarchical redundancy in the related technology can be solved.

Description

Document management methods, devices, systems, and storage media Technical Field

[0001] This application relates to the field of computer technology, and in particular to file management methods, apparatus, systems and storage media. Background Technology

[0002] The file system is a core subsystem in the operating system kernel responsible for managing and storing data on external storage devices of the computer. It is the foundation of modern computer data storage and sharing.

[0003] In related technologies, traditional general-purpose file systems (such as ext4 (fourth-generation extended file system) and NTFS (New Technology File System)) are kernel-mode file systems designed for general-purpose scenarios. However, they have problems such as functional redundancy and hierarchical redundancy. Regarding functional redundancy, traditional general-purpose file systems, designed for general-purpose scenarios, integrate a large number of complex functions (such as complex logs, access control lists, disk quotas, etc.), which are not necessary in specific application scenarios (such as embedded devices), and instead increase performance overhead and consume more resources. Regarding hierarchical redundancy, traditional general-purpose file systems adopt a hierarchical architecture of "user-mode application → kernel-mode file system". Therefore, application operations on files need to frequently switch between user mode and kernel mode, which will cause performance loss, especially in high-frequency, small file scenarios (such as reading and writing a large number of small files), where efficiency is even lower. Summary of the Invention

[0004] To address or partially address the problems existing in related technologies, this application provides a file management method, apparatus, system, and storage medium that can solve the imbalance between performance overhead, resource consumption, and functional complexity in specific application scenarios, and can avoid frequent user-mode-kernel-mode switching, thereby improving the efficiency of high-frequency / small file operations.

[0005] This application provides a file management method applied to a file management system. The file management system comprises multiple lightweight components, including an embedded database, a metadata manager, and a storage abstraction layer. The embedded database stores metadata for each file, the metadata manager manages the metadata for each file, and the storage abstraction layer interfaces with multiple data storage backends. The method includes: when a client requests an operation on a target file, performing a metadata management operation matching the operation request in the embedded database through the metadata manager to obtain the metadata of the target file; the metadata includes a data reference identifier; parsing the data reference identifier to obtain a data storage location; the data storage location indicating the storage location of the target file in a target data storage backend, the target data storage backend being one of the multiple data storage backends; and performing a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location through the storage abstraction layer.

[0006] In one embodiment, the embedded database includes at least one of an embedded relational database and an embedded key-value database, wherein the embedded relational database is used to store metadata that requires complex queries, and the embedded key-value database is used to store metadata of pure key-value pairs.

[0007] In one embodiment, when a client's operation request for a target file is received, the step of obtaining the target file's metadata by performing a metadata management operation matching the operation request in the embedded database through the metadata manager includes: extracting the target file's attribute information from the operation request when the client's operation request for the target file is received; the attribute information includes an operation type, which includes at least one of a write operation and a read operation; and performing a metadata management operation matching the operation type in the embedded database through the metadata manager to obtain the target file's metadata.

[0008] In one embodiment, the attribute information further includes a file path; the step of obtaining the target file's metadata by performing a metadata management operation matching the operation type in the embedded database through the metadata manager includes: when the operation type is the write operation, starting a database transaction; generating a data reference identifier for the target file based on the file path; creating a target record in the embedded database through the metadata manager, and filling the target record with the data reference identifier and the file path as the target file's metadata; wherein the target record is in an uncommitted state in the database transaction.

[0009] In another embodiment, the attribute information further includes a file path; the step of obtaining the metadata of the target file by performing a metadata management operation matching the operation type in the embedded database through the metadata manager includes: when the operation type is the read operation, constructing a query statement based on the file path through the metadata manager, and using the query statement to obtain the metadata of the target file from the embedded database.

[0010] In one embodiment, performing a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location through the storage abstraction layer includes: writing the target file into the target data storage backend based on the data storage location through the storage abstraction layer; the method further includes: committing the database transaction to update the status of the target record in the database transaction to a committed status; returning first response information to the client; the first response information is used to indicate that the target file was successfully written.

[0011] In another embodiment, the step of performing a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location through the storage abstraction layer includes: reading the target file from the target data storage backend based on the data storage location through the storage abstraction layer; the method further includes: returning second response information to the client; the second response information is used to indicate that the target file was successfully read.

[0012] A second aspect of this application provides a file management device applied to a file management system. The file management system comprises multiple lightweight components, including an embedded database, a metadata manager, and a storage abstraction layer. The embedded database stores metadata for each file, the metadata manager manages the metadata for each file, and the storage abstraction layer interfaces with multiple data storage backends. The device includes: a metadata management module, used to, upon receiving an operation request from a client for a target file, perform a metadata management operation matching the operation request in the embedded database through the metadata manager to obtain the metadata of the target file; the metadata includes a data reference identifier; a parsing module, used to parse the data reference identifier to obtain a data storage location; the data storage location indicates the storage location of the target file in a target data storage backend, the target data storage backend being one of the multiple data storage backends; and a file management module, used to, through the storage abstraction layer, perform a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location.

[0013] A third aspect of this application provides a file management system, including: an embedded database, a metadata manager, a storage abstraction layer, and the file management device described above.

[0014] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a file management device of a file management system, causes the file management device to perform the method described above.

[0015] The technical solution provided in this application may include the following beneficial results: The solution provided in this application is applied to a file management system, which consists of multiple lightweight components, including an embedded database, a metadata manager, and a storage abstraction layer. The embedded database is used to store the metadata of each file, the metadata manager is used to manage the metadata of each file, and the storage abstraction layer is used to interface with multiple data storage backends. When a client's operation request for a target file is received, the metadata manager performs a metadata management operation matching the operation request in the embedded database to obtain the metadata of the target file. The metadata includes a data reference identifier. The data reference identifier is parsed to obtain the data storage location. The data storage location is used to indicate the storage location of the target file in the target data storage backend, which is one of multiple data storage backends. The storage abstraction layer performs a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location. The file management system provided in this application is a lightweight design tailored to specific application scenarios. It integrates a small number of lightweight components (embedded database, metadata manager, and storage abstraction layer), retaining only essential functions (such as metadata management and file management), and eliminating redundant general-purpose functions found in traditional file systems (such as complex logs, access control lists, and disk quotas). This significantly reduces performance overhead, resource consumption, and functional complexity. Furthermore, metadata is stored in the embedded database, and file data is stored in the data storage backend connected to the storage abstraction layer. This allows the file management system to be embedded into the application as a library, directly managing files in user space. This avoids frequent user-mode-kernel-mode switching and improves the efficiency of high-frequency / small file operations.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0018] Figure 1 is a flowchart illustrating a file management method according to an embodiment of this application; Figure 2 is another flowchart illustrating a file management method according to an embodiment of this application; Figure 3 is a structural schematic diagram illustrating a file management device according to an embodiment of this application; Figure 4 is a structural schematic diagram illustrating a file management system according to an embodiment of this application. Detailed Implementation

[0019] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0022] In related technologies, traditional general-purpose file systems (such as ext4 and NTFS) are kernel-mode file systems designed for general-purpose scenarios. However, they suffer from problems such as functional redundancy and hierarchical redundancy. Regarding functional redundancy, traditional general-purpose file systems, designed for general-purpose scenarios, integrate a large number of complex functions (such as complex logs, access control lists, disk quotas, etc.), which are not necessary in specific application scenarios (such as embedded devices) and instead increase performance overhead and consume more resources. Regarding hierarchical redundancy, traditional general-purpose file systems adopt a hierarchical architecture of "user-mode application → kernel-mode file system". Therefore, application operations on files need to frequently switch between user mode and kernel mode, which will cause performance loss, especially in high-frequency, small-file scenarios (such as reading and writing a large number of small files), where efficiency is even lower.

[0023] To address the aforementioned issues, this application provides a file management method applied to a file management system. This file management system is a lightweight design tailored to specific application scenarios. For example, it integrates a small number of lightweight components (embedded database, metadata manager, and storage abstraction layer), retaining only essential functions (such as metadata management and file management). Redundant general-purpose functions found in traditional file systems (such as complex logs, access control lists, and disk quotas) are removed, thereby significantly reducing performance overhead, resource consumption, and functional complexity. Furthermore, metadata is stored in the embedded database, and file data is stored in the data storage backend connected to the storage abstraction layer. This allows the file management system to be embedded into the application as a library, directly performing file management in user space, thus avoiding frequent user-mode-kernel mode switching and improving the efficiency of high-frequency / small file operations.

[0024] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 is a flowchart illustrating a file management method according to an embodiment of this application. The file management system consists of multiple lightweight components, including an embedded database, a metadata manager, and a storage abstraction layer. The embedded database is used to store the metadata of each file, the metadata manager is used to manage the metadata of each file, and the storage abstraction layer is used to interface with multiple data storage backends.

[0026] Referring to Figure 1, the file management method of this application may include: S110, when receiving an operation request from a client for a target file, performing a metadata management operation matching the operation request in the embedded database through a metadata manager to obtain the metadata of the target file; the metadata includes a data reference identifier.

[0027] The embodiments of this application can employ components such as embedded databases, metadata managers, and storage abstraction layers to pre-build a lightweight and efficient file management system.

[0028] The core feature of embedded databases is that they run within the application process (user space) and do not require a separate database server process. Therefore, file management systems can use embedded databases to persistently store the metadata of all files.

[0029] The metadata manager can be GORM (Go Object-Relational Mapping), an open-source, full-featured ORM library for Go. GORM maps Go structs to database tables, allowing developers to manipulate relational databases (such as MySQL, PostgreSQL, and SQLite) in an object-oriented manner. It automatically handles tasks such as SQL (Structured Query Language) generation, data association, transactions, and migrations, greatly simplifying the code complexity of database interactions and improving development efficiency.

[0030] The storage abstraction layer can be Go CDK (Go Cloud Development Kit), an open-source project that provides a unified set of APIs (Application Programming Interfaces) and tools, allowing Go developers to write portable cloud applications. By abstracting the differences in underlying cloud services (such as storage, databases, message queues, etc.), it enables the same code to be seamlessly deployed and run on different cloud platforms (such as AWS (Amazon Web Services), Google Cloud, Azure) or on-premises environments, greatly simplifying the complexity of multi-cloud deployment and migration.

[0031] The construction process of the file management system is as follows: First, define the metadata model. Specifically, GORM can be used to create Go structs, mapping file metadata (such as file path, file size, permission flags, etc.) to database tables. Then, initialize the metadata storage engine. Specifically, an appropriate type of embedded database can be selected based on the scenario, and GORM automatically migrates the table structure. Next, abstract the data storage layer. Specifically, the Go CDK interface can be used to dynamically bind local or cloud storage backends by configuring the URL (Uniform Resource Locator). In this way, GORM (for metadata CRUD operations) and Go CDK (for data stream read / write) can be combined in the business logic, and consistency is ensured through a transaction mechanism. After the file management system is built, it can be compiled into a single binary file with no external dependencies and directly deployed to the target environment. Therefore, this embodiment achieves the core value of standardized metadata management, seamless switching of data storage backends, and lightweight delivery through the collaborative efforts of multiple lightweight components.

[0032] The Metadata Manager (GORM) acts as a bridge between the application and the metadata storage engine (embedded database). Developers can use Go structs to define the metadata model. GORM automatically converts all object-oriented operations into SQL or API calls to the underlying database, thereby greatly simplifying database operations and ensuring code conciseness and maintainability.

[0033] The storage abstraction layer (Go CDK) provides a unified interface for manipulating file data itself. It shields the underlying differences between different data storage backends (such as local disks, AWS S3, Azure Blob Storage, and Google Cloud Storage). Therefore, Go CDK enables the portability of data storage backends. For example, a file management system can flexibly switch data storage backends through a single configuration URL without modifying the business logic.

[0034] This application embodiment stores and manages file metadata (file management information / description information) and file data (the actual content of the file) separately. Specifically, metadata is stored through an embedded database, with each record corresponding to a file / directory and including file path, file size, modification time, permission flags, data reference identifiers, etc. File data is stored on a designated data storage backend via Go CDK, in the form of binary large objects (Blobs). As can be seen, this application embodiment fully utilizes the powerful query and management capabilities of the embedded database to process metadata, while leveraging the scalability and reliability of object storage (i.e., the data storage backend) to handle large file data blocks, thereby greatly improving the processing efficiency of the file management system.

[0035] This application embodiment can also construct a file management device in the file management system. The file management method can be specifically applied to the file management device in the file management system. This device does not store data but only manages data, thus satisfying the requirements of lightweight and efficient construction. Data storage depends on the physical storage device (i.e., the data storage backend) itself.

[0036] This application embodiment can utilize various components within a file management system to complete the entire file management process (metadata management, data read / write operations). In its specific implementation, the client can send an operation request for a target file to the file management system. The file management device within the system parses the operation request at the interface layer, extracts the target file's attribute information, and then calls the metadata manager (GORM). The GORM then performs metadata management operations matching the operation request based on the attribute information in the embedded database, thereby obtaining the target file's metadata. This metadata may include, but is not limited to: file path, file size, modification time, permission flags, MIME (Multipurpose Internet Mail Extensions) type, and data reference identifiers.

[0037] S120, parse the data reference identifier to obtain the data storage location; the data storage location is used to indicate the storage location of the target file in the target data storage backend, and the target data storage backend is one of multiple data storage backends.

[0038] A data reference identifier (DATA Identifier) ​​is a resource identifier that can locate a specific data storage backend. However, it is an abstract encapsulation of the data storage location and does not directly expose the specific data storage backend. Therefore, file management devices can obtain the data storage location by parsing the DATA Identifier. The data storage location is the actual physical / logical location of the file data; thus, the parsed data storage location can be used to indicate the storage location of the target file within the target data storage backend.

[0039] The target data storage backend is one of multiple data storage backends that the Storage Abstraction Layer (Go CDK) interfaces with. Specifically, in this embodiment, the Storage Abstraction Layer (Go CDK) abstracts all underlying storage (such as local disks, AWS S3, Azure Blob Storage, Google Cloud Storage, etc.) through its interface, decoupling core business logic from the data storage backend. For example, all methods can be completed by calling the corresponding methods of blob.Bucket (the bucket object for binary large object (Blob) storage services), thereby achieving "write once, run anywhere," allowing switching or adding data storage backends without modifying the code, greatly enhancing the portability and flexibility of the file management system. Therefore, the file management system can be deployed to different data storage backends using the Storage Abstraction Layer (Go CDK). For example, if the file management system is deployed to a local disk, the target data storage backend is the local disk; if the file management system is deployed to AWS S3, the target data storage backend is AWS S3; and so on.

[0040] S130 performs file management operations on the target file based on the data storage location in the target data storage backend, matching the operation request, through the storage abstraction layer.

[0041] As mentioned above, all methods can be completed by calling the corresponding methods of blob.Bucket. Therefore, for a write operation request, the file management device can call the Writer method of the storage abstraction layer (Go CDK), so that the storage abstraction layer (Go CDK) performs file management operations on the target file based on the data storage location in the target data storage backend (such as AWS S3) that match the write operation request; for a read operation request, the file management device can call the Reader method of the storage abstraction layer (Go CDK), so that the storage abstraction layer (Go CDK) performs file management operations on the target file based on the data storage location in the target data storage backend (such as AWS S3) that match the read operation request.

[0042] As can be seen, compared to the traditional general-purpose file system in related technologies, which is a "large and comprehensive" general-purpose scenario design, the file management system provided in this application embodiment is a lightweight design for specific application scenarios (such as embedded devices / IoT (Internet of Things) devices). For example, the number of integrated components (embedded database, metadata manager, and storage abstraction layer) is small and lightweight, and only necessary functions (such as metadata management and file management functions) are retained. Redundant general functions in traditional general-purpose file systems (such as complex logs, access control lists, disk quotas, etc.) are cut off, thereby greatly reducing performance overhead, resource consumption, and functional complexity.

[0043] As can be seen, compared to the traditional general-purpose file system in related technologies, which is a hierarchical architecture of "user mode → kernel mode", the file management system provided in this application is a user mode / library file system. For example, in this application, metadata is stored in an embedded database and file data is stored in a data storage backend that is connected to the storage abstraction layer (Go CDK). This allows the file management system to be embedded into the application as a library and to complete file management directly in user mode, thereby avoiding frequent user mode-kernel mode switching and improving the efficiency of high-frequency / small file operations.

[0044] In addition, with the evolution of hardware technology and application scenarios, traditional general file systems in related technologies are becoming increasingly difficult to adapt to emerging needs: (1) Resource and performance requirements for specific scenarios: In resource-constrained scenarios such as embedded devices and IoT devices, there are extremely low requirements for the memory usage and storage overhead of the file system; in cloud-native and microservice scenarios, container technology requires a lightweight file system that can be started quickly and supports snapshots and isolation; while scenarios such as databases and message queues rely on high-performance single-machine persistent storage capabilities.

[0045] (2) Big data and high-performance computing scenarios: In this scenario, there are extremely high standards for metadata operation performance and data read / write latency. The processing capabilities of traditional general file systems can easily become the performance bottleneck of the data processing process.

[0046] (3) Adaptation requirements of hardware technology iteration: SSD (Solid-State Drive) and NVMe (Non-Volatile Memory Express) hardware have become widely used. Their physical characteristics are fundamentally different from HDD (Hard Disk Drive). Traditional general file systems optimized for HDD cannot adapt to the characteristics of SSD / NVMe. New hardware such as persistent memory combines the high-speed characteristics of memory with the persistence of storage. Traditional general file systems cannot fully realize their performance potential.

[0047] (4) Adaptation limitations of software architecture evolution: In order to reduce development complexity and avoid kernel crash risk, FUSE (Filesystem in Userspace) has been applied, but this type of solution has inherent performance loss.

[0048] The file management system provided in this application embodiment can be further optimized for emerging scenarios, thereby incidentally solving the scalability problem of traditional general-purpose file systems. Specifically, the file management system in this application embodiment can adapt to new hardware (such as SSD / NVMe / persistent memory): the file management system no longer uses the design of traditional general-purpose file systems based on HDD optimization, but combines the physical characteristics of new hardware (such as SSD) (such as fast random read and write), and stores metadata and file data separately (for example: metadata is efficiently queried through an embedded database, and file data is adapted through object storage), thereby giving full play to the performance of new hardware; the file management system in this application embodiment can support lightweight / portable storage switching: the file management system, through the storage abstraction layer (Go CDK), allows data storage backends (such as local disks, AWS S3, etc.) to be flexibly switched, meeting the lightweight and fast startup requirements of cloud-native / container scenarios.

[0049] In summary, the file management system provided in this application, through "lightweight customization + user-space architecture + hardware / scenario adaptation", not only solves the redundancy problem of traditional general-purpose file systems, but also adapts to the needs of emerging scenarios.

[0050] Figure 2 is another flowchart illustrating the file management method of this application, which is applied to a file management system. The file management system consists of multiple lightweight components, including an embedded database, a metadata manager, and a storage abstraction layer. The embedded database is used to store the metadata of each file, the metadata manager is used to manage the metadata of each file, and the storage abstraction layer is used to interface with multiple data storage backends.

[0051] Referring to Figure 2, the file management method of this application may include: S210, when receiving an operation request from a client for a target file, extracting the attribute information of the target file from the operation request; the attribute information includes the operation type, which includes at least one of write operation and read operation.

[0052] In one embodiment, the embedded database includes at least one of an embedded relational database and an embedded key-value database, wherein the embedded relational database is used to store metadata that requires complex queries, and the embedded key-value database is used to store metadata of pure key-value pairs.

[0053] This application embodiment can use an embedded database as a metadata storage engine and operate it through a metadata manager (GORM). The embedded database can be flexibly selected between SQLite (embedded relational database) and BoltDB (embedded key-value database) according to the application scenario to adapt to different needs of complex queries or high-performance KV (key-value) operations.

[0054] SQLite is an open-source, lightweight, self-contained, serverless, zero-configuration embedded relational database. It stores the entire database (including definitions, tables, indexes, and data) as a single file on the host, eliminating the need for a separate database server process. It manages data by directly reading and writing disk files and is widely used in embedded devices, mobile applications, and small to medium-sized desktop / web applications as a local data storage solution.

[0055] BoltDB is an open-source, embedded, key-value store written in Go. It is similar to a standalone, serverless NoSQL database, storing all data in a local file and ensuring data consistency through ACID (A-Atomicity; C-Consistency; I-Isolation; D-Durability) transactions. Its simple and efficient design, using buckets for data grouping, makes it ideal as a local persistent storage or caching engine for Go applications.

[0056] Metadata may include, but is not limited to: file path, file size, modification time, permission flags, MIME type, and data reference identifiers pointing to data blocks (such as keys in object storage, paths in local files, etc.).

[0057] SQLite is suitable for storing metadata that requires complex queries (such as file description information filtered by multiple conditions) and supports operations such as association and statistics. For example, to filter all files that are "modified yesterday, larger than 100MB, and have a video MIME type", SQLite is suitable for scenarios that require querying by multiple conditions such as "modification time + size + MIME type" and may even require sorting / statistics.

[0058] BoltDB is suitable for storing pure key-value metadata (such as file path → data reference identifier mapping) and supports high-frequency read and write operations. For example, to quickly retrieve the corresponding data reference identifier based on a file path, only a direct mapping of "key (file path) → value (data reference identifier)" is needed, and this mapping relationship needs to be read / updated frequently and quickly, BoltDB is suitable for this type of scenario.

[0059] This application embodiment defines and manipulates the metadata model in an object-oriented manner using a Metadata Manager (GORM), avoiding the need for manual SQL writing. Specifically, this application embodiment can define a structure and use GORM tags to annotate the mapping relationship between metadata fields (such as file path, file size, etc.) and database columns. All CRUD operations on metadata can be completed through methods provided by the Metadata Manager (GORM). In addition, this application embodiment can also utilize GORM hooks to automatically update fields such as modification time when creating / updating metadata records, thereby significantly improving development efficiency, making the code clearer, more secure, and easier to maintain and extend.

[0060] This application embodiment enables flexible expansion of metadata through structure and serialization functions without altering the core architecture of the file management system. Specifically, this application embodiment allows adding custom fields (such as tags, comments, etc.) to the metadata structure. Utilizing the serialization function of the metadata manager (GORM), the entire metadata is serialized into JSON (JavaScript Object Notation, a lightweight data interchange format) or Protocol Buffers format and stored as a value in the embedded database. This allows for easy addition of new fields to the serialized data, supporting advanced features such as tags, comments, and ratings, meeting diverse file management needs while maintaining system architecture stability.

[0061] This application embodiment provides a consistent read and write method for upper-layer business logic through a unified interface of the Storage Abstraction Layer (Go CDK). During initialization, this interface dynamically binds to a specific data storage backend driver by parsing a configuration string (such as a URL), thereby achieving complete decoupling between business logic and the underlying physical storage. Traditional general-purpose file systems in related technologies require designing corresponding interfaces or functions based on the storage driver type. However, this application embodiment only needs to call the unified interface of the Storage Abstraction Layer (Go CDK) to automatically adapt to the corresponding data storage backend. For example, when writing data, the Storage Abstraction Layer (Go CDK) interface automatically writes the data to the configured data storage backend, without distinguishing between local and cloud storage.

[0062] This application embodiment constructs a lightweight and efficient file management system using components such as SQLite, BoltDB, GORM, and Go CDK. This allows both large and small projects that require file system integration to choose the construction method provided in this application embodiment. Because the construction method involves fewer components, and the related open-source components used are lightweight, and the data storage backend has been abstracted and adapted, the file management system constructed using this application embodiment has minimal impact on the overall project size.

[0063] This application embodiment can utilize various components within a file management system to complete the entire file management process (metadata management, data read / write operations). In a specific implementation, the client can send an operation request for a target file to the file management system. The file management device within the system parses the operation request at the interface layer, extracting the target file's attribute information. This attribute information may include at least one of the following: file path, operation type, user authentication credentials, and file size. The operation type may include at least one of write and read operations. The file management device can invoke the metadata manager (GORM) to query the user's permission records in the embedded database, thereby verifying the validity of the user's authentication credentials and whether the user has permission to perform the target operation (write / read operation).

[0064] S220 executes a metadata management operation matching the operation type in the embedded database through the metadata manager to obtain the metadata of the target file; the metadata includes data reference identifiers.

[0065] When the operation type is a write operation (such as upload / create), the metadata manager (GORM) can perform metadata management operations matching the write operation in the embedded database to obtain the metadata of the target file; when the operation type is a read operation (such as download / query), the metadata manager (GORM) can perform metadata management operations matching the read operation in the embedded database to obtain the metadata of the target file.

[0066] In one embodiment, the metadata of the target file is obtained by performing a metadata management operation matching the operation type in the embedded database through the metadata manager. This may include: starting a database transaction when the operation type is a write operation; generating a data reference identifier for the target file based on the file path; creating a target record in the embedded database through the metadata manager, and filling the target record with the data reference identifier and the file path as metadata of the target file; wherein the target record is in an uncommitted state in the database transaction.

[0067] For write operations (such as upload / create), to ensure consistency between metadata and file data, when the metadata storage engine (such as SQLite) supports transactions, this embodiment can encapsulate the metadata update operation within a database transaction. When the write operation involves remote storage that does not support transactions (such as AWS S3), this embodiment can adopt a mechanism of writing data first, then writing metadata, with failure compensation to ensure eventual consistency between metadata and file data. Specifically, the transaction processing involves placing file data writing and metadata updates within the same distributed transaction or database transaction for a coordinateable backend such as a local file system. The compensation mechanism involves performing the data upload operation first for remote object storage, updating metadata only after successful upload; if the metadata update fails, a cleanup process is automatically triggered to delete the uploaded data object (i.e., file data) to prevent data leakage.

[0068] Taking uploading a target file as an example, the write operation process is explained as follows: The target file's attribute information may include the file path (e.g., / docs / report.pdf), operation type ("upload"), user authentication credentials (e.g., Token), file size, etc. The file management device calls the metadata manager (GORM) to query the user's permission records in the embedded database (e.g., SQLite) to verify whether the user authentication credentials (Token) are valid and whether the user has permission to upload the target file under the / docs / path. If the verification is successful, a database transaction is started to avoid inconsistencies between metadata and file data. Then, based on the file path and the configured target data storage backend (e.g., AWS), the data is uploaded. S3) Generates a data reference identifier for the target file (e.g., storage: / / s3 / docs / report_20260112.pdf). Then, the metadata manager (GORM) creates a target record in the embedded database (SQLite) and populates the target record with the data reference identifier (storage: / / s3 / docs / report_20260112.pdf), file path ( / docs / report.pdf), file size, MIME type (automatically identified as application / pdf), and current modification time as metadata of the target file. At this time, the target record is in an uncommitted state in the database transaction.

[0069] In another embodiment, the metadata of the target file is obtained by performing a metadata management operation matching the operation type in the embedded database through the metadata manager. This may include: when the operation type is a read operation, constructing a query statement based on the file path through the metadata manager to obtain the metadata of the target file from the embedded database using the query statement.

[0070] For read operations (such as download / query), the file management device can directly construct a query statement based on the file path through the metadata manager (GORM) so that the query statement can be used as a database index to quickly retrieve the metadata of the target file in the embedded database.

[0071] The following describes the read operation process using the download of a target file as an example: The target file's attribute information may include the file path (e.g., / docs / report.pdf), operation type ("download"), user authentication credentials (e.g., Token), file size, etc. The file management device calls the metadata manager (GORM) to query the user's permission records in the embedded database (e.g., SQLite), verifying whether the user's authentication credentials (Token) are valid and whether the user has permission to download the target file under the / docs / path. If the verification is successful, the metadata manager (GORM) constructs a query statement based on the file path. The query statement can be, for example, "WHERE path=' / docs / report.pdf'", so that the query statement can be used as a database index to quickly retrieve the target file's metadata in the embedded database (SQLite). The target file's metadata may include, but is not limited to: file path ( / docs / report.pdf), file size, modification time, permission flags, MIME type, and data reference identifier (storage: / / s3 / docs / report_20260112.pdf).

[0072] S230, parse the data reference identifier to obtain the data storage location; the data storage location is used to indicate the storage location of the target file in the target data storage backend, and the target data storage backend is one of multiple data storage backends.

[0073] This step can be found in the description in S120, and will not be repeated here.

[0074] S240 performs file management operations on the target file based on the data storage location in the target data storage backend, matching the operation request, through the storage abstraction layer.

[0075] This step can be found in the description in S130, and will not be repeated here.

[0076] In one embodiment, performing a file management operation on the target file in the target data storage backend based on the data storage location through the storage abstraction layer, matching the operation request, may include: writing the target file into the target data storage backend based on the data storage location through the storage abstraction layer.

[0077] Continuing with the example of uploading a target file: by parsing the target file's data reference identifier (storage: / / s3 / docs / report_20260112.pdf), the data storage location (e.g., report_20260112.pdf) in the target data storage backend (AWS S3) is obtained. The file management device can then call the Writer method of the storage abstraction layer (Go CDK), passing in the data storage location (report_20260112.pdf), so that the target file can be uploaded to the data storage location (report_20260112.pdf) in the target data storage backend (AWS S3) through the storage abstraction layer (Go CDK).

[0078] In one embodiment, the method may further include: committing a database transaction to update the status of the target record in the database transaction to a committed status; returning a first response message to the client; the first response message being used to indicate that the target file was successfully written.

[0079] After the target file is written to the data storage location of the target data storage backend, the file management device can commit the database transaction that was opened above, so that the previously temporarily stored target record can be formally written into the embedded data (such as SQLite). At this time, the status of the target record in the database transaction is updated from uncommitted to committed, thereby ensuring the consistency between the target file's metadata and the file data.

[0080] The file management device can send an initial response message to the client, allowing the user to clearly know that the target file has been successfully written. Afterward, all query requests based on file paths or attributes (such as listing directories or searching for files) will be directly processed by the embedded database (SQLite / BoltDB), which quickly returns results through its efficient indexing, without needing to traverse the physical storage directory.

[0081] In another embodiment, performing file management operations on the target file in the target data storage backend based on the data storage location through the storage abstraction layer, which matches the operation request, may include: reading the target file from the target data storage backend based on the data storage location through the storage abstraction layer.

[0082] Continuing with the example of downloading the target file: by parsing the target file's data reference identifier (storage: / / s3 / docs / report_20260112.pdf), the data storage location (e.g., report_20260112.pdf) in the target data storage backend (AWS S3) is obtained. The file management device can then call the Reader method of the Storage Abstraction Layer (Go CDK), passing in the data storage location (report_20260112.pdf), so that the target file can be downloaded from the data storage location (report_20260112.pdf) in the target data storage backend (AWS S3) through the Storage Abstraction Layer (Go CDK).

[0083] In another embodiment, the method may further include: returning a second response message to the client; the second response message is used to indicate that the target file was successfully read.

[0084] The file management device can send a second response message to the client, allowing the user to clearly know that the target file has been successfully read. Afterward, all query requests based on file paths or attributes (such as listing directories or searching for files) will be directly processed by the embedded database (SQLite / BoltDB), which quickly returns results through its efficient indexing, without needing to traverse the physical storage directory.

[0085] As can be seen from the two examples above (upload process and download process), the embodiments of this application achieve powerful cross-platform capabilities, excellent performance and reliable data consistency under a lightweight architecture by precisely coordinating "metadata operation first and placed in transaction" and "storage abstraction layer (Go CDK) abstracting data read and write". The upload process is finally atomically synchronized through "transaction commit".

[0086] Furthermore, the embodiments of this application can optimize data storage before and after storage to improve efficiency and save space. For example, before writing file data, the embodiments of this application use an efficient compression algorithm to compress the file data and encrypt it; after reading the file data, the embodiments of this application decrypt it. The key is managed uniformly by the file management system. For large files, the embodiments of this application can implement block-based upload / download logic, transparent to upper-layer services. Therefore, it not only saves storage space and enhances data security, but also improves the large file transfer experience.

[0087] It should be noted that the goal of this application is not to replace traditional general-purpose file systems (such as ext4 and NTFS), but to solve the imbalance between performance overhead, resource consumption and functional complexity of traditional general-purpose file systems in specific application scenarios, so as to achieve a lighter and more efficient system: 1. Improve performance and reduce latency, eliminate the redundant overhead of traditional general-purpose file systems, and achieve extreme read and write speeds and extremely low latency: (1) Database applications (such as SQLite and message queues): Traditional general-purpose file systems are hierarchical architectures of "user mode → kernel mode". Every disk interaction has to go through the kernel path (the kernel path is long). At the same time, the kernel will lock the file. During multi-threaded operations, lock contention is likely to occur (such as multiple requests competing for the lock, resulting in waiting). SQLite and message queues require extremely fast disk read and write (such as the high throughput of message queues and the high frequency of SQLite transactions). Traditional general-purpose file systems have become performance bottlenecks due to the excessively long kernel path and lock contention. In this application, a lightweight and efficient file management system is provided, characterized by "user-space file system + local database (SQLite / BoltDB) for storing metadata". The file management system can perform operations directly in user space, skipping the kernel path and reducing lock contention. Therefore, it can meet the extremely fast disk interaction requirements of database applications such as SQLite and message queues.

[0088] (2) High-frequency small file operations (such as application logs and cache files): The metadata of traditional general-purpose file systems is stored in the kernel's directory structure. When creating / deleting small files frequently, the kernel metadata needs to be frequently manipulated, which is slow. However, application logs and cache files require extremely high metadata operation performance, which cannot be met by the metadata management efficiency of traditional general-purpose file systems. In this application embodiment, the metadata is stored in an embedded database (SQLite / BoltDB), and the high-efficiency query / update capabilities of the embedded database are used to manage the metadata. Therefore, the extremely high metadata operation requirements of high-frequency small files such as application logs and cache files can be met.

[0089] (3) New hardware (such as SSD, NVMe, persistent memory): Traditional general-purpose file systems are designed for HDDs. HDDs are fast for sequential read / write but slow for random read / write. Therefore, traditional general-purpose file systems have algorithms such as prefetching, caching, and head scheduling. However, SSDs / NVMe are fast for random read / write and do not require head scheduling. Persistent memory is as fast as RAM. Therefore, the algorithms designed for HDDs by traditional general-purpose file systems (such as redundant prefetching) will waste the performance of new hardware. In this application, the embodiments are tailored to the characteristics of new hardware such as SSD, NVMe, and persistent memory. For example, the prefetching algorithm of HDD is removed, and "metadata + file data separation storage" is used to adapt to the random read / write advantages of new hardware, thus giving full play to the performance of new hardware.

[0090] 2. Reduce resource consumption and run stably in resource-constrained environments, leaving more CPU, memory and storage space for the application itself: (1) Embedded devices and IoT devices: weak CPU computing power, small memory, limited Flash storage space and lifespan are critical. In this regard, the embodiments of this application provide a lightweight and efficient file management system that can reduce memory consumption and optimize writing strategies to extend Flash lifespan.

[0091] (2) Container environment: The container image should be as small as possible and the startup should be as fast as possible. In this regard, the embodiments of this application provide a lightweight and efficient file management system, which can accelerate container startup and reduce resource quotas.

[0092] 3. Simplify design and enhance reliability. By reducing unnecessary functions, the system complexity is reduced, thereby improving reliability and maintainability: (1) Function tailoring: Many scenarios do not require complex logging, access control lists, or disk quotas. In this regard, the embodiments of this application reduce the amount of code by tailoring redundant functions (such as complex logging), thereby reducing the risk of potential bugs and attack surfaces.

[0093] (2) User-mode implementation: The file management system provided in this application embodiment is implemented in user mode based on FUSE or library, which makes development and debugging simpler and prevents the entire kernel from crashing.

[0094] 4. Implement application-specific optimizations, no longer a "one-size-fits-all" general solution, but provide the best data organization and access mode for specific workloads: (1) Key-value storage: In this application embodiment, pure KV type metadata is stored through BoltDB, which is more efficient than storing KV in traditional general file systems (such as ext4).

[0095] (2) Object storage access: This application embodiment uses the storage abstraction layer (Go CDK) to enable applications to operate cloud storage as seamlessly as local files, which greatly simplifies multi-cloud and hybrid cloud architectures.

[0096] (3) Data model matching: In this application embodiment, the metadata that requires complex queries is stored in SQLite so that the powerful query capabilities of SQL can be used to quickly find files, which is more efficient than traversing the file directory in the traditional general file system.

[0097] 5. Improve portability and deployment convenience, making it easier to migrate and deploy applications and their data storage across platforms and environments: (1) Single file (such as SQLite / BoltDB file) deployment: This application embodiment facilitates backup, copying and moving by including all data in a single file.

[0098] (2) Environment independence: The file management system provided in this application does not depend on the specific file system characteristics of the host machine and its behavior remains consistent on any operating system that supports the library.

[0099] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a file management device, system, and corresponding embodiments.

[0100] Figure 3 is a schematic diagram of the structure of a file management device shown in an embodiment of this application. It is applied to a file management system. The file management system consists of multiple lightweight components, including an embedded database, a metadata manager, and a storage abstraction layer. The embedded database is used to store the metadata of each file, the metadata manager is used to manage the metadata of each file, and the storage abstraction layer is used to interface with multiple data storage backends.

[0101] Referring to Figure 3, a file management device 300 provided in this application may include: a metadata management module 310, used to obtain the metadata of the target file by performing a metadata management operation matching the operation request in an embedded database through a metadata manager when receiving an operation request from a client for a target file; the metadata includes a data reference identifier; a parsing module 320, used to parse the data reference identifier to obtain the data storage location; the data storage location is used to indicate the storage location of the target file in the target data storage backend, and the target data storage backend is one of multiple data storage backends; and a file management module 330, used to perform a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location through a storage abstraction layer.

[0102] In one embodiment, the embedded database includes at least one of an embedded relational database and an embedded key-value database, wherein the embedded relational database is used to store metadata that requires complex queries, and the embedded key-value database is used to store metadata of pure key-value pairs.

[0103] In one embodiment, the metadata management module 310 may include: an attribute information extraction submodule, used to extract attribute information of the target file from the operation request when a client's operation request for the target file is received; the attribute information includes the operation type, which includes at least one of write operation and read operation; and a metadata management submodule, used to perform a metadata management operation matching the operation type in the embedded database through a metadata manager to obtain the metadata of the target file.

[0104] In one embodiment, the attribute information further includes a file path; the metadata management submodule may include: a transaction initiation unit, used to initiate a database transaction when the operation type is a write operation; an identifier generation unit, used to generate a data reference identifier for the target file based on the file path; and a metadata filling unit, used to create a target record in the embedded database through the metadata manager, and fill the target record with the data reference identifier and the file path as metadata of the target file; wherein, the target record is in an uncommitted state in the database transaction.

[0105] In another embodiment, the attribute information also includes the file path; the metadata management submodule may include: a metadata acquisition unit, used to construct a query statement based on the file path through the metadata manager when the operation type is a read operation, so as to obtain the metadata of the target file from the embedded database using the query statement.

[0106] In one embodiment, the file management module 330 may include: a file writing submodule, used to write the target file to the target data storage backend based on the data storage location through the storage abstraction layer; the device may also include: a status update module, used to commit a database transaction to update the status of the target record in the database transaction to a committed status; and a first response information return module, used to return first response information to the client; the first response information is used to indicate that the target file was successfully written.

[0107] In another embodiment, the file management module 330 may include: a file reading submodule, used to read a target file from the target data storage backend based on the data storage location through the storage abstraction layer; the device may also include: a second response information return module, used to return second response information to the client; the second response information is used to indicate that the target file was read successfully.

[0108] As this example demonstrates, the file management system provided in this application is a lightweight design tailored to specific application scenarios. For instance, the number of integrated components (embedded database, metadata manager, and storage abstraction layer) is small and lightweight, retaining only essential functions (such as metadata management and file management). Redundant general-purpose functions found in traditional file systems (such as complex logs, access control lists, and disk quotas) are removed, thereby significantly reducing performance overhead, resource consumption, and functional complexity. Furthermore, metadata is stored in the embedded database, and file data is stored in the data storage backend connected to the storage abstraction layer. This allows the file management system to be embedded into the application as a library, directly performing file management in user space, thus avoiding frequent user-mode-kernel mode switching and improving the efficiency of high-frequency / small file operations.

[0109] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0110] Figure 4 is a schematic diagram of the structure of the file management system shown in an embodiment of this application.

[0111] Referring to Figure 4, the file management system 400 includes: an embedded database 410, a metadata manager 420, a storage abstraction layer 430, and a file management device 300 as described above.

[0112] The embedded database 410 includes at least one of an embedded relational database and an embedded key-value database. The embedded relational database is used to store metadata that requires complex queries, while the embedded key-value database is used to store metadata that is purely key-value pairs. The metadata manager 420 is used to manage the metadata stored in the embedded database 410. Therefore, the embedded database 410 and the metadata manager 420 belong to the metadata management domain.

[0113] Among them, the storage abstraction layer 430 is used to connect to data storage backend 1 to data storage backend N (N is a positive integer). Data storage backend 1 to data storage backend N are all used to store file data. Therefore, the storage abstraction layer 430 belongs to the file data management domain.

[0114] As this example demonstrates, the file management system 400 provided in this application is a lightweight design tailored to specific application scenarios. For instance, the integrated components (embedded database 410, metadata manager 420, and storage abstraction layer 430) are few and lightweight, retaining only essential functions (such as metadata management and file management), while eliminating redundant general-purpose functions found in traditional file systems (such as complex logs, access control lists, and disk quotas), thereby significantly reducing performance overhead, resource consumption, and functional complexity. Furthermore, metadata is stored in the embedded database 410, and file data is stored in the data storage backend connected to the storage abstraction layer 430. This allows the file management system 400 to be embedded into applications as a library, directly performing file management in user space, thus avoiding frequent user-mode to kernel-mode switching and improving the efficiency of high-frequency / small file operations.

[0115] Regarding the system in the above embodiments, the specific manner in which each component performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0116] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0117] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or a computer program or computer instruction code) that, when executed by a file management device of a file management system, causes the file management device to perform part or all of the steps of the methods described above according to this application.

[0118] This application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the method described above.

[0119] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A file management method, characterized in that, An application is provided in a file management system, which comprises multiple lightweight components, including an embedded database, a metadata manager, and a storage abstraction layer. The embedded database stores metadata for each file, the metadata manager manages the metadata of each file, and the storage abstraction layer interfaces with multiple data storage backends. The method includes: when a client requests an operation on a target file, the metadata manager executes a metadata management operation matching the operation request in the embedded database to obtain the target file's metadata; the metadata includes a data reference identifier; the data reference identifier is parsed to obtain the data storage location; the data storage location indicates the storage location of the target file in a target data storage backend, which is one of the multiple data storage backends; and the storage abstraction layer executes a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location.

2. The method according to claim 1, characterized in that, The embedded database includes at least one of an embedded relational database and an embedded key-value database. The embedded relational database is used to store metadata that requires complex queries, and the embedded key-value database is used to store metadata that is purely key-value pairs.

3. The method according to claim 1, characterized in that, When a client's operation request for a target file is received, the metadata manager performs a metadata management operation matching the operation request in the embedded database to obtain the metadata of the target file. This includes: extracting the attribute information of the target file from the operation request; the attribute information includes an operation type, which includes at least one of a write operation and a read operation; and performing a metadata management operation matching the operation type in the embedded database through the metadata manager to obtain the metadata of the target file.

4. The method according to claim 3, characterized in that, The attribute information also includes a file path; the step of obtaining the target file's metadata by performing a metadata management operation matching the operation type in the embedded database through the metadata manager includes: when the operation type is a write operation, starting a database transaction; generating a data reference identifier for the target file based on the file path; creating a target record in the embedded database through the metadata manager, and filling the target record with the data reference identifier and the file path as the target file's metadata; wherein, the target record's status in the database transaction is an uncommitted state.

5. The method according to claim 3, characterized in that, The attribute information also includes the file path; the step of obtaining the target file's metadata by performing a metadata management operation matching the operation type in the embedded database through the metadata manager includes: when the operation type is the read operation, constructing a query statement based on the file path through the metadata manager, and using the query statement to obtain the target file's metadata from the embedded database.

6. The method according to claim 4, characterized in that, The step of performing a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location through the storage abstraction layer includes: writing the target file into the target data storage backend based on the data storage location through the storage abstraction layer; the method further includes: committing the database transaction to update the status of the target record in the database transaction to a committed status; returning a first response information to the client; the first response information is used to indicate that the target file was successfully written.

7. The method according to claim 5, characterized in that, The step of performing a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location through the storage abstraction layer includes: reading the target file from the target data storage backend based on the data storage location through the storage abstraction layer; the method further includes: returning a second response information to the client; the second response information is used to indicate that the target file was successfully read.

8. A document management device, characterized in that, An application is provided in a file management system, which comprises multiple lightweight components, including an embedded database, a metadata manager, and a storage abstraction layer. The embedded database stores metadata for each file, the metadata manager manages the metadata of each file, and the storage abstraction layer interfaces with multiple data storage backends. The device includes: a metadata management module, used to, upon receiving a client's operation request for a target file, execute a metadata management operation matching the operation request in the embedded database through the metadata manager to obtain the metadata of the target file; the metadata includes a data reference identifier; a parsing module, used to parse the data reference identifier to obtain the data storage location; the data storage location indicates the storage location of the target file in a target data storage backend, which is one of the multiple data storage backends; and a file management module, used to, through the storage abstraction layer, execute a file management operation matching the operation request on the target file in the target data storage backend based on the data storage location.

9. A document management system, comprising: An embedded database, a metadata manager, a storage abstraction layer, and a file management device as described in claim 8.

10. A computer-readable storage medium having executable code stored thereon, which, when executed by a file management device of a file management system, causes the file management device to perform the method as described in any one of claims 1-7.

Images