Object protocol access method and device of file system, equipment, medium and product

Abstract

The invention provides an object protocol access method and device for a file system, equipment, a medium and a product, and belongs to the technical field of artificial intelligence and data storage, and the method comprises the steps: receiving a first access request based on an object protocol through an object protocol layer; analyzing the first access request through the request access layer to obtain an analysis result; through an actuator layer, determining a target file system accessed by the first access request based on the routing identification information, converting the first access request into a second access request based on a target storage protocol based on binary storage context information and virtual directory mapping, and determining a target protocol adapter corresponding to the target file system, and accessing the target file system through a standard file operation interface of the target protocol adapter, and executing the second access request. According to the invention, based on a single-storage multi-entry architecture, the access of the object protocol to the file system is realized through the analysis, protocol conversion and routing of the access request by the protocol gateway.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data storage technology, and in particular to a method, apparatus, device, medium and product for accessing object protocols in a file system. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely applied in various fields. File systems and object storage systems are used to store the massive amounts of data required for model training (including large models) on the training side and for model inference on the business side, respectively. In scenarios where large-scale model training and business systems run in parallel, the training side typically uses file protocols to access the storage system, while the business side typically uses object protocols. However, traditional file systems usually only support file protocol access and do not support object protocol access. Summary of the Invention

[0003] This application provides a method, apparatus, device, medium, and product for accessing object protocols in a file system, aiming to solve the problem that file systems do not support object protocol access.

[0004] In a first aspect, this application provides an object protocol access method for a file system, applied to a protocol gateway, the protocol gateway including an object protocol layer, a request access layer, and an executor layer; The object protocol access method includes: The object protocol layer receives the first access request based on the object protocol. The first access request is parsed by the request access layer to obtain the parsing result, which includes routing identification information, binary storage context information, and virtual directory mapping of the target file system to be accessed by the first access request. Through the executor layer, the target file system accessed by the first access request is determined based on the routing identification information. Based on the binary storage context information and the virtual directory mapping, the first access request is converted into a second access request based on the target storage protocol. The target protocol adapter corresponding to the target file system is determined. The target file system is accessed through the standard file operation interface of the target protocol adapter, and the second access request is executed. The target storage protocol is a storage protocol supported by the target file system, the target protocol adapter adopts the target storage protocol, and the object protocol used by the first access request is different from the target storage protocol.

[0005] In one embodiment, the binary storage context information includes first metadata and the raw byte stream from the first access request; The step of converting the first access request into a second access request based on the target storage protocol specifically includes: Determine the protocol conversion rules between the object protocol and the target storage protocol, wherein the protocol conversion rules record all parameters required for protocol conversion; Based on the protocol conversion rules, the first metadata in the binary storage context information is mapped to the request header of the target storage protocol, and the original byte stream in the binary storage context information is mapped to the request body of the target storage protocol; The request header, the request body, and the virtual directory mapping are organized according to the structure of the target storage protocol to obtain the second access request.

[0006] In one embodiment, the protocol gateway further includes a storage protocol layer, the storage protocol layer including at least one file system, the at least one file system including the target file system; the at least one file system including the target file system; the storage protocol layer manages plugins for the protocol adapter corresponding to each file system; The object protocol access method further includes: After determining the target protocol adapter corresponding to the target file system through the executor layer, the storage protocol layer dynamically loads the plugin of the target protocol adapter into the executor layer.

[0007] In one embodiment, the protocol gateway further includes a metadata storage layer, which includes a main layer and a backup layer; The object protocol access method further includes: After obtaining the parsing result, the second metadata of the first access request is synchronously stored in the main layer and backup layer of the metadata storage layer. The second metadata includes the creation time, modification time, permission information, and parsing result of the first access request. Accessing the second metadata specifically includes: In response to a read request, the second metadata required by the read request is read from the main layer; If the primary layer returns an error or access to the primary layer times out, the second metadata required for the read request is read from the backup layer.

[0008] In one embodiment, the request access layer parses the first access request to obtain the binary storage context information, specifically including: Perform signature verification and authentication on the first access request; If the signature verification and authentication pass, extract the operation type from the query string in the Uniform Resource Identifier of the first access request; The storage method corresponding to the operation type is determined based on the mapping table between operation type and storage method; Extract the first metadata and the original byte stream from the request body of the first access request; Based on the storage method, the first metadata and the original byte stream are encapsulated into the binary storage context information, which includes data attributes required by the object protocol and file protocol.

[0009] In one embodiment, authenticating the first access request specifically includes: Map the Uniform Resource Identifier of the first access request to the virtual directory mapping, and use the virtual directory mapping as part of the resolution result; Determine the project identifier and tenant identity bound to the virtual directory mapping; Extract the access key identifier from the request header of the first access request; Query the user database to determine the user identity corresponding to the access key identifier, and determine whether the user identity has access rights to the project identifier under the tenant identity; If the user identity has access rights to the project identifier under the tenant identity, the authentication of the first access request is determined to be successful.

[0010] In one embodiment, updating the protocol adapter specifically includes: After changes to the plugin code or configuration of the protocol adapter, reload the protocol adapter without restarting the process of the protocol gateway.

[0011] In one embodiment, updating the protocol conversion rule specifically includes: Upon receiving an incremental configuration task for the protocol conversion rule, the protocol conversion rule is reloaded without restarting the protocol gateway process.

[0012] Secondly, this application also provides an object protocol access device for a file system, applied to a protocol gateway, the protocol gateway including an object protocol layer, a request access layer and an executor layer; The object protocol access device includes: The receiving module is configured to receive a first access request based on the object protocol through the object protocol layer; The parsing module is used to parse the first access request through the request access layer to obtain the parsing result, which includes routing identification information, binary storage context information, and virtual directory mapping of the target file system to be accessed by the first access request. The protocol conversion module is used to determine the target file system accessed by the first access request based on the routing identification information through the executor layer, convert the first access request into a second access request based on the target storage protocol based on the binary storage context information and the virtual directory mapping, determine the target protocol adapter corresponding to the target file system, access the target file system through the standard file operation interface of the target protocol adapter, and execute the second access request. The target storage protocol is a storage protocol supported by the target file system, the target protocol adapter adopts the target storage protocol, and the object protocol used by the first access request is different from the target storage protocol.

[0013] Thirdly, this application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned file system object protocol access methods.

[0014] Fourthly, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-mentioned file system object protocol access methods.

[0015] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described file system object protocol access methods.

[0016] This application provides a method, apparatus, device, medium, and product for accessing a file system via an object protocol. Based on a single-storage, multi-entry architecture, it receives access requests from the inference-side business system through the object protocol layer of a protocol gateway, parses the access requests through a request access layer, and performs protocol conversion and routing of the access requests through an executor layer. This enables access to the file system via the object protocol, while the training side accesses the file system via the file protocol. Thus, the file protocol and the object protocol can concurrently and transparently access data entities in the same file system, meeting the dual-protocol access requirements for large-scale model training and parallel operation of business systems. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a dual-protocol interoperability access model in a large-scale application scenario; Figure 2 This is a flowchart illustrating the object protocol access method for the file system provided in this application; Figure 3 This is a schematic diagram illustrating the working principle of the protocol gateway provided in this application; Figure 4 This is a schematic diagram of the process provided in this application for parsing the first access request through the request access layer to obtain binary storage context information; Figure 5 This is a flowchart illustrating the authentication process for the first access request provided in this application; Figure 6 This is a schematic diagram of the process provided in this application for converting a first access request into a second access request based on the target storage protocol; Figure 7 This is a schematic diagram of the structure of the object protocol access device for the file system provided in this application; Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0021] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0022] In scenarios where large-scale model training and business systems run in parallel, the training side typically relies on high-throughput file protocols (such as Network File System (NFS) or Portable Operating System Interface (POSIX)), while the business side tends to use object protocols (such as S3) to achieve fine-grained file access and tenant permission isolation.

[0023] Traditional file systems typically only support file protocol access and not object protocol access. Existing technologies address this issue using a "dual storage system + data synchronization" architecture, where the file system and object storage system store data based on the file protocol and object protocol respectively. Clients can write data to the file system using the file protocol and simultaneously upload data to the object storage using the object protocol. A bidirectional synchronization engine ensures that changes on either end are instantly reflected on the other. However, this architecture requires storing data on two different types of storage media, leading to increased costs and resource overhead, latency from data replication and synchronization, conflicts from synchronized modifications, increased complexity in troubleshooting, and reduced reliability.

[0024] Based on the above, this application provides a method, apparatus, device, medium, and product for accessing object protocols in a file system. It should be noted that this application is applicable to file systems that need to process large-scale data files, support multiple access protocols, and have fine-grained permission control capabilities. This file system is suitable for scenarios involving the separate training and inference of all models (including large models), as well as concurrent training and inference, and is particularly suitable for file storage and business data management scenarios related to the training, inference, and data processing of large artificial intelligence models.

[0025] The following is combined with Figures 1 to 8 This application describes the object protocol access methods, apparatus, devices, media, and products for file systems.

[0026] Figure 1This is a schematic diagram of a dual-protocol interoperability access model for a large-scale application scenario provided in this application. (Example:) Figure 1 As shown, this application constructs a unified protocol gateway on the business system side of the large model. The protocol gateway is responsible for object protocol parsing, data routing, protocol conversion, data forwarding, and file system (…). Figure 1 It provides core functions such as access to the storage system (as shown in the diagram), acting as an adaptation layer between the object protocol and the traditional file system. Meanwhile, the training side of the large model can directly access the file system via the file protocol during model training, thus meeting the dual-protocol access requirements for parallel operation of the large model training side and the business side.

[0027] It should be noted that the protocol gateway can receive access requests from multiple business systems and can also interface with multiple file systems, allowing different business systems to send access requests using different object protocols, while also supporting different file system storage protocols.

[0028] In one possible implementation, the protocol gateway abstractly models the data flow process. Specifically, the protocol gateway includes five functional layers: an object protocol layer, a request access layer, an executor layer, a storage protocol layer, and a metadata storage layer. Each layer undertakes specific responsibilities, thereby achieving tenant isolation, protocol decoupling, and efficient data forwarding. The following will describe these functional layers in detail using the object protocol access method of the file system.

[0029] Based on the above, this application also provides a method and apparatus for accessing file systems using object protocols in a protocol gateway. It should be noted that the file system object protocol access method provided in this application is implemented based on a file system object protocol access apparatus. This method is based on a single-storage, multi-entry architecture. It receives access requests from the inference-side business system through the object protocol layer of the protocol gateway, parses the access requests through the request access layer, and performs protocol conversion and routing of the access requests through the executor layer. This enables object protocol access to the file system. Simultaneously, the training side accesses the file system through the file protocol. Therefore, the file protocol and object protocol can concurrently and transparently access data entities in the same file system, meeting the dual-protocol access requirements for large-scale model training and parallel operation of business systems.

[0030] This application describes the object protocol access method of a file system using a file system object protocol access device as the execution subject as an example.

[0031] Figure 2 This is one of the flowcharts illustrating the object protocol access method for the file system provided in this application. For example... Figure 2 As shown, the file system object protocol access method provided in this application includes: S210: Receive a first access request based on the object protocol through the object protocol layer.

[0032] In the field of model training and inference, object protocols are a unified interface specification for storing and accessing binary objects such as large-scale datasets, model files, logs, and checkpoints. During large model inference, business systems obtain the data required for inference from the storage system through access requests based on object protocols, and then perform business inference based on this data. The protocol gateway receives the first access request based on the object protocol from the business system through the unified file operation interface of the object protocol layer. The first access request indicates access to one of the target file systems.

[0033] In one possible implementation, the object protocol layer provides an S3-compliant Representational State Transfer (REST) ​​interface for external business systems. Business systems can access the storage service simply by using the standard S3 Software Development Kit (SDK).

[0034] As a result, business systems can access storage services using a unified file operation interface without needing to be aware of the details of the underlying storage protocol, thus achieving loose coupling between upper-layer business and lower-layer implementation.

[0035] S220: The first access request is parsed by the request access layer to obtain a parsing result. The parsing result includes routing identification information, binary storage context information, and a virtual directory mapping of the target file system that the first access request needs to access. The virtual directory mapping is the path mapping information of the target file system.

[0036] Specifically, the access layer is responsible for parsing access requests (such as S3 requests) entering the protocol gateway, including authentication, tenant-level isolation, and parsing of data to be transmitted. Please refer to [link / reference needed]. Figure 3 The parsing result of the access layer request includes routing identification information, operation type, binary storage context information, and the virtual directory mapping of the target file system that the first access request needs to access.

[0037] The routing identification information includes bucket information, tenant information, project information, user metadata or object key prefixes, etc., and the routing identification information can be determined based on one or more of these.

[0038] Based on this, the request access layer is also used to route the parsing result to the executor in the executor layer corresponding to the operation type based on the operation type. Please refer to [link / reference]. Figure 3 The request routing steps are described in the text. Different executors are used to handle access requests of different operation types.

[0039] In one possible implementation, the request access layer uses Jetty as an embedded web container. Jetty, with its lightweight characteristics and high configurability, is suitable for resource-constrained scenarios or scenarios requiring flexible deployment. Jetty's asynchronous processing model enables non-blocking input / output, thus maintaining good throughput performance in high-concurrency environments.

[0040] S230: Through the executor layer, the target file system to be accessed by the first access request is determined based on the routing identification information. Based on the binary storage context information and the virtual directory mapping, the first access request is converted into a second access request based on the target storage protocol. The target protocol adapter corresponding to the target file system is determined. The target file system is accessed through the standard file operation interface of the target protocol adapter, and the second access request is executed.

[0041] The target storage protocol is a storage protocol supported by the target file system, the target protocol adapter adopts the target storage protocol, and the object protocol used by the first access request is different from the target storage protocol.

[0042] Specifically, the executor layer is responsible for interfacing with the underlying storage protocol, parsing the protocol, mapping protocol conversion rules, and forwarding data. Please refer to [reference needed]. Figure 3 All business requests parsed by the access layer are then processed by an executor in the executor layer to perform protocol parsing and conversion, complete write, read, or metadata operations on the target file system, and ultimately return a unified response. The executor layer is also responsible for managing metadata in the metadata storage layer.

[0043] The executor layer uses a unified file operation interface (such as POSIX-like) to receive data from the access layer, thereby shielding the differences in storage protocols between different file systems and enabling the upper-layer logic to perform file read and write operations in a unified manner.

[0044] Furthermore, the executor layer can load multiple protocol adapters. Different protocol adapters employ standard file operation interfaces for different storage protocols, and each protocol adapter is used to communicate with a file system that supports the same storage protocol. Specifically, the executor layer determines the storage protocol supported by the target file system based on the access request from the business system, selects the protocol adapter corresponding to that storage protocol, and then routes the protocol-converted access request to the target file system through the standard file operation interface of that protocol adapter, executing the second access request, thereby enabling the business system to access the target file system.

[0045] This application embodiment is based on a single-storage, multi-entry architecture. It receives access requests from the inference-side business system through a unified protocol gateway's object protocol layer, parses these requests through a request access layer, and performs protocol conversion and routing through an executor layer. This enables the object protocol to access the file system, while the training side accesses the file system through a file protocol. Thus, the file protocol and object protocol can concurrently and transparently access data entities within the same file system, meeting the dual-protocol access requirements for large-scale model training and parallel operation of business systems. This also avoids the adverse effects of data synchronization between dual storage systems, significantly reducing storage costs and synchronization overhead, and improving the data utilization efficiency of large models. Furthermore, because the protocol gateway decouples the object protocol and file protocol, it significantly reduces the coupling of file system access, maintaining its lightweight and pluggable characteristics, thereby improving system deployment flexibility and operational efficiency.

[0046] The existing storage architecture with a "unified namespace" consists of a unified metadata service layer and a unified data storage layer. The metadata service layer supports both file system directory tree structures and flat object naming conventions. Data blocks in the data storage layer are stored in a distributed object storage pool or block storage pool, independent of the access protocol. However, this architecture requires multi-protocol support to be considered in the underlying design phase of the storage system. Since different storage systems implement this differently, the storage system in this architecture is deeply coupled with specific storage systems, limiting business flexibility and scalability.

[0047] Based on the above considerations, in one possible implementation, the protocol gateway further includes a storage protocol layer, which includes at least one file system, including the target file system. The storage protocol layer manages plugins for the protocol adapters corresponding to each file system. Different protocol adapters can access the file systems via SDK, REST API, or native S3 interface.

[0048] The object protocol access method further includes: After determining the target protocol adapter corresponding to the target file system through the executor layer, the storage protocol layer dynamically loads the plugin of the target protocol adapter into the executor layer.

[0049] Specifically, the storage protocol layer integrates various underlying storage systems such as commercial storage, open-source storage, cloud-native object storage, and traditional file systems.

[0050] In one possible implementation, the protocol adapter can adapt to at least the following file systems: local POSIX file system; network file system (NFS); container storage interface (CSI); object storage service (Amazon S3, BackblazeB2); memory virtual file system, etc.

[0051] Based on the above, the storage protocol layer uniformly manages the protocol adapters, supports the extension of protocol adapters, and loads the corresponding protocol adapters into the executor layer as plug-ins according to the needs of the executor layer.

[0052] Specifically, the executor adopts a plug-in loading framework, with all protocol adapters implemented as independent plug-ins and dynamically loaded at runtime through a unified plug-in management component. All plug-ins use standard file operation interfaces to ensure they can be invoked in a unified manner.

[0053] In actual business scenarios, if the built-in plugins can meet the requirements, the "out-of-the-box" mode can be adopted directly; if the business involves industry-specific protocols or private cloud storage, new plugins can be developed and registered based on the same plugin framework to continuously improve the system's protocol coverage capabilities.

[0054] This application embodiment constructs a universal architecture through plug-in protocol adapters to adapt to various heterogeneous storage environments, ensuring that new storage protocols can be smoothly integrated without modifying the core code, avoiding vendor lock-in, ensuring the portability and scalability of the protocol gateway, enabling business systems to seamlessly access file systems with different storage protocols, and providing a theoretically rigorous and engineering-feasible solution for complex storage needs such as multi-tenancy, large model training and business parallelism.

[0055] In one possible implementation, updating the protocol adapter specifically includes: After changes to the plugin code or configuration of the protocol adapter, reload the protocol adapter without restarting the process of the protocol gateway.

[0056] Specifically, the protocol adapter can be configured online. Administrators can modify plugin parameters (such as endpoint address, credentials, serialization method, etc.) through a visual interface, and the changes take effect immediately. After changes to the plugin code or configuration, the protocol gateway process can be refreshed online by triggering reload() through the internal event bus without restarting the protocol gateway process.

[0057] This application embodiment ensures the continued use of the actuator layer by updating the protocol adapter online.

[0058] In one possible implementation, such as Figure 4 As shown, in step S220, the first access request is parsed through the request access layer to obtain the binary storage context information, specifically including: S2210: Perform signature verification and authentication on the first access request.

[0059] In one possible implementation, for requests conforming to AWS Signature Version 4 (SigV4), key fields in the request header (such as Credential, SignedHeaders, Signature, etc.) are first extracted, and signature verification is completed based on the SigV4 Hash-based Message Authentication Code (HMAC) calculation model. The specific calculation process is as follows: ; ； in, For signature; Indicates derived cipher; (Key) (Request Date) (Regional identification) and The target service name is obtained by parsing the fields in the request header. () is a hash-based message authentication code function; This is a digest of the normalized request to be signed; () converts the binary HMAC result to a hexadecimal string.

[0060] The protocol gateway recalculates the signature using the same algorithm and key, and compares it with the signature sent from the business system. This allows the integrity and legitimacy of the request to be verified without revealing the key.

[0061] If the signature verification passes, the first access request will be authenticated.

[0062] S2220: If the signature verification and authentication are successful, extract the operation type from the query string in the Uniform Resource Identifier (URI) of the first access request.

[0063] Specifically, in an embodiment where the first access request is received via REST conforming to the S3 specification, the S3 operation type of the access request, such as PUT Object, GET Object, DELETE Object, ListObjectsV2, etc., is identified by parsing the query string (e.g., uploadId=…&partNumber=…) in the URI of the first access request.

[0064] S2230: Determine the storage method corresponding to the operation type based on the mapping table between operation type and storage method.

[0065] Specifically, the access layer maintains a mapping table between operation types and storage methods. Different operation types correspond to different storage methods in the mapping table. Therefore, the corresponding storage method can be determined by the operation type. Table 1 shows some mapping tables between storage methods and operation types.

[0066] The executor layer comprises multiple executors, each handling access requests for a specific storage method and mapping the parsed results to access requests based on file system-based storage protocols. Therefore, after the request access layer determines the storage method, when routing the parsed results to the execution layer, the executor layer invokes the executor corresponding to that storage method, which then handles the access request for that operation type.

[0067] Table 1: Correspondence between storage methods and operation types

[0068] Therefore, the storage method abstracts the operation type into the business operation interface inside the protocol gateway. The mapping from operation type to storage method can realize the conversion of access requests from multiple object protocols into unified business logic, thereby ensuring scalability, maintainability and protocol independence.

[0069] S2240: Extract the first metadata and the original byte stream from the request body of the first access request. The metadata in the access request includes object tags, ACLs, encryption information, etc.

[0070] S2250: Based on the storage method, the first metadata and the original byte stream are encapsulated into the binary storage context information, denoted as... 。

[0071] in Represents the collection of first metadata. This represents the raw byte stream.

[0072] The binary storage context information includes the data attributes required by the object protocol and the file protocol, and the interaction between protocols is carried out through the binary storage context.

[0073] As shown in Table 2, the binary storage context structure includes a fixed-length object header, a variable-length extended header, and a data body from the first access request. The object header includes a magic number identifier, version number, object type, creation time, data length, metadata length, and checksum. The extended header includes bucket information, object key, content details, and custom data. The data body includes file data and stream data.

[0074] Table 2: Binary Storage Context Structure

[0075] This application's embodiments obtain a binary storage context based on the storage method corresponding to the operation type, establishing a unified, efficient, secure, and scalable data exchange carrier between different storage protocols and business systems. Subsequently, protocol conversion can be implemented based on this protocol-independent single structure, enabling data transfer from object protocols to the underlying file system, providing both theoretical and practical guarantees for the high reliability of the protocol gateway. Furthermore, encapsulating metadata and the raw byte stream together for transmission ensures they either function simultaneously or fail simultaneously, reducing inconsistencies or state errors caused by the separation of metadata and data streams.

[0076] Existing file system permission control mechanisms are mainly based on a simple user-group-permission model, which cannot meet the complex permission requirements of large-scale scenarios. For example, when the same user participates in different projects simultaneously, the user has multiple identities and needs to have access permissions in each project, but the existing file system cannot meet this complex permission requirement.

[0077] Based on these considerations, in one possible implementation, such as Figure 5 As shown, in step 2210, authenticating the first access request specifically includes: S510: Map the Uniform Resource Identifier of the first access request to the virtual directory mapping, and use the virtual directory mapping as part of the parsing result.

[0078] Specifically, the URI of the first access request is broken down into a two-layer structure: Bucket and Object Key. Let the URI be / {bucket} / {key_1} / {key_2} / … / {key_n}, then the mapping function is: The mapping function maps each level of object keys to a directory or file object in the underlying POSIX file system. Among these, The corresponding bucket directory under the root directory, and subsequent... ( >0) Corresponding subdirectory or file. This mapping is bijective: any file system path can be uniquely reverse-mapped back to the corresponding S3 URL, realizing a one-to-one correspondence between virtual directories and real objects.

[0079] S520: Determine the project identifier and tenant identity bound to the virtual directory mapping.

[0080] The protocol gateway's user database records the mapping between access key identifiers and users, as well as the hierarchical relationships among users, projects, and tenants. Furthermore, the protocol gateway maintains a file system binding table, recording the binding relationships between each virtual directory mapping and the projects and tenants with access rights to that virtual directory mapping. Therefore, upon obtaining a virtual directory mapping, it is possible to determine the project identifiers and tenant identities that can access that virtual directory mapping, as well as the identities of all users with access rights to that virtual directory mapping.

[0081] Specifically, this application's embodiment introduces a three-tiered permission model of "tenant-project-user" above the path hierarchy. Let... U For user identity, P For project identification, T If the user is a tenant, the access control rules can be formalized as follows: 。

[0082] This model achieves fine-grained multi-tenant permission isolation, ensuring that data between different tenants, projects, and users does not interfere with each other and is secure and controllable.

[0083] S530: Extract the AccessKeyId from the request header of the first access request.

[0084] S540: Query the user database to determine the user identity corresponding to the access key identifier, and determine whether the user identity has access rights to the project identifier under the tenant identity.

[0085] After obtaining the access key identifier, the corresponding user identity can be obtained based on the mapping relationship between the access key identifier and the user in the user database. Based on the affiliation relationship between the user, project, and tenant, it can be determined whether the user identity has access rights to the project identifier under the tenant identity determined in step S520.

[0086] S550: If the user identity has access rights to the above-mentioned project identifier under the above-mentioned tenant identity, determine that the authentication of the first access request is successful.

[0087] This application embodiment constructs an extensible multi-tenant access control framework through structured path mapping, providing both theoretical and implementation-based protection for the security of the protocol gateway. Furthermore, this application embodiment also achieves cross-tenant security isolation through a unified permission model.

[0088] In one possible implementation, in step S230, the target file system accessed by the first access request can be determined based on the bucket information in the URI of the first access request (see step S510). Other methods in the prior art can also be used to determine the target file system accessed by the first access request, and this application does not impose any restrictions.

[0089] In one possible implementation, the binary storage context information includes the first metadata and the raw byte stream from the first access request. Based on this, as... Figure 6 As shown, in step S230, converting the first access request into a second access request based on the target storage protocol specifically includes: S610: Determine the protocol conversion rules between the object protocol and the target storage protocol, wherein the protocol conversion rules record all parameters required for protocol conversion.

[0090] To achieve visualized management and unified configuration of rules, the protocol gateway introduces a rule configuration center. This center records all parameters required for protocol conversion in the form of structured metadata. Table 3 shows all the parameters required for protocol conversion, including execution mode, storage address, protocol type, plugin type, protocol header, interface mapping, and sequence number format.

[0091] Table 3: All parameters required for protocol conversion

[0092] The above configuration is modeled using a unified Rule Description Language (RDL) and submitted to the executor when the protocol gateway starts up or the rules are updated.

[0093] The executor layer enables business systems to seamlessly access file systems with different storage protocols through a unified protocol conversion rule management mechanism, providing a theoretically rigorous and engineering-feasible solution for complex storage needs such as multi-tenancy, large model training, and business parallelism.

[0094] S620: Based on the protocol conversion rules, map the first metadata in the binary storage context information to the request header of the target storage protocol, and map the original byte stream in the binary storage context information to the request body of the target storage protocol.

[0095] Specifically, when the executor calls the SDK or REST interface of the target storage protocol, it maps the information in the binary storage context information to the different parts required by the target storage protocol, thereby realizing protocol-independent unified business logic.

[0096] S630: Organize the request header, the request body, and the virtual directory mapping according to the structure of the target storage protocol to obtain the second access request.

[0097] This application embodiment converts the parsing result into a second access request based on protocol conversion rules, ensuring the accuracy of the conversion.

[0098] In one possible implementation, updating the protocol conversion rule specifically includes: Upon receiving an incremental configuration task for the protocol conversion rule, the protocol conversion rule is reloaded without restarting the protocol gateway process.

[0099] For details, please refer to Figure 3 The executor layer itself follows the standard file operation interface. Based on this, the protocol conversion rules can implement the following lifecycle methods: Initialization: When the protocol gateway starts, the executor calls pull() to retrieve the latest protocol conversion rules from the rule configuration center. Based on the retrieved protocol conversion rules, the executor executes init() to complete resource allocation (thread pool, connection pool, etc.) and internal state initialization.

[0100] After initialization, the executor registers itself to the internal service directory using register() for use in subsequent request routing.

[0101] Hot update: When the rule configuration center publishes a new rule, the protocol gateway triggers `push()` to push the incremental configuration to the executor. Without restarting the protocol gateway process, the executor executes `reload()` to complete the online reloading of the protocol conversion rules, achieving zero-downtime updates. This online update of protocol conversion rules ensures the continuous use of the executor.

[0102] This application embodiment ensures the continued use of the executor layer by updating the protocol conversion rules online.

[0103] Some storage systems may have unreliable metadata storage mechanisms. If a problem occurs with the metadata storage, it may lead to the loss of all data in the entire storage system.

[0104] Based on the above considerations, in one possible implementation, the protocol gateway further includes a metadata storage layer, which comprises a main layer and a backup layer. The metadata storage layer stores secondary metadata during the access process, including the creation time, modification time, permission information, and parsing results of access requests based on object protocols. This secondary metadata is crucial for implementing file-level access control, tenant isolation, and efficient directory traversal. The metadata storage layer provides highly available and high-performance metadata persistence services for the entire system.

[0105] In production environments with extremely high performance and consistency requirements, a single storage medium often cannot meet both needs. Based on this consideration, this application introduces a multi-level metadata storage model, which combines two metadata storage solutions.

[0106] Specifically, the metadata storage layer includes a primary layer and a backup layer. The primary layer uses high-performance, low-latency storage media to handle daily read and write requests. The backup layer uses highly reliable, persistent storage media to provide a degradation path when the primary layer experiences anomalies (such as service downtime, network partitioning, or data errors). Understandably, the system periodically performs consistency checks on the data between the two layers and performs compensatory writes or rollbacks for inconsistencies.

[0107] To meet the diverse performance, consistency, and availability requirements of different business scenarios, please combine the following possible implementation methods: Figure 3 This application provides three types of metadata persistence schemes: File extended attributes: Metadata is viewed as extended attributes at the file system level, with each attribute having a...<key, value> Key-value pairs are stored in the inode of the corresponding file or directory. The advantage of the file extended attribute mechanism lies in its strong binding; metadata is automatically carried along with the file during migration or copying. The disadvantage is that it is limited by the underlying file system's support for extended attributes and the I / O throughput limit of a single node. Typical implementations of the file extended attribute mechanism include the fourth-generation extended file system (ext4), the X File System (XFS), and Network File System version 4 (NFSv4).

[0108] Key-value storage services: Independent distributed key-value databases offering strong or eventual consistency options, providing independent high-concurrency read / write interfaces, maintaining a consistency model while scaling horizontally, and featuring low latency and elastic scaling. Key-value storage services include transactions for remote dictionary servers (Redis) and Raft consensus for etcd.

[0109] Bucket Cache Pool: A hierarchical cache based on a multi-layered tree hash structure, designed for bucket-level metadata aggregation. It employs a three-layer tree hash cache (L1-L2-L3). Its core idea is to distribute metadata across the parent directory through two hash operations, while ensuring that all child objects within the same parent directory reside in the same L2 subtree, thus achieving locality of reference optimization. The working mechanism of this structure can be summarized as follows: L1 Sharding: The parent directory is hashed and moduloed once for global hashing, reducing hotspot collisions. L2 Subtree: A second division and modulo operation is performed on top of L1, concentrating all child objects within the same parent directory on the same L2 node, improving the local cache hit rate during directory traversal. L3 Leaves: File handles are hashed again, ensuring that a large number of files within a single directory are evenly distributed across different leaf nodes, preventing excessive load on a single node.

[0110] The specific mapping formula for the three-level tree hash cache is as follows: in, Represents the parent directory identifier (such as a path string or directory inode). Represents a file handle or a unique identifier (such as an object key or inode). and The number of L1 and L2 fragments is determined by a pre-configured constant.

[0111] In the storage bucket cache pool, when the number of cache modifications exceeds the threshold N, the system automatically triggers persistence and synchronizes the cached content to the disk.

[0112] Based on the above, in one possible implementation, the main layer of the metadata storage layer uses a key-value storage service, while the backup layer uses a metadata persistence medium with a file-extended attribute mechanism. The key-value storage service provides high-speed querying, while the file-extended attribute mechanism achieves persistent fault tolerance.

[0113] In one possible implementation, the primary metadata storage layer uses a key-value storage service, while the backup layer uses a bucket cache pool. The key-value storage service provides high-speed querying, while the bucket cache pool enables persistent fault tolerance.

[0114] Based on the above, the object protocol access method further includes: After obtaining the parsing result, the second metadata of the first access request is synchronously stored in the main layer and backup layer of the metadata storage layer.

[0115] In one possible implementation, storing the second metadata of the first access request in the metadata storage layer specifically includes: After successfully writing the second metadata to the main layer, the second metadata is synchronously written to the backup layer.

[0116] Based on the above, accessing the second metadata specifically includes: In response to a read request, the second metadata required by the read request is read from the main layer; If the primary layer returns an error or access to the primary layer times out, the second metadata required for the read request is read from the backup layer.

[0117] This application embodiment constructs a distributed metadata management mechanism, which ensures strong consistency of metadata while providing a degradation path to achieve fault self-healing and automatic recovery of metadata services, thereby realizing a fault tolerance mechanism and ensuring the continuous availability of the file system when some nodes fail, thus guaranteeing the reliability of metadata.

[0118] Based on the above, this application also provides a file system object protocol access device. The file system object protocol access device and the aforementioned file system object protocol access method can be referred to in correspondence.

[0119] As an example, such as Figure 7 As shown, the file system object protocol access device provided in this application includes: The receiving module 710 is configured to receive a first access request based on the object protocol through the object protocol layer; The parsing module 720 is used to parse the first access request through the request access layer to obtain a parsing result. The parsing result includes routing identification information, binary storage context information, and a virtual directory mapping of the target file system that the first access request needs to access. The protocol conversion module 730 is used to determine the target file system accessed by the first access request based on the routing identification information through the executor layer, convert the first access request into a second access request based on the target storage protocol based on the binary storage context information and the virtual directory mapping, determine the target protocol adapter corresponding to the target file system, access the target file system through the standard file operation interface of the target protocol adapter, and execute the second access request. The target storage protocol is a storage protocol supported by the target file system, the target protocol adapter adopts the target storage protocol, and the object protocol used by the first access request is different from the target storage protocol.

[0120] This application embodiment enables object protocol access to the file system through protocol conversion by the protocol gateway, eliminating access barriers between protocols. As a result, file protocol and object protocol can concurrently and transparently access data entities in the same file system, realize cross-protocol data sharing, solve the adverse effects of data synchronization in dual storage systems, significantly reduce storage costs and synchronization overhead, and improve data utilization efficiency.

[0121] In one possible implementation, the protocol gateway further includes a metadata storage layer, which comprises a primary layer and a backup layer. Furthermore, the file system object protocol access device further includes a metadata storage module 740 and a metadata access module 750.

[0122] The metadata storage module 740 is used to synchronously store the second metadata of the first access request to the main layer and backup layer of the metadata storage layer. The second metadata includes the creation time, modification time, permission information, and parsing result of the first access request.

[0123] Metadata access module 750 is specifically used for: In response to a read request, the second metadata required by the read request is read from the main layer; If the primary layer returns an error or access to the primary layer times out, the second metadata required for the read request is read from the backup layer.

[0124] This application embodiment constructs a distributed metadata management mechanism, which ensures strong consistency of metadata while providing a degradation path to achieve fault self-healing and automatic recovery of metadata services, thus ensuring the continuous availability of the file system when some nodes fail.

[0125] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can invoke logical instructions in the memory 830 to execute a file system object protocol access method. This method includes: receiving a first access request based on an object protocol through the object protocol layer; parsing the first access request through the request access layer to obtain a parsing result, the parsing result including routing identification information, binary storage context information, and a virtual directory mapping of the target file system to be accessed by the first access request; determining the target file system to be accessed by the first access request based on the routing identification information through the executor layer, converting the first access request into a second access request based on a target storage protocol based on the binary storage context information and the virtual directory mapping, determining the target protocol adapter corresponding to the target file system, accessing the target file system through the standard file operation interface of the target protocol adapter, and executing the second access request; wherein the target storage protocol is a storage protocol supported by the target file system, the target protocol adapter uses the target storage protocol, and the object protocol used by the first access request is different from the target storage protocol.

[0126] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the object protocol access method for the file system provided in the above embodiments. The method includes: receiving a first access request based on an object protocol through the object protocol layer; parsing the first access request through the request access layer to obtain a parsing result, the parsing result including routing identification information, binary storage context information, and a virtual directory mapping of the target file system to be accessed by the first access request; determining the target file system to be accessed by the first access request based on the routing identification information through the executor layer, converting the first access request into a second access request based on the target storage protocol based on the binary storage context information and the virtual directory mapping, and determining the target protocol adapter corresponding to the target file system, accessing the target file system through the standard file operation interface of the target protocol adapter, and executing the second access request; wherein the target storage protocol is a storage protocol supported by the target file system, the target protocol adapter adopts the target storage protocol, and the object protocol adopted by the first access request is different from the target storage protocol.

[0128] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an object protocol access method for the file system provided in the above embodiments. The method includes: receiving a first access request based on an object protocol through the object protocol layer; parsing the first access request through the request access layer to obtain a parsing result, the parsing result including routing identification information, binary storage context information, and a virtual directory mapping of the target file system to be accessed by the first access request; determining the target file system to be accessed by the first access request based on the routing identification information through the executor layer, converting the first access request into a second access request based on a target storage protocol based on the binary storage context information and the virtual directory mapping, determining a target protocol adapter corresponding to the target file system, accessing the target file system through the standard file operation interface of the target protocol adapter, and executing the second access request; wherein the target storage protocol is a storage protocol supported by the target file system, the target protocol adapter adopts the target storage protocol, and the object protocol adopted by the first access request is different from the target storage protocol.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for accessing an object protocol in a file system, characterized in that, Applied to a protocol gateway, the protocol gateway includes an object protocol layer, a request access layer, and an executor layer; the object protocol access method includes: The object protocol layer receives the first access request based on the object protocol. The first access request is parsed by the request access layer to obtain the parsing result, which includes routing identification information, binary storage context information, and virtual directory mapping of the target file system to be accessed by the first access request. Through the executor layer, the target file system accessed by the first access request is determined based on the routing identification information. Based on the binary storage context information and the virtual directory mapping, the first access request is converted into a second access request based on the target storage protocol. The target protocol adapter corresponding to the target file system is determined. The target file system is accessed through the standard file operation interface of the target protocol adapter, and the second access request is executed. The target storage protocol is a storage protocol supported by the target file system, the target protocol adapter adopts the target storage protocol, and the object protocol used by the first access request is different from the target storage protocol.

2. The file system object protocol access method according to claim 1, characterized in that, The binary storage context information includes the first metadata and the raw byte stream in the first access request; The step of converting the first access request into a second access request based on the target storage protocol specifically includes: Determine the protocol conversion rules between the object protocol and the target storage protocol, wherein the protocol conversion rules record all parameters required for protocol conversion; Based on the protocol conversion rules, the first metadata in the binary storage context information is mapped to the request header of the target storage protocol, and the original byte stream in the binary storage context information is mapped to the request body of the target storage protocol; The request header, the request body, and the virtual directory mapping are organized according to the structure of the target storage protocol to obtain the second access request.

3. The file system object protocol access method according to claim 1, characterized in that, The protocol gateway further includes a storage protocol layer, the storage protocol layer including at least one file system, the at least one file system including the target file system; The storage protocol layer manages the plugins for the protocol adapters corresponding to each file system; The object protocol access method further includes: After determining the target protocol adapter corresponding to the target file system through the executor layer, the storage protocol layer dynamically loads the plugin of the target protocol adapter into the executor layer.

4. The file system object protocol access method according to claim 1, characterized in that, The protocol gateway also includes a metadata storage layer, which includes a main layer and a backup layer. The object protocol access method further includes: After obtaining the parsing result, the second metadata of the first access request is synchronously stored in the main layer and backup layer of the metadata storage layer. The second metadata includes the creation time, modification time, permission information, and parsing result of the first access request. Accessing the second metadata specifically includes: In response to a read request, the second metadata required by the read request is read from the main layer; If the primary layer returns an error or access to the primary layer times out, the second metadata required for the read request is read from the backup layer.

5. The file system object protocol access method according to claim 2, characterized in that, The first access request is parsed by the request access layer to obtain the binary storage context information, specifically including: Perform signature verification and authentication on the first access request; If the signature verification and authentication pass, extract the operation type from the query string in the Uniform Resource Identifier of the first access request; The storage method corresponding to the operation type is determined based on the mapping table between operation type and storage method; Extract the first metadata and the original byte stream from the request body of the first access request; Based on the storage method, the first metadata and the original byte stream are encapsulated into the binary storage context information, which includes data attributes required by the object protocol and file protocol.

6. The file system object protocol access method according to claim 5, characterized in that, The authentication of the first access request specifically includes: Map the Uniform Resource Identifier of the first access request to the virtual directory mapping, and use the virtual directory mapping as part of the resolution result; Determine the project identifier and tenant identity bound to the virtual directory mapping; Extract the access key identifier from the request header of the first access request; Query the user database to determine the user identity corresponding to the access key identifier, and determine whether the user identity has access rights to the project identifier under the tenant identity; If the user identity has access rights to the project identifier under the tenant identity, the authentication of the first access request is determined to be successful.

7. The file system object protocol access method according to claim 3, characterized in that, Updating the protocol adapter specifically includes: After changes to the plugin code or configuration of the protocol adapter, reload the protocol adapter without restarting the process of the protocol gateway.

8. The file system object protocol access method according to claim 2, characterized in that, Updating the protocol conversion rules specifically includes: Upon receiving an incremental configuration task for the protocol conversion rule, the protocol conversion rule is reloaded without restarting the protocol gateway process.

9. An object protocol access device for a file system, characterized in that, Applied to a protocol gateway, the protocol gateway includes an object protocol layer, a request access layer, and an executor layer; The object protocol access device includes: The receiving module is configured to receive a first access request based on the object protocol through the object protocol layer; The parsing module is used to parse the first access request through the request access layer to obtain the parsing result, which includes routing identification information, binary storage context information, and virtual directory mapping of the target file system to be accessed by the first access request. The protocol conversion module is used to determine the target file system accessed by the first access request based on the routing identification information through the executor layer, convert the first access request into a second access request based on the target storage protocol based on the binary storage context information and the virtual directory mapping, determine the target protocol adapter corresponding to the target file system, access the target file system through the standard file operation interface of the target protocol adapter, and execute the second access request. The target storage protocol is a storage protocol supported by the target file system, the target protocol adapter adopts the target storage protocol, and the object protocol used by the first access request is different from the target storage protocol.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the file system object protocol access method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium, wherein a computer program is stored on the non-transitory computer-readable storage medium, characterized in that, When the computer program is executed by the processor, it implements the file system object protocol access method as described in any one of claims 1 to 8.

12. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the file system object protocol access method as described in any one of claims 1 to 8.

Images