A data processing method and device based on object storage system
By setting object and event interfaces in the object storage system, parallel processing and mapping of objects into file system views is solved, and the problem that traditional storage systems cannot meet the semantic requirements of large-scale data storage and file system is improved performance and compatibility , reducing management complexity.
Patent Information
- Application Number
- CN202411008803.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-07-26
AI Technical Summary
Traditional storage systems are difficult to meet the needs of large-scale and diversified data storage, especially applications that cannot directly use distributed object storage systems require file system semantics and strong consistency support, resulting in increased performance overhead and management complexity.
By setting up object storage interfaces and event interfaces in the object storage system, parallel processing of objects is realized, and objects are mapped into file system views, file system directory tree structure is provided, and user-defined rule collections and event monitoring modules are supported to ensure data Consistent and efficient storage of IO.
Improves the compatibility and performance of object storage systems, solves the consistency problem between file semantics and object storage systems, reduces management costs and complexity, and is suitable for large-scale small file storage and access.
Smart Images

Figure CN118535579B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of storage technology, and in particular to a data processing method and device based on an object storage system. Background Art
[0002] With the rapid growth of data volume and data types, traditional storage systems can no longer meet the large-scale and diverse data storage needs. Distributed object storage systems have become an indispensable part of the storage infrastructure of today's enterprises and cloud service providers through their high scalability, flexibility and adaptability to big data. The rise of big data and cloud computing technologies has provided more application scenarios for distributed object storage systems and promoted the development of distributed object storage system technology.
[0003] Distributed object storage system is an important part of today's large-scale data storage field. It has significant differences and advantages over traditional file systems and block storage systems. Object storage uses objects as the basic storage unit for flat storage. Each object contains data, metadata and a unique identifier. Compared with traditional file systems and block storage, the complexity of object storage does not increase when objects are added, and object storage has good scalability; the flat storage format also reduces maintenance costs. Therefore, object storage is more suitable for large-scale data, elastic expansion and cloud environment applications.
[0004] Although distributed object storage has high scalability, flat hierarchical structure and low maintenance overhead, many applications cannot use distributed object storage systems directly. These applications need to rely on file semantics, such as the file system semantics of POSIX (Portable Operating System Interface). For example, many traditional enterprise applications (such as database systems and file management systems) rely on the POSIX file system interface to ensure data consistency and reliability. These applications usually require the strong consistency semantics and transaction support provided by the file system to ensure the correctness and recoverability of data operations. Summary of the invention
[0005] In view of the above problems, a data processing method and device based on an object storage system are proposed to overcome the above problems or at least partially solve the above problems, including:
[0006] A data processing method based on an object storage system, wherein the object storage system stores data using objects as basic storage units, the object storage system is provided with an object storage interface and an event interface, the object storage interface and the event interface process requests in parallel, the method comprising:
[0007] In response to a first request for an object sent by a client, processing the object stored in the object storage system through an object storage interface in the object storage system;
[0008] In response to a second request for a file sent by the client, the object stored in the object storage system is mapped to a file in the file system through an event interface in the object storage system, and a file system view of the file system is presented to provide a file system directory tree structure.
[0009] Optionally, mapping the objects stored in the object storage system to files in a file system and presenting a file system view of the file system to provide a file system directory tree structure includes:
[0010] The object required for the second request is determined from the objects stored in the object storage system, the object required for the second request is stored in a directory of the file system in the form of a file, and a file system view of the file system is presented to provide a file system directory tree structure.
[0011] Optionally, the event interface is provided with an event handler component, and the determining the object required for the second request from the objects stored in the object storage system, and storing the object required for the second request in a directory of the file system in the form of a file, comprises:
[0012] Through the event handler component, using the rule set of the event corresponding to the second request, the object required by the second request is determined from the objects stored in the object storage system, and the object required by the second request is stored in the directory of the file system in the form of a file.
[0013] Optionally, presenting the file system view of the file system to provide a file system directory tree structure includes:
[0014] The metadata of the object required by the second request is obtained through the event handler component, and the file system view of the file system is presented according to the metadata to provide a file system directory tree structure.
[0015] Optionally, before determining, by the event handler component, an object required by the second request from objects stored in the object storage system using a rule set of the event corresponding to the second request, and storing the object required by the second request in a directory of the file system in the form of a file, the method further includes:
[0016] A rule set for the event corresponding to the second request is obtained from the client through the event handler component. The client stores the rule set and can be customized by the user.
[0017] Optionally, the rule set includes one or more of the following rule categories: object name, object size, time of accessing the object, and object metadata attributes.
[0018] Optionally, the event handler component sets an event monitoring module, and the method further includes:
[0019] The event monitoring module is used to monitor changes of objects stored in the object storage system.
[0020] Optionally, the event handler component is provided with a cache acceleration module, and the method further comprises:
[0021] The metadata is cached in a preset database in the form of key-value pairs through the cache acceleration module, so that the metadata in the preset database can be requested for acquisition.
[0022] Optionally, the method further comprises:
[0023] When the requested metadata does not exist in the preset database, the metadata is obtained from the object storage system through the event handler component, and the obtained metadata is stored in the preset database.
[0024] Optionally, the event handler component is provided with a remote procedure call service, and the method further comprises:
[0025] Remote communication is performed with the client via the remote procedure call service.
[0026] Optionally, the event corresponding to the second request includes any one or more of the following: an object creation event, an object deletion event, and an object attribute update event.
[0027] Optionally, the method further comprises:
[0028] When a write operation to a file is detected and the file is closed, the most recently written data is flushed to disk;
[0029] When it is detected that the file is opened again, the most recently written data is read.
[0030] Optionally, the object storage system is a distributed object storage system, which includes multiple storage pools, different storage pools store different types of resources, and each storage pool includes multiple node storage devices.
[0031] Optionally, the method further comprises:
[0032] Slice the data to be stored according to a preset size to obtain multiple objects;
[0033] Grouping the multiple objects to obtain multiple object groups;
[0034] The plurality of object groups are stored in the object storage system.
[0035] Optionally, after grouping the multiple objects to obtain multiple object groups, the method further includes:
[0036] The identifiers of the object groups are determined according to the number of the plurality of object groups and the objects in the object groups.
[0037] Optionally, the first request includes a request for an object creation event and an object deletion event, and the second request includes a request for an object attribute update event.
[0038] A data processing device based on an object storage system, wherein the object storage system stores data using objects as basic storage units, the object storage system is provided with an object storage interface and an event interface, the object storage interface and the event interface process requests in parallel, and the object storage system stores data using objects as basic storage units, the device comprising:
[0039] an object request response module, configured to process the object stored in the object storage system through an object storage interface in the object storage system in response to a first request for an object sent by a client;
[0040] The file request response module is used to respond to the second request for the file sent by the client, map the object stored in the object storage system to a file in the file system through the event interface in the object storage system, and present a file system view of the file system to provide a file system directory tree structure.
[0041] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method described above when executed by the processor.
[0042] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.
[0043] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described above.
[0044] The embodiments of the present invention have the following advantages:
[0045] In an embodiment of the present invention, in response to a first request for an object sent by a client, the object stored in the object storage system is processed through the object storage interface in the object storage system; in response to a second request for a file sent by the client, the object stored in the object storage system is mapped to a file in a file system through an event interface in the object storage system, and a file system view of the file system is presented to provide a file system directory tree structure, thereby providing a file system in the object storage system and improving the compatibility of the object storage system. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0047] Figure 1 is a flowchart of a data processing method based on an object storage system provided by some embodiments of the present invention;
[0048] Figure 2a is a schematic diagram of a system architecture provided by some embodiments of the present invention;
[0049] Figure 2b is a schematic diagram of another system architecture provided by some embodiments of the present invention;
[0050] Figure 3 is a structural block diagram of a data processing device based on an object storage system provided by some embodiments of the present invention;
[0051] Figure 4 is a structural block diagram of an electronic device provided by some embodiments of the present invention;
[0052] Figure 5 is a structural block diagram of a storage medium provided by some embodiments of the present invention;
[0053] Figure 6 It is a structural block diagram of a program product provided by some embodiments of the present invention. DETAILED DESCRIPTION
[0054] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] At present, some applications cannot directly use the semantics of object interfaces such as S3 (the interface of cloud storage service) and Swift (the interface of open source cloud storage service), but need to use the POSIX file system semantic interface for storage IO. The relevant technology integrates the POSIX file system semantics with the underlying distributed object storage, and maps the file system interface to a distributed storage system based on object storage, so that applications can use distributed object storage just like using traditional local file systems.
[0056] The technology designs a Metadata server to store file system metadata information such as directory structure, file attributes, permissions, etc. The Metadata server is responsible for coordinating and managing the file system's namespace and metadata operations.
[0057] The application initiates a file system semantic request through the file system client. The file semantic request is processed by the Metadata server and then converted into the form of object storage to interact with the distributed object storage cluster, providing users with a method compatible with standard file systems to access object storage.
[0058] Although the relevant technology has realized that distributed object storage provides POSIX file system semantics, this method also has some disadvantages. There is a performance overhead when mapping the file system interface to a distributed storage system based on object storage, especially for the storage and access of a large number of small files, which will cause a large overhead. Object storage is usually more suitable for large files and sequential I / O operations, and massive small files are not suitable for this interface mapping conversion.
[0059] Moreover, ensuring the consistency and data integrity of a distributed file system requires additional overhead and complexity. Due to the mapping and conversion between file semantics and the underlying object storage system, object deletion, creation, and object metadata updates are not synchronized with the file system's directory tree, which introduces some consistency issues.
[0060] Furthermore, integrating the file system interface and the object storage system requires additional management and maintenance costs, and maintaining components such as the management nodes and metadata servers of the distributed file system will also increase complexity.
[0061] In an embodiment of the present invention, a system is provided that uses distributed object storage as a base and supports both object interface and file interface semantics. The system may be referred to as Object And File System (OAFS). The system ensures eventual consistency by processing events of the object storage. The events describe changes in the distributed object storage, such as operations such as object creation and deletion.
[0062] In order to meet the needs of some applications that rely on the directory tree structure of the file system, such as enterprise applications (such as database systems, file management systems) that rely on the POSIX file system interface to ensure data consistency and reliability, these applications usually require the strong consistency semantics and transaction support provided by the file system to ensure the correctness and recoverability of data operations.
[0063] The embodiment of the present invention can provide a file system view customized by a user program in a flat namespace of a distributed object storage system, and the view provides a file system directory tree structure for a client application under a special configuration of a rule set. In the embodiment of the present invention, the user-defined semantic file system view can meet the file semantic requirements of the user application and can perform storage IO efficiently.
[0064] Among them, according to the needs of client application users, a file system semantic view that meets user needs based on distributed object storage can be presented through a set of rules. The view can be flexibly configured through fields such as object name, object size, access time or object attributes, that is, metadata extended attributes in the file system.
[0065] Specifically, an embodiment of the present invention designs a custom rule set, the rules including: filtering objects to allow users to access only the objects they are interested in, then filtering the object name, object size, object access time and object attributes in the object namespace (wherein the object name filtering rule allows not only precise name search, but also allows objects to be filtered based on special prefixes or suffixes), and finally, presenting the filtered objects to the user in a directory tree result view of the file system.
[0066] Furthermore, the system of the embodiment of the present invention is based on close-to-open semantics, and the interface is designed on this basis to ensure data consistency.
[0067] Through the embodiments of the present invention, in special scenarios, a high-performance, low-latency file system can be provided for flattened object storage, and a corresponding file system view can be constructed through a rule set, each view containing a set of object filters for related needs, and then these related objects are stored as files in a directory defined by the rule set, which can meet the requirements of efficient file directory operations.
[0068] Moreover, the embodiment of the present invention also supports running on multiple application servers for concurrent access, and each application server can have overlapping file system views, which is beneficial to improving the concurrent read and write performance of the file system, while supporting concurrent access based on its own file system view and object protocol interface.
[0069] Specifically, the changes of objects are mapped to files in the file system through events. This mapping process is different from related technologies. This mapping process formulates a file system view that meets the needs of customer applications through a set of rules, rather than mapping all object data to files in the file system. This avoids the problem that the file system metadata performance cannot meet the needs of customer application queries and other operations when multiple objects in a single bucket are mapped to files. When some applications rely on file semantics, the embodiments of the present invention formulate a unique file system view that complies with the rule set for the application, rather than scanning and mapping all distributed object storage system objects, which can quickly meet the data and metadata required by customers and make storage IO more efficient.
[0070] Compared with the above related technologies, the embodiments of the present invention also have the following advantages:
[0071] It solves the problem of providing object storage interfaces in distributed object storage clusters while supporting file semantic processing, and does not affect storage performance when storing and accessing a large number of small files.
[0072] It solves some consistency issues introduced by the mapping and conversion between file semantics and the underlying object storage system.
[0073] The solution solves the problem of integrating the file system interface and the object storage system, which requires additional management and maintenance costs. Maintaining components such as the management node and metadata server of the distributed file system will also increase the complexity.
[0074] The following further describes the embodiments of the present invention in conjunction with the accompanying drawings.
[0075] Reference Figure 1 , shows a flowchart of a data processing method based on an object storage system provided by some embodiments of the present invention, the object storage system uses objects as basic storage units to store data, and the object storage system is a distributed object storage system, such as Figure 2a and Figure 2b ,The object storage system includes multiple storage pools, different storage pools store different ,types of resources, and each storage pool includes multiple node storage ,devices.
[0076] In some embodiments of the present invention, the object storage system is provided with an object storage interface and an event interface, and the object storage interface and the event interface process requests in parallel.
[0077] The design concept of the distributed object storage system is based on scalability and fault tolerance, eliminating performance bottlenecks such as single-point metadata, adopting a distributed and decentralized design architecture, slicing data and distributing it in the form of objects on multiple nodes, and ensuring that the load and capacity of each node are balanced. The technology of data access and management is carried out through network connections. In the storage pool, the client's data is stored in slices, which are fixed to 4MB objects by default. The objects are grouped, and the group id is the remainder of the Hash (the hash value of the corresponding object) and the number of groups, and finally stored in the underlying device.
[0078] The distributed object storage system architecture of the present invention includes the following key components:
[0079] (1) The distributed storage client includes object semantics and file semantics, which correspond to the object storage interface provided by the object gateway service and the OAFS event handler service provided by the present invention. The above services are set on each storage cluster node of the distributed object storage system.
[0080] (2) Unified, self-controlled, and scalable distributed storage consistency management system: This component is the core component of the distributed storage cluster. It provides distributed storage services in the form of objects. In this component, data is stored as objects, and each object has a unique identifier and related data. It is responsible for storing objects in a distributed manner on each node of the storage cluster and provides functions such as data replication, recovery, and load balancing to ensure data reliability and high-performance access.
[0081] (3) Storage pool: A distributed storage cluster consists of multiple storage nodes, each of which can contain multiple hard disks or storage devices. The storage pool divides storage resources into different pools to meet different storage needs.
[0082] (4) Controllable, scalable, distributed data balanced placement algorithm: This component evenly distributes data objects across the nodes of the storage cluster to avoid data hotspots and improve system performance.
[0083] (5) Monitoring service: The monitoring service cluster is responsible for monitoring the status and configuration information of the cluster. Its technical principles include maintaining the status information, configuration information, health status, etc. of the cluster to ensure the consistency and availability of the cluster.
[0084] (6) Event interface: Requests for file semantics are forwarded and processed by the OAFS event handler, and finally the event interface performs storage IO in the distributed object storage according to the rule set. Consistent with the OAFS event handler, an event interface module is set up on each storage node.
[0085] Specifically, the following steps may be included:
[0086] Step 101: In response to a first request for an object sent by a client, an object stored in the object storage system is processed through an object storage interface in the object storage system.
[0087] In some embodiments of the present invention, the method further comprises:
[0088] The data to be stored is sliced according to a preset size to obtain a plurality of objects; the plurality of objects are grouped to obtain a plurality of object groups; and the plurality of object groups are stored in the object storage system.
[0089] In some embodiments of the present invention, after grouping the multiple objects to obtain multiple object groups, the method further includes:
[0090] The identifiers of the object groups are determined according to the number of the plurality of object groups and the objects in the object groups.
[0091] In practical applications, the design concept of the distributed object storage system is based on scalability and fault tolerance, eliminating performance bottlenecks such as single-point metadata, adopting a distributed and decentralized design architecture, slicing data and distributing it in the form of objects on multiple nodes, and ensuring that the load and capacity of each node are balanced. The technology of data access and management is performed through network connections. In the storage pool, the client's data is stored in slices, fixed to 4MB objects by default, and the objects are grouped. The group id is the remainder of the Hash (the hash value of the corresponding object) and the number of groups, and finally stored in the underlying device.
[0092] Step 102, in response to a second request for a file sent by the client, the object stored in the object storage system is mapped to a file in the file system through an event interface in the object storage system, and a file system view of the file system is presented to provide a file system directory tree structure.
[0093] In some embodiments of the present invention, the first request includes a request for an object creation event and an object deletion event, and the second request includes a request for an object attribute update event.
[0094] For the client: A client based on the semantics of the OAFS file system is designed. The client initiates object and file semantic requests (i.e., the first request and the second request) to the distributed object storage system by mounting the OAFS type. The OAFS client communicates with the OAFS event handler through RPC remote procedure calls.
[0095] When a user process initiates a request to access a file and determines through the mount type that the file belongs to the OAFS file system type of the present invention, the request (ie, the second request) is forwarded to the process for file system protocol conversion and processing.
[0096] The file system client consists of four parts:
[0097] (1) The underlying API (Application Programming Interface) based on the Filesystem in Userspace (FUSE);
[0098] (2) A memory cache module for storing metadata and data locally on the client system;
[0099] (3) Object storage S3 or Swift SDK (Software Development Kit) that communicates with the distributed object storage system;
[0100] (4) RPC (Remote Procedure Call Protocol) client communication module, which interacts with the event handler to process events.
[0101] For the system side: the distributed object storage system is composed of multiple storage server nodes, each storage node deploys and designs object storage interfaces (Object store interfaces) and event interfaces (Event interfaces), and these two interfaces simultaneously and in parallel process requests for object protocol (such as S3 or Swift SDK) semantics and requests for Posix file system semantics. The OAFS file system interface of the present invention and the interface of the distributed object storage can be used simultaneously and in parallel. In addition, multiple file system clients can concurrently operate and access the distributed object storage.
[0102] Among them, the client user's request interacts concurrently with the object storage interface and the event interface. In particular, the data management of object creation and deletion event requests interacts with the distributed object storage interface through the client S3 or Swift SDK. The object attribute update request is processed through RPC and the OAFS event handler component. The file system view is constructed through the rule set of client file semantics to prevent conflicts between the file system semantics and the object storage event handler.
[0103] In some embodiments of the present invention, mapping the objects stored in the object storage system to files in a file system and presenting a file system view of the file system to provide a file system directory tree structure includes: determining the objects required for the second request from the objects stored in the object storage system, storing the objects required for the second request in a directory of the file system in the form of files, and presenting a file system view of the file system to provide a file system directory tree structure.
[0104] In some embodiments of the present invention, the event interface is provided with an event handler component, and the determining the object required for the second request from the objects stored in the object storage system, and storing the object required for the second request in a directory of the file system in the form of a file, comprises:
[0105] Through the event handler component, using the rule set of the event corresponding to the second request, the object required by the second request is determined from the objects stored in the object storage system, and the object required by the second request is stored in the directory of the file system in the form of a file.
[0106] In some embodiments of the present invention, the event corresponding to the second request includes any one or more of the following: an object creation event, an object deletion event, and an object attribute update event.
[0107] In some embodiments of the present invention, presenting the file system view of the file system to provide a file system directory tree structure includes: obtaining metadata of the object required for the second request through the event handler component, and presenting the file system view of the file system based on the metadata to provide the file system directory tree structure.
[0108] In some embodiments of the present invention, before determining the object required for the second request from the objects stored in the object storage system through the event handler component and using the rule set of the event corresponding to the second request, and storing the object required for the second request in the form of a file in a directory of the file system, it also includes: obtaining the rule set of the event corresponding to the second request from the client through the event handler component, and the client stores the rule set and can be customized by the user.
[0109] In some embodiments of the present invention, the rule set includes one or more of the following rule categories: object name, object size, time of accessing the object, and object metadata attributes.
[0110] At the upper layer of each event interface, the present invention deploys and designs an event handler (Event-Handler) component for processing events in the file system and triggering corresponding operations, specifically processing events in the file system, such as file writing, querying, creating, modifying, deleting, reading and other operations.
[0111] During file system processing, event handlers can be used to monitor and respond to various events occurring in the file system, as well as filter events based on user-defined rules to perform corresponding processing operations, such as triggering certain actions, recording logs, and issuing notifications.
[0112] The event handler component designs a file system view based on distributed object storage, and builds the corresponding file system view through the rule set Rulesets (rule setting). Each view contains a set of object filters for related needs, and then stores these related objects as files in the directory defined by the rule set Rulesets. The file life cycle requested by the user starts with the creation of the file, and then the metadata of the file is forwarded to the event handler component through the event interface, and the event handler updates the file system view.
[0113] To maintain consistency, the present invention proposes three event types: object creation, deletion, and object attribute update. Object storage attributes refer to metadata information stored in the object, including the object's name, size, type, creation time, owner, etc.
[0114] In order to effectively manage and organize these object storage properties, you can filter and filter out objects that meet specific conditions. Set filtering rules for object creation and deletion events, and create custom indexes to speed up retrieval and filtering operations. By creating indexes, you can quickly locate and filter objects that meet the conditions and improve query performance.
[0115] In actual applications, the file system has designed a set of custom rules, which include: filtering objects, allowing users to access only the objects they are interested in, and then filtering the object name, object size, object access time, and object attributes in the object namespace. The object name filtering rules not only allow precise name searches, but also allow objects to be filtered based on special prefixes or suffixes. Finally, the filtered objects are presented to the user in the file system's directory tree result view.
[0116] In an embodiment of the present invention, a method for rule set Rulesets is proposed. When an application client initiates a request, it is necessary to obtain a path and rule set Rulesets configuration. The rule set Rulesets is based on YAML (a markup language) syntax. Each rule set consists of four rule categories, including object name, size, time to access the object, and object metadata attributes.
[0117] Rulesets are configured and saved on the file system client. When an application initiates a request, it is sent to the event handler component, which evaluates it and responds to the result, and returns the metadata of all matching rule set objects.
[0118] In some embodiments of the present invention, the event handler component sets an event monitoring module, and the method further includes: monitoring changes of objects stored in the object storage system through the event monitoring module.
[0119] In actual applications, the event monitoring module is responsible for extracting events during client request processing and processing them in conjunction with the user-defined rule engine. During request processing, each event is checked to see if it is applicable to the user-defined rule set. If the event is applicable to the user-defined rule set, the Key-Value storage and rule set are updated accordingly.
[0120] In some embodiments of the present invention, the event handler component is provided with a cache acceleration module, and the method further comprises: caching metadata in a preset database in the form of key-value pairs through the cache acceleration module, so as to request retrieval through the metadata in the preset database.
[0121] In some embodiments of the present invention, the method further comprises:
[0122] When the requested metadata does not exist in the preset database, the metadata is obtained from the object storage system through the event handler component, and the obtained metadata is stored in the preset database.
[0123] In actual applications, the event handler stores object metadata in the form of Key-Value through the preset database of the distributed object storage node (such as RocksDB database). The Key-Value cache acceleration module is used to quickly update the object attribute status of the object storage.
[0124] When a client initiates a request, it is first processed in the database. If the object's metadata is not in the database, the event handler goes to the distributed object store and obtains all object data that conforms to the client's rule set. Then, the object data that conforms to the client's rule set is cached in the database for quick retrieval on the next request.
[0125] In some embodiments of the present invention, the event handler component is provided with a remote procedure call service, and the method further comprises: performing remote communication with the client via the remote procedure call service.
[0126] In actual applications, the communication between the file system client and the server event handler (Event-Handler) component is carried out through RPC. The RPC communication library is a communication library used to implement remote procedure calls, allowing cross-network communication and calling of remote functions between different computers or processes.
[0127] In some embodiments of the present invention, the method further comprises:
[0128] When a write operation is detected on a file and the file is closed, the latest written data is refreshed to the disk; when it is detected that the file is opened again, the latest written data is read.
[0129] In actual applications, the file system is based on close-to-open semantics, and the OAFS interface is designed on this basis. After writing to a file and closing the file, the next time the file is opened for reading, the latest written data will be seen. This means that closing the file will flush the data to the disk so that it can read the latest data in subsequent open operations to ensure data consistency.
[0130] In an embodiment of the present invention, in response to a first request for an object sent by a client, the object stored in the object storage system is processed through the object storage interface in the object storage system; in response to a second request for a file sent by the client, the object stored in the object storage system is mapped to a file in a file system through an event interface in the object storage system, and a file system view of the file system is presented to provide a file system directory tree structure, thereby providing a file system in the object storage system and improving the compatibility of the object storage system.
[0131] For the client:
[0132] A client based on OAFS file system semantics is designed. The client initiates object and file semantic requests to the distributed object storage system by mounting the OAFS type. The OAFS client communicates with the OAFS event handler through RPC remote procedure calls.
[0133] When a user process initiates a request to access a file and determines through the mount type that the file belongs to the OAFS file system type of the present invention, the request is then forwarded to the daemon process for file system protocol conversion and processing.
[0134] The file system client consists of four parts:
[0135] (1) The underlying API (Application Programming Interface) based on the Filesystem in Userspace (FUSE);
[0136] (2) A memory cache module for storing metadata and data locally on the client system;
[0137] (3) Object storage S3 or Swift SDK (Software Development Kit) that communicates with the distributed object storage system;
[0138] (4) RPC (Remote Procedure Call Protocol) client communication module, which interacts with the event handler to process events.
[0139] For the system side:
[0140] The distributed object storage system is composed of multiple storage server nodes, each storage node deploys and designs object storage interfaces Object store interfaces and event interfaces Event interfaces, which simultaneously and in parallel process requests of object protocol (such as S3 or Swift SDK) semantics and requests of Posix file system semantics. The OAFS file system interface of the present invention and the interface of the distributed object storage can be used simultaneously and in parallel. In addition, multiple file system clients can concurrently operate and access the distributed object storage.
[0141] Among them, client user requests interact concurrently with object storage interfaces and event interfaces. In particular, data management of object creation and deletion event requests interacts with the distributed object storage interface through the client S3 or Swift SDK. Object attribute update requests are processed through RPC and OAFS event handler components. The file system view is built through a set of client file semantic rules to prevent conflicts between file system semantics and object storage event handlers.
[0142] For event handling components and views:
[0143] At the upper layer of each event interface Event interfaces, the present invention deploys and designs an event handler (Event-Handler) component for processing events in the file system and triggering corresponding operations, specifically processing events in the file system, such as file writing, querying, creating, modifying, deleting, reading and other operations.
[0144] During file system processing, event handlers can be used to monitor and respond to various events occurring in the file system, as well as filter events based on user-defined rules to perform corresponding processing operations, such as triggering certain actions, recording logs, and issuing notifications.
[0145] The event handler component designs a file system view based on distributed object storage, and builds the corresponding file system view through the rule set Rulesets. Each view contains a set of object filters for related needs, and then stores these related objects as files in the directory defined by the rule set Rulesets. The file life cycle requested by the user starts with the creation of the file, and then the metadata of the file is forwarded to the event handler component through the event interface, and the event handler updates the file system view.
[0146] To maintain consistency, the present invention proposes three event types: object creation, deletion, and object attribute update. Object storage attributes refer to metadata information stored in the object, including the object's name, size, type, creation time, owner, etc.
[0147] In order to effectively manage and organize these object storage properties, you can filter and filter out objects that meet specific conditions. Set filtering rules for object creation and deletion events, and create custom indexes to speed up retrieval and filtering operations. By creating indexes, you can quickly locate and filter objects that meet the conditions and improve query performance.
[0148] For the specific structure of the event handler component:
[0149] The present invention proposes a file system event handler (Event-Handler) to maintain the metadata status of the management object (metadata management refers to the client initiating a request to the event handler through RPC to retrieve the latest status of the directory, which is the directory view of the rule set, including directory items and nodes in the file system, and is stored in the memory medium cache).
[0150] Specifically, it includes a directory structure and is composed of the following components: an event monitoring module and a user-defined rule set, whose subcomponents include a Key-Value cache acceleration module and an RPC server.
[0151] (1) Event monitoring module and user-defined rule set: monitors the state changes of objects in the object storage and processes the customized rules for each event.
[0152] The event monitoring module is responsible for extracting events during client request processing and processing them in conjunction with the user-defined rule engine. During request processing, each event is checked to see if it is applicable to the user-defined rule set. If the event is applicable to the user-defined rule set, the Key-Value storage and rule set are updated accordingly.
[0153] (2) Key-Value cache acceleration module, used to quickly update the object attribute status of object storage.
[0154] The event handler stores object metadata in the form of Key-Value through the preset database of the distributed object storage node (such as RocksDB database). The Key-Value cache acceleration module is used to quickly update the object attribute status of the object storage.
[0155] When a client initiates a request, it is first processed in the database. If the object's metadata is not in the database, the event handler goes to the distributed object store and obtains all object data that conforms to the client's rule set. Then, the object data that conforms to the client's rule set is cached in the database for quick retrieval on the next request.
[0156] (3) RPC server, used to communicate with the file system client.
[0157] The communication between the file system client and the server event handler (Event-Handler) component of the present invention is carried out by RPC. The RPC communication library is a communication library for implementing remote procedure calls, allowing cross-network communication and calling of remote functions between different computers or processes.
[0158] For custom rule sets and views:
[0159] The file system of the present invention designs a set of custom rules, which include: filtering and screening objects, allowing users to access only the objects they are concerned about, and then filtering the object name, object size, object access time and object attributes in the object namespace. The filtering rules of object names not only allow accurate name searches, but also allow objects to be filtered according to special prefixes or suffixes. Finally, the filtered objects are presented to the user in a directory tree result view of the file system.
[0160] In an embodiment of the present invention, a method for rule set Rulesets is proposed. When an application client initiates a request, it is necessary to obtain a path and rule set Rulesets configuration. The rule set Rulesets is based on YAML (a markup language) syntax. Each rule set consists of four rule categories, including object name, size, time to access the object, and object metadata attributes.
[0161] Rulesets are configured and saved on the file system client. When an application initiates a request, it is sent to the event handler component, which evaluates it and responds to the result, and returns the metadata of all matching rule set objects.
[0162] The following shows a rule set named RulesetsTest.yml. All objects matching the rule are placed in a directory named after the rule set. For example, objects matching share*.zip, with a size of 10MB, an access time of 2024-06-2108:42:11036041672-0700, and an attribute type of NFS (Network File System) or CIFS (Common Internet File System) shared data are concentrated in a directory view for storage.
[0163] Rule set Rulesets configuration file example RulesetsTest.yml:
[0164] Rulesets1:
[0165] Object: "share*.zip"
[0166] Size: "10MB"
[0167] Access time: “2024-06-21 08:42:11036041672-0700”Properties:
[0168] Type: "NFS"
[0169] Type: "CIFS"
[0170] In addition, the OAFS file system of the present invention is based on close-to-open semantics, and the OAFS interface is designed on this basis. After writing to a file and closing the file, the next time the file is opened for reading, the latest written data will be seen. This means that closing the file will refresh the data to the disk so that it can read the latest data in the subsequent opening operation, ensuring data consistency.
[0171] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0172] Reference Figure 3 , shows a schematic diagram of the structure of a data processing device based on an object storage system provided by some embodiments of the present invention, wherein the object storage system uses objects as basic storage units to store data, and the object storage system is provided with an object storage interface and an event interface, and the object storage interface and the event interface process requests in parallel, and specifically may include the following modules:
[0173] The object request response module 301 is used to process the object stored in the object storage system through the object storage interface in the object storage system in response to a first request for an object sent by a client.
[0174] The file request response module 302 is used to respond to the second request for the file sent by the client, map the object stored in the object storage system to a file in the file system through the event interface in the object storage system, and present a file system view of the file system to provide a file system directory tree structure.
[0175] In some embodiments of the present invention, mapping the objects stored in the object storage system to files in a file system and presenting a file system view of the file system to provide a file system directory tree structure includes:
[0176] The object required for the second request is determined from the objects stored in the object storage system, the object required for the second request is stored in a directory of the file system in the form of a file, and a file system view of the file system is presented to provide a file system directory tree structure.
[0177] In some embodiments of the present invention, the event interface is provided with an event handler component, and the determining the object required for the second request from the objects stored in the object storage system, and storing the object required for the second request in a directory of the file system in the form of a file, comprises:
[0178] Through the event handler component, using the rule set of the event corresponding to the second request, the object required by the second request is determined from the objects stored in the object storage system, and the object required by the second request is stored in the directory of the file system in the form of a file.
[0179] In some embodiments of the present invention, presenting the file system view of the file system to provide a file system directory tree structure includes:
[0180] The metadata of the object required by the second request is obtained through the event handler component, and the file system view of the file system is presented according to the metadata to provide a file system directory tree structure.
[0181] In some embodiments of the present invention, before determining the object required by the second request from the objects stored in the object storage system by using the rule set of the event corresponding to the second request through the event handler component, and storing the object required by the second request in a directory of the file system in the form of a file, the method is further configured to:
[0182] A rule set for the event corresponding to the second request is obtained from the client through the event handler component. The client stores the rule set and can be customized by the user.
[0183] In some embodiments of the present invention, the rule set includes one or more of the following rule categories: object name, object size, time of accessing the object, and object metadata attributes.
[0184] In some embodiments of the present invention, the event handler component is provided with an event monitoring module, and the device is further used for:
[0185] The event monitoring module is used to monitor changes of objects stored in the object storage system.
[0186] In some embodiments of the present invention, the event handler component is provided with a cache acceleration module, and the method further comprises:
[0187] The metadata is cached in a preset database in the form of key-value pairs through the cache acceleration module, so that the metadata in the preset database can be requested for acquisition.
[0188] In some embodiments of the present invention, the device is also used for:
[0189] When the requested metadata does not exist in the preset database, the metadata is obtained from the object storage system through the event handler component, and the obtained metadata is stored in the preset database.
[0190] In some embodiments of the present invention, the event handler component is provided with a remote procedure call service, and the method further comprises:
[0191] Remote communication is performed with the client via the remote procedure call service.
[0192] In some embodiments of the present invention, the event corresponding to the second request includes any one or more of the following: an object creation event, an object deletion event, and an object attribute update event.
[0193] In some embodiments of the present invention, the device is also used for:
[0194] When a write operation to a file is detected and the file is closed, the most recently written data is flushed to disk;
[0195] When it is detected that the file is opened again, the most recently written data is read.
[0196] In some embodiments of the present invention, the object storage system is a distributed object storage system, which includes multiple storage pools, different storage pools store different types of resources, and each storage pool includes multiple node storage devices.
[0197] In some embodiments of the present invention, the device is also used for:
[0198] Slice the data to be stored according to a preset size to obtain multiple objects;
[0199] Grouping the multiple objects to obtain multiple object groups;
[0200] The plurality of object groups are stored in the object storage system.
[0201] In some embodiments of the present invention, after grouping the multiple objects to obtain multiple object groups, the apparatus is further configured to:
[0202] The identifiers of the object groups are determined according to the number of the plurality of object groups and the objects in the object groups.
[0203] In some embodiments of the present invention, the first request includes a request for an object creation event and an object deletion event, and the second request includes a request for an object attribute update event.
[0204] Reference Figure 4 , shows an electronic device provided by some embodiments of the present invention, including a processor 401, a memory 402, and a computer program stored in the memory 402 and capable of running on the processor 401, and the computer program implements the above method when executed by the processor.
[0205] Reference Figure 5, shows a computer-readable storage medium 500 provided in some embodiments of the present invention, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.
[0206] Reference Figure 6 , shows a computer program product 600 provided by some embodiments of the present invention, including a computer program, which implements the above method when executed by a processor.
[0207] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0208] The above provides a detailed introduction to a data processing method and device based on an object storage system. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A data processing method based on an object storage system, characterized in that: The object storage system stores data using objects as basic storage units. The object storage system is provided with an object storage interface and an event interface. The object storage interface and the event interface process requests in parallel. The method includes: In response to a first request for an object sent by a client, processing the object stored in the object storage system through an object storage interface in the object storage system; In response to a second request for a file sent by the client, mapping the object stored in the object storage system to a file in a file system through an event interface in the object storage system, and presenting a file system view of the file system to provide a file system directory tree structure; The mapping of the objects stored in the object storage system to files in a file system and presenting a file system view of the file system to provide a file system directory tree structure includes: Determining an object required by the second request from objects stored in the object storage system, storing the object required by the second request in a directory of a file system in the form of a file, and presenting a file system view of the file system to provide a file system directory tree structure; The event interface is provided with an event handler component, and the determining the object required by the second request from the objects stored in the object storage system, and storing the object required by the second request in a directory of the file system in the form of a file, includes: Through the event handler component, using the rule set of the event corresponding to the second request, the object required by the second request is determined from the objects stored in the object storage system, and the object required by the second request is stored in the directory of the file system in the form of a file.
2. The method according to claim 1, characterized in that The presenting of the file system view of the file system to provide a file system directory tree structure includes: The metadata of the object required by the second request is obtained through the event handler component, and the file system view of the file system is presented according to the metadata to provide a file system directory tree structure.
3. The method according to claim 1 or 2, characterized in that: Before determining the object required by the second request from the objects stored in the object storage system by using the event handler component and using the rule set of the event corresponding to the second request, and storing the object required by the second request in a directory of the file system in the form of a file, the method further includes: A rule set for the event corresponding to the second request is obtained from the client through the event handler component. The client stores the rule set and can be customized by the user.
4. The method according to claim 3, characterized in that The rule set includes one or more of the following rule categories: object name, object size, time of accessing the object, and object metadata attributes.
5. The method according to claim 1 or 2, characterized in that: The event handler component sets an event monitoring module, and the method further includes: The event monitoring module is used to monitor changes of objects stored in the object storage system.
6. The method according to claim 1 or 2, characterized in that: The event handler component is provided with a cache acceleration module, and the method further comprises: The metadata is cached in a preset database in the form of key-value pairs through the cache acceleration module, so that the metadata in the preset database can be requested for acquisition.
7. The method according to claim 6, characterized in that The method further comprises: When the requested metadata does not exist in the preset database, the metadata is obtained from the object storage system through the event handler component, and the obtained metadata is stored in the preset database.
8. The method according to claim 1 or 2, characterized in that: The event handler component is provided with a remote procedure call service, and the method further comprises: Remote communication is performed with the client via the remote procedure call service.
9. The method according to claim 1 or 2, characterized in that: The event corresponding to the second request includes any one or more of the following: an object creation event, an object deletion event, and an object attribute update event.
10. The method according to any one of claims 1 to 2, characterized in that: The method further comprises: When a write operation to a file is detected and the file is closed, the most recently written data is flushed to disk; When it is detected that the file is opened again, the most recently written data is read.
11. The method according to any one of claims 1 to 2, characterized in that: The object storage system is a distributed object storage system, which includes multiple storage pools. Different storage pools store different types of resources, and each storage pool includes multiple node storage devices.
12. The method according to claim 11, characterized in that The method further comprises: Slice the data to be stored according to a preset size to obtain multiple objects; Grouping the multiple objects to obtain multiple object groups; The plurality of object groups are stored in the object storage system.
13. The method according to claim 12, characterized in that After grouping the multiple objects to obtain multiple object groups, the method further includes: The identifiers of the object groups are determined according to the number of the plurality of object groups and the objects in the object groups.
14. The method according to claim 1, characterized in that The first request includes a request for an object creation event and an object deletion event, and the second request includes a request for an object attribute update event.
15. A data processing device based on an object storage system, characterized in that: The object storage system stores data using objects as basic storage units. The object storage system is provided with an object storage interface and an event interface. The object storage interface and the event interface process requests in parallel. The device comprises: an object request response module, configured to process the object stored in the object storage system through an object storage interface in the object storage system in response to a first request for an object sent by a client; a file request response module, configured to respond to a second request for a file sent by a client, map an object stored in the object storage system to a file in a file system through an event interface in the object storage system, and present a file system view of the file system to provide a file system directory tree structure; The mapping of the objects stored in the object storage system to files in a file system and presenting a file system view of the file system to provide a file system directory tree structure includes: Determining an object required by the second request from objects stored in the object storage system, storing the object required by the second request in a directory of a file system in the form of a file, and presenting a file system view of the file system to provide a file system directory tree structure; The event interface is provided with an event handler component, and the determining the object required by the second request from the objects stored in the object storage system, and storing the object required by the second request in a directory of the file system in the form of a file, includes: Through the event handler component, using the rule set of the event corresponding to the second request, the object required by the second request is determined from the objects stored in the object storage system, and the object required by the second request is stored in the directory of the file system in the form of a file.
16. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 14 when executed by the processor.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.
18. A computer program product, characterized in that A computer program is included which, when executed by a processor, implements the method according to any one of claims 1 to 14.
