Method and apparatus for separate storage of object storage file data and metadata

Abstract

This application relates to the field of object storage technology, specifically disclosing a method and apparatus for separating the storage of object storage file data and metadata. This application achieves separate storage of data and metadata, reducing the amount of low-level operations during object file uploads, significantly improving program execution efficiency and object storage efficiency. Furthermore, by decoupling data and metadata, it enables horizontal scaling of the data cluster, alleviating access and storage pressure on a single data cluster, further improving the input / output efficiency of the entire resource pool, achieving fault isolation of the data cluster, and enhancing cluster stability.

Description

Technical Field

[0001] This application relates to the field of object storage technology, and in particular to a method, apparatus, computer device and storage medium for the separate storage of object storage file data and metadata. Background Technology

[0002] Metadata, or data describing attributes, is primarily used to support functions such as indicating storage location, historical data, data retrieval, and data recording. With the rapid development of the internet (especially mobile internet), the Internet of Things (IoT), cloud computing, and big data, data is growing explosively. Data generated by various applications such as social networks, mobile communications, online video and audio, e-commerce, sensor networks, and scientific experiments not only has enormous storage capacity but also exhibits significant characteristics such as diverse data types, large variations in data size, and rapid flow. It often generates massive amounts of files, ranging from tens of millions to hundreds of millions, even billions or tens of billions, posing a significant challenge to metadata management. Therefore, file metadata management has become a recognized problem in both industry and academia. Based on the object storage service of Ceph 14.2.8, the current storage process faces the following two problems in metadata management:

[0003] 1. Once the number of files reaches billions, the data balancing speed decreases significantly during scaling. This is because in Ceph, user-uploaded files may generate multiple rados objects (one header object and multiple tail objects) based on sharding rules. File data is sliced ​​and stored on these rados objects. The header object stores the file's xattr metadata, and each rados object corresponds to a file in a Linux filesystem. The xattr is stored through the file's extended attributes. Since a single user-uploaded file may correspond to multiple Linux files in the backend, as the number of files increases, the efficiency of enumerating files in the XFS filesystem decreases, leading to a decrease in data balancing efficiency and object storage efficiency.

[0004] 2. Currently, each resource pool of Ceph object storage can only correspond to one data cluster. As the cluster is gradually expanded, issues such as single resource pool capacity bottlenecks, hardware differences, and failure rates will arise, leading to problems with cluster stability.

[0005] Therefore, improving the efficiency of object storage and the stability of the cluster has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides a method, apparatus, computer device, and storage medium for the separate storage of object storage file data and metadata, so as to improve the efficiency of object storage and the stability of the cluster.

[0007] Firstly, this application provides a method for separating the storage of object storage file data and metadata, the method comprising:

[0008] Based on preset configuration parameters, the file attributes and read / write information are configured to obtain migration configuration information;

[0009] Based on the migration configuration information, a preset metadata migration process is executed to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage, so as to achieve separate storage of object storage file data and metadata.

[0010] Furthermore, after executing a preset metadata migration process based on the migration configuration information to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage, thereby achieving separate storage of object storage file data and metadata, the process further includes:

[0011] Upon receiving a file uploaded by a user, the target upload type of the file to be stored is determined between normal upload and overwrite upload.

[0012] Based on the target upload type, determine the target upload process;

[0013] Based on the target upload process, the file to be stored is uploaded and stored.

[0014] Furthermore, the step of uploading and storing the file to be stored based on the target upload process includes:

[0015] When the target upload type is normal upload, the file to be stored is written to the data pool;

[0016] Metadata is generated based on the file to be stored, and the metadata is stored in the bucket.index data pool to achieve separate storage of object storage file data and metadata.

[0017] Furthermore, the step of uploading and storing the file to be stored based on the target upload process includes:

[0018] When the target upload type is overwrite upload, the file to be stored is written to the data pool;

[0019] Retrieve the extended attribute omap of the file to be stored from the bucket.index data pool;

[0020] Based on the extended attribute omap of the file to be stored, the extended attribute omap of the historical file to be stored is overridden to update the metadata of the file to be stored and realize the separate storage of object storage file data and metadata.

[0021] Furthermore, after uploading and storing the file to be stored based on the target upload process, the method further includes:

[0022] When the target upload type is normal upload and the file to be stored fails to upload, or when the target upload type is overwrite upload and the file to be stored fails to upload, the file to be stored is sent to the garbage collection queue to wait for recycling.

[0023] When the target upload type is overwrite upload and the file to be stored is successfully uploaded, the historical header object of the historical stored file corresponding to the file to be stored and the historical stored file are added to the garbage collection queue to wait for recycling.

[0024] Furthermore, based on the migration configuration information, the step of executing a preset metadata migration process to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage includes:

[0025] When the object to be migrated is a single file object, the address tag (idtag) of the object to be migrated is obtained based on the identifier of the object to be migrated input by the user;

[0026] The location of the metadata is obtained based on the idtag, and the location of the metadata is determined in omap based on the migration configuration information;

[0027] When the location of the metadata is not in the extended property omap of the bucket.index data pool, the metadata is written to the extended property omap, and it is checked whether the object to be migrated has been deleted;

[0028] If the migrated object has not been deleted, check whether the idtag has been updated;

[0029] If the idtag is not updated, the location of the metadata is modified to achieve the migration of the metadata.

[0030] Furthermore, the step of executing a preset metadata migration process based on the migration configuration information to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage also includes:

[0031] When the object to be migrated is a cluster, read the migration status of the metadata;

[0032] When the migration status is in progress, obtain the list of buckets to be migrated, and lock the first bucket in the list of buckets to be migrated;

[0033] Obtain the bucket information of the first bucket, and based on the bucket information, obtain the first bucket metadata migration stage;

[0034] When the metadata migration phase is completed and the migration time is no more than a preset time, the first bucket is unlocked and removed from the list of buckets to be migrated. Then, all buckets in the list of buckets to be migrated are traversed in turn.

[0035] When the list to be migrated is empty, the migration status of the metadata is modified to complete, so that the metadata of the object to be migrated in the data pool is migrated to the bucket.index data pool for storage.

[0036] Secondly, this application also provides a separate storage device for object storage file data and metadata, the device comprising:

[0037] The migration configuration information acquisition module is used to configure the file attributes and read / write information based on preset configuration parameters to obtain migration configuration information.

[0038] The separate storage module is used to execute a preset metadata migration process based on the migration configuration information, and migrate the metadata of the object to be migrated in the data pool to the bucket.index data pool for storage, so as to realize the separate storage of object storage file data and metadata.

[0039] Thirdly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the object storage file data and metadata separation storage method as described above.

[0040] Fourthly, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the above-described method for separating the storage of object storage file data and metadata.

[0041] This application discloses a method, apparatus, computer device, and storage medium for separating object storage file data and metadata. Based on preset configuration parameters, file attributes and read / write information are configured to obtain migration configuration information. Based on this migration configuration information, a preset metadata migration process is executed to migrate the metadata of the object to be migrated from the `data` data pool to the `bucket.index` data pool for storage, thereby achieving separate storage of object storage file data and metadata. This application achieves separate storage of data and metadata, reducing a large number of operations on the underlying layers during object file uploads, greatly improving program execution efficiency and object storage efficiency. Furthermore, by decoupling data and metadata, it enables horizontal scaling of the data cluster, alleviating access and storage pressure on a single data cluster, further improving the input / output efficiency of the entire resource pool, achieving fault isolation of the data cluster, and improving cluster stability. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic flowchart of a first embodiment of a method for separating object storage file data and metadata, provided by an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of the distribution of object data and metadata before an improvement to the object storage file data and metadata separation storage method provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram illustrating the distribution of object data and metadata, provided in an embodiment of this application, representing an improved method for separating and storing object storage file data and metadata.

[0046] Figure 4 This is a flowchart illustrating the migration process of a single object's metadata, provided in an embodiment of this application, for a method of separating object storage file data and metadata.

[0047] Figure 5 This is a flowchart illustrating the metadata migration process of a method for separating object storage file data and metadata, provided in an embodiment of this application.

[0048] Figure 6 This is a schematic diagram of the metadata separation and upgrade process of an object storage file data and metadata separation storage method provided in an embodiment of this application;

[0049] Figure 7 This is a schematic flowchart of a second embodiment of a method for separating object storage file data and metadata, provided by an embodiment of this application;

[0050] Figure 8 This is an improved flowchart of the file upload process for separating object storage file data and metadata, provided by an embodiment of this application.

[0051] Figure 9 A schematic block diagram of a separate storage device for object storage file data and metadata provided for embodiments of this application;

[0052] Figure 10 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0055] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0056] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0057] Embodiments of this application provide a method, apparatus, computer device, and storage medium for separating object storage file data and metadata. This method for separating object storage file data and metadata can be applied to servers, improving object storage efficiency and cluster stability by storing data and metadata separately. The server can be a standalone server or a server cluster.

[0058] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0059] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a method for separating object storage file data and metadata, provided in an embodiment of this application. This method can be applied to servers to improve object storage efficiency and cluster stability by storing data and metadata separately.

[0060] like Figure 1 As shown, the method for separating the storage of object storage file data and metadata specifically includes steps S101 to S102.

[0061] S101. Based on preset configuration parameters, configure the file attributes and read / write information to obtain migration configuration information;

[0062] In one embodiment, Ceph is a unified distributed file system with excellent performance, high availability, and scalability. Ceph's unification lies in its ability to provide file system, block storage, and object storage, while its distributed nature is reflected in its dynamic scalability. All storage—whether block devices, object storage, or file storage—ultimately transforms into underlying objects. Each object contains three elements: data, xattr, and omap. Data stores the object's data, while xattr and omap store extended attributes of the object, stored in Ceph's built-in key-value database, rocksDB, primarily used to enumerate files within a bucket. Due to the characteristics of xattr and omap, both single and multiple versions have corresponding version metadata. Therefore, xattr can be stored in omap, and the xattr for a specific version can be stored in the corresponding version, effectively migrating xattr from the Linux file system's extended attributes to rocksDB.

[0063] In one embodiment, to separate metadata on the original cluster without affecting normal file upload and download services, the omap attribute of the file is redesigned. In addition to migrating the xattr attribute to the omap attribute, a read / write flag needs to be added to ensure normal file read and write operations at different upgrade stages. For this purpose, the xattr_pos extended attribute is added, which records the storage location of file metadata. The default value is 0, where: 0 indicates that the metadata of the object exists in the header object of the Data pool; 1 indicates that the metadata of the object is located in the omap attribute of the rados object in the Index pool; 2 indicates that the metadata of the object is located in both the header object and the omap.

[0064] In one embodiment, is_complete: a write completion flag, exists in the omap index data structure. This field is used in the double-write process (writing to both omap and xattr) to update omap, and is used to resolve the data inconsistency caused by the failure to update omap when writing the header object successfully but updating omap fails during the double-write process, which affects the read results. The default value is true.

[0065] In one embodiment, to ensure the metadata separation write process after upgrading on the original cluster, two configuration items are designed: rgw_xattrs_write_flag and rgw_xattrs_read_flag.

[0066] Among them, rgw_xattrs_write_flag: the default value is 1. A value of 1 indicates that only the header object is written. If xattr_pos is found to be 1, the object is double-written. A value of 2 indicates that only the omap is written. A value of 3 indicates that the omap is written first and then the header object is written. rgw_xattrs_read_flag: the default value is 1. A value of 1 indicates that only the header object is read. A value of 2 indicates that only the omap is read. A value of 3 indicates that the omap is read first, and the header object is read only if it is not found.

[0067] In one embodiment, a migration tool is provided to support migrating the metadata of old objects from the xattr of the head object in the data pool to the omap of the bucket.index pool during the upgrade process. Therefore, the xattrs_pos_status parameter is added to RGWBucketInfo to identify whether the object has been migrated. The specific values ​​and meanings of the parameter are as follows:

[0068] 0: xattr is in the head object, not migrated, bucket created by the old cluster, default value is 0.

[0069] 1: xattr is in the head object or omap during migration.

[0070] 2: xattr in omap indicates that a new bucket has been created after migration or upgrade.

[0071] In one embodiment, to record the migration status of each object in each bucket in real time, a special namespace `xattr_migration` is added to the log pool. Only one rados object is created in this namespace, named `xattrs_migration`. Its OMAP header records the migration status of the `xattr` object and maintains an OMAP list recording the buckets currently being migrated. The `omap_key` is `bucket_name`, and the `omap_value` is a structure `schedule_migration`, defined as follows:

[0072] 0: uninitial, uninitialized, the initial value after the upgrade, meaning that xattr data migration was not performed using the command-line tool after the upgrade. When the current migration status is this value, a background thread needs to check the migration status periodically;

[0073] 1: Processing, in progress. After using the command-line tool to execute the migration of xattr data, the migration status changes to this value. When the background thread detects that the migration status is this value, it starts executing the xattr migration process.

[0074] 2: processed, migration complete. When the background thread detects this migration status, it can exit the thread to save CPU resources.

[0075] In one embodiment, to avoid affecting normal business operations, two configuration items are designed to specify the migration work period and interval: rgw_xattrs_migration_work_time, the working time period of the metadata migration thread, which defaults to 00:00-06:00; and rgw_xattrs_migration_debug_interval, the execution interval of the metadata migration thread, which defaults to 10 minutes.

[0076] S102. Based on the migration configuration information, execute the preset metadata migration process to migrate the metadata of the object to be migrated in the data pool to the bucket.index data pool for storage, so as to realize the separate storage of object storage file data and metadata.

[0077] In one embodiment, as shown in 2, Figure 2The left side shows the OMAP metadata information of files in the index pool. This metadata is stored in the OMAP of a specific bucket shard. A bucket shard can store OMAP metadata for multiple files, thus forming a one-to-many relationship. The right side shows the metadata of files in the data pool, which is mainly stored in the header object. As shown in the diagram, for a segmented uploaded file object obj1, an underlying layer generates one header object, multiple multipart segment header objects, and multiple shadow segment footer objects. The header object only stores the xattr metadata information of the object and does not store any data information.

[0078] In one embodiment, based on the aforementioned metadata storage method, it's conceivable that if the xattr metadata information stored in the head object of obj1 is migrated to the corresponding omap attribute of obj1 in the index pool, then one less rados object will be generated at the underlying level, thus reducing one underlying op operation. For storage scenarios with massive amounts of objects, the reduction in the number of rados will significantly improve storage efficiency. The improved object metadata-data distribution is as follows: Figure 3 As shown.

[0079] Furthermore, the step of executing a preset metadata migration process based on the migration configuration information to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage further includes: when the object to be migrated is a cluster, reading the migration status of the metadata; when the migration status is in progress, obtaining a list of buckets to be migrated and locking the first bucket in the list; obtaining the bucket information of the first bucket and, based on the bucket information, obtaining the metadata migration stage of the first bucket; when the metadata migration stage is that the migration is complete and the migration time is no greater than a preset time, unlocking the first bucket and deleting the first bucket from the list of buckets to be migrated, and traversing all buckets in the list of buckets to be migrated in sequence; when the list of buckets to be migrated is empty, modifying the migration status of the metadata to complete, so as to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage.

[0080] In one embodiment, the migration configuration information as described above includes the working period and interval for data migration, and the data migration work is performed based on the working period and interval.

[0081] In a specific embodiment, the object to be migrated can be the entire cluster, and the migration process of the entire cluster is as follows: Figure 4As shown, when cluster metadata migration begins, the xattr migration status is read. If the migration status is "processing", the bucket list is obtained, the first bucket in the list is locked, and the bucket info is obtained. If xattrs_pos_status does not equal 2, the bucket is added to the migration list, and the migration progress is read. If xattrs_pos_status equals 1, it is checked whether time_now - last_time is greater than 1 hour. If time_now - last_time is less than 1 hour, the bucket is unlocked and removed from the migration list. The second bucket in the bucket list is obtained, and the above operations are performed on the second bucket. All buckets in the bucket list are traversed. After the traversal is completed, it is checked whether the migration list is empty. If the migration list is empty, the xattr migration status is modified to "processed".

[0082] In another embodiment, when xattrs_pos_status is not equal to 1, xattrs_pos_status is set to 1, and the bucket is unlocked. 1000 objects are obtained based on the Starmark. The xattr position of the first object is obtained, and it is determined whether the position is in omap. If not, the xattr is read from the head object and written to omap, and the position of the xattr data in omap is modified. When all 1000 objects have been traversed, the migration progress of the bucket for the current object is recorded, and the last_time is recorded. When all objects in the bucket have been traversed, the bucket is locked. When xattrs_pos_status = 2, the bucket is unlocked, the bucket is deleted from the migration list, and the second bucket is obtained. When all buckets have been traversed and the migration list is empty, the xattr migration status is modified to processed.

[0083] Further, the step of executing a preset metadata migration process based on the migration configuration information to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage includes: when the object to be migrated is a single file object, obtaining the address tag idtag of the object to be migrated based on the identifier of the object to be migrated input by the user; obtaining the location of the metadata based on the idtag, and determining whether the location of the metadata is in omap based on the migration configuration information; when the location of the metadata is not in the extended attribute omap of the bucket.index data pool, writing the metadata into the extended attribute omap, and checking whether the object to be migrated has been deleted; if the object to be migrated has not been deleted, checking whether the idtag has been updated; if the idtag has not been updated, modifying the location of the metadata to achieve the migration of the metadata.

[0084] In a specific embodiment, the object to be migrated can also be a single file object, and the migration process for a single file object is as follows: Figure 5 As shown, the address tag of the object to be migrated is obtained based on the identifier of the object to be migrated, and the storage location of the metadata is obtained based on the address tag. It is then determined whether the storage location of the metadata is in omap. If it is not in omap, the metadata is obtained and written to omap. After successful writing, the address tag of the metadata is verified to determine whether the address tag has been updated. If it has not been updated, the location information of the metadata is modified to complete the metadata migration.

[0085] In one embodiment, the complete metadata separation upgrade process is as follows: Figure 6 As shown, upgrade CEPH to a metadata separation version, enabling single-write and single-read header objects to maintain consistency with the previous version. Then, enable dual-read and dual-write, and conduct gray-scale verification of read-write OMAP. Next, enable dual-read and single-write OMAP and perform metadata migration. After migrating the old data, enable single-read and single-write OMAP. At any stage of this process, if instability is detected, the read-write configuration items can be modified promptly to switch back to the original read-write header object process, ensuring that the business can quickly return to stable operation.

[0086] Please see Figure 7 , Figure 7 This is a schematic flowchart illustrating a method for separating object storage file data and metadata, provided in an embodiment of this application. This method can be applied to servers to improve object storage efficiency and cluster stability by storing data and metadata separately.

[0087] like Figure 7As shown, the method for separating the storage of object storage file data and metadata specifically includes steps S201 to S203.

[0088] S201. Upon receiving a file to be stored uploaded by a user, determine the target upload type of the file to be stored from the perspectives of normal upload and overwrite upload;

[0089] S202. Based on the target upload type, determine the target upload process;

[0090] S203. Based on the target upload process, upload and store the file to be stored.

[0091] Furthermore, the step of uploading and storing the file to be stored based on the target upload process includes: when the target upload type is a normal upload, writing the file to be stored into the data pool; generating metadata based on the file to be stored, and storing the metadata in the bucket.index data pool, so as to achieve separate storage of object storage file data and metadata.

[0092] Furthermore, the step of uploading and storing the file to be stored based on the target upload process includes: when the target upload type is overwrite upload, writing the file to be stored into the data pool; obtaining the extended attribute omap of the file to be stored from the bucket.index data pool; and overwriting the extended attribute omap of the historical file to be stored based on the extended attribute omap of the file to be stored, so as to update the metadata of the file to be stored and realize the separate storage of object storage file data and metadata.

[0093] Furthermore, after uploading and storing the file to be stored based on the target upload process, the method further includes: when the target upload type is a normal upload and the file to be stored fails to upload, or when the target upload type is an overwrite upload and the file to be stored fails to upload, sending the file to be stored to a garbage collection queue to await recycling; when the target upload type is an overwrite upload and the file to be stored is successfully uploaded, adding the historical header object of the historical storage file corresponding to the file to be stored and the historical storage file to the garbage collection queue to await recycling.

[0094] In one embodiment, a normal upload is the upload of a new file to be stored, while an overwrite upload is the upload of a file that has already been uploaded.

[0095] In one embodiment, such as Figure 8 As shown, for a gateway that has already enabled metadata separation configuration, the specific steps for uploading files to be stored are as follows:

[0096] Step 1: Write the tail object to the data pool;

[0097] Step 2: Retrieve the omap from bucket.index. Since the omap for the object uploaded for the first time has not yet been generated in the bucket.index pool, it is generated and written to the index pool, completing this upload process. At this point, the tail object is still stored in the data pool, while the metadata is stored in the index pool. If this upload fails, the tail object is added to the garbage collection queue to await reclaiming.

[0098] In one embodiment, for an overlay upload scenario, the specific process is as follows:

[0099] Step 1: Write the tail object to the data pool;

[0100] Step 2: Retrieve the omap object from bucket.index. For overwrite uploads, the omap object will always be available, and it will contain the object's xattr attribute.

[0101] Step 3: Overwrite the omap property of the object to update the metadata;

[0102] Step 4: Determine if the file migration is complete. Metadata separation and upgrade completion is based on the assumption that the old object's metadata has already been migrated. Therefore, the migration must be complete at this point. Add the old object's head and tail objects to the GC object. If writing to omap fails, add the current tail object to the GC queue for collection. Here, GC refers to the garbage collection queue.

[0103] The above embodiments provide a method for separating the storage of object storage file data and metadata, providing two different processing logics for uploading, downloading and deleting new and old data. This allows for the gradual migration of the xattr metadata of old file objects from the head object to the omap attribute of the index pool while maintaining compatibility with old file objects, thus providing better technical support for subsequent unified metadata cluster services.

[0104] Please see Figure 9 , Figure 9 This is a schematic block diagram of an embodiment of the present application providing a separate storage device for object storage file data and metadata. This separate storage device is used to execute the aforementioned separate storage method for object storage file data and metadata. The separate storage device can be configured on a server.

[0105] like Figure 9 As shown, the object storage file data and metadata separate storage device 300 includes:

[0106] The migration configuration information acquisition module 301 is used to configure the file attributes and read / write information based on preset configuration parameters to obtain migration configuration information;

[0107] The separate storage module 302 is used to execute a preset metadata migration process based on the migration configuration information, and migrate the metadata of the object to be migrated in the data pool to the bucket.index data pool for storage, so as to realize the separate storage of object storage file data and metadata.

[0108] Furthermore, the object storage file data and metadata separate storage device 300 further includes: a file upload storage module, the file upload storage module including:

[0109] The target upload type determination unit is used to determine the target upload type of the file to be stored when receiving a file uploaded by a user, in the context of normal upload and overwrite upload;

[0110] The target upload process determination unit is used to determine the target upload process based on the target upload type.

[0111] The file upload unit is used to upload and store the file to be stored based on the target upload process.

[0112] Furthermore, the file upload unit includes:

[0113] The storage file writing subunit is used to write the file to be stored into the data pool when the target upload type is normal upload;

[0114] The metadata generation subunit is used to generate metadata based on the file to be stored and store the metadata in the bucket.index data pool to achieve separate storage of object storage file data and metadata.

[0115] Furthermore, the file upload unit includes:

[0116] The storage file writing subunit is used to write the file to be stored into the data pool when the target upload type is overwrite upload;

[0117] The extended attribute omap is obtained as a sub-unit, used to retrieve the extended attribute omap of the file to be stored from the bucket.index data pool;

[0118] The extended attribute omap override subunit is used to override the extended attribute omap of the historical file to be stored based on the extended attribute omap of the file to be stored, so as to update the metadata of the file to be stored and realize the separate storage of object storage file data and metadata.

[0119] Furthermore, the file upload and storage module also includes:

[0120] The waiting-for-recycling unit is used to send the file to be stored to the garbage collection queue to wait for recycling when the target upload type is normal upload and the file to be stored fails to upload, or when the target upload type is overwrite upload and the file to be stored fails to upload.

[0121] The waiting-for-recycling unit is used to add the historical header object of the historical storage file corresponding to the file to be stored and the historical storage file to the garbage collection queue when the target upload type is overwrite upload and the file to be stored is successfully uploaded, so as to wait for recycling.

[0122] Furthermore, the separate storage module 302 includes:

[0123] The address tag idtag acquisition unit is used to acquire the address tag idtag of the object to be migrated based on the identifier of the object to be migrated input by the user when the object to be migrated is a single file object;

[0124] The location determination unit is used to obtain the location of the metadata based on the idtag, and determine whether the location of the metadata is in omap based on the migration configuration information;

[0125] The metadata writing unit is used to write the metadata to the extended attribute omap when the location of the metadata is not in the bucket.index data pool, and to check whether the object to be migrated has been deleted.

[0126] The idtag verification unit is used to verify whether the idtag has been updated when the migrated object has not been deleted.

[0127] The location modification unit is used to modify the location of the metadata if the idtag is not updated, so as to realize the migration of the metadata.

[0128] Furthermore, the separate storage module 302 also includes:

[0129] The migration status reading unit is used to read the migration status of metadata when the object to be migrated is a cluster;

[0130] A bucket locking unit is used to obtain a list of buckets to be migrated when the migration status is in progress, and to lock the first bucket in the list of buckets to be migrated.

[0131] The migration phase acquisition unit is used to acquire the bucket information of the first bucket and, based on the bucket information, acquire the first bucket metadata migration phase.

[0132] The bucket unlocking unit is used to unlock the first bucket and delete the first bucket from the list of buckets to be migrated when the metadata migration phase is completed and the migration time is no greater than a preset time, and then traverse all buckets in the list of buckets to be migrated in turn.

[0133] The migration status modification unit is used to modify the migration status of the metadata to "completed" when the list to be migrated is empty, so as to migrate the metadata of the object to be migrated in the data pool to the bucket.index data pool for storage.

[0134] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0135] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 10 It runs on the computer device shown.

[0136] Please see Figure 10 , Figure 10 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0137] See Figure 10 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0138] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any method of separating object storage file data from metadata.

[0139] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0140] Internal memory provides an environment for the execution of computer programs on non-volatile storage media. When the computer program is executed by the processor, it enables the processor to perform any method of separating object storage file data and metadata.

[0141] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0143] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0144] Based on preset configuration parameters, the file attributes and read / write information are configured to obtain migration configuration information;

[0145] Based on the migration configuration information, a preset metadata migration process is executed to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage, so as to achieve separate storage of object storage file data and metadata.

[0146] In one embodiment, after the processor performs a preset metadata migration process based on the migration configuration information to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage, thereby achieving separate storage of object storage file data and metadata, it is also used to implement:

[0147] Upon receiving a file uploaded by a user, the target upload type of the file to be stored is determined between normal upload and overwrite upload.

[0148] Based on the target upload type, determine the target upload process;

[0149] Based on the target upload process, the file to be stored is uploaded and stored.

[0150] In one embodiment, when the processor implements the process of determining the target upload process based on the target upload type, it is configured to:

[0151] When the target upload type is normal upload, the file to be stored is written to the data pool;

[0152] Metadata is generated based on the file to be stored, and the metadata is stored in the bucket.index data pool to achieve separate storage of object storage file data and metadata.

[0153] In one embodiment, when the processor uploads and stores the file to be stored based on the target upload process, it is configured to:

[0154] When the target upload type is overwrite upload, the file to be stored is written to the data pool;

[0155] Retrieve the extended attribute omap of the file to be stored from the bucket.index data pool;

[0156] Based on the extended attribute omap of the file to be stored, the extended attribute omap of the historical file to be stored is overridden to update the metadata of the file to be stored and realize the separate storage of object storage file data and metadata.

[0157] In one embodiment, after the processor uploads and stores the file to be stored based on the target upload process, it is further configured to:

[0158] When the target upload type is normal upload and the file to be stored fails to upload, or when the target upload type is overwrite upload and the file to be stored fails to upload, the file to be stored is sent to the garbage collection queue to wait for recycling.

[0159] When the target upload type is overwrite upload and the file to be stored is successfully uploaded, the historical header object of the historical stored file corresponding to the file to be stored and the historical stored file are added to the garbage collection queue to wait for recycling.

[0160] In one embodiment, when the processor performs a preset metadata migration process based on the migration configuration information to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage, it is used to:

[0161] When the object to be migrated is a single file object, the address tag (idtag) of the object to be migrated is obtained based on the identifier of the object to be migrated input by the user;

[0162] The location of the metadata is obtained based on the idtag, and the location of the metadata is determined in omap based on the migration configuration information;

[0163] When the location of the metadata is not in the extended property omap of the bucket.index data pool, the metadata is written to the extended property omap, and it is checked whether the object to be migrated has been deleted;

[0164] If the migrated object has not been deleted, check whether the idtag has been updated;

[0165] If the idtag is not updated, the location of the metadata is modified to achieve the migration of the metadata.

[0166] In one embodiment, when the processor performs a preset metadata migration process based on the migration configuration information to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage, it is also configured to:

[0167] When the object to be migrated is a cluster, read the migration status of the metadata;

[0168] When the migration status is in progress, obtain the list of buckets to be migrated, and lock the first bucket in the list of buckets to be migrated;

[0169] Obtain the bucket information of the first bucket, and based on the bucket information, obtain the first bucket metadata migration stage;

[0170] When the metadata migration phase is completed and the migration time is no more than a preset time, the first bucket is unlocked and removed from the list of buckets to be migrated. Then, all buckets in the list of buckets to be migrated are traversed in turn.

[0171] When the list to be migrated is empty, the migration status of the metadata is modified to complete, so that the metadata of the object to be migrated in the data pool is migrated to the bucket.index data pool for storage.

[0172] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the object storage file data and metadata separation storage methods provided in the embodiments of this application.

[0173] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for separating and storing object storage file data and metadata, characterized in that, include: Based on preset configuration parameters, the file attributes and read / write information are configured to obtain migration configuration information; Based on the migration configuration information, a preset metadata migration process is executed to migrate the metadata of the object to be migrated in the data pool to the bucket.index data pool for storage, so as to achieve separate storage of object storage file data and metadata; The step of executing a preset metadata migration process based on the migration configuration information, migrating the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage, includes: When the object to be migrated is a single file object, the address tag (idtag) of the object to be migrated is obtained based on the identifier of the object to be migrated input by the user; The location of the metadata is obtained based on the idtag, and the location of the metadata is determined in omap based on the migration configuration information; When the location of the metadata is not in the extended property omap of the bucket.index data pool, the metadata is written to the extended property omap, and it is checked whether the object to be migrated has been deleted; If the migrated object has not been deleted, check whether the idtag has been updated; If the idtag is not updated, the location of the metadata is modified to achieve the migration of the metadata.

2. The method for separating and storing object storage file data and metadata according to claim 1, characterized in that, Based on the migration configuration information, a preset metadata migration process is executed to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage, thereby achieving separate storage of object storage file data and metadata. The process also includes: Upon receiving a file uploaded by a user, the target upload type of the file to be stored is determined between normal upload and overwrite upload. Based on the target upload type, determine the target upload process; Based on the target upload process, the file to be stored is uploaded and stored.

3. The method for separating and storing object storage file data and metadata according to claim 2, characterized in that, Based on the target upload process, the file to be stored is uploaded and stored, including: When the target upload type is normal upload, the file to be stored is written to the data pool; Metadata is generated based on the file to be stored, and the metadata is stored in the bucket.index data pool to achieve separate storage of object storage file data and metadata.

4. The method for separating and storing object storage file data and metadata according to claim 2, characterized in that, Based on the target upload process, the file to be stored is uploaded and stored, including: When the target upload type is overwrite upload, the file to be stored is written to the data pool; Retrieve the extended attribute omap of the file to be stored from the bucket.index data pool; Based on the extended attribute omap of the file to be stored, the extended attribute omap of the historical file to be stored is overridden to update the metadata of the file to be stored and realize the separate storage of object storage file data and metadata.

5. The method for separating and storing object storage file data and metadata according to claim 2, characterized in that, Based on the target upload process, after uploading and storing the file to be stored, the process further includes: When the target upload type is normal upload and the file to be stored fails to upload, or when the target upload type is overwrite upload and the file to be stored fails to upload, the file to be stored is sent to the garbage collection queue to wait for recycling. When the target upload type is overwrite upload and the file to be stored is successfully uploaded, the historical header object of the historical stored file corresponding to the file to be stored and the historical stored file are added to the garbage collection queue to wait for recycling.

6. The method for separating and storing object storage file data and metadata according to any one of claims 1 to 5, characterized in that, Based on the migration configuration information, a preset metadata migration process is executed to migrate the metadata of the object to be migrated from the data pool to the bucket.index data pool for storage. This process also includes: When the object to be migrated is a cluster, read the migration status of the metadata; When the migration status is in progress, obtain the list of buckets to be migrated, and lock the first bucket in the list of buckets to be migrated; Obtain the bucket information of the first bucket, and based on the bucket information, obtain the first bucket metadata migration stage; When the metadata migration phase is completed and the migration time is no more than a preset time, the first bucket is unlocked and removed from the list of buckets to be migrated. Then, all buckets in the list of buckets to be migrated are traversed in turn. When the list of buckets to be migrated is empty, the migration status of the metadata is modified to "completed" to migrate the metadata of the objects to be migrated in the data pool to the bucket.index data pool for storage.

7. A separate storage device for object storage file data and metadata, characterized in that, include: The migration configuration information acquisition module is used to configure the file attributes and read / write information based on preset configuration parameters to obtain migration configuration information. The separate storage module is used to execute a preset metadata migration process based on the migration configuration information, and migrate the metadata of the object to be migrated in the data pool to the bucket.index data pool for storage, so as to realize the separate storage of object storage file data and metadata; The separate storage module is further used for: When the object to be migrated is a single file object, the address tag (idtag) of the object to be migrated is obtained based on the identifier of the object to be migrated input by the user; The location of the metadata is obtained based on the idtag, and the location of the metadata is determined in omap based on the migration configuration information; When the location of the metadata is not in the extended property omap of the bucket.index data pool, the metadata is written to the extended property omap, and it is checked whether the object to be migrated has been deleted; If the migrated object has not been deleted, check whether the idtag has been updated; If the idtag is not updated, the location of the metadata is modified to achieve the migration of the metadata.

8. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, when executing the computer program, implement the method for separating object storage file data and metadata as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method for separating object storage file data and metadata as described in any one of claims 1 to 6.

Images