A method, apparatus and medium for storing object data

Abstract

The application discloses a kind of object data storage method, device and medium, applicable to data storage technical field.AEP file is written into AEP file with the help of the characteristics of low latency, non-volatile, long life of object data to be aggregated, control aggregation thread traverses the object data in AEP file;When object data meets the aggregation requirement, the object data that meets the aggregation requirement is aggregated into rados object, and is flushed to HDD disk to complete the storage of object data.The existing offline aggregation is adjusted from NVME disk to HDD disk data writing to AEP file, which alleviates the write amplification, flushing problem of existing offline aggregation mode, improves the writing speed and aggregation efficiency of object, reduces the number of rados objects through object aggregation, improves the reconstruction speed and system performance, and improves the space utilization and fault recovery speed of HDD disk.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a method, apparatus and medium for storing object data. Background Technology

[0002] With the development of the Internet, the amount of Internet data is growing larger and larger, and distributed object storage has emerged. This new storage method is suitable for storing unstructured data such as pictures and videos. It also has the characteristics of high-speed direct access to disk in block storage and distributed sharing in file storage, so it has been widely developed and applied.

[0003] Currently, in distributed object storage, the massive amount of small object data presents challenges. Recovering data from these small objects individually during storage failures leads to lengthy recovery times and impacts user operations. To address this, a method called small object aggregation is implemented, combining multiple small objects into a single large object. This large object is then written to a cache pool on a solid-state drive (SSD) using the Non-Volatile Memory Express (NVMe) protocol. An aggregation thread then scans the small object data offline, aggregating it into a large object before flushing it to a data pool on a mechanical hard drive (HDD). However, NVMe exhibits significant write-erase cycles; frequent writes can cause write amplification during aggregation, potentially affecting the lifespan of the NVMe SSD by deleting data from it. Furthermore, the offline aggregation method requires frequent scanning of small objects. Given the massive amount of small object data, the flushing process may consume threads from other business processes, leading to delayed flushing, reduced storage reliability, and a negative impact on user experience.

[0004] Therefore, finding a method for storing object data is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, and medium for storing object data, which alleviates the problems of write amplification and untimely flushing in existing offline aggregation methods, and improves the writing speed and aggregation efficiency of objects.

[0006] To address the aforementioned technical problems, the present invention provides a method for storing object data, comprising:

[0007] Retrieve the object data to be aggregated that the front-end business logic has written to the local system;

[0008] Save the data portion of the object data to an AEP file, where the object data includes the data portion and the metadata portion;

[0009] Control the aggregation thread to traverse object data within the AEP file;

[0010] When object data meets the aggregation requirements, the object data that meets the aggregation requirements is aggregated into a rados object and flushed to the HDD disk to complete the storage of the object data.

[0011] Preferably, before obtaining the object data to be aggregated that the front-end business writes to the local system, the method further includes:

[0012] Receive the initial object data written to the local system and the corresponding storage bytes;

[0013] Determine whether the storage bytes of the initial object data are less than or equal to the preset storage bytes;

[0014] If so, then the initial object data is determined to be the object data to be aggregated.

[0015] Preferably, the aggregation requirement is that the number of bytes stored in the object data equals a storage threshold, wherein the storage threshold is greater than a preset storage byte. Determining that the object data meets the aggregation requirement includes:

[0016] Retrieve the storage bytes corresponding to the object data in the AEP file;

[0017] Select the first N object data and count the number of bytes stored in the first N object data;

[0018] Determine whether the storage bytes of the first N object data have reached the storage threshold;

[0019] If so, then the first N object data are determined to meet the aggregation requirements;

[0020] If not, select the (N+1)th object data, and use the previous (N+1)th object data as the new previous N object data, and return to the step of selecting the previous N object data and counting the storage bytes of the previous N object data, until the storage bytes of the object data in the aggregation task meet the aggregation requirements.

[0021] Preferably, it further includes:

[0022] Obtain the location and offset of the data portion of the object data within the AEP file;

[0023] Store the location information, offset, and metadata of the object data to the NVME disk;

[0024] The metadata portion is stored on the NVMe disk, including:

[0025] The metadata portion is stored on the NVME disk according to the mapping method between bucket ID and object name.

[0026] Preferably, it further includes:

[0027] Retrieve the data of the object to be read;

[0028] Find the corresponding radios object in the NVME disk based on the bucket ID and object name of the object data to be read;

[0029] Find the location information of the data part of the radios object data to be read based on the metadata part of the radios object data to be read.

[0030] Determine whether the current state of the object data to be read is an aggregated state;

[0031] If so, then read and write the data portion of the object to be read in the HDD disk according to the location information;

[0032] If not, then read and write the data portion of the object to be read in the AEP file based on the location information.

[0033] Preferably, after saving the data portion of the object data to the AEP file, the method further includes:

[0034] Save the record information corresponding to the object data to the AEP file so that the object data storage process can be restarted after a power failure, and the record information of the AEP file can be traversed and read into memory.

[0035] Preferably, after aggregating the object data that meets the aggregation requirements into a rados object and flushing it to the HDD, the method further includes:

[0036] Modify the metadata, location information, and offset of the object data corresponding to the rados object and store them to the NVME disk;

[0037] Delete the object data and record information of the rados object in the AEP file.

[0038] To address the aforementioned technical problems, the present invention also provides a storage device for object data, comprising:

[0039] The acquisition module is used to acquire object data to be aggregated that the front-end business writes to the local system;

[0040] The save module is used to save the data portion of object data to an AEP file, where the object data includes the data portion and the metadata portion;

[0041] The traversal module is used to control the aggregation thread to traverse the object data within the AEP file;

[0042] The aggregation module is used to aggregate object data that meets the aggregation requirements into rados objects and then flush them to the HDD disk to complete the storage of the object data.

[0043] To address the aforementioned technical problems, the present invention also provides a storage device for object data, comprising:

[0044] Memory, used to store computer programs;

[0045] A processor is a step used to implement methods for storing object data as described above when executing a computer program.

[0046] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the object data storage method described above.

[0047] This invention provides a method for storing object data, comprising: acquiring object data to be aggregated written to the local system by front-end business; saving the data portion of the object data to an AEP file, wherein the object data includes a data portion and a metadata portion; controlling an aggregation thread to traverse the object data within the AEP file; when the object data meets the aggregation requirements, aggregating the object data that meets the aggregation requirements into a rados object, and flushing it to the HDD disk to complete the storage of the object data. This method leverages the low latency, non-volatility, and long lifespan characteristics of AEP files to write the object data to be aggregated into the AEP file, adjusting the existing offline aggregation method from writing data from the NVMe disk to the HDD disk to writing data from the AEP file to the HDD disk. This alleviates the write amplification and untimely flushing problems of existing offline aggregation methods, improves the object writing speed and aggregation efficiency, reduces the number of rados objects through object aggregation, improves reconstruction speed and system performance, and simultaneously improves HDD disk space utilization and fault recovery speed.

[0048] In addition, the present invention also provides an object data storage device and medium, which have the same beneficial effects as the object data storage method described above. Attached Figure Description

[0049] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating an object data storage method provided in an embodiment of the present invention;

[0051] Figure 2 A structural diagram of an object data storage device provided in an embodiment of the present invention;

[0052] Figure 3This is a structural diagram of another object data storage device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0054] The core of this invention is to provide a method, apparatus, and medium for storing object data, which alleviates the problems of write amplification and untimely flushing in existing offline aggregation methods, and improves the writing speed and aggregation efficiency of objects.

[0055] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] It should be noted that the object data storage method provided by this invention is mainly for the storage of small object aggregation. Small object aggregation involves aggregating small objects smaller than 512K within the same bucket into a large data object and storing it in the radios resource pool. Object storage is a new network storage architecture that combines the high-speed direct disk access of Storage Area Network (SAN) with the distributed sharing characteristics of Network-Attached Storage (NAS).

[0057] Figure 1 A flowchart illustrating an object data storage method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0058] S11: Obtain the object data to be aggregated that the front-end business writes to the local system;

[0059] S12: Save the data portion of the object data to an AEP file, where the object data includes the data portion and the metadata portion;

[0060] S13: Controls the aggregation thread to traverse object data within the AEP file;

[0061] S14: When the object data meets the aggregation requirements, the object data that meets the aggregation requirements is aggregated into a rados object and flushed to the HDD disk to complete the storage of the object data.

[0062] Specifically, in distributed object storage, aggregation functionality needs to be enabled on the premise of tiered processing. When front-end business logic writes data to the local system, massive amounts of object data are generated. Since this invention primarily addresses the storage problem of small object data, it is necessary to perform a preliminary assessment of the object data to determine which data is to be aggregated and which data does not require aggregation.

[0063] In a preferred embodiment, before obtaining the object data to be aggregated written to the local system by the front-end business in step S11, the following steps are included in determining the object data to be aggregated:

[0064] Receive the initial object data written to the local system and the corresponding storage bytes;

[0065] Determine whether the storage bytes of the initial object data are less than or equal to the preset storage bytes;

[0066] If so, then the initial object data is determined to be the object data to be aggregated.

[0067] For example, if the preset storage size is 512KB, it checks whether the storage size of the obtained initial object data is less than or equal to the preset storage size. If so, it indicates that the storage size of the initial object data is small and requires subsequent aggregation, thus it is determined as object data to be aggregated. If not, it indicates that the storage size of the initial object data is large, and aggregation is not required; it can be stored directly as the current initial object data. The metadata portion of the initial object data is stored on the NVMe disk, while the data portion is written directly to the HDD disk.

[0068] After obtaining the object data to be aggregated in step S11, the data portion of the object data is saved to the AEP file. It should be noted that the object data consists of two parts: a data portion and a metadata portion. The metadata portion is data about the data itself. In some cases, it doesn't specifically refer to a single piece of data; it can be understood as a group of information / data used to describe the data. All the data and information in this group / data describes / reflects a certain aspect of the data, and thus this group / data can be called metadata.

[0069] Metadata can describe the elements or attributes of data (name, size, data type, etc.), or its structure (length, fields, data columns), or its related data (where it is located, how to contact it, owner).

[0070] The data is partially saved to the AEP file. Apache Pass is a new type of non-volatile Optance Memory device introduced by Intel. Located on the memory bus, it's a device between DRAM and hard disks, offering persistence, large capacity, and nanosecond-level read / write speeds. Currently, Linux-created AEP device nodes are also called pmem (e.g., / dev / pmem0), primarily treating PMEM as a block device similar to a disk. Therefore, file systems can be created on PMEM devices, making them appear no different from regular disks. However, the specific physical attributes of the devices are completely different. For example, in terms of read / write latency, PMEM can achieve levels close to DRAM, which is far beyond the reach of disks.

[0071] After the object data is stored in the AEP file, the AEP saves the record information of the object data so that if a power failure occurs during the storage process and a restart occurs, the record information in the AEP file can be directly traversed and read into memory. To reduce memory usage, only the metadata information of the object to be aggregated and its location information in the AEP are recorded in memory.

[0072] The aggregation thread is controlled to traverse the object data within the AEP file. Specifically, the aggregation thread is not specifically limited in this embodiment. It can be a regular thread used for subsequent aggregation processes, a thread that performs other business operations in non-aggregation situations, or a specially set-up aggregation thread that only provides aggregation.

[0073] When the traversed object data meets the aggregation requirements, it is aggregated into a rados object. In this embodiment, the aggregation requirements can be set by the number of stored bytes of the object data reaching a certain threshold, or by the number of aggregated object data and the number of stored bytes both reaching certain preset conditions. It is understood that if there is a large amount of object data in the AEP file, the object data that meets the aggregation requirements will be aggregated. For example, if there are 100 object data, but the first 20 object data meet the aggregation requirements, then the first 20 object data will be aggregated. The subsequent 80 object data will be checked to see if they meet the aggregation requirements; if so, they will be aggregated. Aggregation will continue as long as the traversed object data meets the aggregation requirements, until the remaining object data is insufficient to reach the threshold and cannot meet the aggregation requirements. In this case, it will wait for newly written object data to the AEP file.

[0074] For massive amounts of object data, aggregating them into a single large object (RADO object) reduces the amount of object data in the RADO system, accelerates cluster refactoring, and improves system performance. An object is the basic unit of data storage in an object storage system. Each object is a combination of data and a set of data attributes. Data attributes can be set according to application needs, including data distribution and quality of service. Each object maintains its own attributes, simplifying storage system management. Objects can vary in size. Object storage is a non-hierarchical data storage method, typically used in cloud computing environments. Unlike other data storage methods, object-based storage does not use a directory tree: data is stored as individual objects. Data is not placed in a directory hierarchy but exists at the same level within a flat address space. Applications identify each individual data object through a unique address. Each object can contain metadata that aids in retrieval and is designed for application-level (rather than user-level) access using APIs. After aggregation into a RADO object, it is flushed to the HDD disk to complete the storage of the object data.

[0075] This invention provides a method for storing object data, comprising: acquiring object data to be aggregated written to the local system by front-end business; saving the data portion of the object data to an AEP file, wherein the object data includes a data portion and a metadata portion; controlling an aggregation thread to traverse the object data within the AEP file; when the object data meets the aggregation requirements, aggregating the object data that meets the aggregation requirements into a rados object, and flushing it to the HDD disk to complete the storage of the object data. This method leverages the low latency, non-volatility, and long lifespan characteristics of AEP files to write the object data to be aggregated into the AEP file, adjusting the existing offline aggregation method from writing data from the NVMe disk to the HDD disk to writing data from the AEP file to the HDD disk. This alleviates the write amplification and untimely flushing problems of existing offline aggregation methods, improves the object writing speed and aggregation efficiency, reduces the number of rados objects through object aggregation, improves reconstruction speed and system performance, and simultaneously improves HDD disk space utilization and fault recovery speed.

[0076] Based on the above embodiments, the aggregation requirement is that the storage bytes of the object data are equal to a storage threshold, wherein the storage threshold is greater than a preset storage byte. Determining that the object data meets the aggregation requirement includes:

[0077] Retrieve the storage bytes corresponding to the object data in the AEP file;

[0078] Select the first N object data and count the number of bytes stored in the first N object data;

[0079] Determine whether the storage bytes of the first N object data have reached the storage threshold;

[0080] If so, then the first N object data are determined to meet the aggregation requirements;

[0081] If not, select the (N+1)th object data, and use the previous (N+1)th object data as the new previous N object data, and return to the step of selecting the previous N object data and counting the storage bytes of the previous N object data, until the storage bytes of the object data in the aggregation task meet the aggregation requirements.

[0082] Specifically, the storage threshold is greater than a preset storage byte limit. The preset storage byte limit is a standard for determining the object data to be aggregated. If the storage byte limit of the object data is less than or equal to the preset storage byte limit, it is determined to be object data to be aggregated. In this embodiment, the storage threshold is set according to the aggregation requirements, and it is much greater than the preset storage byte limit.

[0083] The process involves retrieving the storage bytes corresponding to the object data in the AEP file. Since the specific storage byte value varies for each object data item, it's necessary to first select the first N object data items and count their storage bytes. If the counted storage bytes of the first N object data items reach the storage threshold, then aggregation is possible, meaning the aggregation requirement is met, and the first N object data items are aggregated. If the threshold is not reached, it means the storage bytes of the current first N object data items are insufficient, and more object data items need to be selected to make up the difference. Therefore, the N+1 object data items are selected, meaning the current selection of the first N+1 object data items is used as the new first N object data items, and the process continues until the object data in the AEP file meets the aggregation requirement. If the current object data in the AEP file does not meet the aggregation requirement, the process continues to wait for new object data to be written to the AEP file and then continues the process.

[0084] For example, if an AEP file contains 100 object data entries, and the storage bytes of the first 20 object data entries do not reach the storage threshold, then the first 21 object data entries are selected for further evaluation until object data that meets the aggregation requirements is found. If the current 50 object data entries meet the aggregation requirements, then the aggregation of the first 50 object data entries is performed. The remaining 50 object data entries are then evaluated to see if they meet the aggregation requirements for the next rados object. This process continues until all object data in the AEP file is aggregated. If the storage threshold is 4MB, the aggregated rados object is then flushed to the HDD.

[0085] The process for determining whether object data meets aggregation requirements provided in this embodiment of the invention facilitates aggregation into rados objects, reduces the number of rados objects, and improves reconstruction speed and system performance.

[0086] Based on the above embodiments, the method further includes:

[0087] Obtain the location and offset of the data portion of the object data within the AEP file;

[0088] Store the location information, offset, and metadata of the object data to the NVME disk;

[0089] The metadata portion is stored on the NVMe disk, including:

[0090] The metadata portion is stored on the NVME disk according to the mapping method between bucket ID and object name.

[0091] Specifically, the location information and offset of the data portion within the AEP file are stored on the NVME disk for easy access to the object later. Simultaneously, the metadata portion of the object data is also stored on the NVME disk.

[0092] NVMe stands for Non-Volatile Memory express, a protocol that enables solid-state drives (SSDs) to run faster and is becoming increasingly popular among enterprise users.

[0093] NVMe (Non-Volatile Memory Fast) is a host controller interface and storage protocol used to accelerate data transfer between enterprise and client systems and solid-state drives (SSDs) via PCIe (High-Speed ​​Interconnection) for high-speed peripheral components in computers. Compared to SAS and SATA-based SSDs, the main advantages of NVMe-based PCIe SSDs are reduced latency in the host software stack, increased input / output operations per second (IOPS), and potentially lower power consumption, depending on the form factor and the number of PCIe lanes used.

[0094] The specific advantages of NVMe include:

[0095] ① Performance is improved several times over;

[0096] ② It can significantly reduce latency;

[0097] ③NVMe can increase the maximum queue depth from 32 to 64,000, and the IOPS capability of SSDs will also be greatly improved;

[0098] ④ Automatic power consumption state switching and dynamic energy management functions greatly reduce power consumption;

[0099] ⑤ The emergence of the NVMe standard solved the driver compatibility problem between different PCIe SSDs.

[0100] Correspondingly, the metadata portion is stored in the NVME disk, specifically named according to the bucket ID plus the object name, while the data portion is empty.

[0101] After aggregating the object data that meets the aggregation requirements into a rados object and flushing it to the HDD, the method also includes:

[0102] Modify the metadata, location information, and offset of the object data corresponding to the rados object and store them to the NVME disk;

[0103] Delete the object data and record information of the rados object in the AEP file.

[0104] To reduce memory usage and maintain real-time updates, after the rados object is flushed to the HDD, the metadata, location information, and offset of the object data corresponding to the rados object are modified and transferred to the current HDD. The modified metadata, location information, and offset are then stored on the NVME disk.

[0105] At the same time, delete the object data and record information of the rados object in the AEP file. The record information, as mentioned in the above embodiment, is for easy reading during the restart and power-down process. At this time, the object data has been aggregated into the rados object, and its corresponding object information is not used as a reference, so it is deleted.

[0106] The object writing process and the corresponding deletion content in the subordinate processes provided in this embodiment of the invention facilitate timely reading of updated content.

[0107] Based on the above embodiments, the method further includes:

[0108] Retrieve the data of the object to be read;

[0109] Find the corresponding radios object in the NVME disk based on the bucket ID and object name of the object data to be read;

[0110] Find the location information of the data part of the radios object data to be read based on the metadata part of the radios object data to be read.

[0111] Determine whether the current state of the object data to be read is an aggregated state;

[0112] If so, then read and write the data portion of the object to be read in the HDD disk according to the location information;

[0113] If not, then read and write the data portion of the object to be read in the AEP file based on the location information.

[0114] Specifically, in the above embodiment, writing was performed. The object reading process requires finding the rados object in the NVME disk based on the object's bucket ID and object name, and then locating the object data based on the rados object's metadata information.

[0115] At the same time, it is necessary to check the current aggregation status of the object data. If it is aggregated, the data portion is read from the HDD disk according to the location information; if it is not aggregated, the data portion of the object data to be read is read and written from the AEP file according to the location information.

[0116] The object reading process provided in this embodiment improves the reading speed through the above-described writing process.

[0117] The foregoing has described in detail various embodiments corresponding to the object data storage method. Based on this, the present invention also discloses an object data storage device corresponding to the above-described method. Figure 2 This is a structural diagram of an object data storage device provided in an embodiment of the present invention. Figure 2 As shown, the object data storage device includes:

[0118] Module 11 is used to obtain the object data to be aggregated written to the local system by the front-end business.

[0119] The storage module 12 is used to save the data portion of the object data to an AEP file, wherein the object data includes a data portion and a metadata portion;

[0120] Traversal module 13 is used to control the aggregation thread to traverse object data within the AEP file;

[0121] The aggregation module 14 is used to aggregate the object data that meets the aggregation requirements into a rados object and flush it to the HDD disk to complete the storage of the object data.

[0122] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0123] This invention provides an object data storage device, comprising: acquiring object data to be aggregated that is written to the local system by front-end business; saving the data portion of the object data to an AEP file, wherein the object data includes a data portion and a metadata portion; controlling an aggregation thread to traverse the object data within the AEP file; and when the object data meets the aggregation requirements, aggregating the object data that meets the aggregation requirements into a rados object and flushing it to the HDD disk to complete the storage of the object data. This device leverages the low latency, non-volatility, and long lifespan characteristics of AEP files to write the object data to be aggregated into the AEP file, adjusting the existing offline aggregation data writing from the NVMe disk to the HDD disk to AEP file to the HDD disk. This alleviates the write amplification and untimely flushing problems of existing offline aggregation methods, improves the object writing speed and aggregation efficiency, reduces the number of rados objects through object aggregation, improves reconstruction speed and system performance, and simultaneously improves HDD disk space utilization and fault recovery speed.

[0124] Figure 3 A structural diagram of another object data storage device provided in an embodiment of the present invention is shown below. Figure 3 As shown, the device includes:

[0125] Memory 21 is used to store computer programs;

[0126] Processor 22 is used to implement the steps of storing object data when executing a computer program.

[0127] The storage device for object data provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0128] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 22 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 22 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0129] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 21 is used to store at least the following computer program 211, which, after being loaded and executed by the processor 22, is capable of implementing the relevant steps of the object data storage method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. The operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the object data storage method, etc.

[0130] In some embodiments, the object data storage device may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0131] Those skilled in the field can understand, Figure 3 The structure shown does not constitute a limitation on the storage device for object data and may include more or fewer components than illustrated.

[0132] The processor 22 implements the object data storage method provided in any of the above embodiments by calling instructions stored in the memory 21.

[0133] This invention provides an object data storage device, comprising: acquiring object data to be aggregated that is written to the local system by front-end business; saving the data portion of the object data to an AEP file, wherein the object data includes a data portion and a metadata portion; controlling an aggregation thread to traverse the object data within the AEP file; and when the object data meets the aggregation requirements, aggregating the object data that meets the aggregation requirements into a rados object and flushing it to the HDD disk to complete the storage of the object data. This device leverages the low latency, non-volatility, and long lifespan characteristics of AEP files to write the object data to be aggregated into the AEP file, adjusting the existing offline aggregation data writing from the NVMe disk to the HDD disk to AEP file to the HDD disk. This alleviates the write amplification and untimely flushing problems of existing offline aggregation methods, improves the object writing speed and aggregation efficiency, reduces the number of rados objects through object aggregation, improves reconstruction speed and system performance, and simultaneously improves HDD disk space utilization and fault recovery speed.

[0134] Furthermore, the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor 22, it implements the steps of the object data storage method described above.

[0135] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above-described object data storage method.

[0137] The foregoing has provided a detailed description of a method for storing object data, a device for storing object data, and a medium for storing object data provided by the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0138] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A storage method of object data, characterized by, The method comprises: acquiring object data to be aggregated written by a front-end service to a local system; storing a data part of the object data into an AEP file, wherein the object data comprises the data part and a metadata part; controlling an aggregation thread to traverse the object data in the AEP file; when the object data meets an aggregation requirement, aggregating the object data meeting the aggregation requirement into a rados object and flushing the rados object to an HDD disk to complete storage of the object data.

2. The object data storage method of claim 1, wherein, Before the acquiring, the method further comprises: receiving initial object data written to the local system and corresponding storage bytes; determining whether the storage bytes of the initial object data are less than or equal to a preset storage byte; if yes, determining that the initial object data is the object data to be aggregated.

3. The object data storage method according to claim 2, wherein The aggregation requirement is that the storage bytes of the object data are equal to a storage threshold, wherein the storage threshold is greater than the preset storage byte, and determining that the object data meets the aggregation requirement comprises: acquiring storage bytes corresponding to the object data of the AEP file; selecting the first N object data and counting the storage bytes of the first N object data; determining whether the storage bytes of the first N object data reach the storage threshold; if yes, determining that the first N object data meet the aggregation requirement; if no, selecting the (N+1)th object data, taking the first N+1 object data as new first N object data, and returning to the step of selecting the first N object data and counting the storage bytes of the first N object data until the storage bytes of the object data in the aggregation task meet the aggregation requirement.

4. The object data storage method according to any one of claims 1 to 3, characterized by, The method further comprises: acquiring location information and an offset of the data part of the object data in the AEP file; storing the location information, the offset and the metadata part of the object data into an NVME disk; wherein storing the metadata part into the NVME disk comprises: storing the metadata part into the NVME disk according to a mapping mode of a bucket ID and an object name.

5. The object data storage method according to claim 4, wherein The method further comprises: acquiring object data to be read; finding a corresponding rados object in the NVME disk according to a bucket ID and an object name of the object data to be read; finding location information of a data part of the object data to be read according to a metadata part of the rados object of the object data to be read; determining whether a current state of the object data to be read is an aggregated state; if yes, reading and writing the data part of the object data to be read in the HDD disk according to the location information; if no, reading and writing the data part of the object data to be read in the AEP file according to the location information.

6. The object data storage method according to claim 5, wherein After the storing, the method further comprises: storing record information corresponding to the object data into the AEP file, so that the storage process of the object data can restart after power failure, and the record information in the AEP file is traversed and read into a memory.

7. The object data storage method according to claim 6, wherein After the object data meeting the aggregation requirement is aggregated into the rados object and flushed to the HDD disk, further comprising: modifying the metadata part of the object data corresponding to the rados object, the corresponding location information and the offset and storing to the NVME disk; deleting the object data and the record information of the rados object in the AEP file.

8. A storage apparatus of object data, characterized by, Comprising: an acquisition module, configured to acquire object data to be aggregated written by a front-end service into a local system; a saving module, configured to save a data part of the object data into an AEP file, wherein the object data comprises the data part and a metadata part; a traversal module, configured to control an aggregation thread to traverse the object data in the AEP file; an aggregation module, configured to aggregate object data meeting an aggregation requirement into a rados object when the object data meets the aggregation requirement, and flush the object data to an HDD disk to complete storage of the object data.

9. A storage apparatus of object data, characterized by, Comprising: a memory, configured to store a computer program; a processor, configured to execute the computer program to implement the steps of the object data storage method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the object data storage method according to any one of claims 1 to 7.

Images