An object storage method and apparatus

Abstract

The application provides an object storage method and device. In the method, each storage block in a storage device corresponds to usage state information belonging to the storage block. When storing an object, a determined bucket name for storing the object to be stored, an object name of the object to be stored, an object version number of the object to be stored, and a data size of the object to be stored are obtained. A target data disk for uploading the object to be stored is determined according to the bucket name, the object version number, and the data size. Object data of the object to be stored is flushed into the target data disk according to the usage state information corresponding to the target data disk. An index disk for storing object metadata of the object to be stored is determined according to the bucket name, and the object metadata of the object to be stored is flushed into the index disk. It can be seen that the technical solution provided by the embodiment can improve the system storage performance and further reduce the disk space overhead.

Description

Technical Field

[0001] This application relates to the field of storage, and more particularly to an object storage method and apparatus. Background Technology

[0002] Currently, when storing objects, whether uploaded via PUT or in segments, they are stored as files. Specifically, a file system is created on each disk, and each object is stored as a file within the file system. The management of individual files relies on the inherent characteristics of the file system, which are: before reading or writing a file, its location must be determined. For example, the parent directory of a single file stores its name and inode code. Each inode code corresponds to information such as the file's size, permissions, and the physical address of the data within the file. For instance, to read file c in a folder, where file c is within file b, and file b is within file a, with the path a / b / c, the search process is as follows: first, obtain the folder name a and its corresponding inode_a from the root directory; then, obtain the folder name b and its corresponding inode_b from inode_a; finally, obtain the file name c and its corresponding inode_c from inode_b; and finally, query the specific information of file c from inode_c.

[0003] As seen above, for a file, its parent directory stores the filenames of all files within that directory, along with the corresponding inode codes. The file's metadata information can be found based on the inode code. This metadata information includes the object's data size, modification time, number of data blocks, and their locations, and is stored in a fixed-size metadata space configured within the file system. On one hand, when the number of objects and files is too large, the inode information may not be contiguous, potentially leading to a large amount of random I / O. Excessive random I / O degrades storage performance. On the other hand, when the number of files is very large, storing the inode codes consumes a significant amount of disk space. If these files are relatively small, the metadata space used to store the inode codes may be full while the data space remains available, preventing further write operations to the disk containing the files. Furthermore, since the inode codes implicitly store information such as file permissions and creation / modification times, this information is useless to the stored objects themselves but still occupies disk space, resulting in some disk waste. Summary of the Invention

[0004] In view of this, this application provides an object storage method and apparatus to improve system storage performance while further reducing disk space overhead.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] In a first aspect, embodiments of this application provide an object storage method applied to a storage device. The storage device is configured with a data disk for storing object data and an index disk for storing object metadata. The disk space of each data disk is mapped into N consecutive storage blocks of the same size. Each storage block corresponds to a usage status information for that storage block. When the usage status information is a first value, it indicates that the storage block is in an available state; when the usage status information is a second value, it indicates that the storage block is in an unavailable state. The method includes:

[0007] Obtain the determined bucket name, object name, and version number of the object to be stored;

[0008] The target data disk for uploading the object to be stored is determined based on the bucket name, the object name, and the version number; and the object data of the object to be stored is flushed to the target disk according to the usage status information corresponding to the target data disk.

[0009] The index disk for storing the object metadata of the object to be stored is determined based on the bucket name, and the object metadata of the object to be stored is flushed to the index disk based on the bucket name, object name, and version number.

[0010] Secondly, embodiments of this application provide an object storage device applied to a storage device. The storage device is configured with a data disk for storing object data and an index disk for storing object metadata. The disk space of each data disk is mapped into N consecutive storage blocks of the same size. Each storage block corresponds to a usage status information for that storage block. When the usage status information is a first value, it indicates that the storage block is in an available state; when the usage status information is a second value, it indicates that the storage block is in an unavailable state. The device includes:

[0011] The information acquisition unit is used to acquire the determined bucket name, object name, and version number of the object to be stored, which are used to store the object to be stored.

[0012] The data storage unit is used to determine the target data disk for uploading the object to be stored based on the bucket name, the object name, and the version number; and to flush the object data of the object to be stored to the target disk according to the usage status information corresponding to the target data disk.

[0013] The metadata storage unit is used to determine the index disk for storing the object metadata of the object to be stored based on the bucket name, and to flush the object metadata of the object to be stored to the index disk based on the bucket name, object name, and version number.

[0014] Therefore, the object storage method and apparatus provided in this application embodiment utilize a storage device configured with a data disk for storing object data and an index disk for storing object metadata. The disk space of each data disk is mapped into N consecutive storage blocks of the same size. Each storage block corresponds to a usage status information. When the usage status information is a first value, it indicates that the storage block is available; when the usage status information is a second value, it indicates that the storage block is unavailable. When storing an object, the method obtains the determined bucket name, object name, and version number of the object to be stored; determines the target data disk for uploading the object to be stored based on the bucket name, version number, and object name; and flushes the object data of the object to be stored to the target disk according to the usage status information corresponding to the target data disk; determines the index disk for storing the object metadata of the object to be stored based on the bucket name, and flushes the object metadata of the object to be stored to the index disk according to the bucket name, object name, and version number. As can be seen, the method provided in this embodiment no longer uses the method of creating a file system to store objects, but instead directly stores objects in mapped storage blocks. This avoids the additional disk space overhead caused by inode encoding. In addition, by recording the available disk space according to the usage status information of each storage block, and sequentially acquiring the available disk space before writing objects, the disk seek time can be reduced when reading and writing objects, thereby improving the system's storage performance. It is evident that the technical solution provided in this embodiment can improve system storage performance while further reducing disk space overhead. Attached Figure Description

[0015] Figure 1 This is a flowchart of a data storage method provided by existing technology;

[0016] Figure 2 This is a flowchart illustrating an exemplary embodiment of the present application of an object storage method;

[0017] Figure 3 This is a flowchart illustrating an exemplary embodiment of the present application of an object reading method;

[0018] Figure 4 This is a schematic diagram of the structure of an object storage device shown in an exemplary embodiment of this application;

[0019] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0023] The two upload methods for the above objects are PUT upload and segmented upload. In the PUT upload method, each object corresponds to a unique file. The object's metadata and the data to be uploaded are stored in a single file named Hash(bucket name + object name).

[0024] The segmented upload method generally involves three steps, specifically:

[0025] Step 1: Initialize the segmented task and generate an UploadId that uniquely identifies the segmented task.

[0026] Step 2: Upload each segment, and identify each segment using PartNumber.

[0027] Step 3: Merge the segments, combining all segments into a single object.

[0028] To improve upload efficiency, instead of merging the various uploaded segments into a single file, an index file is used to logically organize the segments into a single object.

[0029] As can be seen, when storing objects, whether uploaded via PUT or in segments, they are stored as files. Specifically, a file system is created on each disk, and each object is stored as a file within that file system. The management of individual files relies on the inherent characteristics of the file system, which specifically require locating the file's position before reading or writing it. For a detailed example of a single file, please refer to [link to relevant documentation]. Figure 1 The parent directory of the file stores the file's name and its inode encoding, such as... Figure 1 As shown, the filenames in the parent directory and their corresponding inode codes are a.txt inode1, b.txt inode1, and c.txt inode1. Each inode code corresponds to information such as the file size, permissions, and physical address of the data within the file. Based on the file's inode1, we can obtain the file size, permissions, and physical address of the data within the file corresponding to that inode code. Based on the obtained information, we can obtain the data block stored in the data space corresponding to a.txt inode1, b.txt inode1, and c.txt inode1.

[0030] Taking multiple files as an example, for instance, if we want to read file c in a folder, where file c is in file b and file b is in file a, and the path is a / b / c, the search process is as follows: First, get the folder name a and its corresponding inode_a from the parent directory. Then, get the folder name b and its corresponding inode_b from inode_a. Next, get the file name c and its corresponding inode_c from inode_b. Finally, we can query the specific information of file c from inode_c. It can be seen that the deeper the directory, the more paths there are to find the file information, and the longer it takes.

[0031] As seen above, for a file, its parent directory stores the filenames of all files within that directory, along with the corresponding inode codes. The file's metadata information can be found based on the inode code. This metadata information includes the object's data size, modification time, number of data blocks, and their locations, and is stored in a fixed-size metadata space configured within the file system. On one hand, when the number of objects and files is too large, the inode information may not be contiguous, potentially leading to a large amount of random I / O. Excessive random I / O degrades storage performance. On the other hand, when the number of files is very large, storing the inode codes consumes a significant amount of disk space. If these files are relatively small, the metadata space used to store the inode codes may be full while the data space remains available, preventing further write operations to the disk containing the files. Furthermore, since the inode codes implicitly store information such as file permissions and creation / modification times, this information is useless to the stored objects themselves but still occupies disk space, resulting in some disk waste.

[0032] To address the aforementioned technical problems, this application provides an object storage method and apparatus. The method is applied to a storage device configured with a data disk for storing object data and an index disk for storing object metadata. The disk space of each data disk is mapped to N consecutive storage blocks of the same size. Each storage block corresponds to a usage status information. When the usage status information is a first value, it indicates that the storage block is available; when the usage status information is a second value, it indicates that the storage block is unavailable. The method includes: obtaining a determined bucket name, object name, and version number of the object to be stored; determining a target data disk for uploading the object to be stored based on the bucket name, version number, and object name; and flushing the object data of the object to be stored to the target disk according to the usage status information corresponding to the target data disk; determining an index disk for storing the object metadata of the object to be stored based on the bucket name, and flushing the object metadata of the object to be stored to the index disk according to the bucket name, object name, and version number.

[0033] Therefore, the method provided in this embodiment no longer uses the method of creating a file system to store objects, but instead directly stores objects in mapped storage blocks. This avoids the additional disk space overhead caused by inode encoding. In addition, by recording the available disk space according to the usage status information of each storage block, and sequentially acquiring the available disk space before writing objects, the disk addressing time can be reduced when reading and writing objects, thereby improving the system's storage performance. It is evident that the technical solution provided in this embodiment can improve system storage performance while further reducing disk space overhead.

[0034] The above object storage method will now be described with reference to specific embodiments, as follows:

[0035] See Figure 2 , Figure 2 This is a flowchart illustrating an object storage method according to an embodiment of this application. The method is applied to a storage device configured with a data disk for storing object data and an index disk for storing object metadata. The data disk and index disk can be deployed on different disks or on the same disk. However, for the same disk, since the storage space ratio between the data disk and the index disk is fixed, it is difficult to flexibly expand the index disk or data disk individually when the storage space corresponding to the index disk containing the metadata or the data disk containing the data is exhausted. Therefore, as an embodiment, the data disk and the index disk are configured on two different disks. The technical solution provided by this embodiment can isolate object metadata from object data. When the storage space corresponding to the index disk containing the object metadata or the data disk containing the object data is exhausted, the index disk or data disk can be flexibly expanded individually. Furthermore, by considering factors such as disk performance and price, it is possible to choose which type or model of disk to use as the index disk or the data disk, offering greater freedom and flexibility compared to using a single disk.

[0036] In this embodiment, the disk space of each data disk is mapped into N consecutive storage blocks of the same size. Each storage block corresponds to a usage status information of that storage block. When the usage status information is a first value, it indicates that the storage block is available. When the usage status information is a second value, it indicates that the storage block is unavailable.

[0037] In this embodiment, the first value and the second value do not specifically refer to two fixed values, but can refer to two different values. For example, the first value is 0 and the second value is 1, or the first value is 1 and the second value is 0, or the first value is A and the second value is B. This embodiment does not impose a fixed limitation on the first value and the second value.

[0038] The above Figure 2 The flowchart shown includes the following steps:

[0039] Step 201: Obtain the bucket name, object name, and version number of the bucket to be stored for storing the object to be stored.

[0040] A bucket is a basic storage entity in object storage, used as a container to hold objects. Therefore, the bucket name mentioned above is the name used to represent the bucket where the object to be stored is located; knowing the bucket name allows you to find that bucket.

[0041] An object is the basic entity stored in object storage. An object is a collection of data from a file and its related attribute information, including the object's key, object data, and object metadata. The object name mentioned above is the name of the object to be stored. The key, also called the object name, uniquely identifies the object in the storage bucket. Object metadata is a set of name-value pairs, including the object's system metadata and user-defined metadata. Object metadata can be set when uploading an object; it cannot be modified after uploading. The only way to modify object metadata is to create a copy of the object and set its metadata. Object data corresponds to the actual data of the object. Object data metadata is a set of name-value pairs, including information such as the disk name, size, and location where the object data resides.

[0042] It should be noted that the bucket in this embodiment enables multiple versions, which can prevent objects with the same name from being overwritten. The system will randomly generate a version number for each uploaded object, and the version number will be used in the calculation when storing the object.

[0043] Step 202: Determine the target data disk for uploading the object to be stored based on the bucket name, the object name, and the version number; and according to the usage status information corresponding to the target data disk, flush the object data of the object to be stored to the target disk.

[0044] In this embodiment, one way to determine the target data disk for uploading the object to be stored based on the bucket name, the version number, and the object name is to map the data disk name where the object is located using a Hash(bucket name + object name + version number) algorithm.

[0045] The target data disk is mapped into N storage blocks, each corresponding to its own usage status information. In this embodiment, the usage status information corresponding to the target data disk can be understood as the usage status information corresponding to each storage block belonging to the target data disk. Based on the location of the storage block in the target disk whose usage status information represents the first value of availability, the target storage block on the target disk where the object data will be flushed is determined. That is, the starting position and offset of the target data in the target disk are confirmed, and the data is directly flushed to the target storage block in the target disk. These target storage blocks can be one or multiple, and the number of storage blocks is related to the size of the object data. When a target storage block is occupied, its corresponding usage status information is updated to the second value. The specific implementation method will be described in detail later and will not be repeated here.

[0046] Step 203: Determine the index disk for storing the object metadata of the object to be stored based on the bucket name, and flush the object metadata of the object to be stored to the index disk based on the bucket name, object name, and version number.

[0047] One way to determine the index disk where the object metadata of the object to be stored is based on the bucket name in this step is to map the name of the index disk where the object metadata is located using a Hash(bucket name) algorithm, and record the object metadata of the object data as index information in the index disk.

[0048] This concludes the process. Figure 2 The description shown.

[0049] Therefore, in the technical solution of this application embodiment, the object storage method provided in this embodiment no longer uses the method of creating a file system to store objects, but directly stores objects in mapped storage blocks. This can avoid the additional disk space overhead caused by inode encoding. In addition, the available disk space is recorded according to the usage status information of each storage block, and the available disk space is obtained sequentially before writing objects. This can reduce disk addressing time when reading and writing objects, thereby improving the storage performance of the system. It can be seen that the technical solution provided in this embodiment can improve the storage performance of the system and further reduce the disk space overhead.

[0050] After completion Figure 2 Following the flowchart shown, as an example, before step 202, the following steps A to B are also included:

[0051] Step A: Determine the target upload method for uploading the object to be stored to the data disk.

[0052] In this embodiment, the target upload method can be either a PUT upload method or a segmented upload method.

[0053] As an example, the target upload method for uploading the object to be stored to the data disk is determined based on the data size of the object to be stored.

[0054] For the PUT upload method, there is a data limit threshold. If the data size exceeds this threshold, the upload will be segmented. For example, if the data size is less than or equal to the threshold, the user-selected PUT upload method or segmented upload method will be the target upload method. If the data size is greater than the threshold, the segmented upload method will be the target upload method.

[0055] The implementation method of flushing the object data of the object to be stored to the target disk according to the usage status information corresponding to the target data disk in step 202 includes step B:

[0056] Step B: According to the usage status information corresponding to the target data disk, use the target upload method to flush the object data of the object to be stored to the target disk.

[0057] This embodiment can flexibly select the optimal target upload method based on the data size, avoiding frequent operations caused by incorrect target upload method selection due to large data volumes. This avoids both wasting time and providing a poor user experience. Therefore, the technical solution provided in this embodiment can save time and deliver a better user experience.

[0058] As an example, when the target upload method is PUT upload, the implementation method in step 202 above, which involves using the target upload method to flush the object data of the object to be stored to the target disk according to the usage status information corresponding to the target data disk, includes:

[0059] Step C: Query the usage status information corresponding to each storage block according to the position order of each storage block in the target disk. When the usage status information is the first value, proceed to step D.

[0060] This step involves storing the object data into the first available storage blocks in the order they are located on the target disk.

[0061] Step D: Based on the data size of the object data of the object to be stored, the object data of the object to be stored is flushed to the target storage block or the target storage block and the storage block after the target storage block, and the usage status information corresponding to the storage block storing the object data of the object to be stored is updated to the second value; the target storage block is the storage block corresponding to the query usage status information being the first value.

[0062] In this way, each storage block stores object data sequentially, and there will be no idle storage blocks between storage blocks that have already stored data, thus improving the storage efficiency of storage blocks.

[0063] As another embodiment, when the target upload method is segmented upload, the implementation method in step 202 of flushing the object data of the object to be stored to the target disk according to the usage status information corresponding to the target data disk using the target upload method includes:

[0064] Step E: Obtain the nth segment of the object data of the object to be stored.

[0065] The above n is a positive integer representing the sequence number.

[0066] Step F: For each segment of data, query the usage status information corresponding to each storage block according to the position order of each storage block in the target disk. When the usage status information is found to be the first value, based on the data size of the segment, the segment data is flushed to the target storage block or the target storage block and the storage blocks after the target storage block, and the usage status information corresponding to the storage block storing the segment data is updated to the second value.

[0067] The target storage block mentioned above is the storage block corresponding to the first value when the usage status information is found.

[0068] As can be seen, by applying the technical solution provided in this embodiment, when other segmented tasks and this segmented task are written to the same target disk at the same time, this embodiment can avoid the problem of write conflicts when writing to storage blocks, and can also avoid the situation where storage blocks with stored data become idle and have not been written to for a long time after the storage object is deleted.

[0069] As an example, when the target upload method is PUT upload, the object metadata information includes at least the disk name of the data disk where the object data is located, the starting position of the object data on the data disk and its offset.

[0070] As another embodiment, when the target upload method is segmented upload, the object metadata includes at least the segment identifier corresponding to each segment of the object data, the segment task identifier, the disk name of the data disk where the object data is located, the starting position of each segment on the data disk, and its offset. In this embodiment, the segment task identifier is used to uniquely identify each segment task. Each segment task corresponds to multiple segments of data, and each segment of data has a unique segment identifier.

[0071] As an example, such as Figure 3 As shown, after step 203, the following steps are also included:

[0072] Step 301: Obtain the bucket name of the bucket containing the object to be read, the object name of the object to be read, and the version number of the object to be read.

[0073] Step 302: Determine the location of the index disk where the object metadata of the object to be read is located based on the bucket name.

[0074] Step 303: Based on the bucket name, object name, and version number, obtain the metadata information of the object to be read from the index disk according to the index disk location.

[0075] In this embodiment, the bucket name, object name, and version number are associated with the metadata information of the object to be read. In other words, with the bucket name, object name, and version number, the metadata information of the object to be read can be determined.

[0076] Step 304: Read the object to be read from the target data disk according to the index information of the object to be read recorded in the metadata information.

[0077] There is a correspondence between the index information of the object to be read and the object stored on the target disk. The index information may include the name of the data disk where the object data is stored, the starting position of the data disk and its offset, etc.

[0078] As can be seen, by applying the technical solution provided in this embodiment, the data to be read can be quickly and accurately read from the storage block of the target disk where the data to be read is located.

[0079] As an example, when the metadata information includes a segment identifier and a segment task identifier, the target upload method is determined to be segmented upload; the implementation method of step 304 includes: according to the disk name of the data disk where the object data is located, the segment identifier corresponding to each segment data, the starting position of each segment data on the data disk and its offset, and according to the storage order of each segment data on the data disk, sequentially reading the object to be read from the data disk corresponding to the disk name recorded in the metadata information.

[0080] As another embodiment, when the metadata information does not contain segment identifier and segment task identifier, the target upload method is determined to be PUT upload; the implementation method of step 304 includes: according to the disk name of the data disk where the object data is located, the starting position of the object data on the data disk and its offset recorded in the metadata information, and according to the storage order of the object data on the data disk, the object to be read is read from the data disk corresponding to the disk name recorded in the metadata information.

[0081] As one embodiment, after step 203, the method further includes:

[0082] Step H: Obtain the bucket name, object name, and version number of the object to be deleted.

[0083] Step 1: Determine the location of the index disk containing the object metadata information based on the bucket name.

[0084] As an example, the location of the index disk where the object data metadata is stored can be located by using the hash (bucket name).

[0085] Step J: Based on the obtained bucket name, object name, and version number, retrieve the object metadata information of the object to be deleted from the index disk according to the index disk location.

[0086] As an example, the object metadata information of the object to be deleted is obtained based on the key value (bucket name + object name + version number).

[0087] Step K: Based on the object metadata information, delete the object to be deleted from the data disk, and update the usage status information of the storage block where the deleted object is located to the second value.

[0088] After deleting the object to be deleted, this step releases the storage space corresponding to the storage block where the object is located, and updates the usage status information of the storage block to an available state, so that other objects can continue to use the storage block in the future.

[0089] This concludes the description of the embodiments.

[0090] The apparatus provided in this application is described below:

[0091] See Figure 4 , Figure 4 This application provides a schematic diagram of the structure of an object storage device 400, applied to a storage device. The storage device is configured with a data disk for storing object data and an index disk for storing object metadata. The disk space of each data disk is mapped into N consecutive storage blocks of the same size. Each storage block corresponds to a usage status information. When the usage status information is a first value, it indicates that the storage block is available; when the usage status information is a second value, it indicates that the storage block is unavailable. The device includes:

[0092] The information acquisition unit 401 is used to acquire the determined bucket name for storing the object to be stored, the object name of the object to be stored, and the version number of the object to be stored;

[0093] Data storage unit 402 is used to determine the target data disk for uploading the object to be stored based on the bucket name, the object name, and the version number; and to flush the object data of the object to be stored to the target disk according to the usage status information corresponding to the target data disk.

[0094] Metadata storage unit 403 is used to determine the index disk for storing the object metadata of the object to be stored based on the bucket name, and to flush the object metadata of the object to be stored to the index disk based on the bucket name, object name and version number.

[0095] As one embodiment, the device further includes:

[0096] The upload method determination unit is used to determine the target upload method for uploading the object to be stored to the data disk;

[0097] The data storage unit 402, according to the usage status information corresponding to the target data disk, flushes the object data of the object to be stored to the target disk, specifically for:

[0098] According to the usage status information corresponding to the target data disk, the object data of the object to be stored is flushed to the target disk using the target upload method.

[0099] As an example, when the target upload method is PUT upload, the data storage unit 402, according to the usage status information corresponding to the target data disk, flushes the object data of the object to be stored to the target disk, specifically for:

[0100] According to the position order of each storage block in the target disk, the usage status information corresponding to each storage block is queried. When the usage status information is found to be a first value, the object data of the object to be stored is flushed to the target storage block or the target storage block and the storage blocks after the target storage block, based on the data size of the object data of the object to be stored. The usage status information corresponding to the storage block storing the object data of the object to be stored is updated to a second value. The target storage block is the storage block corresponding to the first value when the usage status information is found.

[0101] As an example, when the target upload method is segmented upload, the data storage unit 402, according to the usage status information corresponding to the target data disk, flushes the object data of the object to be stored to the target disk, specifically for:

[0102] Retrieve the nth segment of the object data to be stored;

[0103] For each segment of data, according to the position order of each storage block in the target disk, the usage status information corresponding to each storage block is queried. When the usage status information is found to be the first value, the segment of data is flushed to the target storage block or the target storage block and the storage blocks after the target storage block based on the data size of the segment. The usage status information corresponding to the storage block storing the segment of data is updated to the second value. The target storage block is the storage block corresponding to the first value when the usage status information is found.

[0104] As an example, the object metadata information includes at least the disk name of the data disk where the object data is located, the starting position of the object data on the data disk and its offset.

[0105] As an example, the object metadata includes at least the segment identifier corresponding to each segment of the object data, the segment task identifier, the disk name of the data disk where the object data is located, the starting position of each segment of data on the data disk and its offset.

[0106] As one embodiment, the device further includes: an object data reading unit, specifically used for:

[0107] Get the bucket name of the bucket containing the object to be read, the object name of the object to be read, and the version number of the object to be read;

[0108] The location of the index disk where the object metadata of the object to be read is located is determined based on the bucket name;

[0109] Based on the bucket name, object name, and version number, the metadata information of the object to be read is obtained from the index disk according to the index disk location;

[0110] Based on the index information of the object to be read recorded in the metadata information, the object to be read is read from the target data disk.

[0111] As an example, the first upload method determination unit is used to determine that the target upload method is segmented upload when the metadata information includes a segment identifier and a segmented task identifier;

[0112] The object data reading unit reads the object to be read from the target data disk according to the index information of the object to be read recorded in the metadata information, including:

[0113] Based on the disk name of the data disk where the object data is located, the segment identifier corresponding to each segment of data, the starting position and offset of each segment of data on the data disk, and in accordance with the storage order of each segment of data on the data disk, the object to be read is sequentially read from the data disk corresponding to the disk name recorded in the metadata information.

[0114] As an example, the first upload method determination unit is used to determine the target upload method as PUT upload when the metadata information does not contain segment identifier and segment task identifier;

[0115] The object data reading unit reads the object to be read from the target data disk according to the index information of the object to be read recorded in the metadata information, including:

[0116] Based on the disk name of the data disk where the object data is located, the starting position of the object data on the data disk and its offset recorded in the metadata information, the object to be read is read from the data disk corresponding to the disk name recorded in the metadata information, according to the storage order of the object data on the data disk.

[0117] As one embodiment, the device further includes: a deletion unit, specifically used for:

[0118] Get the bucket name, the object name, and the version number of the object to be deleted from the bucket where the object is stored.

[0119] Based on the bucket name, the location of the index disk containing the object metadata information is determined;

[0120] Based on the obtained bucket name, object name, and version number, the object metadata information of the object to be deleted is retrieved from the index disk according to the index disk location;

[0121] Based on the object metadata information, the object to be deleted is deleted from the data disk, and the usage status information corresponding to the storage block where the deleted object is located is updated to the second value.

[0122] As an example, the data disk and the index disk are configured on two different disks.

[0123] This concludes the process. Figure 4 The description shown.

[0124] Therefore, in the technical solution of this application embodiment, the object storage method provided in this embodiment no longer uses the method of creating a file system to store objects, but directly stores objects in mapped storage blocks. This can avoid the additional disk space overhead caused by inode encoding. In addition, the available disk space is recorded according to the usage status information of each storage block, and the available disk space is obtained sequentially before writing objects. This can reduce disk addressing time when reading and writing objects, thereby improving the storage performance of the system. It can be seen that the technical solution provided in this embodiment can improve the storage performance of the system and further reduce the disk space overhead.

[0125] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0126] The electronic device provided in this application, from a hardware perspective, can be found in the hardware architecture diagram. Figure 5 As shown, it includes: a machine-readable storage medium and a processor, wherein: the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the object storage operations disclosed in the above example.

[0127] The machine-readable storage medium provided in this application embodiment stores machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to perform the object storage operations disclosed in the above examples.

[0128] Here, a machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, a machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0129] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0130] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0131] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0136] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An object storage method, characterized in that, The method is applied to a storage device configured with a data disk for storing object data and an index disk for storing object metadata. The disk space of each data disk is mapped to N consecutive storage blocks of the same size. Each storage block corresponds to a usage status information value. When the usage status information is a first value, it indicates that the storage block is available; when the usage status information is a second value, it indicates that the storage block is unavailable. Obtain the determined bucket name, object name, and version number of the object to be stored; the bucket name is used to represent the name of the bucket storing the object to be stored; the bucket represents the container used to store objects in object storage. The target data disk for uploading the object to be stored is determined based on the bucket name, the object name, and the version number; and the object data of the object to be stored is flushed to the target data disk according to the usage status information corresponding to each storage block belonging to the target data disk. The bucket name is used to map the name of the index disk where the object's metadata is located, thereby determining the index disk where the object's metadata is stored. The object's metadata is then flushed to the index disk based on the bucket name, object name, and version number.

2. The method according to claim 1, characterized in that, Before determining the target data disk for uploading the object to be stored based on the bucket name, the object name, and the version number, the method further includes: Determine the target upload method for uploading the object to be stored to the data disk; The step of flushing the object data of the object to be stored to the target data disk according to the usage status information corresponding to each storage block belonging to the target data disk includes: According to the usage status information corresponding to each storage block belonging to the target data disk, the object data of the object to be stored is flushed to the target data disk using the target upload method.

3. The method according to claim 2, characterized in that, When the target upload method is PUT upload, the step of flushing the object data of the object to be stored to the target data disk using the target upload method according to the usage status information corresponding to each storage block belonging to the target data disk includes: According to the position order of each storage block in the target data disk, the usage status information corresponding to each storage block is queried. When the usage status information is found to be a first value, the object data of the object to be stored is flushed to the target storage block or the target storage block and the storage blocks after the target storage block, based on the data size of the object data of the object to be stored. The usage status information corresponding to the storage block storing the object data of the object to be stored is updated to a second value. The target storage block is the storage block corresponding to the first value when the usage status information is found.

4. The method according to claim 2, characterized in that, When the target upload method is segmented upload, the step of flushing the object data of the object to be stored to the target data disk according to the usage status information corresponding to each storage block belonging to the target data disk using the target upload method includes: Retrieve the nth segment of the object data to be stored; For each segment of data, according to the position order of each storage block in the target data disk, the usage status information corresponding to each storage block is queried. When the usage status information is found to be a first value, the segment of data is flushed to the target storage block or the target storage block and the storage blocks after the target storage block based on the data size of the segment. The usage status information corresponding to the storage block storing the segment of data is updated to a second value. The target storage block is the storage block corresponding to the first value when the usage status information is found.

5. The method according to claim 3, characterized in that, The object metadata information includes at least the disk name of the data disk where the object data is located, the starting position of the object data on the data disk and its offset.

6. The method according to claim 4, characterized in that, The object metadata includes at least the segment identifier, segment task identifier, disk name of the data disk where the object data is located, starting position and offset of each segment on the data disk.

7. The method according to claim 1, characterized in that, After flushing the object metadata information of the object to be stored to the index disk, the method further includes: Get the bucket name of the bucket containing the object to be read, the object name of the object to be read, and the version number of the object to be read; The location of the index disk where the object metadata of the object to be read is located is determined based on the bucket name; Based on the bucket name, object name, and version number, the metadata information of the object to be read is obtained from the index disk according to the index disk location; Based on the index information of the object to be read recorded in the metadata information, the object to be read is read from the target data disk.

8. The method according to claim 6, characterized in that, If the metadata information includes a segment identifier and a segment task identifier, then the target upload method is determined to be segmented upload; The step of reading the object to be read from the target data disk based on the index information of the object recorded in the metadata information includes: Based on the disk name of the data disk where the object data is located, the segment identifier corresponding to each segment of data, the starting position and offset of each segment of data on the data disk, and in accordance with the storage order of each segment of data on the data disk, the object to be read is sequentially read from the data disk corresponding to the disk name recorded in the metadata information.

9. The method according to claim 5, characterized in that, If the metadata information does not contain segment identifier and segment task identifier, then the target upload method is determined to be PUT upload; The step of reading the object to be read from the target data disk based on the index information of the object recorded in the metadata information includes: Based on the disk name of the data disk where the object data is located, the starting position of the object data on the data disk and its offset recorded in the metadata information, the object to be read is read from the data disk corresponding to the disk name recorded in the metadata information, according to the storage order of the object data on the data disk.

10. The method according to claim 1, characterized in that, This method is followed by: Get the bucket name, the object name, and the version number of the object to be deleted from the bucket where the object is stored. Based on the bucket name, the location of the index disk containing the object metadata information is determined; Based on the obtained bucket name, object name, and version number, the object metadata information of the object to be deleted is obtained from the index disk according to the index disk location; Based on the object metadata information, the object to be deleted is deleted from the data disk, and the usage status information corresponding to the storage block where the deleted object is located is updated to the first value.

11. The method according to any one of claims 1-10, characterized in that, The data disk and the index disk are configured on two different disks.

12. An object storage device, characterized in that, An apparatus is applied to a storage device configured with a data disk for storing object data and an index disk for storing object metadata. The disk space of each data disk is mapped to N consecutive storage blocks of the same size. Each storage block corresponds to a usage status information value. When the usage status information is a first value, it indicates that the storage block is available; when the usage status information is a second value, it indicates that the storage block is unavailable. The apparatus includes: The information acquisition unit is used to acquire the bucket name for storing the object to be stored, the object name of the object to be stored, and the version number of the object to be stored; the bucket name is used to represent the name of the bucket storing the object to be stored; the bucket represents a container in object storage used to store objects. The data storage unit is used to determine the target data disk for uploading the object to be stored based on the bucket name, the object name, and the version number; and to flush the object data of the object to be stored to the target data disk according to the usage status information corresponding to each storage block belonging to the target data disk. The metadata storage unit is used to map the name of the index disk where the object metadata is located according to the bucket name, so as to determine the index disk where the object metadata of the object to be stored is stored, and to flush the object metadata of the object to be stored to the index disk according to the bucket name, object name and version number.

Images