Object segmentation uploading method, device and equipment

By generating globally unique prefixes and slicing data blocks in the object storage system, the merged segmentation rule information is constructed, which solves the problem of too large mapping tables when uploading segments, and achieves more efficient data storage and deletion.

CN118981276APending Publication Date: 2024-11-19XINHUASAN INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410959246.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In object storage systems, the mapping table generated during segment upload is too large, resulting in an increase in disk space and memory usage, and a large resource consumption is consumed when deleting objects.

Method used

By generating a globally unique prefix corresponding to the object one by one, and dividing the segmented data into data blocks, constructing single segmented rule information and merging segmented rule information, reducing records in the mapping table and storing merges only in the object mapping table The segmentation rule information and data block size afterwards.

Benefits of technology

It greatly reduces the size of the object map table, reduces the disk space and memory usage, and improves the efficiency of data deletion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981276A_ABST
    Figure CN118981276A_ABST
Patent Text Reader

Abstract

The invention provides an object segmentation uploading method, device and equipment, which are used for solving the technical problems of overlarge mapping table and the like in a technical object storage system. According to the method, object segmentation rule information uploaded in a segmented mode is merged based on the segmentation number, the coverage number and the segmentation size, and multiple pieces of segmentation rule information with the continuous segmentation number, the same coverage number and the segmentation size are merged and stored, so that the size of an object mapping table is greatly reduced, and the segmentation efficiency is improved. And the disk space occupied by object metadata persistence and the memory space in the cache are reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a method, device and equipment for uploading objects in segments. Background Art

[0002] High-availability cloud storage services, such as Simple Storage Service (S3), often use multipart upload when uploading large objects. Many storage service clients use multipart upload when uploading objects larger than 10MB. The S3 protocol stipulates that objects larger than 5GB must be uploaded in multipart. A complete multipart upload usually requires three types of requests:

[0003] The first type of request is an initial multipart upload request, which generates an upload ID.

[0004] The second type of request is a multiple-segment upload request. In the multiple-segment upload request, the upload identifier uploadID and the segment number are specified. The data of each segment is uploaded through multiple multi-segment upload requests, one request for each segment, and the data is uploaded to the object storage.

[0005] The third type of request is to complete a multipart upload request, where the object storage merges the parts into an object visible to the S3 client.

[0006] Compared with ordinary upload, multipart upload divides an object into multiple requests for uploading, which can bring the following benefits: it is easy to implement concurrent upload and improve upload performance. If an exception occurs during the upload of a segment (such as a network exception), only the abnormal segment needs to be retransmitted, and there is no need to retransmit the complete object data. If the upload is paused halfway through, you can continue uploading from the last segment when needed, realizing a function similar to breakpoint resume.

[0007] In addition, in the multipart upload of the S3 protocol, it is stipulated that the size range of each part except the last part is [5MB~5GB], the size range of the last part is [0~5GB], and each object has a maximum of 10,000 parts. The protocol allows each part to be uploaded with different sizes, and does not require the order of multipart upload.

[0008] In order to persist and read objects uploaded in parts, a data organization scheme describing the objects uploaded in parts needs to be designed.

[0009] Since the final data of multi-part upload is composed of data uploaded multiple times, in the conventional solution, each multi-part upload will independently persist the data of each part. In distributed storage, in order to achieve better data balancing, each part data will also be divided into smaller granularity data blocks according to a fixed size. When the multi-part upload is finally completed, a data mapping table is constructed to record the correspondence between each part data and the offset, and the mapping table is persisted in the object metadata.

[0010] When reading data, obtain the mapping table from the object metadata, find the segmented data where the data is located in the mapping table according to the current read offset, calculate the offset of the required data in the segmented data, then calculate the data block where the required data is located and the corresponding offset, and read the corresponding data.

[0011] The above traditional multi-part upload solution needs to record all multi-part upload data information. When the number of parts is large, the mapping table will be large. Since the mapping table needs to be persisted in the object metadata and usually needs to be cached in memory, it not only increases the disk space required for persistence, but also consumes more memory. At the same time, when deleting the object, it is necessary to traverse each segment data in the mapping table for deletion, which occupies relatively more resources. Summary of the invention

[0012] In view of this, the present invention provides a method, device and equipment for uploading objects in segments, which are used to solve technical problems such as too large mapping tables in technical object storage systems.

[0013] Based on one aspect of an embodiment of the present invention, the present invention provides a method for uploading an object in segments, which is applied to a storage node in an object storage system. The method includes:

[0014] During the initialization of multi-part upload, a globally unique prefix corresponding to the object is generated, and the size of the data block used to split the object segment data is determined;

[0015] In the process of uploading segmented data, the segmented data is divided into data blocks and a data block name is generated for each data block, wherein the data block name includes the prefix and the segment number, the overlay number and the data block number;

[0016] After each segment is uploaded, construct the single segment rule information of each segment, which includes: segment number, overlay number, and segment size;

[0017] After all segments of the object are uploaded, the single segment complete rule information of each segment is constructed based on the single segment rule information of each segment, and the single segment complete rule information is merged to generate merged segment rule information; the single segment complete rule information includes the single segment rule information and the segment start offset address;

[0018] An object mapping table is generated based on the merge segmentation rule information, and the mapping table is persisted as object metadata; the object mapping table at least includes the merge segmentation rule information, prefix, and data block size.

[0019] Furthermore, the method for generating merged segment rule information by merging the single segment complete rule information is:

[0020] Sort the single segment complete rule information of all segments of the object by segment number;

[0021] All single-segment complete information with consecutive segment numbers, the same coverage number and the same segment size are merged, and the field values ​​of the merged merged segment rule information are consistent with the single-segment complete information with the smallest segment number among the multiple merged single-segment complete information.

[0022] Furthermore, the prefix in the mapping table is a 64-bit integer, the data block size is represented by the power n of 2 to the nth power, and the data block size in the mapping table only stores the power n.

[0023] Furthermore, the segment number is 2 bytes, the coverage number is 3 binary bits, the data block number is 13 binary bits, and the segment size is 45 binary bits.

[0024] Furthermore, in the case that, except for the last segment of the object, the segment numbers of the other segments are continuous, the coverage numbers are the same, and the segment sizes are also the same, the object mapping table only includes two pieces of merging segment rule information.

[0025] Based on another aspect of an embodiment of the present invention, the present invention further provides a device for uploading object segments, which is applied to a storage node in an object storage system, and includes:

[0026] An initialization module, used to generate a globally unique prefix corresponding to an object one by one during the initialization of multi-part upload, and determine the size of a data block used to segment the object data;

[0027] The segmented upload module is used to divide the segmented data into data blocks and generate a data block name for each data block during the process of uploading segmented data, wherein the data block name includes the prefix and the segment number, the overwrite number and the data block number; after each segment is uploaded, construct the single segment rule information of each segment, wherein the single segment rule information includes: the segment number, the overwrite number and the segment size;

[0028] A segment merging module is used to construct the single segment complete rule information of each segment based on the single segment rule information of each segment after all segments of the object are uploaded, and merge the single segment complete rule information to generate merged segment rule information; the single segment complete rule information includes the single segment rule information and the segment start offset address;

[0029] The mapping table generation module is used to generate an object mapping table based on the merge segmentation rule information and persist the mapping table as object metadata; the object mapping table at least includes the merge segmentation rule information, prefix, and data block size.

[0030] Furthermore, the segment merging module further comprises:

[0031] A sorting unit, used to sort the single segment complete rule information of all segments of the object according to the segment number;

[0032] The merging unit is used to merge all single-segment complete information with consecutive segment numbers, the same coverage number and the same segment size. The field values ​​of the merged merged segment rule information are consistent with the single-segment complete information with the smallest segment number among the multiple single-segment complete information merged.

[0033] Furthermore, the prefix in the mapping table is a 64-bit integer, the data block size is represented by the power n of 2 to the power of n, and the data block size in the mapping table only stores the power n;

[0034] The segment number is 2 bytes, the overlay number is 3 binary bits, the data block number is 13 binary bits, and the segment size is 45 binary bits.

[0035] The present invention merges the object segmentation rule information uploaded in segments based on the segment number, overwrite number and segment size, and merges and stores multiple segmentation rule information with consecutive segment numbers, the same overwrite number and segment size, thereby greatly reducing the size of the object mapping table and reducing the disk space occupied by the persistence of object metadata and the memory space in the cache. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments of the present invention or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present invention.

[0037] Figure 1 A schematic diagram provided for an embodiment of the present invention;

[0038] Figure 2This is an example diagram of an application scenario of object storage multipart upload in one embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the steps of a method for uploading multiple objects in an object storage system according to an embodiment of the present invention;

[0040] Figure 4 This is an example of the field structure of a data block name used in an embodiment of the present invention;

[0041] Figure 5 It is a before-and-after comparison diagram of merging single segment complete rule information of all segments of an object according to segment number, coverage number and segment size in one embodiment of the present invention;

[0042] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of the present invention for implementing the object segmented uploading method provided in the present invention. DETAILED DESCRIPTION

[0043] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, rather than limiting the embodiments of the present invention. The singular forms of "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. Although the embodiments of the present invention may be described in a first, second, or third manner, this description is only used to distinguish similar information, entities, or steps, rather than to describe a specific order or sequence. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. For another example, in some scenarios, the first information may refer to one information or to multiple information of the same type. In addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination". The "and / or" in the present invention is only a kind of association relationship describing the associated objects, indicating that there may be three relationships, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B may be singular or plural. In addition, in the description of the present invention, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0044] The present invention provides a solution for uploading objects in segments in an object storage system. The present invention can reduce the mapping records of a mapping table, realize complete mapping of data, support the coexistence of different segment sizes, and also realize a more efficient data deletion method. The present invention does not limit the data persistence method, and can be persisted through files or through the underlying object storage system provided by a distributed storage system.

[0045] The specific implementation process of the present invention is described in detail below in conjunction with the drawings and specific embodiments. It should be noted that the steps shown in the processes of the drawings and embodiments can be executed in a computer system such as a set of computer executable instructions, and one or more of the exemplified method steps can be single or multiple or aggregated together and run in the computer system in the form of a software module or a virtual device. Moreover, although the exemplified step process has time and sequential logic, the present invention is not limited to these steps in certain cases. They can be executed in an order different from that here. Under the guidance of the purpose of the present invention, simple changes or combinations of the execution order of these steps should also fall within the scope of the disclosure of the embodiments of the present invention.

[0046] Figure 1 The figure is a schematic diagram of the segmentation of the object to be uploaded when uploading the object in segments in one embodiment of the present invention. Each segment has multiple data blocks. Usually, except for the last data block of the last segment, the data blocks in the remaining segments are of the same size. The data block size is usually determined according to the system configuration. According to the S3 protocol, the sizes of the segments can be different, but for ease of processing, the sizes of the segments except the last segment are usually set to be the same.

[0047] Figure 2 This is an example diagram of an application scenario of object storage segmented upload in an embodiment of the present invention. The distributed storage in this embodiment is a Ceph distributed storage system. The S3 protocol is used between the storage node and the storage client. The storage client (S3 Client) uploads the object segment to the object storage gateway (OBS gateway) in the storage node through the S3 protocol. After the object storage gateway receives the S3 object uploaded by the client, it divides the object data into data blocks according to the pre-configured object size (for example, 4MB), persists the data blocks to the storage pool, and constructs the object metadata and persists it to the storage pool. The object storage device / service (OSD) is responsible for persisting the data and metadata to a non-volatile storage medium (for example, a disk). The object metadata includes a mapping table. The object storage service gateway RGW and the object storage device / service OSD are both service processes running in the storage node.

[0048] Figure 3 A schematic diagram of the steps of a method for uploading objects in multiple parts in an object storage system provided in an embodiment of the present invention. The method is applied to a storage node, and the method includes:

[0049] Step 301. During the initialization of multi-part upload, a globally unique prefix corresponding to the object is generated, and a data block size for segmenting the object data is determined;

[0050] When initializing multipart upload, the object storage gateway (OBS gateway) in the storage node generates a globally unique prefix that corresponds to the object one by one when receiving the initial multipart upload request from the storage client (S3 Client). The prefix is ​​used to identify the object.

[0051] After receiving the segments uploaded by the storage client, the object storage gateway needs to split the object segments into data blocks for persistence. Therefore, during the initialization of the multi-part upload process, the size of the data blocks for splitting the object segments will also be determined.

[0052] In one embodiment of the present invention, the prefix is ​​a 64-bit integer, which only takes up 8 bytes. In other embodiments of the present invention, the prefix may also be other globally unique field types, such as a globally unique string.

[0053] When determining the size of the data blocks of the object to be uploaded, in order to reduce the space occupied, the data block size is represented by the power of the data block size. For example, the data block size is 2 n Bytes, the data block size in the mapping table only stores the power n. After this processing, the mapping table only needs 1 byte to store the data block size, and the maximum power can reach 31, so the maximum data block size can reach 2GB. Taking the common 4MB data block size as the power, the power is 22.

[0054] Typically the data block size is determined based on configuration and can be read from the system configuration file.

[0055] Step 302: In the process of uploading segmented data, the segmented data is divided into data blocks and a data block name is generated for each data block, wherein the data block name includes the prefix and the segment number, the overlay number and the data block number;

[0056] After initializing the multipart upload process, the storage client uploads each object segment to the object storage gateway through multiple multipart upload requests. The object storage gateway divides the segments according to the determined data block size and generates a data block name for each data block after segmentation. The data block name is used to uniquely identify the data block persisted in the storage pool.

[0057] Figure 4This is an example of the field structure of the data block name used in an embodiment of the present invention. In this example, the data block name consists of a prefix and a 4-byte segment block identifier, wherein the first 16 bits of the 4-byte segment block identifier are the segment number, which can represent a maximum of 65536 segments, which can meet the specification of the S3 protocol for a maximum of 10,000 segments. The last 13 bits of the segment block identifier are the data block number, which is used to identify the sequence number of the data block in the segment, and can represent a maximum of 8192 data blocks. Calculated based on 1MB per data block, the size of each segment can reach 8GB, which can meet the specification of the maximum 5GB of a single segment of the S3 protocol. Since the S3 protocol supports overwriting the uploaded segments when the segment upload is not completed, in order to support the scenario of concurrent uploading of the same segment, the middle three bits are used to represent the overwrite number, and each segment upload uses a unique overwrite number for the segment, which can support up to 8 concurrent uploads of the same segment. Since the scenario of concurrently uploading the same segment is not common, the maximum support of 8 concurrent uploads can basically meet various scenarios.

[0058] Step 303. After each segment is uploaded, construct the single segment rule information of each segment, the single segment rule information includes: segment number, coverage number, segment size;

[0059] The single segment rule information is used to describe the composition rule of each object segment.

[0060] In one embodiment of the present invention, the single segment rule information includes at least the segment number, the coverage number, and the segment size field. The segment number occupies 16 binary bits, the coverage number occupies 3 binary bits, and the segment size occupies 45 binary bits. The three fields are 8 bytes in total. The 45-bit segment size can represent a size far exceeding the maximum 5GB specification of a single segment specified by the S3 protocol.

[0061] Step 304: After all segments of the object are uploaded, merge based on the single segment complete rule information of each segment; the single segment complete rule information includes the single segment rule information and the segment start offset address;

[0062] After all segments of the object are uploaded, a list of complete rule information for all segments will be saved in the storage node, where each list item is a single segment complete rule information. The single segment complete rule information includes the single segment rule information generated in the previous step and the segment starting offset address (referred to as the starting offset), which is the starting position address of the segment stored in the persistent storage.

[0063] In one embodiment of the present invention, the segment start offset address is 8 bytes, totaling 64 bits, plus the 8 bytes of single segment rule information generated in the previous step, totaling 16 bytes.

[0064] In one embodiment of the present invention, the method for merging based on the single segment complete rule information of each segment is:

[0065] Step 3041. Sort the single segment complete rule information of all segments of the object by segment number;

[0066] Step 3042. All single-segment complete information with consecutive segment numbers, the same coverage number and the same segment size are merged, and the field values ​​of the merged merged segment rule information are consistent with the single-segment complete information with the smallest segment number among the multiple merged single-segment complete information.

[0067] Figure 5 The figure is a before-and-after comparison diagram of merging the single segment complete rule information of all segments of an object according to the segment number, overwrite number and segment size in one embodiment of the present invention. The exemplified object in the figure includes 7 segments, and the segment numbers in the single segment complete rule information of the first 3 segments (segment numbers 1, 2 and 3) are continuous, the overwrite numbers are the same and the segment sizes are also the same, which meets the merging strategy, and therefore can be merged into one merged segment rule information, and the segment number, overwrite number, segment size and segment start offset address of the merged segment rule information are the same as the fields of the segment rule information with the smallest segment number (segment number 1) among the 3 single segment complete rule information merged this time.

[0068] Figure 5 In order to facilitate the demonstration of the merging effect, the given examples use a variety of segment sizes for object segmentation in the figure. However, in actual usage scenarios, except for the last segment (because most objects are not aligned by segment size), the sizes of other segments are usually the same. When there is no segment overlap (that is, the overlap numbers of all segments are the same and the sizes of all segments except the last segment are the same), all single-segment complete rule information can be eventually merged into two merged segment rule information, namely, one merged segment rule information merged from the single-segment complete rule information of all segments before the last segment and the last single-segment complete rule information. If the stored objects (such as files) are aligned by segment size, all single-segment complete rule information will eventually be merged into one merged segment rule information.

[0069] Step 305: Generate an object mapping table based on the merge segmentation rule information, and persist the mapping table as object metadata; the object mapping table at least includes the merge segmentation rule information, prefix, and data block size.

[0070] like Figure 5In the example, each single-segment complete rule information is 16 bytes. If calculated based on the scenario where the coverage number and block size are the same, the final merged rule information is only 2 pieces, occupying a total of 32 bytes. Adding the 8-byte object prefix and 1-byte data block size information, the mapping table is only 41 bytes in a normal scenario.

[0071] Based on the solution for merging segment rule information provided by the present invention, when a storage node receives an I / O request from a storage client, a method for reading / writing object data is as follows:

[0072] Step S01. According to the offset address N of the I / O request, match the record whose first segment starting offset address is smaller than the offset address N of the I / O request in the merge segment rule information in the mapping table;

[0073] Step S02: Use the segment start offset address M, the start segment number P and the segment size S in the matching merge segment rule information to calculate the actual segment number where the data of the I / O request is located. The calculation method is:

[0074] The actual segment number of the requested data = P + (NM) / S

[0075] Step S03. According to the data block size B in the mapping table, the data block number of the requested data and the offset address in the data block are calculated as follows:

[0076] The data block number where the requested data is located = ((NM)%S) / B

[0077] The offset address of the requested data in the data block = ((NM)%S)%B

[0078] Step S03: construct a data block name of the data block where the I / O request data is located according to the data block name naming rule, and the required data can be read based on the constructed data block name and the offset address in the data block.

[0079] The embodiment of the present invention can achieve the use of a very small mapping table to describe the data organization method of the segmented upload object in most scenarios by simplifying the field expression in the segmentation rule and adopting a merging strategy for the segmentation rule information, while also supporting data organization methods of multiple different segment sizes. By using the present invention, the size of the mapping table describing the data organization method of the segmented upload object can be reduced, and the disk space occupied by persistence and the memory space in the cache can be reduced. At the same time, since all data blocks of a segmented upload object use the same globally unique prefix, they can be deleted in batches by prefix filtering in the deletion scenario, thereby improving deletion efficiency.

[0080] Figure 6A schematic diagram of the structure of an electronic device for implementing the object segment upload method provided by the present invention is provided in one embodiment of the present invention. The device 600 includes: a processor 610 such as a central processing unit (CPU), a communication bus 620, a communication interface 640, and a memory 630. The processor 610 and the memory 630 can communicate with each other through the communication bus 620. The memory 630 stores a computer program, and when the computer program is executed by the processor 610, the function of one or more steps in the object segment upload method provided by the present invention can be implemented.

[0081] Memory refers to a device based on a certain storage medium for storing computer programs and / or data. It can be a volatile memory (VM, often called memory) or a non-volatile memory (NVM). Memory refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a very fast speed. It is used as a storage medium for temporary data of the operating system and other running programs. Memory can be synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), etc. Non-volatile memory refers to a memory that uses a persistent storage medium. It has the characteristics of large capacity and persistent data storage. It can be a storage class memory (SCM), a solid state disk (SSD), a NAND flash memory, a disk, etc. SCM is the industry's general term for new storage media between memory and flash memory. It is a composite storage technology that combines persistent storage characteristics and memory characteristics. Its access speed is slower than DRAM and faster than SSD hard disks.

[0082] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0083] It should be appreciated that embodiments of the present invention may be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory (or non-persistent) memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, wherein the storage medium configured in this way causes the computer to operate in a specific and predefined manner. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose. In addition, the operation of the process described in the present invention may be performed in any suitable order, unless the present invention otherwise indicates or otherwise clearly contradicts the context. The process described in the present invention (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0084] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in the present invention includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0085] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for uploading an object in multiple parts, characterized in that: The method is applied to a storage node in an object storage system, and the method includes: During the initialization of multi-part upload, a globally unique prefix corresponding to the object is generated, and the size of the data block used to split the object segment data is determined; In the process of uploading segmented data, the segmented data is divided into data blocks and a data block name is generated for each data block, wherein the data block name includes the prefix and the segment number, the overlay number and the data block number; After each segment is uploaded, construct the single segment rule information of each segment, which includes: segment number, overlay number, and segment size; After all segments of the object are uploaded, the single segment complete rule information of each segment is constructed based on the single segment rule information of each segment, and the single segment complete rule information is merged to generate merged segment rule information; the single segment complete rule information includes the single segment rule information and the segment start offset address; An object mapping table is generated based on the merge segmentation rule information, and the mapping table is persisted as object metadata; the object mapping table at least includes the merge segmentation rule information, prefix, and data block size.

2. The method according to claim 1, characterized in that The method for generating merged segment rule information by merging the single segment complete rule information is as follows: Sort the single segment complete rule information of all segments of the object by segment number; All single-segment complete information with consecutive segment numbers, the same coverage number and the same segment size are merged, and the field values ​​of the merged merged segment rule information are consistent with the single-segment complete information with the smallest segment number among the multiple merged single-segment complete information.

3. The method according to claim 2, characterized in that The prefix in the mapping table is a 64-bit integer, the data block size is represented by the power n of 2 to the power of n, and the data block size in the mapping table only stores the power n.

4. The method according to claim 3, characterized in that: The segment number is 2 bytes, the overlay number is 3 binary bits, the data block number is 13 binary bits, and the segment size is 45 binary bits.

5. The method according to claim 4, characterized in that In the case that, except for the last segment of the object, the segment numbers of the other segments are continuous, the coverage numbers are the same, and the segment sizes are also the same, the object mapping table only includes two pieces of segment merging rule information.

6. An object segment uploading device, characterized in that: The device is applied to a storage node in an object storage system, and includes: An initialization module, used to generate a globally unique prefix corresponding to an object one by one during the initialization of multi-part upload, and determine the size of a data block used to segment the object data; The segmented upload module is used to divide the segmented data into data blocks and generate a data block name for each data block during the process of uploading segmented data, wherein the data block name includes the prefix and the segment number, the overwrite number and the data block number; after each segment is uploaded, construct the single segment rule information of each segment, wherein the single segment rule information includes: the segment number, the overwrite number and the segment size; A segment merging module is used to construct the single segment complete rule information of each segment based on the single segment rule information of each segment after all segments of the object are uploaded, and merge the single segment complete rule information to generate merged segment rule information; the single segment complete rule information includes the single segment rule information and the segment start offset address; The mapping table generation module is used to generate an object mapping table based on the merge segmentation rule information and persist the mapping table as object metadata; the object mapping table at least includes the merge segmentation rule information, prefix, and data block size.

7. The device according to claim 6, characterized in that The segment merging module further comprises: A sorting unit, used to sort the single segment complete rule information of all segments of the object according to the segment number; The merging unit is used to merge all single-segment complete information with consecutive segment numbers, the same coverage number and the same segment size. The field values ​​of the merged merged segment rule information are consistent with the single-segment complete information with the smallest segment number among the multiple single-segment complete information merged.

8. The device according to claim 7, characterized in that The prefix in the mapping table is a 64-bit integer, the data block size is represented by the power n of 2, and the data block size in the mapping table only stores the power n; The segment number is 2 bytes, the overlay number is 3 binary bits, the data block number is 13 binary bits, and the segment size is 45 binary bits.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium communicate with each other via the communication bus; Storage medium for storing computer programs; A processor, for implementing the method according to any one of claims 1 to 5 when executing a computer program stored on a storage medium.

10. A storage medium having a computer program stored thereon, characterized in that: The computer program implements the method of any one of claims 1 to 5 when executed by a processor.