Method and device for updating same-name large objects in distributed object storage

Abstract

The invention relates to the technical field of distributed storage, and discloses a method and device for updating distributed object storage homonymous large objects, and the method comprises the steps: obtaining an initial serial number of each segment in an initial large object stored in a distributed system, and a first verification value of the initial segment corresponding to each initial serial number; obtaining the data volume of a first segment in the initial large object, and dividing the to-be-uploaded large object into multiple segments based on the data volume to obtain multiple to-be-uploaded segments; calculating a verification value of each to-be-uploaded segment to obtain a to-be-uploaded serial number of the to-be-uploaded segment and a second verification value of the to-be-uploaded segment corresponding to each to-be-uploaded serial number; and under the condition that the target to-be-uploaded serial number is the same as the numerical value represented by the initial serial number, if the second verification value of the to-be-uploaded segment corresponding to the target to-be-uploaded serial number is different from the first verification value of the initial segment corresponding to the initial serial number, uploading the to-be-uploaded segment corresponding to the target to-be-uploaded serial number.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a method and device for updating large objects with the same name in distributed object storage. Background Art

[0002] In the related art, for the way of uploading large objects in distributed object storage, you can choose to upload them in parts or as a whole. After a local large object is uploaded to the storage, when you need to modify the large object, since it does not support online modification of the large object content directly in the storage, you can only modify the original large object locally and upload it to the storage again. If you choose to upload in parts, the entire multipart upload process will be re-executed. After the upload is completed, the new large object with the same name will overwrite the existing large object, and the update operation of the large object is completed.

[0003] If a network anomaly or program anomaly occurs during the segmented upload process, the upload will be interrupted. Distributed object storage in related technologies already supports breakpoint resumption or concurrent upload to improve transmission efficiency. However, whether it is a whole upload or a segmented upload, it is necessary to upload the complete large object data to overwrite the data already stored in the cluster. In the field of video storage applications, the size of each video object is basically at the G level or even larger. If repeated editing, updating and uploading are required, many unmodified segmented contents in the large video object will be uploaded repeatedly, which may increase bandwidth pressure and increase storage and transmission costs. Summary of the invention

[0004] In view of this, the present invention provides a method and device for updating large objects with the same name in distributed object storage, so as to solve the problem that many unchanged segmented contents are uploaded repeatedly during the large object update process, which may increase bandwidth pressure and increase storage and transmission costs.

[0005] In a first aspect, the present invention provides a method for updating large objects with the same name in distributed object storage, the method comprising: obtaining an initial serial number of each segment in an initial large object stored in a distributed system, and a first check value of the initial segment corresponding to each of the initial serial numbers; obtaining the data volume of the first segment in the initial large object, and dividing the large object to be uploaded into multiple segments based on the data volume to obtain multiple segments to be uploaded; calculating the check value of each of the segments to be uploaded to obtain the serial number to be uploaded of the segment to be uploaded, and a second check value of the segment to be uploaded corresponding to each of the serial numbers to be uploaded; when the target serial number to be uploaded is the same as the numerical value represented by the initial serial number, if the second check value of the segment to be uploaded corresponding to the target serial number to be uploaded is different from the first check value of the initial segment corresponding to the initial serial number, uploading the segment to be uploaded corresponding to the target serial number to be uploaded.

[0006] In an optional embodiment, the aforementioned method for updating large objects with the same name in distributed object storage further includes: if the value represented by the target to-be-uploaded sequence number does not exist in the value represented by the initial sequence number, uploading the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number.

[0007] In an optional embodiment, the aforementioned method for updating large objects with the same name in distributed object storage also includes: determining the unmodified reused segments in the initial large object; if the segment to be uploaded corresponding to the target sequence number to be uploaded is uploaded successfully, merging the segment to be uploaded corresponding to the target sequence number to be uploaded and the reused segment to complete the update of the initial large object.

[0008] In an optional embodiment, the aforementioned method for updating large objects with the same name in distributed object storage also includes: if the upload of the segment to be uploaded corresponding to the target sequence number to be uploaded fails, re-uploading the segment to be uploaded corresponding to the target sequence number to be uploaded; if the re-upload fails, adding a deletion tag to the segment to be uploaded corresponding to the target sequence number to be uploaded.

[0009] In an optional implementation, the determining the unmodified reuse segments in the initial large object includes:

[0010] When the target initial sequence number is the same as the numerical value represented by the sequence number to be uploaded, if the second check value of the segment to be uploaded corresponding to the sequence number to be uploaded is the same as the first check value of the target initial segment corresponding to the target initial sequence number, the target initial segment is the multiplexed segment.

[0011] In a second aspect, the present invention provides a device for updating large objects with the same name in distributed object storage, the device comprising: a metadata management module, used to obtain the initial serial number of each segment in the initial large object stored in the distributed system, and a first check value of the initial segment corresponding to each of the initial serial numbers; a difference identification and analysis module, used to obtain the data volume of the first segment in the initial large object, and divide the large object to be uploaded into multiple segments based on the data volume to obtain multiple segments to be uploaded; calculate the check value of each of the segments to be uploaded to obtain the serial number to be uploaded of the segment to be uploaded, and a second check value corresponding to each of the serial numbers to be uploaded; a segment upload module, used to upload the segment to be uploaded corresponding to the target serial number to be uploaded if the second check value of the segment to be uploaded corresponding to the target serial number to be uploaded is different from the first check value of the initial segment corresponding to the initial serial number when the target serial number to be uploaded is the same as the numerical value represented by the initial serial number.

[0012] In an optional embodiment, the metadata management module includes: a large object metadata management unit, used to record the number of segments in the initial large object based on the segment number field, and to record the key-value pair information of each segment based on the metadata key-value pair field; a segment metadata management unit, used to record the serial number to be uploaded based on the segment identifier field, and to record the second check value based on the segment check value field.

[0013] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for updating large objects with the same name in distributed object storage according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for updating large objects with the same name in distributed object storage according to the first aspect or any corresponding embodiment thereof.

[0015] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the method for updating large objects with the same name in distributed object storage according to the first aspect or any corresponding embodiment thereof.

[0016] In the process of updating large objects stored in distributed objects, by comparing the first check value of each segment of the initial large object stored in the distributed system with the second check value of each segment of the large object to be uploaded, the updated segments are judged and analyzed, and the updated segments are uploaded, which reduces the data transmission volume and saves transmission time and bandwidth costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic flow chart of a method for updating large objects with the same name in distributed object storage provided by an embodiment of the present invention is shown;

[0019] Figure 2 A schematic diagram of the flow chart of a method for identifying differences in large object segments provided by an embodiment of the present invention is shown;

[0020] Figure 3A schematic diagram showing the comparison process between the initial large object and the large object to be uploaded is shown;

[0021] Figure 4 Another schematic diagram of the comparison process between the initial large object and the large object to be uploaded is shown;

[0022] Figure 5 Another schematic diagram of the comparison process between the initial large object and the large object to be uploaded is shown;

[0023] Figure 6 A schematic diagram showing the process of updating multiple large objects with the same name;

[0024] Figure 7 Another method flow chart for updating large objects with the same name in distributed object storage is shown;

[0025] Figure 8 A schematic diagram showing the structure of a device for updating a large object in a distributed object storage provided by an embodiment of the present invention is shown;

[0026] Fig. 9 A schematic diagram of the working process of a device for updating a large object in a distributed object storage is shown;

[0027] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] Multipart Upload: In distributed object storage, for objects that need to be uploaded, the entire object is divided into multiple data blocks (called Parts) according to a certain segment size. These data blocks are numbered based on consecutive serial numbers. When uploading, each Part can be uploaded independently or in any order. Finally, based on the distributed object storage system, the original large object is merged and combined according to the Part numbering order.

[0030] Large objects with the same name. In distributed object storage, if the size of the object to be uploaded exceeds 100MB, it is considered a large object, and it is recommended to use multi-part upload. If multiple versions are not enabled, if the same object is uploaded repeatedly, the object will be overwritten with the same name.

[0031] Metadata, in distributed object storage, includes system metadata and user-defined metadata. Metadata refers to the descriptive information of an object, including the name of the object, the amount of data contained in the object, the creation time, the owner and other key-value information.

[0032] According to an embodiment of the present invention, a method embodiment for updating large objects with the same name in distributed object storage is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] In this embodiment, a method for updating large objects with the same name in distributed object storage is provided, which can be used in mobile terminals, such as mobile phones, tablet computers, desktop computers, or servers. Figure 1 A flow chart of a method for updating large objects with the same name in distributed object storage provided by an embodiment of the present invention is shown. Figure 1 As shown, the process includes the following steps:

[0034] Step S101, obtaining the sequence number of each segment in the initial large object stored in the distributed system, and the first check value of the initial segment corresponding to each sequence number.

[0035] Figure 2 The following is a flow chart of a method for identifying large object segment differences provided by an embodiment of the present invention. Figure 2 As shown, in this step, the client uses multi-threaded concurrency locally, and in step S201, the main thread obtains the check value of each segment of the record, that is, the first check value, by reading the metadata of the initial large object. The first check value can be the fifth version of the message digest algorithm (Message-Digest Algorithm, MD5) or a cyclic redundancy check code (CyclicRedundancy Check, referred to as CRC).

[0036] Step S102, obtain the data volume of the first segment in the initial large object, divide the large object to be uploaded into multiple segments based on the data volume, and obtain multiple segments to be uploaded; calculate the check value of each segment to be uploaded, obtain the to-be-uploaded serial number of the to-be-uploaded segment, and the second check value corresponding to each to-be-uploaded serial number.

[0037] In this step, the initial large object includes multiple segments. Among the multiple segments of the initial large object, the data volume of the segment corresponding to the maximum sequence number may be the same as or different from the data volume of other segments. The data volume of the first segment is the same as the data volume of other segments in the initial large object. The large object to be uploaded is divided using the same segment size as the initial large object to obtain N segments to be uploaded. M sub-threads are created to calculate the check values ​​of the segments to be uploaded. Step S202, each sub-thread calculates and compares the second check values ​​of N / M segments to be uploaded. N represents the number of segments, and M represents the number of threads. The number of threads can be 8 to 64, and can be set to 32. In this way, a multi-concurrency method can be used to compare the segment MD5 of the uploaded large object and the local large object to be uploaded, judge and analyze the updated segments, manage and record the results of the identification through a temporary kv data table, and reorganize and generate a new PartID sequence.

[0038] Step S103, when the target sequence number to be uploaded is the same as the value represented by the sequence number, if the second check value of the segment to be uploaded corresponding to the target sequence number to be uploaded is different from the first check value of the initial segment corresponding to the sequence number, upload the segment to be uploaded corresponding to the target sequence number to be uploaded.

[0039] This step is described by way of example. Figure 3 The figure shows the comparison process between the initial large object and the large object to be uploaded. Figure 3 As shown, the size of the large object to be uploaded is the same as the size of the existing large object. Since the same segment size is used, the number of segments is the same, and the boundary range (range length) corresponding to each segment is also the same.

[0040] The initial large object 31 includes: Part_1 segment 311, Part_2 segment 312, Part_3 segment 313, Part_4 segment 314 and Part_5 segment 315, and the large object 32 to be uploaded includes: Part_1 segment 321, Part_2 segment 322, Part_3 segment 323, Part_4 segment 324 and Part_5 segment 325. The content corresponding to some segments may be updated, which may span one or more segments, and the segments with updated content are recorded.

[0041] The initial large object 31 and the large object to be uploaded 32 are of the same size and have five segments. After difference identification, it is found that the contents of the second segment and the third segment are different. The identified difference segments can be recorded in a temporary key-value pair data table. The created temporary key-value pair data table can be stored in memory. If the first check value is the same as the second check value, it means that the segment content has not been modified. If the first check value is different from the second check value, it means that the corresponding segment content has been updated, and the updated segment information is recorded, and the updated segment information is recorded in the temporary key-value pair data table.

[0042] The main fields of the temporary key-value data table include: Object_Name: the name of the large object; Object_ID: the ID corresponding to the large object in storage; Part_Name: the name of the segment; Part_ID: the ID corresponding to the segment; Part_MD5: the MD5 value of the updated segment; Part_Size: records the size of each segment of the large object, where the size of the Nth segment is not necessarily the same as the size of the N-1th segment; Updated: 0 means the content has not been updated, 1 means it has been updated; BucketID: the ID of the bucket where the object is located.

[0043] The second check value corresponding to the part_2 segment in the segments to be uploaded is different from the first check value corresponding to the part_2 segment in the initial large object, so the part_2 segment in the large object to be uploaded is uploaded; the second check value corresponding to the part_3 segment in the segments to be uploaded is different from the first check value corresponding to the part_3 segment in the initial large object, so the part_3 segment in the large object to be uploaded is uploaded. Step S203, obtain the segment lists Part_2 and Part_3 that need to be updated, and the reused segment lists Part_1, Part_4 and Part_5 that have not been modified. Set Updated of part_2 and part_3 to 1 in the key-value pair data table, and record the new check value, which can be an MD5 value, and other field information.

[0044] The method for updating large objects with the same name in distributed object storage provided in this embodiment, during the process of updating large objects in distributed object storage, compares the first check value of each segment of the initial large object stored in the distributed system with the second check value of each segment of the large object to be uploaded, judges and analyzes the updated segments, and uploads the updated segments, thereby reducing the data transmission volume and saving transmission time and bandwidth costs.

[0045] Figure 4 Another schematic diagram of the comparison process between the initial large object and the large object to be uploaded is shown. Figure 4As shown, the initial large object 41 includes: Part_1 segment 411, Part_2 segment 412, Part_3 segment 413, Part_4 segment 414 and Part_5 segment 415, and the large object 42 to be uploaded includes: Part_1 segment 421, Part_2 segment 422, Part_3 segment 423 and Part_4 segment 424.

[0046] The size of the large object 42 to be uploaded is smaller than the size of the existing initial large object 41. Since the same segment size is used, the number of segments of the large object to be uploaded is 4, which is less than the number of segments of the recorded initial large object, 5. In addition to setting the Updated mark of the segment with data update, the number of segments also needs to be recorded.

[0047] The initial large object 41 has 5 segments, and the large object to be uploaded 42 has 4 segments. The size of the large object to be uploaded is smaller than the size of the initial large object. After identifying the difference in segment content, the first check value corresponding to the first segment of the initial large object is different from the second check value corresponding to the first segment of the large object to be uploaded, and the first check value corresponding to the third segment of the initial large object is different from the second check value corresponding to the third segment of the large object to be uploaded. The content of part1 and part3 is different. In the temporary key-value pair data table, set Updated of part1 and part3 to 1, and record the new MD5 values ​​of part_1 segment 421 and part_3 segment 423. The metadata of the large object to be uploaded will also record the number of segments.

[0048] Figure 5 FIG. 4 shows another schematic diagram of the comparison process between the initial large object and the large object to be uploaded. Figure 5 As shown, the initial large object 51 includes: Part_1 segment 511, Part_2 segment 512, Part_3 segment 513, Part_4 segment 514 and Part_5 segment 415, and the large object 52 to be uploaded includes: part_1 segment 521, part_2 segment 522, part_3 segment 523, part_4 segment 524, part_5 segment 525, part_6 segment 526 and part_7 segment 527.

[0049] The size of the large object 52 to be uploaded is larger than the size of the existing initial large object 51. Since the same segment size is used, the number of segments 7 of the large object to be uploaded is more than the number of segments 5 of the recorded initial large object. In addition to setting the Updated mark of the segment with data update, it is also necessary to record the number of segments and the MD5 of the difference segments.

[0050] In some optional implementations, the aforementioned method for updating large objects with the same name in distributed object storage further includes: if the value represented by the target to-be-uploaded sequence number does not exist in the value represented by the initial sequence number, uploading the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number.

[0051] In this embodiment, the value represented by the target to-be-uploaded sequence number does not exist in the value represented by the initial sequence number, that is, the size of the large object 52 to be uploaded is larger than the size of the initial large object 51. After identifying the difference in segmented content, the first check value corresponding to the second segment of the initial large object is different from the second check value corresponding to the second segment of the large object to be uploaded, the first check value corresponding to the fourth segment of the initial large object is different from the second check value corresponding to the fourth segment of the large object to be uploaded, and the first check value corresponding to the fifth segment of the initial large object is different from the second check value corresponding to the fifth segment of the large object to be uploaded. The contents of part2, part4 and part5 are different. In the temporary key-value pair data table, Updated of part2, part4 and part5 is set to 1, and the new MD5 value is recorded. At the same time, the MD5 values ​​of the newly added part6 and part7 need to be recorded, and the metadata of the large object to be uploaded records the number of segments.

[0052] In this way, when the amount of large object data to be uploaded increases during the large object update process, segment reuse and large object data update can also be well performed.

[0053] According to the above logic, if there are multiple uploads of a large object with the same name, each time after comparing all segments, the Part IDs of the segments need to be reorganized to ensure the continuity of all IDs. Figure 6 The following is a schematic diagram showing the process of updating multiple large objects with the same name. Figure 6 As shown, the initial large object 61 includes: Part1 segment 611, Part2 segment 612, Part3 segment 613, Part4 segment 614 and Part5 segment 615. After the first data update, Part2, Part4 and Part5 are updated and the size increases, and Part6 and Part7 are added. The first large object to be uploaded 62 includes: Part1 segment 611, 1_Part2 segment 622, Part3 segment 613, 1_Part4 segment 624, 1_Part5 segment 625, Part6 segment 626 and Part7 segment 627. Among them, 1_Part2 segment 622, 1_Part4 segment 624 and 1_Part5 segment 625 represent the updated segments uploaded for the first time with the same name overwriting.

[0054] After the second data update, the 2_Part1 segment, 2_Part2 segment and 2_Part4 segment are updated, and the size of the large object to be uploaded is reduced. The second large object to be uploaded 63 includes: 2_Part1 segment 631, 2_Part2 segment 632, Part3 segment 613, 2_Part4 segment 634, 1_Part5 segment 625 and Part6 segment 626. Among them, the 2_Part1 segment, 2_Part2 segment and 2_Part4 segment are the second overwrite upload with the same name, and are the updated segments compared with the first overwrite upload with the same name.

[0055] After the third data update, the 3_Part2 segment and the 3_Part6 segment are updated, and the size of the large object to be uploaded does not change. The third large object to be uploaded 64 includes: 2_Part1 segment 631, 3_Part2 segment 642, Part3 segment 613, 2_Part4 segment 634, 1_Part5 segment 625 and 3_Part6 segment 646. Among them, the 3_Part2 segment and the 3_Part6 segment are the third overwrite upload with the same name, and the segments are updated compared with the second overwrite upload with the same name.

[0056] In the above three data update processes, the Part3 segment 613 has not been modified. Therefore, in the large object update process, the content corresponding to the Part3 segment 613 in the initial large object can be used to merge and update with other segments.

[0057] In some optional embodiments, the aforementioned method for updating large objects with the same name in distributed object storage also includes: determining the reused segments that have not been modified in the initial large object; if the segment to be uploaded corresponding to the target sequence number to be uploaded is uploaded successfully, merging the segment to be uploaded corresponding to the target sequence number to be uploaded and the reused segment to complete the update of the initial large object.

[0058] In this embodiment, Figure 7 Another method flow chart for updating large objects with the same name in distributed object storage is shown. Figure 7 As shown,

[0059] Step S701, initialize fast multipart upload of large objects; add a custom parameter fast-put-object in the Hypertext Transfer Protocol (http) request header to identify the fast multipart upload of large objects. When the upload method is specified as fast upload, the client obtains the size, version, and storage category of the object by reading the object metadata interface.

[0060] Step S702, segment difference identification and analysis: by introducing the segment difference identification and analysis method, the difference segments between the large object to be uploaded and the large object already existing in the cluster are identified and recorded in a temporary key-value pair data table. The segment difference identification and analysis can be performed through the steps recorded in step S103, which will not be repeated here.

[0061] Step S703, metadata management of large objects and segmented objects; compared with metadata management in related technologies, four new fields are added to the metadata of large objects in the embodiment of the present invention, including: Put_Type field, the x-put-type tag value set in the corresponding request is fast-put-object; Part_kv_info field: metadata KV information of each segment recorded, including segment MD5 value and length; Part_Num field: record the number of all segments of the large object; Part_Size field: record the size of each segment of the large object. Five new fields are added to the segment metadata, including: segment size, segment Part ID, segment Range, MD5 and ETag information, etc.

[0062] Step S704, segment multiplexing and segment uploading; when performing segment uploading, although the entire large object is uploaded, since the updated segments have been identified and the metadata of the segments have been updated, only the segments with updated content can be uploaded. For the segments that have not changed, the segment multiplexing method is adopted, and the range length of the segment objects to be reused is specified, and the X-Amz-Copy-Source method is called to copy the data content.

[0063] Step S705, merging segments: calling the standard segment upload interface to perform the upload operation, after the operation is successful, recording the ETag of each segment, updating and recording the metadata information of each segment. Finally, calling the standard again to merge all the segments.

[0064] Step S706, determine whether the multi-part upload is successful, if successful, proceed to step S708, if failed, proceed to step S707;

[0065] Step S707, exception handling and junk object marking; in this step, if the uploading of the segment to be uploaded corresponding to the target sequence number to be uploaded fails, the segment to be uploaded corresponding to the target sequence number to be uploaded is re-uploaded; if the re-upload fails, a deletion tag is added to the segment to be uploaded corresponding to the target sequence number to be uploaded.

[0066] Step S708, updating the metadata information of the existing large object, and updating the recorded large object segment metadata information.

[0067] A segment reuse strategy is introduced to obtain the unupdated segments based on the metadata information of the large object to be uploaded. Since the unupdated segments already exist, they can be reused repeatedly, thereby reducing data redundancy and transmission overhead during upload.

[0068] In some optional embodiments, determining the unmodified reused segments in the initial large object includes: when the target initial sequence number is the same as the numerical value represented by the sequence number to be uploaded, if the second check value of the segment to be uploaded corresponding to the sequence number to be uploaded is the same as the first check value of the target initial segment corresponding to the target initial sequence number, the target initial segment is a reused segment.

[0069] In this embodiment, if the second check value of the to-be-uploaded segment corresponding to the to-be-uploaded sequence number is the same as the first check value of the target initial segment corresponding to the target initial sequence number, it indicates that the to-be-uploaded segment has not changed compared to the target initial segment. Then the target initial segment does not need to be updated.

[0070] In this way, using the segment multiplexing method, only the updated segments are uploaded, and the non-updated segment contents are reused, which reduces transmission redundancy, improves data transmission efficiency, and reduces storage costs.

[0071] In this embodiment, a device for updating a large object in a distributed object storage is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0072] This embodiment provides a device for updating a large object in a distributed object storage. Figure 8 The structure diagram of the device for updating a large object in a distributed object storage provided by an embodiment of the present invention is shown as follows: Figure 8 As shown, including:

[0073] The metadata management module 801 is used to obtain the initial sequence number of each segment in the initial large object stored in the distributed system, and the first check value of the initial segment corresponding to each initial sequence number.

[0074] The difference identification and analysis module 802 is used to obtain the data volume of the first segment in the initial large object, divide the large object to be uploaded into multiple segments based on the data volume, and obtain multiple segments to be uploaded; calculate the check value of each segment to be uploaded, obtain the to-be-uploaded serial number of the to-be-uploaded segment, and the second check value corresponding to each to-be-uploaded serial number.

[0075] The segmented upload module 803 is used to upload the segment to be uploaded corresponding to the target sequence number to be uploaded when the target sequence number to be uploaded and the initial sequence number represent the same value, if the second check value of the segment to be uploaded corresponding to the target sequence number to be uploaded is different from the first check value of the initial segment corresponding to the initial sequence number.

[0076] Based on the metadata management module 801, the metadata information of the initial large object and the temporary key-value pair data table information of each segment of the large object to be uploaded generated by the difference identification and analysis module 802 are obtained. The metadata of the large object to be uploaded and the segment metadata are updated and solidified in the memory. The main fields of the metadata structure of the large object to be uploaded are shown in Table 1.

[0077] Table 1 Main fields of metadata of large objects to be uploaded

[0078] BucketID Name Length Date Last-Modified ETag Version_id Put_Type Part_kv_info …

[0079] Among them, BucketID is the identifier of the bucket where the large object is located; Name is the name of the large object; Length is used to indicate the length of the data; Date is a date identifier; Last-Modified indicates the last modification time of the resource in the server; ETag is used to identify the specific version of the resource; Version_id is the object version identifier; Put_Type is a customized fast multipart upload method; Part_kv_info is used to record the metadata key-value pair information of each part. The metadata structure of each part object is shown in Table 2.

[0080] Table 2 Metadata structure of each segment object

[0081] Object_Name Part_ID Range Obj_Version_id ETag MD5 Shared_Tag Part_Size …

[0082] Among them, Object_Name is the name of the large object; Part_ID is the ID corresponding to the segment; Range is the boundary range; Obj_Version_id is the object version identifier; ETag is used to identify the specific version of the resource; MD5 is the checksum; Shared_Tag is the reuse tag; Part_Size is used to record the size of each segment of the large object.

[0083] The metadata management module is used to record and maintain the metadata of the segments. It adds metadata management fields and promptly updates the segment metadata information of the large object to be uploaded after segment comparison and analysis to ensure data consistency during upload and final merging.

[0084] Fig. 9 FIG. 2 shows a schematic diagram of the working process of a device for updating a large object in a distributed object storage. Fig. 9As shown, the difference identification and analysis module 802 is also used to obtain the Part_ID of the difference segment and the Part_ID that has not been updated in the initial large object; the large object content reading module 901 is used to directly read the segment data corresponding to the new Part_ID according to the Range based on the Part_ID of the difference segment; the large object reading module 901 is also used to set the segment reuse flag and call X-Amz-Copy-Source according to the Range to read the unupdated segment data; step S91: upload the data read by the large object content reading module (Upload Parts); step S92: determine whether the segment upload is successful; step S93: if unsuccessful, do not update the metadata information of the segment object, try to retry, and if the retry fails, add a delete tag to each segment object; step S94: if successful, generate a new ETag for each segment, and go to step S95; step S95: update the metadata information of each segment; the segment merging module 902 is used to update the metadata information of the large object.

[0085] In some optional implementations, the metadata management module 801 includes:

[0086] The large object metadata management unit is used to record the number of segments in the initial large object based on the segment number field, and to record the key-value pair information of each segment based on the metadata key-value pair field.

[0087] The segment metadata management unit is used to record the to-be-uploaded sequence number based on the segment identifier field, and to record the second check value based on the segment check value field.

[0088] In some optional implementations, the aforementioned apparatus for updating a large object in a distributed object storage further includes:

[0089] The first uploading module is used for uploading the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number if the value represented by the target to-be-uploaded sequence number does not exist in the value represented by the initial sequence number.

[0090] In some optional implementations, the aforementioned apparatus for updating a large object in a distributed object storage further includes:

[0091] The first update module is used to determine the unmodified reused segments in the initial large object; if the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number is uploaded successfully, merge the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number and the reused segment to complete the update of the initial large object.

[0092] In some optional implementations, the aforementioned apparatus for updating a large object in a distributed object storage further includes:

[0093] The second updating module is used for re-uploading the segment to be uploaded corresponding to the target sequence number to be uploaded if the upload fails; if the re-upload fails, adding a deletion tag to the segment to be uploaded corresponding to the target sequence number to be uploaded.

[0094] In some optional implementations, the second update module includes:

[0095] The first unit of the second update module is used to, when the target initial sequence number is the same as the numerical value represented by the sequence number to be uploaded, if the second check value of the segment to be uploaded corresponding to the sequence number to be uploaded is the same as the first check value of the target initial segment corresponding to the target initial sequence number, the target initial segment is a multiplexed segment.

[0096] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0097] The device for updating large objects in distributed object storage in this embodiment is presented in the form of functional units, where the units refer to application specific integrated circuits (ASICs), processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0098] The embodiment of the present invention also provides a computer device having the above Figure 8 An apparatus for updating large objects in a distributed object store is shown.

[0099] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0100] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0101] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.

[0102] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0103] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0104] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0105] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0106] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0107] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0108] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for updating large objects with the same name in distributed object storage, characterized in that: The method comprises: Obtaining an initial sequence number of each segment in an initial large object stored in a distributed system, and a first check value of the initial segment corresponding to each of the initial sequence numbers; Obtaining the data volume of the first segment in the initial large object, dividing the large object to be uploaded into multiple segments based on the data volume, and obtaining multiple segments to be uploaded; calculating the check value of each of the segments to be uploaded, obtaining the to-be-uploaded serial number of the to-be-uploaded segment, and the second check value of the to-be-uploaded segment corresponding to each of the to-be-uploaded serial number; When the target sequence number to be uploaded is the same as the value represented by the initial sequence number, if the second check value of the segment to be uploaded corresponding to the target sequence number to be uploaded is different from the first check value of the initial segment corresponding to the initial sequence number, upload the segment to be uploaded corresponding to the target sequence number to be uploaded.

2. The method according to claim 1, characterized in that The method further comprises: If the value represented by the target to-be-uploaded sequence number does not exist in the value represented by the initial sequence number, the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number is uploaded.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Determining unmodified reuse segments in the initial large object; If the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number is successfully uploaded, the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number and the multiplexed segment are merged to complete the update of the initial large object.

4. The method according to claim 3, characterized in that The method further comprises: If the uploading of the segment to be uploaded corresponding to the target uploading sequence number fails, re-uploading the segment to be uploaded corresponding to the target uploading sequence number; If the re-upload fails, a deletion tag is added to the to-be-uploaded segment corresponding to the target to-be-uploaded sequence number.

5. The method according to claim 3, characterized in that: The determining of the unmodified reused segments in the initial large object comprises: When the target initial sequence number is the same as the numerical value represented by the sequence number to be uploaded, if the second check value of the segment to be uploaded corresponding to the sequence number to be uploaded is the same as the first check value of the target initial segment corresponding to the target initial sequence number, the target initial segment is the multiplexed segment.

6. A device for updating large objects with the same name in distributed object storage, characterized in that: The device comprises: A metadata management module, used to obtain an initial sequence number of each segment in an initial large object stored in a distributed system, and a first check value of the initial segment corresponding to each of the initial sequence numbers; The difference identification and analysis module is used to obtain the data volume of the first segment in the initial large object, divide the large object to be uploaded into multiple segments based on the data volume, and obtain multiple segments to be uploaded; calculate the check value of each of the segments to be uploaded, and obtain the to-be-uploaded serial number of the to-be-uploaded segment, and the second check value corresponding to each of the to-be-uploaded serial numbers; The segmented upload module is used to upload the segment to be uploaded corresponding to the target sequence number to be uploaded if the target sequence number to be uploaded is the same as the value represented by the initial sequence number and if the second check value of the segment to be uploaded corresponding to the target sequence number to be uploaded is different from the first check value of the initial segment corresponding to the initial sequence number.

7. The device according to claim 6, characterized in that The metadata management module includes: A large object metadata management unit, configured to record the number of segments in the initial large object based on a segment number field, and to record key-value pair information of each segment based on a metadata key-value pair field; The segment metadata management unit is used to record the to-be-uploaded sequence number based on the segment identifier field, and to record the second check value based on the segment check value field.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for updating large objects with the same name in distributed object storage as described in any one of claims 1 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for updating large objects with the same name in distributed object storage according to any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the method for updating large objects with the same name in distributed object storage according to any one of claims 1 to 5.

Images