Object merging method and device

Abstract

The present invention discloses an object merging method and device. The method comprises: receiving a target object whose data volume is less than a preset threshold; caching the target object and writing the target object into metadata; writing the target object into one of a plurality of merged objects; when the total data volume of the target objects in the merged objects reaches a first preset data volume, writing the target objects of the preset data volume into a data pool; when the total data volume of the merged objects reaches a second preset data volume, writing the merged objects into the data pool and updating the metadata of all target objects of the merged objects, wherein the second preset data volume is greater than the first preset data volume. The present invention solves the technical problem that the object merging method in the related art needs to be written into the cache pool, then read from the cache pool, and written into the merged object for merging, which not only occupies too many cache pool resources but also has low efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data storage, and in particular, to an object merging method and apparatus. Background Art

[0002] With the high-speed development of Internet applications, object storage has been widely used due to its ease of use and almost infinite flat scalability. Among them, the demand for small file storage has become increasingly prominent. However, limited by the allocation granularity of the storage backend space, such as the allocation granularity of some backend SSDs is 256K and that of HDDs is 1M, directly storing a large number of small files will cause huge space waste. Moreover, when writing small files to HDDs, limited by the HDD IOPS performance bottleneck, the writing performance is relatively low.

[0003] In the face of the above difficulties, the currently commonly used optimization method in the industry is to merge small files. First, write small files into a high cache pool (SSD) and record logs, and then aggregate a batch of small files and read them out to form a large file and then rewrite it into the backend data pool (HDD). By using the merging method, the problems of huge space waste in small file services and backend performance can be significantly improved.

[0004] The existing commonly used merging method is to adopt a background merging method, that is, first write small files into the cache SSD and then record logs. The background merging module scans the logs, aggregates an appropriate number, and then reads out small objects from the cache SSD again and aggregates them into a merged object and writes it to the HDD backend. This background merging method has the following disadvantages:

[0005] Each uploaded small file will go through one read and two writes, which is equivalent to an extra read for each IO. For the cache SSD, when performing background merging, reading small objects will generate a large number of read IOs, which will affect the front-end writing performance.

[0006] Since the background merging is performed by scanning the logs in the background, if the front-end IO volume is large but the background merging is not timely, a large amount of data will accumulate in the cache SSD, which will increase the space occupancy of the cache disk and even fill up the cache space.

[0007] Since the background merging is independent of the front-end IO, it cannot have a good feedback adjustment effect on the front-end IO, which is likely to cause a large accumulation of merged small objects and it is difficult to adjust the front-end processing rate.

[0008] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0009] An embodiment of the present invention provides an object merging method and apparatus, at least to solve the technical problem that the object merging method in the related art needs to be written into the cache pool, then read from the cache pool, and written into the merged object for merging, which not only occupies too much cache pool resources but also has low efficiency.

[0010] According to one aspect of the embodiments of the present invention, an object merging method is provided, including: receiving a target object with a data volume less than a preset threshold; caching the target object and writing the target object into metadata, where the metadata is used to record the storage path of the target object; writing the target object into one of a plurality of merged objects; when the total data volume of the target objects in the merged object reaches a first preset data volume, writing the target objects with the preset data volume into the data pool, where the preset data volume is the maximum write data volume of the input interface of the data pool; when the total data volume of the merged object reaches a second preset data volume, writing the merged object into the data pool and updating the metadata of all target objects of the merged object, where the second preset data volume is greater than the first preset data volume.

[0011] Optionally, writing the target object into one of a plurality of merged objects includes: determining a merged object in an idle state among the plurality of merged objects, where the merged object in the idle state can be written or cached waiting to be written; suspending the writing of the target object when the number of merged objects in the idle state is zero; writing the target object into the merged object when the number of merged objects in the idle state is one; and randomly selecting one of the merged objects and writing the target object into the merged object when the number of merged objects in the idle state is multiple.

[0012] Optionally, writing the target object into the merged object includes: locking the target object by the merged object to be written; writing the locked target object into the cache space set by the merged object; and migrating the target object from the cache space to the merged object when the merged object is currently writable; where the merged object is in a state of not being cacheable and waiting to be written when the number of target objects in the cache space exceeds a preset number.

[0013] Optionally, when the total data volume of the merging object reaches a second preset data volume, write the merging object into the data pool and update the metadata of all target objects of the merging object, including: determining the total data volume of the merging object according to the total data volume of the target objects merged in the merging object, the metadata parameters of the target objects, and the attribute parameters of the merging object; when the total data volume reaches the second preset data volume, write the metadata parameters and the attribute parameters in the merging object, as well as the data of the remaining target objects, into the data pool; update the metadata parameters to update the metadata of the target objects.

[0014] Optionally, when the total data volume of the merging object reaches a second preset data volume, after writing the merging object into the data pool and updating the metadata of all target objects of the merging object, it further includes: creating a new merging object at the original position of the merging object; wherein, the new merging object is in a non-writable state during the creation process.

[0015] Optionally, it further includes: when the total data volume of the merging object reaches a second preset data volume, determining that the merging of the merging object is completed; selecting the last merged target object in the merging object and executing the completion state of the merging object; updating the metadata of the target object and executing the updated metadata state and the completed metadata update state of the target object; wherein, in the completion state, the updated metadata state and the completed metadata update state are both writable idle states.

[0016] Optionally, the method further includes: after receiving the target object, creating a first merging log of the target object, where the first merging log is used to record the target object, and after the target object completes merging and updates the metadata, deleting the first merging log; when re-entering after a merging interruption, scanning the first merging log to determine the target objects that have not completed merging and performing re-merging.

[0017] Optionally, it further includes: after creating the merging object, creating a second merging log of the merging object, where the second merging log is used to record the data storage location and the merging state of the merging object, and after the merging object is successfully written into the data pool and the metadata is updated, deleting the second merging log; when re-entering after a merging interruption, scanning the second merging log to perform re-merging on the target objects whose metadata has been updated; recycling the merging object corresponding to the second merging log.

[0018] According to another aspect of the embodiments of the present invention, there is also provided an object merging device, including: a receiving module, configured to receive a target object whose data volume is less than a preset threshold; a caching module, configured to cache the target object and write the target object into metadata, where the metadata is used to record the storage path of the target object; a writing module, configured to write the target object into one of a plurality of merged objects; a first merging module, configured to, when the total data volume of the target objects in the merged object reaches a first preset data volume, write the target objects with the preset data volume into a data pool, where the preset data volume is the maximum write data volume of the input interface of the data pool; a second merging module, configured to, when the total data volume of the merged object reaches a second preset data volume, write the merged object into the data pool and update the metadata of all the target objects in the merged object, where the second preset data volume is greater than the first preset data volume.

[0019] According to another aspect of the embodiments of the present invention, there is also provided a computer storage medium, where the computer storage medium includes a stored program, and when the program runs, it controls the device where the computer storage medium is located to execute the object merging method described in any one of the above.

[0020] According to another aspect of the embodiments of the present invention, there is also provided a processor, where the processor is used to run a program, and when the program runs, it executes the object merging method described in any one of the above.

[0021] In an embodiment of the present invention, a target object with a received data volume less than a preset threshold is adopted; the target object is cached, and the target object is written into metadata, where the metadata is used to record the storage path of the target object; the target object is written into one of a plurality of merge objects; when the total data volume of the target objects in the merge object reaches a first preset data volume, a preset data volume of target objects is written into a data pool, where the preset data volume is the maximum write data volume of the input interface of the data pool; when the total data volume of the merge object reaches a second preset data volume, the merge object is written into the data pool, and the metadata of all target objects in the merge object is updated, where the second preset data volume is greater than the first preset data volume. By directly writing the target object into the merge object and writing it into the data pool when the data volume in the merge object reaches the first preset data volume, and completing the merge when the merge object reaches the second preset data volume and updating the metadata of the target object, the purpose of quickly and timely writing the target object into the data pool is achieved, and the time stored in the cache pool is short, and the time in the merge object is also short. Thus, the technical effect of reducing the burden on the cache pool, greatly reducing the occupation of memory resources in the cache pool, and improving the efficiency of target object merging is achieved, and further the technical problem in the related art that the object merging method needs to be written into the cache pool, then read from the cache pool, and written into the merge object for merging, which not only occupies too many cache pool resources but also has low efficiency is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0023] Figure 1 is a flowchart of an object merging method according to an embodiment of the present invention;

[0024] Figure 2 is a flowchart of object merging according to an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of the data structure of a merge object according to an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of the working process of a state machine according to an embodiment of the present invention;

[0027] Figure 5 is a schematic diagram of an object merging device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data used in appropriate cases can be interchanged so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] According to an embodiment of the present invention, a method embodiment of an object merging method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0031] Figure 1 is a flowchart of an object merging method according to an embodiment of the present invention. As Figure 1 shown, the method includes the following steps:

[0032] Step S102, receiving a target object with a data volume less than a preset threshold;

[0033] Step S104, caching the target object and writing the target object into metadata, where the metadata is used to record the storage path of the target object;

[0034] Step S106, writing the target object into one of a plurality of merged objects;

[0035] Step S108, when the total data volume of the target objects in the merged object reaches a first preset data volume, writing a preset data volume of the target objects into the data pool, where the preset data volume is the maximum write data volume of the input interface of the data pool;

[0036] Step S110, when the total data volume of the merging object reaches the second preset data volume, write the merging object into the data pool and update the metadata of all target objects of the merging object, where the second preset data volume is greater than the first preset data volume.

[0037] Through the above steps, adopt target objects with the received data volume less than the preset threshold; cache the target objects and write the target objects into the metadata, where the metadata is used to record the storage path of the target objects; write the target objects into one of multiple merging objects; when the total data volume of the target objects in the merging object reaches the first preset data volume, write the target objects with the preset data volume into the data pool, where the preset data volume is the maximum write data volume of the input interface of the data pool; when the total data volume of the merging object reaches the second preset data volume, write the merging object into the data pool and update the metadata of all target objects of the merging object, where the second preset data volume is greater than the first preset data volume. By directly writing the target objects into the merging object and writing them into the data pool when the data volume in the merging object reaches the first preset data volume, and completing the merging when the merging object reaches the second preset data volume and updating the metadata of the target objects, the purpose of quickly and timely writing the target objects into the data pool is achieved. The target objects are not only stored in the cache pool for a short time, but also in the merging object for a short time, thus realizing the technical effects of reducing the burden on the cache pool, greatly reducing the occupation of the memory resources of the cache pool, and improving the efficiency of target object merging. Furthermore, the technical problem that the object merging method in the related art needs to be written into the cache pool, then read from the cache pool, and written into the merging object for merging, which not only occupies too much cache pool resources but also has low efficiency, is solved.

[0038] The execution subject of the above steps can be a storage device with a merging function, such as a server or a memory. The above target object is less than the preset threshold, indicating that the data volume of the object is small and needs to be merged. The above preset threshold can be set according to experience or modified according to requirements. In implementation, usually 1M or 4M can be used as the above preset threshold to determine whether the received object is a target object.

[0039] After receiving the target object, first write the target object into the cache pool for caching to avoid the loss of the target object due to the interruption of the merging process. When writing the target object into the cache, automatically update the metadata of the target object, thereby updating the storage path of the target object in the metadata to facilitate system query.

[0040] While writing the target object into the cache pool, the target object is sent to the merging object for merging. The execution subject of the above steps S106 to S108 can be the merging module set by the above storage device for performing the merging action. Therefore, before sending the target object to the merging object, the target object can be sent to the merging module first, and then sent by the merging module to the above merging object for merging.

[0041] It should be noted that in one embodiment, after receiving the target object, the target object can be copied, one is sent to the cache pool, and the other is sent to the merging object for merging. In the process of sending the two target objects to the cache pool respectively, it can be done simultaneously, or first sent to the cache pool to ensure the effective caching of the target object, and then sent to the merging object for merging.

[0042] The above merging objects are multiple, and the merging processes of the multiple merging objects are independent of each other and do not interfere with each other. The merging module allocates the target objects to be merged to the multiple merging objects. By adopting the method of parallel merging of multiple merging objects, the merging efficiency can be greatly improved. It should be noted that when allocating the merging object for the target object to perform merging, one of the multiple executable merging objects can be randomly selected, or the multiple merging objects can be traversed in sequence until a merging object that can perform merging is found.

[0043] The above merging object cannot perform merging in some states. For example, when the previous merging object has completed merging and written to the data pool, it cannot perform merging for the target object, and a new merging object will be generated at the position of the original merging object. If the new merging object is still in the creation process, then the new merging object cannot perform merging for the target object either. Therefore, when determining the merging object for the target object, it is necessary to select a merging object that can perform the merging operation from multiple merging objects. When there are multiple merging objects that can perform the merging operation, select one of them according to the above selection method as the merging object of the target object, and then send the target object to the merging object for merging.

[0044] During the process of the merging object performing merging, target objects are continuously written into the merging object. In order to improve the merging efficiency and reduce the memory occupied by the multiple target objects to be merged in the merging object, when the total data volume of the target objects in the merging object reaches the first preset data volume, the target objects of the first preset data volume are written into the data pool. In order to ensure that the target object can be written into the data pool at one time to improve the writing efficiency, the first preset data volume can be set to the single - time maximum writing data volume of the input interface of the data pool. Specifically, in implementation, when the single - time maximum writing data volume of the data interface of the data pool is 1M, the above first preset data volume is set to 1M.

[0045] It should be noted that since the size of the target object is random and uncontrollable, there is a situation where the total data volume of the target object is slightly larger than the above-mentioned first preset data volume. At this time, when writing, select the data of the first preset data volume from the target object with the total data volume for writing, and the remaining data is still stored in the merging object, waiting for the subsequent writing of the target object. When it accumulates to the first preset data volume, perform the next writing.

[0046] In addition, since the merging object cannot be written infinitely, when the total data volume of the merging object reaches the second preset data volume, write the merging object into the data pool and update the metadata of all target objects in the merging object to complete the merging of the target objects. The above-mentioned second preset data volume is usually much larger than the above-mentioned first preset data volume and can be an integer multiple of the above-mentioned first preset data volume. For example, in implementation, the above-mentioned first preset data volume is 1M, and the above-mentioned second preset data volume is 48M.

[0047] Optionally, writing the target object into one of multiple merging objects includes: determining the merging object in the idle state among multiple merging objects, where the merging object in the idle state can be written or cached waiting for writing; in the case where the number of merging objects in the idle state is zero, pause writing the target object; in the case where the number of merging objects in the idle state is one, write the target object into the merging object; in the case where the number of merging objects in the idle state is multiple, randomly select one merging object and write the target object into the merging object.

[0048] When writing the target object into one of multiple merging objects, considering the non-mergable state of the merging object, it is necessary to first determine the merging object in the idle state among multiple merging objects, that is, the mergable merging object.

[0049] Considering that the merging object writing mechanism in this embodiment is to first write the target object into the cache space and then migrate it into the merging object, it can be considered that the merging object being writable or cacheable waiting for writing are both states that can perform the merging operation on the target object, that is, the above-mentioned idle state.

[0050] In the case where the number of merging objects in the idle state is zero, it indicates that the rate of receiving the target object is too fast at this time, and the merging efficiency of the system reaches the maximum and still cannot meet the requirement. It is necessary to control the merging of the target object and generate feedback to the system to pause receiving the target object.

[0051] In the case where the number of merging objects in the idle state is one, write the target object into the merging object; in the case where the number of merging objects in the idle state is multiple, randomly select one merging object and write the target object into the merging object. The above-mentioned random selection method can also select the merging object by the above-mentioned method of traversing in order.

[0052] Optionally, writing the target object into the merging object includes: locking the target object by the merging object to be written; writing the locked target object into the cache space set by the merging object; when the merging object is currently writable, migrating the target object from the cache space to the merging object; wherein, when the number of target objects in the cache space exceeds a preset number, the merging object is in a non-cacheable waiting-to-write state.

[0053] When determining the merging object into which the target object is to be written, first lock the target object with the merging object to prevent the target object from being written into other merging objects, resulting in redundancy and wasting resources. It can also prevent data interaction from being chaotic and affecting the merging efficiency.

[0054] When writing the target object into the merging object, first write the locked target object into the cache space set by the merging object, and when the merging object is currently writable, migrate the target object from the cache space to the merging object; wherein, when the number of target objects in the cache space exceeds a preset number, the merging object is in a non-cacheable waiting-to-write state, that is, a non-idle state.

[0055] Optionally, when the total data volume of the merging object reaches a second preset data volume, write the merging object into the data pool and update the metadata of all target objects of the merging object, including: determining the total data volume of the merging object according to the total data volume of the target objects merged in the merging object, the metadata parameters of the target objects, and the attribute parameters of the merging object; when the total data volume reaches the second preset data volume, write the metadata parameters and attribute parameters in the merging object, as well as the data of the remaining target objects, into the data pool; update the metadata parameters to update the metadata of the target objects.

[0056] When the total data volume of the merging object reaches a second preset data volume, write the merging object into the data pool. It should be noted that the total data volume of the merging object here not only includes multiple target objects written into the merging object, but also includes other data contents of the merging object.

[0057] The merging object can also include simple metadata of all target objects, metadata parameters of the target objects such as the position of the metadata, for example, Figure 3 meta and index in. It can also include attribute parameters of the merging object such as the version number of the merging object, the number of target objects, and the metadata size, for example, Figure 3 head in. The above metadata parameters of the target objects and the attribute parameters of the merging object are written into the data pool together after the merging of the merging object is completed.

[0058] When the total data volume reaches the second preset data volume, write the metadata parameters and attribute parameters in the merging object, as well as the data of the remaining target objects, into the data pool. And update the metadata parameters to update the metadata of the target objects.

[0059] Optionally, when the total data volume of the merging object reaches the second preset data volume, after writing the merging object into the data pool and updating the metadata of all target objects of the merging object, it further includes: creating a new merging object at the original position of the merging object; wherein, the new merging object is in a non-writable state during the creation process.

[0060] Optionally, it further includes: when the total data volume of the merging object reaches the second preset data volume, determining that the merging of the merging object is completed; selecting the last merged target object in the merging object, and executing the completion state of the merging object; updating the metadata of the target object, and executing the updated metadata state and the completed metadata update state of the target object; wherein, in the completion state, the updated metadata state and the completed metadata update state are both writable idle states.

[0061] When the merging of the merging object is completed, the last target object will be detached and subsequent processing will be performed, including processing Figure 4 the COMPLETE, UPDATING, and UPDATED states. This can prevent the merging object from blocking other target objects from being merged when doing time-consuming finishing work (updating the metadata of all target objects takes a long time).

[0062] Optionally, the method further includes: after receiving the target object, creating a first merging log of the target object, wherein the first merging log is used to record the target object, and after the target object is merged and its metadata is updated, deleting the first merging log; when re-entering after the merging is interrupted, scanning the first merging log to determine the target objects that have not been merged, and performing re-merging.

[0063] After the merging module receives the above-mentioned target object, it creates a first merging log of the target object to record the merging process of the target object. After the target object is merged and its metadata is updated, the first merging log is deleted. In this way, when re-entering after the merging process is interrupted, by reading the existing first merging log, it can be determined which target objects have not been merged, and the target objects are taken out from the cache pool and re-merged, effectively preventing the loss of target objects due to the terminal.

[0064] Optionally, it further includes: after creating a merge object, creating a second merge log for the merge object, where the second merge log is used to record the data storage location and merge status of the merge object, and deleting the second merge log after the merge object is successfully written into the data pool and the metadata is updated; in the case of re-entering after a merge interruption, scanning the second merge log, and re-merging the target objects for which the metadata has been successfully updated; recycling the merge object corresponding to the second merge log.

[0065] After creating a merge object, establish a second merge log for the merge object, and delete it after the merge object is successfully written into the data pool and the metadata is updated. In this way, when re-entering after a merge interruption, read the second merge log to determine the merge objects that have not been successfully written into the data pool. Since the metadata of all target objects of the merge object needs to be updated during the process of writing the merge object into the data pool, and it also needs to update the metadata of each target object one by one and cannot be completed simultaneously at one time. However, due to the merge interruption, the metadata update of the target objects of this merge object is interrupted, and after re-entering, it cannot continue to be updated, and only the target objects in this merge object can be re-merged. However, in the merge object, there are actually two types of target objects. One is the target object for which the metadata update has not been completed, that is, the target object for which the merge has not been completed. Such target objects still have the first merge log and can be re-merged in the above manner. The other is the target object for which the metadata update has been completed. This target object has no first log, but because the merge object cannot continue to be merged, this type of target object also needs to be re-merged. Therefore, determine the target objects for which the metadata has been updated through the second merge log, and perform re-merging. And recycle this merge object. To ensure that all target objects are re-merged when re-entering after a merge interruption, and avoid data loss caused by the merge interruption.

[0066] It should be noted that this embodiment also provides an optional implementation manner, which will be described in detail below.

[0067] This embodiment provides a real-time online merging method to solve various problems encountered in the above-mentioned background merging. This online merging method directly adds a merging module to the front-end IO process. When writing small files, it first writes to the cache SSD to ensure data security and improve the response speed, and then returns the user request without increasing the external IO latency. Then, a copy of the small file data is retained in memory and sent to the merging module. After the merging module accumulates a certain number of small files, it writes them to the HDD backend without the need to read data from the cache SSD. Moreover, in this method, the front-end IO process and the merging process are executed using the same context. Before the merging process ends, this context will not process the next IO, which can play a good role in regulating the front-end IO and prevent a large number of front-end IOs from piling up and the background merging from being unable to process in time. Small files uploaded can be directly merged through the online merging module without the need to read small file data from the cache SSD, and the SSD usage space can be greatly reduced.

[0068] Figure 2 is a flowchart of object merging according to an embodiment of the present invention. As Figure 2 shown, the specific steps of this embodiment are as follows:

[0069] (1) A user uploads a small object, which is also the above-mentioned target object and can be an object with a data volume less than 1M or 4M.

[0070] (2) The object gateway first processes the object storage protocol, then writes metadata, and writes data to the cache SSD to improve the processing speed.

[0071] (3) In order not to reduce the user IO response latency, IO SUCCESS is returned to the CLIENT before the online merging process.

[0072] (4) Send the data of the small object to the merging module.

[0073] (5) The merging module first records a compact_log of the small object (which is also the above-mentioned first merging log) for the merging recovery replay function when exiting abnormally, ensuring that each small object can complete the merging.

[0074] (6) To improve the merging concurrency and reduce the conflict probability, multiple merging objects are used to collect data simultaneously. Each time a small object is sent for merging, an idle merging object is selected for online merging.

[0075] (7) To prevent excessive memory occupation, after the merging object receives the small object data, as long as the remaining data in memory reaches 1M, it writes to the backend data pool.

[0076] When the merging object reaches the set upper limit (e.g., 48M), the merging of this merging object ends. The metadata of the merging object is written, the compact_log (the first merging log) of all small objects corresponding to this merging object is deleted, and then the metadata of all small objects is updated so that their data storage locations point to this merging object. Finally, the data of the original small objects written in the cache pool is deleted to free up space.

[0077] Figure 3 It is a schematic diagram of the data structure of the merging object according to the embodiment of the present invention. As Figure 3 shown, to meet the requirement of being able to write merged data to the backend HDD at any time while uploading small objects, a Figure 3 data structure in is designed to save the backend data of the merging object.

[0078] Putting data at the front can facilitate writing to the backend directly when the data reaches 1M, reducing the memory occupancy. Other meta, index, and head are temporarily stored in memory and written together when the merging of the merging object is completed.

[0079] 3. To better control each processing flow of the merging object, the following state machine is designed for the merging object in this embodiment, and corresponding processing is performed according to the state.

[0080] Figure 4 It is a schematic diagram of the working process of the state machine according to the embodiment of the present invention. As Figure 4 shown, the following processing is performed on the state of the merging object:

[0081] Since online merging is a one-to-many process, that is, when small objects are written by multiple threads, the same merging object may be used. Therefore, a lock needs to be added for protection during each merging to achieve mutual exclusion and ensure the consistency and integrity of the data and metadata of the merging object.

[0082] In order to minimize the waiting time of front-end IO merging and reduce the impact on front-end IO performance, only CREATE and CHUNK2 in the above states need to be waited for by the front-end, and all other states do not need to be waited. Among them, the CREATE state is to write compact_big_log (the second merge log) when creating a new merge object, which is used to recycle merged objects in abnormal situations; CHUNK is a data unit written to the back-end HDD each time. After the data to be merged is sent to the merge module, it cannot be cached indefinitely (it will take up a lot of memory space), nor can it be written to the back-end every time a CHUNK (1M) is gathered, blocking other IO merges (low execution efficiency). Therefore, a CHUNK cache space is designed, namely the CHUNK1 state, which still accepts subsequent small objects to add merged data when writing data. If there are already two CHUNKs (CHUNK2 state) waiting to be sent for writing, the front-end IO merge will be blocked. At this time, the front-end rate can also be feedback-adjusted to prevent excessive pressure.

[0083] When the merged objects are merged, the subsequent processing will be separated and continued in the IO context of the last merged small object, processing the COMPLETE / UPDATING / UPDATED status. This will prevent the merged object from blocking other small objects from being merged when doing time-consuming finishing work (updating all small object metadata takes a long time).

[0084] Exception handling process: When an abnormal exit occurs during the merge, the following process is used to merge and replay after the gateway is restarted:

[0085] Since compact_log (the first merge log) is recorded at the beginning of small object merging, it will be deleted only when the merge metadata is updated. Therefore, compact_log (the first merge log) is scanned first, and the unfinished merged small objects are read out and resent to the merge module for merging.

[0086] All unfinished merged objects will have compact_big_log (that is, the second merge log mentioned above), so these unfinished merged objects are rescanned based on these logs, and the small objects with updated metadata are re-read and added to the merge module for re-merging, and the unfinished merged objects are recycled.

[0087] This implementation adopts an online merging method in which data is directly sent to the merging module after being uploaded, which can reduce the process of reading from the cache SSD. It is convenient to directly append the back-end merge object data structure of the written data. A set of merging state machine mechanism can better control the various processes of online merging. When the merging is not completed, there are corresponding log records for large and small objects, which ensures the integrity of the merging, and can facilitate replay and recovery in abnormal situations.

[0088] The online merging method for object storage in this embodiment can provide an efficient, fast, and stable merging function. The advantages over the previous background merging are as follows: Online merging reduces the need for each small object IO in background merging from one read and two writes to only two writes, eliminating the need to read data from the cache SSD again, greatly reducing the pressure on the cache pool. After merging is completed, the data is deleted from the cache SSD, and online merging can be executed directly without waiting. Only a small number of small objects being merged in memory need to be retained in the cache pool, greatly saving the space of the cache pool. Online merging directly performs merging processing in the front-end IO context, which can provide good feedback to the front-end IO and prevent the front-end IO from being blocked due to excessive pressure on the back-end.

[0089] Figure 5 It is a schematic diagram of an object merging device according to an embodiment of the present invention, as Figure 5 shown. According to another aspect of the embodiment of the present invention, an object merging device is further provided, including: a receiving module 50, a caching module 52, a writing module 54, a first merging module 56, and a second merging module 58. The device will be described in detail below.

[0090] The receiving module 50 is configured to receive target objects with a data volume less than a preset threshold; the caching module 52 is connected to the receiving module 50 and is configured to cache the target objects and write the target objects into metadata, where the metadata is used to record the storage paths of the target objects; the writing module 54 is connected to the caching module 52 and is configured to write the target objects into one of a plurality of merging objects; the first merging module 56 is connected to the writing module 54 and is configured to write a preset data volume of target objects into the data pool when the total data volume of the target objects in the merging object reaches a first preset data volume, where the preset data volume is the maximum write data volume of the input interface of the data pool; the second merging module 58 is connected to the first merging module 56 and is configured to write the merging object into the data pool and update the metadata of all target objects in the merging object when the total data volume of the merging object reaches a second preset data volume, where the second preset data volume is greater than the first preset data volume.

[0091] Through the above device, the receiving module 50 receives a target object with a data volume less than a preset threshold; the caching module 52 caches the target object and writes the target object into metadata, where the metadata is used to record the storage path of the target object; the writing module 54 writes the target object into one of a plurality of merging objects; the first merging module 56 writes a preset data volume of target objects into the data pool when the total data volume of the target objects in the merging object reaches a first preset data volume, where the preset data volume is the maximum writing data volume of the input interface of the data pool; the second merging module 58 writes the merging object into the data pool when the total data volume of the merging object reaches a second preset data volume and updates the metadata of all the target objects in the merging object, where the second preset data volume is greater than the first preset data volume. By directly writing the target object into the merging object, writing it into the data pool when the data volume in the merging object reaches the first preset data volume, completing the merging when the merging object reaches the second preset data volume, and updating the metadata of the target object, the purpose of quickly and timely writing the target object into the data pool is achieved. It not only has a short storage time in the cache pool but also a short time in the merging object, thus achieving the technical effects of reducing the burden on the cache pool, greatly reducing the occupation of memory resources in the cache pool, and improving the efficiency of target object merging. Furthermore, it solves the technical problem in the related art that the object merging method needs to be written into the cache pool, read from the cache pool, and written into the merging object for merging, which not only occupies too many cache pool resources but also has low efficiency.

[0092] According to another aspect of the embodiments of the present invention, there is also provided a computer storage medium. The computer storage medium includes a stored program, where, when the program runs, it controls the device where the computer storage medium is located to execute the object merging method in any one of the above.

[0093] According to another aspect of the embodiments of the present invention, there is also provided a processor. The processor is used to run a program, where, when the program runs, it executes the object merging method in any one of the above.

[0094] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0095] In the above embodiments of the present invention, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0096] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0097] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0099] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks or optical discs that can store program codes.

[0100] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An object merging method, characterized in that: include: Receive a target object whose data volume is less than a preset threshold; wherein, after receiving the target object, duplicate the target object, send one copy to the cache pool, and send the other copy to the merge object for merging; caching the target object and writing the target object into metadata, wherein the metadata is used to record a storage path of the target object; Writing the target object into one of the multiple merged objects; When the total data volume of the target objects in the merged object reaches a first preset data volume, the target objects of the preset data volume are written into the data pool, wherein the preset data volume is the maximum write data volume of the input interface of the data pool; when writing, data of the first preset data volume is selected from the target objects of the total data volume for writing, and the remaining data is still stored in the merged object, waiting for the subsequent target objects to be written, and the next writing is performed when the data volume reaches the first preset data volume; When the total data volume of the merged object reaches a second preset data volume, the merged object is written into the data pool, and the metadata of all target objects of the merged object is updated, wherein the second preset data volume is greater than the first preset data volume; the total data volume of the merged object includes multiple target objects written into the merged object, metadata of all target objects, and metadata parameters of all target objects.

2. The method according to claim 1, characterized in that Writing the target object into one of the plurality of merged objects comprises: Determine a merged object in an idle state among a plurality of merged objects, wherein the merged object in the idle state can be written or can be cached and wait for writing; When the number of merged objects in the idle state is zero, suspending writing to the target object; When there is only one merged object in the idle state, writing the target object into the merged object; When there are multiple merged objects in the idle state, a merged object is randomly selected and the target object is written into the merged object.

3. The method according to claim 2, characterized in that Writing the target object into the merged object comprises: The target object is locked through the merged object to be written; Writing the locked target object into the cache space set by the merged object; If the merged object is currently writable, migrating the target object from the cache space to the merged object; Wherein, when the number of target objects in the cache space exceeds a preset number, the merged objects are in a non-cacheable waiting-to-write state.

4. The method according to claim 3, characterized in that When the total data volume of the merged object reaches a second preset data volume, writing the merged object into the data pool and updating metadata of all target objects of the merged object includes: Determining the total data volume of the merged object according to the total data volume of the target object merged in the merged object, the metadata parameter of the target object, and the attribute parameter of the merged object; When the total data volume reaches the second preset data volume, writing the metadata parameters and the attribute parameters in the merged object, as well as the remaining data of the target object, into the data pool; The metadata parameters are updated to update the metadata of the target object.

5. The method according to claim 2, characterized in that: When the total data volume of the merged object reaches a second preset data volume, after writing the merged object into the data pool and updating metadata of all target objects of the merged object, the method further includes: Creating a new merged object at the original position of the merged object; The new merged object is in a non-writable state during the creation process.

6. The method according to claim 2, characterized in that Also includes: When the total data volume of the merged objects reaches a second preset data volume, determining that the merge of the merged objects is completed; Selecting a target object that was merged last among the merged objects, and executing the completion state of the merged object; Updating the metadata of the target object, and executing the updated metadata state and the completed metadata update state of the target object; Among them, in the completion state, the metadata update state and the metadata update completion state are both writable idle states.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: After receiving the target object, creating a first merge log of the target object, wherein the first merge log is used to record the target object, and after the target object completes the merge and updates the metadata, deleting the first merge log; When the merge is re-entered after being interrupted, the first merge log is scanned to determine the target object that has not been merged, and then the merge is performed again.

8. The method according to claim 7, characterized in that Also includes: After the merged object is created, a second merge log of the merged object is created, wherein the second merge log is used to record the data storage location and merge status of the merged object, and after the merged object is successfully written into the data pool and the metadata is updated, the second merge log is deleted; In the case of re-entering after merging is interrupted, scanning the second merge log, and re-merging the target object whose metadata has been updated; The merged object corresponding to the second merged log is recovered.

9. An object merging device, characterized in that: include: A receiving module, used for receiving a target object whose data volume is less than a preset threshold; wherein, after receiving the target object, the target object is copied, one is sent to the cache pool, and the other is sent to the merge object for merging; A cache module, used to cache the target object and write the target object into metadata, wherein the metadata is used to record the storage path of the target object; A writing module, used for writing the target object into one of the multiple merged objects; A first merging module is used to write the target objects of the preset data volume into the data pool when the total data volume of the target objects in the merged objects reaches a first preset data volume, wherein the preset data volume is the maximum write data volume of the input interface of the data pool; when writing, the data of the first preset data volume is selected from the target objects of the total data volume for writing, and the remaining data is still stored in the merged object, waiting for the subsequent target objects to be written, and the next writing is performed when the data volume reaches the first preset data volume; The second merging module is used to write the merged object into the data pool and update the metadata of all target objects of the merged object when the total data volume of the merged object reaches a second preset data volume, wherein the second preset data volume is greater than the first preset data volume; the total data volume of the merged object includes multiple target objects written into the merged object, metadata of all target objects, and metadata parameters of all target objects.

10. A processor, characterized in that: The processor is used to run a program, wherein the program executes the object merging method according to any one of claims 1 to 8 when running.

Images