Small object storage method and related equipment thereof
By obtaining the offset address and storage balance in RocksDB and writing data entities into the merged object according to the data entity size, the performance problem of RocksDB when managing data of super-large scale and achieving more efficient storage performance.
Patent Information
- Application Number
- CN202510198899.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-21
- Publication Date
- 2025-07-08
AI Technical Summary
Existing RocksDBs have degraded performance when managing data of super-large-scale small objects and cannot meet business needs.
By obtaining the data entity and data entity size of the small object to be written, and obtaining the offset address and storage balance in RocksDB, writing data entities in the target merged object based on this information, avoiding writing index information and metadata, and using different storage media to improve performance.
Improves the performance of RocksDB, reduces the pressure of reading index information and metadata when reading small objects, and improves overall storage performance.
Smart Images
Figure CN120276665A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed cloud storage, and particularly to a small object storage method and related devices thereof. Background Art
[0002] Ceph distributed object storage adopts a decentralized storage architecture, and its performance can increase linearly with the number of nodes, and is widely used in cloud computing, network disks, mirror repositories, and view storage.
[0003] In the view service scenario, the scale of picture storage is generally above tens of billions. In the ultra-large-scale picture storage scenario, since the existing RocksDB stores the data entities, metadata, and index information of small objects, it is unable to manage ultra-large-scale small object data, resulting in a reduction in small object storage performance and an inability to meet business requirements. Summary of the Invention
[0004] This application provides a small object storage method and related devices thereof for improving the performance of RocksDB.
[0005] The first aspect of this application provides a small object storage method applied to a first RocksDB. The first RocksDB includes a plurality of merge objects for writing data entities of a plurality of small objects, and includes:
[0006] Obtain the data entity and data entity size of the small object to be written, and obtain a first offset address and a first storage balance in the first RocksDB. The first offset address represents the write address of the latest small object written in the first RocksDB, and the first storage balance represents the remaining writable data capacity of the first merge object to which the latest small object belongs in the first RocksDB; write the data entity into a target merge object in the first RocksDB according to the data entity size, the first offset address, and the first storage balance, where the target merge object is the first merge object or the next merge object adjacent to the first merge object.
[0007] In some embodiments, the small object storage method further includes:
[0008] Obtain the write address of the small object to be written after it is written into the first RocksDB, and obtain the metadata and index information of the small object to be written; merge the write address, the data entity size, the metadata, and the index information to obtain merged merge information, and write the merge information into a second RocksDB.
[0009] In some embodiments, writing the data entity into a target merge object of the first RocksDB according to the data entity size, the first offset address, and the first storage balance includes:
[0010] Obtaining a target bit determination result for determining the target merge object, where the target bit determination result is determined according to the data entity size and the first storage balance; in response to the target bit determination result indicating that the target merge object is the first merge object, writing the data entity at the next increment address of the first offset address of the first merge object, where the first offset address is one of multiple write addresses of the first merge object; in response to the target bit determination result indicating that the target merge object is not the first merge object, writing the data entity at the starting write address of the next merge object of the first merge object.
[0011] In some embodiments, obtaining the target bit determination result for determining the target merge object, where the target bit determination result is determined according to the data entity size and the first storage balance, includes:
[0012] In response to the data entity size being less than or equal to the first storage balance, obtaining the target bit determination result that the target merge object is the first merge object; in response to the data entity size being greater than the first storage balance, obtaining that the target merge object is not the first merge object.
[0013] In some embodiments, the method further includes a total storage amount of merge objects. After responding that the target bit determination result indicates that the target merge object is not the first merge object and before writing the data entity at the starting write address of the next merge object of the first merge object, it includes:
[0014] In the first RocksDB, creating the next merge object according to the ending write address of the first merge object and the total storage amount of merge objects.
[0015] In some embodiments, creating the next merge object according to the ending write address of the first merge object and the total storage amount of merge objects includes:
[0016] Obtaining the next write address adjacent to and incrementing from the ending write address of the first merge object, and then determining the next write address as the starting write address of the next merge object; starting from the starting write address of the next merge object, incrementing by an address step corresponding to the total storage amount of merge objects to obtain the ending write address of the next merge object, and then creating the next merge object including multiple write addresses.
[0017] In some embodiments, the total storage amount of the merged objects can be adjusted.
[0018] In some embodiments, the first RocksDB is applied to an HDD storage medium, and the second RocksDB is applied to an SDD storage medium.
[0019] The second aspect of the present application provides an electronic device, which includes a memory and a processor coupled to each other. The processor is configured to execute program instructions stored in the memory to implement the small object storage method of the first aspect described above.
[0020] The third aspect of the present application provides a computer-readable storage medium, which stores program instructions therein. The program instructions are executed by a processor to implement the small object storage method of the first aspect described above.
[0021] Compared with the prior art, the technical solution of the present application obtains the data entity and data entity size of the small object to be written, and obtains the first offset address and the first storage balance in the first RocksDB, and then writes the data entity into the target merged object in the first RocksDB according to the data entity size, the first offset address, and the first storage balance. Among them, the first offset address represents the write address of the latest small object written in the first RocksDB, the first storage balance represents the remaining writable data capacity of the first merged object to which the latest small object belongs in the first RocksDB, and the target merged object is the first merged object or the next merged object adjacent to the first merged object. Compared with the existing method of storing small objects in RocksDB, by the above means, the data entity of the small object to be stored is written into the first RocksDB without writing index information and metadata in RocksDB. Then, when reading later, only the data entity of the small object needs to be read in RocksDB, avoiding the need to additionally read its index information and metadata when reading and writing small objects in the existing RocksDB, and improving the performance of RocksDB. Description of the Drawings
[0022] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without any creative effort, other drawings can also be obtained based on these drawings.
[0023] Figure 1 It is a schematic flowchart of the small object storage method according to an embodiment of the present application;
[0024] Figure 2Schematic flowchart of a small object storage method according to an embodiment of the present application;
[0025] Figure 3 Schematic flowchart of writing a data entity of a small object to be written based on a target position determination result according to an embodiment of the present application;
[0026] Figure 4 Schematic flowchart of determining a target position determination result according to an embodiment of the present application;
[0027] Figure 5 Schematic flowchart of creating a merged object according to an embodiment of the present application;
[0028] Figure 6a Schematic structural diagram of a small object storage device according to an embodiment of the present application;
[0029] Figure 6b Schematic structural diagram of a small object storage device according to an embodiment of the present application;
[0030] Figure 7 Schematic structural diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners
[0031] The solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings of the specification.
[0032] In the following description, specific details such as specific system architectures, interfaces, and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the present application.
[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.
[0034] The terms "first", "second", and "third" in this application are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", and "third" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship and movement conditions between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.
[0035] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.
[0036] Please refer to Figure 1 , Figure 1 which shows a schematic flow diagram of a small object storage method applied to a first RocksDB, specifically including:
[0037] 101: Obtain the data entity and data entity size of the small object to be written, and obtain a first offset address and a first storage balance in the first RocksDB, where the first offset address represents the write address of the latest small object written in the first RocksDB, and the first storage balance represents the remaining writable data capacity of the first merge object to which the latest small object belongs in the first RocksDB;
[0038] Obtain the data entity of the small object to be written and the data entity size of the small object to be written, and obtain the write address of the latest small object written in the first RocksDB, and the first storage balance representing the remaining writable data capacity of the first merge object corresponding to the latest small object.
[0039] It should be noted that the merged object represents a relatively independent storage space in the first RocksDB, which is used to store the data entities of multiple small objects respectively. Therefore, when the merged object stores a sufficient number of small objects, the multiple small objects as a whole can be regarded as a merged object. Thus, the first RocksDB includes multiple merged objects, and each merged object is used to store the respective data entities of multiple small objects.
[0040] The merged object has a preset total storage capacity, that is, the total storage capacity of the merged object, and the storage balance representing the remaining storage capacity decreases during the process of writing multiple small objects into the merged object.
[0041] The merged object includes multiple continuously increasing write addresses, and the write addresses are used to indicate the positions where the data entities of the small objects are written.
[0042] The composition of the small object includes a data entity, metadata including small object attribute information, and index information. Among them, the index information is used to locate the data entity; the metadata is used to record the attribute information of the data entity, such as the size, creation time, version, and / or checksum of the data entity, etc., which are not specifically limited here; the data entity represents streaming unstructured business data, where streaming unstructured refers to real-time data streams without a fixed structure pattern. For example, videos, voices, and pictures can be used as streaming unstructured business data, which are not specifically limited here.
[0043] The data entity size represents the data size of the data entity of the small object, which is different from the overall data size of the small object.
[0044] It can be understood that when writing a small object into the first RocksDB, the small object is the small object to be written, which has a corresponding data entity and data entity size, as well as a first offset address and a first storage balance.
[0045] 102: Write the data entity into the target merged object in the first RocksDB according to the data entity size, the first offset address, and the first storage balance, where the target merged object is the first merged object or the next merged object adjacent to the first merged object.
[0046] After obtaining the data entity size of the small object to be written and the first offset address and the first storage balance returned by the first RocksDB, write the data entity of the small object to be written into the target merged object in the first RocksDB according to the data entity size, the first offset address, and the first storage balance.
[0047] Since the merge object has a total storage capacity and can only write multiple small objects that match the total storage capacity, the target merge objects where the small objects to be written can be written include the first merge object to which the latest small object that has been written belongs or the next merge object adjacent to the first merge object.
[0048] It can be understood that in this application, by writing the data entity of the small object to be written into the first RocksDB without writing additional metadata and index information therein, the performance pressure of RocksDB caused by the resource allocation structure that the existing RocksDB needs to allocate write / read data entities, metadata, and index information simultaneously and the write / read of oversized small objects is reduced, and the performance of RocksDB is improved.
[0049] Please refer to Figure 2 , Figure 2 which shows writing the write address, metadata, and index information of the small object to be written into the second RocksDB, specifically including:
[0050] 201: Obtain the write address after the small object to be written is written into the first RocksDB, and obtain the metadata and index information of the small object to be written;
[0051] Obtain the write address after the small object to be written is written into the target merge object of the first RocksDB, and obtain the metadata and index information of the small object to be written.
[0052] 202: Merge the write address and the data entity size, and the metadata and the index information to obtain the merged merge information, and write the merged information into the second RocksDB.
[0053] Merge the obtained write address, data entity size, metadata, and index information to obtain the merged information, and then write the merged information into the second RocksDB.
[0054] It can be understood that compared with the existing RocksDB storage method, in this application, the data entity of the small object to be written is written into the first RocksDB, and the merged information obtained by merging the data entity size, metadata, and index information of the small object to be written and the write address written into the first RocksDB is written into the second RocksDB, completing the storage of the small object to be written. That is, by writing the data of the small object into two RocksDBs, each RocksDB is responsible for processing the corresponding data of the small object, improving the performance of RocksDB.
[0055] In some embodiments, to further improve the overall write and read performance of two RocksDBs, the first RocksDB for writing the data entity of the to-be-written small object is applied to the first storage medium, and the second RocksDB for writing the merge information of the to-be-written small object is applied to the second storage medium, where the performance of the first storage medium is lower than that of the second storage medium.
[0056] Preferably, the first storage medium is an HDD storage medium, and the second storage medium is an SDD storage medium.
[0057] Please refer to Figure 3 , Figure 3 which shows writing the data entity of the to-be-written small object based on the target bit determination result, where the target bit determination result is used to determine the target merge object for writing the data entity of the to-be-written small object in the first RocksDB, specifically including:
[0058] 301: Obtain the target bit determination result for determining the target merge object, where the target bit determination result is determined according to the data entity size and the first storage balance;
[0059] Obtain the target bit determination result, where the target bit determination result is determined according to the data entity size of the to-be-written small object and the first storage balance.
[0060] The target bit determination result includes that the target merge object is the first merge object, or the target merge object is not the first merge object. Wherein, the target merge object not being the first merge object may also be the next merge object of the target merge object being the first merge object.
[0061] 302: In response to the target bit determination result that the target merge object is the first merge object, write the data entity at the next increment address of the first offset address of the first merge object, where the first offset address is one of the multiple write addresses of the first merge object;
[0062] In response to the target bit determination result that the target merge object is the first merge object, increment the first offset address of the first merge object to obtain the next increment address (i.e., the target write address), and then write the data entity at the target write address in the first merge object.
[0063] It can be understood that the first offset address indicating the write address of the latest small object that has been written, and the target write address indicating the write address of the to-be-written small object are adjacent in ascending order and belong to the multiple write addresses of the first merge object.
[0064] 303: If the target bit determination result is that the target merging object is not the first merging object, write the data entity at the starting write address of the next merging object of the first merging object.
[0065] If the target bit determination result is that the target merging object is not the first merging object, determine the next merging object of the first merging object, then determine the starting write address of the next merging object, and finally write the data entity of the small object to be written at the starting write address of the next merging object.
[0066] Please refer to Figure 4 , Figure 4 which shows the process of determining the target bit determination result. Among them, the specific target bit determination result is obtained by comparing the size of the data entity of the small object to be written with the first storage balance of the first merging object to which the latest written small object belongs. Specifically, it includes:
[0067] 401: If the size of the data entity is less than or equal to the first storage balance, obtain the target bit determination result that the target merging object is the first merging object;
[0068] If the size of the data entity of the small object to be written is less than or equal to the first storage balance of the first merging object, obtain the target bit determination result that the generated target merging object is the first merging object.
[0069] 402: If the size of the data entity is greater than the first storage balance, obtain that the target merging object is not the first merging object.
[0070] If the size of the data entity of the small object to be written is greater than the first storage balance of the first merging object, obtain that the generated target merging object is not the first merging object, that is, the target merging object is the next merging object of the first merging object.
[0071] In some embodiments, when the first storage balance of the first merging object cannot store the data entity of the small object to be written, the next merging object adjacent to the first merging object can be created according to the ending write address of the first merging object and the total storage amount of the merging object, so as to write the data entity of the small object to be written into the created next merging object. Specifically, please refer to Figure 5 , specifically including:
[0072] 501: Obtain the next write address adjacent to and incrementing from the ending write address of the first merging object, and then determine the next write address as the starting write address of the next merging object;
[0073] Obtain the end write address of the first merge object in the first RocksDB, and then determine the next write address adjacent to and incrementing from the end write address, so as to determine it as the start write address of the next merge object adjacent to and incrementing from the first merge object.
[0074] 502: Starting from the start write address of the next merge object, incrementally increase the address step corresponding to the total storage amount of the merge object to obtain the end write address of the next merge object, and then create the next merge object including multiple write addresses.
[0075] Determine the start write address of the next merge object, and determine the address step corresponding to the total storage amount of the merge object. Then, starting from the start write address, incrementally increase the address step to obtain the end write address of the next merge object, so as to create in the first RocksDB the next merge object including multiple write addresses starting from the start write address and ending at the end write address.
[0076] Please refer to Figure 6a , Figure 6a which shows a small object storage device, including:
[0077] The first acquisition module 601 is used to acquire the data entity and data entity size of the small object to be written, and acquire the first offset address and the first storage balance in the first RocksDB, where the first offset address represents the write address of the latest small object written in the first RocksDB, and the first storage balance represents the remaining writable data capacity of the first merge object to which the latest small object belongs in the first RocksDB;
[0078] The first write module 602 is used to write the data entity into the target merge object in the first RocksDB according to the data entity size, the first offset address, and the first storage balance, where the target merge object is the first merge object or the next merge object adjacent to the first merge object.
[0079] Please refer to Figure 6b , Figure 6b which shows another structural schematic diagram of a small object storage device. In addition to including the first acquisition module 601 and the first write module 602, it further includes:
[0080] The second acquisition module 603 is used to acquire the write address of the small object to be written after it is written into the first RocksDB, and acquire the metadata and index information of the small object to be written.
[0081] A merging module 604, configured to merge the write address, the data entity size, the metadata, and the index information to obtain merged information, and write the merged information into a second RocksDB.
[0082] In some embodiments, Figure 6a or / and Figure 6b It further includes a plurality of merging object modules, configured to serve as write recipients of the first write module 602.
[0083] This application further includes an electronic device. Please refer to Figure 7 including: a memory 701 and a processor 702; wherein, the memory is used to store programs; the processor is used to execute the programs in the memory, including executing the methods of any of the foregoing Figures 1 to 5 embodiment methods;
[0084] Optionally, the electronic device further includes a bus system for connecting the memory and the processor, so that the memory and the processor can communicate.
[0085] The processor may be a Central Processing Unit (CPU), and the processor may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0086] In some embodiments, the memory may be an internal storage unit of a cloud device, such as the hard disk or memory of a cloud device. In other embodiments, the memory may also be an external storage device of a cloud device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the cloud device. Further, the memory may also include both the internal storage unit and the external storage device of the cloud device. The memory is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of a computer program. The memory may also be used to temporarily store data that has been output or will be output.
[0087] This application also includes a computer-readable storage medium that stores program instructions internally, and the program instructions are executed by a processor to implement the foregoing Figures 1 to 5 method of any one of the embodiments.
[0088] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented.
[0089] Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc.
[0090] The above are only the embodiments of this application, and do not limit the patent protection scope of this application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of this application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of this application.
Claims
1. A small object storage method, characterized in that, Applied to the first RocksDB, the first RocksDB includes a plurality of merge objects for writing data entities of a plurality of small objects, including: Obtain the data entity and data entity size of the small object to be written, and obtain the first offset address and the first storage balance in the first RocksDB, where the first offset address represents the write address of the latest small object written in the first RocksDB, and the first storage balance represents the remaining writable data capacity of the first merge object to which the latest small object belongs in the first RocksDB; Write the data entity in the target merge object of the first RocksDB according to the data entity size, the first offset address, and the first storage balance, where the target merge object is the first merge object or the next merge object adjacent to the first merge object.
2. The small object storage method according to claim 1, wherein The small object storage method further includes: Obtain the write address after the small object to be written is written into the first RocksDB, and obtain the metadata and index information of the small object to be written; Merge the write address, the data entity size, the metadata, and the index information to obtain merged merge information, and write the merge information into the second RocksDB.
3. The small object storage method according to claim 1 or 2, wherein The writing the data entity in the target merge object of the first RocksDB according to the data entity size, the first offset address, and the first storage balance includes: Obtain a target bit determination result for determining the target merge object, where the target bit determination result is determined according to the data entity size and the first storage balance; In response to the target bit determination result that the target merge object is the first merge object, write the data entity at the next increment address of the first offset address of the first merge object, where the first offset address is one of the multiple write addresses of the first merge object; In response to the target bit determination result that the target merge object is not the first merge object, write the data entity at the starting write address of the next merge object of the first merge object.
4. The small object storage method according to claim 3, wherein The obtaining the target bit determination result for determining the target merge object, where the target bit determination result is determined according to the data entity size and the first storage balance, includes: In response to the data entity size being less than or equal to the first storage balance, obtain the target bit determination result that the target merge object is the first merge object; In response to the data entity size being greater than the first storage balance, obtain that the target merge object is not the first merge object.
5. The small object storage method according to claim 1 or 2, characterized in that, The method further includes a merge object storage total for indicating the total storage amount of the merge object. After the response that the target bit determination result is that the target merge object is not the first merge object and before writing the data entity at the starting write address of the next merge object of the first merge object, it includes: In the first RocksDB, the next merge object is created according to the end write address of the first merge object and the total storage amount of the merge object.
6. The small object storage method according to claim 5, wherein The creating of the next merge object according to the end write address of the first merge object and the total storage amount of the merge object includes: Obtaining a next write address adjacent to and incrementing from the end write address of the first merge object, and then determining the next write address as the start write address of the next merge object; Taking the start write address of the next merge object as a starting point, incrementally increasing an address step corresponding to the total storage amount of the merge object to obtain the end write address of the next merge object, and then creating the next merge object including multiple write addresses.
7. The small object storage method according to claim 5, wherein The total storage amount of the merge object is adjustable.
8. The small object storage method according to claim 2, wherein The first RocksDB is applied to an HDD storage medium, and the second RocksDB is applied to an SDD storage medium.
9. An electronic device, characterized in that, The electronic device includes: a memory and a processor coupled to each other, and the processor is configured to execute program instructions stored in the memory to implement the small object storage method according to any one of claims 1 to 8.
10. A computer-readable storage medium, characterized in that, It internally stores program instructions, and the program instructions are executed by the processor to implement the small object storage method according to any one of claims 1 to 8.