A method, device, equipment and medium for optimizing a polymeric large object write process

Abstract

The application discloses a method and device for optimizing a polymeric large object write process, equipment and medium, and relates to the technical field of distributed storage. The method comprises the following steps: receiving an instruction for modifying and writing a target small object; obtaining the target small object in a polymeric large object in a mechanical hard disk data pool according to the instruction, and modifying and writing the target small object; obtaining the size relationship before and after the modification and writing of the target small object; and placing the target small object after the modification and writing into the original position of the target small object in the polymeric large object or after the polymeric large object strip according to the size relationship. Therefore, the size relationship before and after the modification and writing of the target small object in the polymeric large object is used to place the modified target small object back to the original position or to the polymeric large object strip, so that the number of times of creating a new space of the polymeric large object is reduced, the invalid utilization rate of the storage space is reduced, and the storage efficiency of the mechanical hard disk data pool is improved.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a method, apparatus, device and medium for optimizing the write process of aggregated large objects. Background Technology

[0002] In distributed storage, to improve the reconstruction speed of massive amounts of small object data (data less than or equal to 512KB), object storage uses a "caching + aggregation" scheme instead of the original direct storage method. Specifically, small objects are preferentially written to a solid-state drive (SSD) cache pool; when the SSD cache pool reaches a certain level, these small objects are aggregated into large objects and flushed to the hard disk drive (HDD) data pool. By adopting this cache aggregation approach, the number of Reliable, Autonomic Distributed Object Store (RADOS) objects in the cluster is significantly reduced, thereby improving data reconstruction speed.

[0003] Currently, when modifying small objects within an aggregate object, the existing mechanism first removes the small object to be modified from the aggregate object, then invalidates the space where the original small object resides, and creates a new space after the modified object to record the modified object information. However, when the invalidated space is insufficient to trigger garbage collection by the HDD data pool's aggregate garbage management module, a large amount of invalidated space remains in the HDD data pool, resulting in a waste of storage space.

[0004] Given the above issues, optimizing the write process of aggregated large objects and avoiding excessive invalidation space in aggregated large objects that leads to wasted storage space is a problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device, and medium for optimizing the write process of aggregated large objects, so as to optimize the write process of aggregated large objects and avoid excessive invalidation space in aggregated large objects, which would lead to a waste of storage space.

[0006] To address the aforementioned technical problems, this application provides a method for optimizing the write process of aggregated large objects, including:

[0007] Receive instructions to modify and write a target small object; wherein the size of the target small object is no greater than 512K;

[0008] According to the instructions, the target small object is obtained from the aggregated large object in the mechanical hard disk data pool, and the target small object is modified and written; wherein, the aggregated large object contains multiple small objects;

[0009] Obtain the size relationship of the target small object before and after the modification;

[0010] Based on the size relationship, the modified target small object is placed in the original position of the target small object in the aggregated large object or after the strip of the aggregated large object.

[0011] Preferably, the step of placing the modified target small object into the original position of the target small object in the aggregated large object or into the strip of the aggregated large object according to the size relationship includes:

[0012] When the size of the target small object after modification is smaller than the size of the target small object before modification, data is filled into the target small object after modification until the size of the target small object is equal to the size of the target small object before modification. Then, the target small object after data filling is placed into the original position of the target small object in the aggregated large object.

[0013] When the size of the target small object after modification is equal to the size of the target small object before modification, the target small object after modification is directly placed into the original position of the target small object in the aggregated large object;

[0014] When the size of the target small object after modification is greater than the size of the target small object before modification, the space of the original position of the target small object in the aggregated large object is invalidated, and space is created after the original strip of the aggregated large object to put the target small object after modification.

[0015] Preferably, before modifying the target small object, the method further includes:

[0016] Create a temporary space to store the modified target small object.

[0017] Preferably, the step of filling the modified target small object with data includes:

[0018] Invalid data is filled into the modified target small object within the temporary space.

[0019] Preferably, before retrieving the target small object from the aggregated large object in the hard disk data pool according to the instruction, and after receiving the instruction to modify and write the target small object, the method further includes:

[0020] Determine whether the target small object is aggregated into the aggregate large object;

[0021] If so, proceed to the step of obtaining the target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction;

[0022] If not, then the target small object is directly modified and written in the solid-state drive cache pool according to the instructions.

[0023] Preferably, it further includes:

[0024] The remaining space of the hard disk data pool is obtained according to a preset period;

[0025] When the remaining space does not meet the preset threshold, the invalid space in the mechanical hard disk data pool is reclaimed.

[0026] Preferably, after reclaiming the invalid space in the hard disk data pool, the method further includes:

[0027] Output the current remaining space of the mechanical hard disk data pool.

[0028] To address the aforementioned technical problems, this application also provides an apparatus for optimizing the write process of aggregated large objects, comprising:

[0029] A receiving module is used to receive instructions to modify or write a target small object; wherein the size of the target small object is no greater than 512K;

[0030] The first acquisition module is used to acquire the target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction, and modify and write the target small object; wherein, the aggregated large object contains multiple small objects;

[0031] The second acquisition module is used to acquire the size relationship of the target small object before and after modification.

[0032] The object storage module is used to place the modified target small object into the original position of the target small object in the aggregated large object or after the strip of the aggregated large object according to the size relationship.

[0033] To address the aforementioned technical problems, this application also provides an optimization device for aggregated large object write processes, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor is used to implement the steps of the above-described method for optimizing the aggregated large object write process when executing the computer program.

[0036] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for optimizing the write process of aggregated large objects.

[0037] The method for optimizing the write process of aggregated large objects provided in this application involves receiving an instruction to modify and write a target small object, wherein the size of the target small object is no greater than 512K; retrieving the target small object from the aggregated large object in the hard disk data pool according to the instruction, and modifying and writing the target small object; wherein the aggregated large object contains multiple small objects; obtaining the size relationship of the target small object before and after the modification and write; and placing the modified target small object into its original position within the aggregated large object or after the aggregated large object stripe according to the size relationship. Therefore, the above solution reduces the number of times the aggregated large object creates new space and lowers the ineffective utilization rate of storage space by adding a judgment and comparison mechanism to the write process of aggregated large objects. Specifically, based on the size relationship of the target small object before and after the modification and write in the aggregated large object, it chooses to put the modified target small object back in its original position or append it to the aggregated large object stripe, thereby improving the storage efficiency of the hard disk data pool.

[0038] In addition, this application also provides an apparatus, device and medium for optimizing the writing process of aggregated large objects, with the same effect as above. Attached Figure Description

[0039] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A schematic diagram of the object aggregation process provided in the embodiments of this application;

[0041] Figure 2 A flowchart illustrating an optimization method for writing aggregated large objects, provided in an embodiment of this application;

[0042] Figure 3 A schematic diagram of an apparatus for optimizing the write process of aggregated large objects provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of an aggregated large object write process optimization device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0045] The core of this application is to provide a method, apparatus, device, and medium for optimizing the write process of aggregated large objects, so as to optimize the write process of aggregated large objects and avoid excessive invalidation space in aggregated large objects, which would lead to a waste of storage space.

[0046] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] With the rapid development of the internet industry, the world has entered the era of big data. In this era, various types of information need to be stored, such as identity information, educational information, and other data. To better record and manage this information, the system responsible for information storage must have massive storage capabilities. In distributed storage clusters, data reconstruction due to disk failures is performed at the object level. Data reconstruction is primarily time-consuming during disk input / output (IN / OUT, IO), and reconstructing massive amounts of object data takes even longer. Therefore, the time spent on data reconstruction poses a significant risk to the reliability of distributed storage clusters.

[0048] To improve the speed of reconstructing massive amounts of small object data, object storage uses a "caching + aggregation" approach to replace the original direct storage method. Figure 1 This is a schematic diagram illustrating the object aggregation process provided in an embodiment of this application. Figure 1 As shown in the manual, users deploy an object storage cluster and enable cache aggregation. Small objects (data size no larger than 512K) uploaded via the cosbench or S3browser tools are preferentially written to the hot pool of the SSD cache pool. Figure 1 The metadata of Obj1, Obj2, and Obj3 in the SSD cache pool is written to the SSD cache pool's metadata pool (Omap). When the SSD cache pool's storage capacity reaches the user-defined watermark (i.e., storage threshold), the data in the SSD cache pool will be migrated to the HDD data pool. During the migration process, the cluster aggregates multiple small objects into a single large aggregate object (Omap). Figure 1 Migration is performed within the Agg-Obj aggregate to reduce the number of underlying objects, thereby improving data reconstruction performance and disk utilization. It's important to note that each small object has specific attributes within its parent aggregate, such as offset, length, and the aggregate's object identifier (oid). These attributes determine the small object's position within the aggregate, facilitating subsequent access to the corresponding small object within the aggregate.

[0049] Currently, when modifying small objects within a large aggregate object, the small object to be modified is first removed from the aggregate object, then the space where the original small object resides is invalidated, and a new space is created after the object modification to record the modified object information. However, when the invalidated space is insufficient to trigger garbage collection by the HDD data pool's aggregate garbage management module, a large amount of invalidated space exists in the HDD data pool, resulting in wasted storage space. Therefore, this application provides an optimization method for writing large aggregate objects to optimize the writing process of large aggregate objects and avoid excessive invalidation space in the large aggregate object, thus avoiding wasted storage space. It is understood that the application scenario of the optimization method for writing large aggregate objects provided in this application is a distributed object storage cluster. It should be noted that this application does not limit the application scenario of the optimization method for writing large aggregate objects; in addition to distributed object storage clusters, it can also be applied to other application scenarios involving aggregated object storage, depending on the specific implementation.

[0050] Figure 2 This is a flowchart of an optimization method for writing aggregated large objects, provided in an embodiment of this application.

[0051] like Figure 2 As shown, the optimization methods for writing large aggregated objects include:

[0052] S10: Receive instructions to modify or write the target small object.

[0053] The size of the target small object is no more than 512K.

[0054] S11: According to the instruction, retrieve the target small object from the aggregated large object in the mechanical hard disk data pool, and modify and write the target small object.

[0055] The aggregated large object contains multiple smaller objects.

[0056] Specifically, based on a distributed object storage cluster, instructions to modify and write target small objects are received. The target small object resides within an aggregated large object in a hard disk data pool, which contains multiple aggregated large objects. It can be understood that the size of the target small object is no greater than 512KB, and the size of the aggregated large object is greater than 512KB; for example, multiple small objects are aggregated into a 4MB aggregated large object.

[0057] Furthermore, based on the offset and length of the target small object and the object identifier (oid) of the aggregated large object, the target small object is obtained from the aggregated large object, and the target small object is modified and written according to the instructions.

[0058] It should be noted that this embodiment does not restrict the specific content that can be modified in the target small object; it depends on the specific implementation.

[0059] S12: Get the size relationship of the target small object before and after modification.

[0060] Furthermore, after modifying the target small object, the size relationship between the target small object before and after the modification is obtained. It can be understood that there are three possible size relationships: the size of the target small object after modification is less than the size of the target small object before modification, the size of the target small object after modification is equal to the size of the target small object before modification, and the size of the target small object after modification is greater than the size of the target small object before modification.

[0061] S13: Based on the size relationship, place the modified target small object into the original position of the target small object in the aggregate large object or after the strip of the aggregate large object.

[0062] Finally, based on the size relationship of the target small object before and after the modification, the modified target small object is placed in its original position within the aggregated large object or after a stripe of the aggregated large object. It is important to note that within the aggregated large object, the small objects are arranged in stripes. When a new small object is added to the aggregated large object, new space needs to be created after the aggregated large object's stripes to store the new small object. In this embodiment, the specific method of placing the modified target small object in its original position within the aggregated large object or after a stripe of the aggregated large object is not limited and depends on the specific implementation.

[0063] In this embodiment, an instruction to modify and write a target small object is received; wherein the size of the target small object is no greater than 512K; according to the instruction, the target small object is obtained from the aggregated large object in the mechanical hard disk data pool, and the target small object is modified and written; wherein the aggregated large object contains multiple small objects; the size relationship of the target small object before and after the modification and write is obtained; according to the size relationship, the modified target small object is placed in the original position of the target small object in the aggregated large object or appended to the aggregated large object stripe. Therefore, the above scheme reduces the number of times the aggregated large object creates new space and reduces the ineffective utilization rate of storage space by adding a judgment and comparison mechanism to the aggregated large object write process. Specifically, based on the size relationship of the target small object before and after the modification and write in the aggregated large object, it chooses to put the modified target small object back in its original position or append it to the aggregated large object stripe, thereby improving the storage efficiency of the mechanical hard disk data pool.

[0064] Based on the above embodiments, as a preferred embodiment, placing the modified target small object into the original position of the target small object in the aggregated large object or into the aggregated large object strip according to the size relationship includes:

[0065] When the size of the target small object after modification is smaller than the size of the target small object before modification, fill the target small object with data until the size of the target small object is equal to the size of the target small object before modification, and then put the target small object after data filling into the original position of the target small object in the aggregate large object.

[0066] When the size of the target small object after modification is equal to the size of the target small object before modification, the target small object after modification is directly placed into the original position of the target small object in the aggregate large object.

[0067] When the size of the target small object after modification is larger than the size of the target small object before modification, the space of the original position of the target small object in the aggregate large object is invalidated, and space is created after the original strip of the aggregate large object to put the target small object after modification.

[0068] In practice, after retrieving the target small object from the aggregated large object and modifying it, if the size of the modified target small object is smaller than the original target small object, data needs to be padded to the modified target small object until its size equals the original target small object's size. Finally, the modified target small object's content overwrites the original target small object's space within the aggregated large object. This embodiment does not restrict the specific data padded to the target small object; it can be data related to the target small object or invalid data, depending on the specific implementation.

[0069] If the size of the modified target small object is equal to the size of the original target small object, no padding is needed; the modified target small object is simply overwritten in the original location, and the space occupied by the original target small object does not need to be invalidated. If the size of the modified target small object is larger than the original target small object, the space occupied by the original target small object is insufficient, so the space information of the original target small object needs to be invalidated, and a new space is created after the aggregated large object stripe to store the information of the modified target small object. Understandably, the aggregated large object after the above process can still perform normal upload, download, copy, and delete operations, and object access control lists (ACLs) and object append operations can still be used normally.

[0070] This optimizes the write process under different modification scenarios, reduces the number of write operations, reduces the ineffective utilization of space, and further improves storage performance.

[0071] To better facilitate the modification and writing of the target small object, based on the above embodiments, as a preferred embodiment, before modifying and writing the target small object, the following is also included:

[0072] Create a temporary space to store the modified target small object.

[0073] Specifically, to better facilitate modifications to the target object, a temporary space can be created first. This temporary space stores the modified target object, preventing situations where the modified target object cannot be saved. It should be noted that this embodiment does not limit the size of the temporary space; it depends on the specific implementation.

[0074] Furthermore, as a preferred embodiment, if the size of the modified target small object is smaller than the size of the original target small object, and data needs to be padded to the modified target small object, invalid data can be padded to the modified target small object in a temporary space. That is, invalid data is padded to the insufficient part in the temporary space, thereby covering the original space of the target small object in the aggregated large object with the padded target small object. In this way, the storage efficiency of the storage space is improved.

[0075] Based on the above embodiments, as a preferred embodiment, before retrieving the target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction, after receiving the instruction to modify and write the target small object, the method further includes:

[0076] Determine whether the target small object is aggregated into an aggregate large object;

[0077] If so, proceed to the step of retrieving the target small object from the aggregated large object in the mechanical hard disk data pool according to the instructions;

[0078] If not, the target small object will be modified and written directly in the solid-state drive cache pool according to the instructions.

[0079] In the above embodiments, the modification write to the target small object occurs in the mechanical hard disk data pool storing the aggregated large object. However, in specific implementations, the target small object is not aggregated in the solid-state drive cache pool before migration. To better implement the modification write to the target small object, as a preferred embodiment, in this embodiment, before retrieving the target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction, after receiving the instruction to modify the target small object, it is necessary to determine whether the target small object has been aggregated into an aggregated large object; if so, the step of retrieving the target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction is entered, and the above-mentioned modification write and object content overwriting operations are performed. If not, the target small object is directly modified and written in the solid-state drive cache pool according to the instruction to facilitate the subsequent aggregation of the target small object.

[0080] To better monitor the storage space of the hard disk drive data pool, in addition to the above embodiments, as a preferred embodiment, the following is also included:

[0081] The remaining space in the hard drive data pool is obtained according to a preset cycle;

[0082] When the remaining space does not meet the preset threshold, the invalid space in the mechanical hard drive data pool is reclaimed.

[0083] In practice, the remaining space of the hard disk drive data pool is obtained according to a preset period. This embodiment does not limit the preset period; it depends on the specific implementation. Furthermore, when the remaining space does not meet a preset threshold, invalid space in the hard disk drive data pool is reclaimed to facilitate the storage of other large aggregated objects or other data. It should be noted that this embodiment does not limit the preset threshold; it depends on the specific implementation.

[0084] In addition, as a preferred embodiment, after reclaiming invalid space in the hard disk drive data pool, the current remaining space of the hard disk drive data pool can be output so that users can monitor the current storage status of the hard disk drive data pool in real time and make decisions on subsequent data storage.

[0085] In the above embodiments, the method for optimizing the write process of aggregated large objects has been described in detail. This application also provides embodiments of the apparatus for optimizing the write process of aggregated large objects.

[0086] Figure 3 This is a schematic diagram of an apparatus for optimizing the write process of aggregated large objects, provided in an embodiment of this application.

[0087] like Figure 3 As shown, the aggregation large object write process optimization device includes:

[0088] The receiving module 10 is used to receive instructions to modify or write the target small object; wherein the size of the target small object is no greater than 512K.

[0089] The first acquisition module 11 is used to acquire a target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction, and modify and write the target small object; wherein, the aggregated large object contains multiple small objects.

[0090] The second acquisition module 12 is used to acquire the size relationship of the target small object before and after modification.

[0091] The object storage module 13 is used to place the modified target small object into the original position of the target small object in the aggregated large object or after the strip of the aggregated large object according to the size relationship.

[0092] In a preferred embodiment, the object storage module is specifically used for:

[0093] When the size of the target small object after modification is smaller than the size of the target small object before modification, fill the target small object with data until the size of the target small object is equal to the size of the target small object before modification, and then put the target small object after data filling into the original position of the target small object in the aggregate large object.

[0094] When the size of the target small object after modification is equal to the size of the target small object before modification, the target small object after modification is directly placed into the original position of the target small object in the aggregate large object.

[0095] When the size of the target small object after modification is larger than the size of the target small object before modification, the space of the original position of the target small object in the aggregate large object is invalidated, and space is created after the original strip of the aggregate large object to put the target small object after modification.

[0096] As a preferred embodiment, it also includes:

[0097] The space creation module is used to create temporary spaces to store modified target small objects.

[0098] In one preferred embodiment, the object storage module specifically fills the modified target small object with invalid data in the temporary space.

[0099] As a preferred embodiment, it also includes:

[0100] The judgment module is used to determine whether the target small object has been aggregated into an aggregated large object before obtaining the target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction, after receiving the instruction to modify and write the target small object; if so, the first acquisition module is triggered.

[0101] If not, the modification-write module is triggered; the modification-write module is used to directly modify and write the target small object in the solid-state drive cache pool according to the instructions.

[0102] As a preferred embodiment, it also includes:

[0103] The third acquisition module is used to acquire the remaining space of the mechanical hard disk data pool according to a preset period.

[0104] The space reclamation module is used to reclaim invalid space in the hard disk data pool when the remaining space does not meet a preset threshold.

[0105] As a preferred embodiment, it also includes:

[0106] The generation module is used to output the current remaining space of the hard disk data pool after reclaiming invalid space in the hard disk data pool.

[0107] In this embodiment, the aggregated large object write process optimization device includes a receiving module, a first acquisition module, a second acquisition module, and an object storage module. The aggregated large object write process optimization device can implement all the steps of the above-described aggregated large object write process optimization method during operation. It receives an instruction to modify and write a target small object; wherein the size of the target small object is no greater than 512K; according to the instruction, it acquires the target small object from the aggregated large object in the hard disk data pool and modifies and writes the target small object; wherein the aggregated large object contains multiple small objects; it acquires the size relationship of the target small object before and after the modification and write; and according to the size relationship, it places the modified target small object back into its original position within the aggregated large object or appends it to the aggregated large object stripe. Therefore, the above solution reduces the number of times the aggregated large object creates new space and lowers the ineffective utilization rate of storage space by adding a judgment and comparison mechanism to the aggregated large object write process. Specifically, based on the size relationship of the target small object before and after the modification and write in the aggregated large object, it selects whether to put the modified target small object back in its original position or append it to the aggregated large object stripe, thereby improving the storage efficiency of the hard disk data pool.

[0108] Figure 4 This is a schematic diagram of an aggregated large object write process optimization device provided in an embodiment of this application.

[0109] like Figure 4 As shown, the device for optimizing the large object write process includes:

[0110] Memory 20 is used to store computer programs;

[0111] Processor 21 is used to implement the steps of the method for optimizing the aggregated large object write process as mentioned in the above embodiments when executing a computer program.

[0112] The device for optimizing the aggregated large object write process provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0113] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0114] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the aggregated large object write process optimization method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the aggregated large object write process optimization method.

[0115] In some embodiments, the aggregated large object write process optimization device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0116] Those skilled in the art will understand that Figure 4The structure shown does not constitute a limitation on the device for optimizing the aggregate large object write process and may include more or fewer components than shown.

[0117] In this embodiment, the aggregated large object write process optimization device includes a memory and a processor. The memory stores a computer program; the processor executes the computer program to implement the steps of the aggregated large object write process optimization method mentioned in the above embodiment. The method involves receiving an instruction to modify and write a target small object; wherein the size of the target small object is no greater than 512K; retrieving the target small object from the aggregated large object in the hard disk data pool according to the instruction, and modifying and writing the target small object; wherein the aggregated large object contains multiple small objects; obtaining the size relationship of the target small object before and after the modification and writing; and placing the modified target small object into its original position within the aggregated large object or after the aggregated large object stripe according to the size relationship. Therefore, the above solution reduces the number of times the aggregated large object creates new space and lowers the ineffective utilization rate of storage space by adding a judgment and comparison mechanism to the aggregated large object write process. Specifically, based on the size relationship of the target small object before and after the modification and writing in the aggregated large object, it selects whether to put the modified target small object back in its original position or append it to the aggregated large object stripe, thereby improving the storage efficiency of the hard disk data pool.

[0118] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0119] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] In this embodiment, a computer program is stored on a computer-readable storage medium. When the computer program is executed by a processor, it implements the steps described in the above method embodiment. The process involves receiving an instruction to modify and write a target small object; wherein the size of the target small object is no greater than 512KB; retrieving the target small object from the aggregated large object in the hard disk data pool according to the instruction, and modifying and writing the target small object; wherein the aggregated large object contains multiple small objects; obtaining the size relationship of the target small object before and after the modification and write; and placing the modified target small object into its original position within the aggregated large object or after the aggregated large object stripe according to the size relationship. Therefore, the above solution reduces the number of times the aggregated large object creates new space and lowers the ineffective utilization rate of storage space by adding a judgment and comparison mechanism to the aggregated large object write process. Specifically, based on the size relationship of the target small object before and after the modification and write in the aggregated large object, it selects whether to put the modified target small object back in its original position or append it to the aggregated large object stripe, thereby improving the storage efficiency of the hard disk data pool.

[0121] The foregoing provides a detailed description of a method, apparatus, device, and medium for optimizing the large object write process provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0122] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for optimizing the write process of aggregated large objects, characterized in that, include: Receive instructions to modify and write a target small object; wherein the size of the target small object is no greater than 512K; According to the instructions, the target small object is obtained from the aggregated large object in the mechanical hard disk data pool, and the target small object is modified and written; wherein, the aggregated large object contains multiple small objects; Obtain the size relationship of the target small object before and after the modification; Based on the size relationship, the modified target small object is placed into the original position of the target small object in the aggregated large object or into the strip of the aggregated large object; The step of placing the modified target small object into the original position of the target small object in the aggregated large object or into the strip of the aggregated large object according to the size relationship includes: When the size of the target small object after modification is smaller than the size of the target small object before modification, data is filled into the target small object after modification until the size of the target small object is equal to the size of the target small object before modification. Then, the target small object after data filling is placed into the original position of the target small object in the aggregated large object. When the size of the target small object after modification is equal to the size of the target small object before modification, the target small object after modification is directly placed into the original position of the target small object in the aggregated large object, and the space of the original position is not invalidated. When the size of the target small object after modification is greater than the size of the target small object before modification, the space of the original position of the target small object in the aggregated large object is invalidated, and space is created after the original strip of the aggregated large object to put the target small object after modification.

2. The method for optimizing the write process of aggregated large objects according to claim 1, characterized in that, Before modifying the target small object, the method further includes: Create a temporary space to store the modified target small object.

3. The method for optimizing the write process of aggregated large objects according to claim 2, characterized in that, The process of filling the modified target small object with data includes: Invalid data is filled into the modified target small object within the temporary space.

4. The method for optimizing the write process of aggregated large objects according to claim 1, characterized in that, Before retrieving the target small object from the aggregated large object in the hard disk data pool according to the instruction, and after receiving the instruction to modify and write the target small object, the method further includes: Determine whether the target small object is aggregated into the aggregate large object; If so, proceed to the step of obtaining the target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction; If not, then the target small object is directly modified and written in the solid-state drive cache pool according to the instructions.

5. The method for optimizing the write process of aggregated large objects according to any one of claims 1 to 4, characterized in that, Also includes: The remaining space of the hard disk data pool is obtained according to a preset period; When the remaining space does not meet the preset threshold, the invalid space in the mechanical hard disk data pool is reclaimed.

6. The method for optimizing the write process of aggregated large objects according to claim 5, characterized in that, After reclaiming the invalid space in the hard disk data pool, the method further includes: Output the current remaining space of the mechanical hard disk data pool.

7. A device for optimizing the write process of aggregated large objects, characterized in that, include: A receiving module is used to receive instructions to modify or write a target small object; wherein the size of the target small object is no greater than 512K; The first acquisition module is used to acquire the target small object from the aggregated large object in the mechanical hard disk data pool according to the instruction, and modify and write the target small object; wherein, the aggregated large object contains multiple small objects; The second acquisition module is used to acquire the size relationship of the target small object before and after modification. The object storage module is used to place the modified target small object into the original position of the target small object in the aggregated large object or after the strip of the aggregated large object according to the size relationship; The object storage module is specifically used for: when the size of the modified target small object is smaller than the size of the target small object before modification, filling the modified target small object with data until the size of the target small object is equal to the size of the target small object before modification, and placing the data-filled target small object into the original position of the target small object in the aggregated large object; when the size of the modified target small object is equal to the size of the target small object before modification, directly placing the modified target small object into the original position of the target small object in the aggregated large object, without invalidating the space of the original position; when the size of the modified target small object is larger than the size of the target small object before modification, invalidating the space of the original position of the target small object in the aggregated large object, and creating space after the original stripe of the aggregated large object to place the modified target small object.

8. A device for optimizing the write process of aggregated large objects, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the aggregated large object write process optimization method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the aggregated large object write process optimization method as described in any one of claims 1 to 6.

Images