Object data storage method, electronic equipment and storage medium

Abstract

The invention discloses an object data storage method, electronic equipment and a storage medium. The object data storage method comprises the steps of obtaining a plurality of data blocks of at least one piece of object data; determining the category of each data block based on the position of each data block in each object data; for each piece of object data, in response to the fact that the number of the target data blocks belonging to the category with the highest aggregation priority in the object data is greater than or equal to a preset number, aggregating the target data blocks to obtain at least one aggregation object of the object data; and based on the at least one aggregation object corresponding to each object data, determining a plurality of to-be-stored objects, and storing each to-be-stored object in a preset data pool. According to the scheme, it can be guaranteed that the data blocks belonging to the same object data are aggregated in fewer aggregation objects more intensively, and therefore the storage efficiency of the object data is improved.

Description

Technical Field

[0001] The present application relates to the field of storage technologies, and in particular, to a method for storing object data, an electronic device, and a storage medium. Background Art

[0002] The distributed file storage system (Ceph) is a widely adopted distributed file storage system that can automatically rebalance and automatically recover. It can provide block device storage, file system storage, and object storage functions. Among them, Ceph can provide an object storage interface through its object storage gateway (Rados Gateway, RGW), providing users with convenient and fast object data operations, and writing and deleting object data to the object storage device (Object Storage Device, OSD) through the RGW. In the scenario where object data is docked to RADOS for object data storage, the RGW gateway first splits the object data into multiple RADOS objects. The sizes of the RADOS objects are different, forming a natural situation of mixed upper and lower cases. Usually, the small RADOS objects are directly aggregated, and the RADOS objects split from the same object data may be aggregated into different large RADOS objects, resulting in the large RADOS objects aggregated from the same object data being too scattered and having low predictability, thus causing excessive computing power resources to be consumed when operating or reading the entire object data subsequently, increasing the pressure on the storage system.

[0003] In view of the existing technical deficiencies, how to provide a solution to improve the storage efficiency of object data is a technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0004] The present application provides at least a method for storing object data, an electronic device, and a storage medium.

[0005] The present application provides a method for storing object data, including: obtaining a plurality of data blocks of at least one object data; determining the category of each data block based on the position of each data block in each object data; for each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to a preset number, aggregating each target data block to obtain at least one aggregated object of the object data; determining a plurality of objects to be stored based on the at least one aggregated object corresponding to each object data, and storing each object to be stored in a preset data pool.

[0006] The present application provides a storage device for object data, including: an acquisition module, a first determination module, an aggregation module, and a second determination module; the acquisition module is configured to acquire a plurality of data blocks of at least one object data; the first determination module is configured to determine the category of each data block based on the position of each data block in each object data; the aggregation module is configured to, for each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to a preset number, aggregate each target data block to obtain at least one aggregated object of the object data; the second determination module is configured to determine a plurality of objects to be stored based on at least one aggregated object corresponding to each object data, and store each object to be stored in a preset data pool.

[0007] The present application provides an electronic device, including a memory and a processor, and the processor is configured to execute program instructions stored in the memory to implement the above-mentioned object data storage method.

[0008] The present application provides a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the above-mentioned object data storage method is implemented.

[0009] In the above solution, based on the positions of the acquired data blocks in each object data, the category of each data block is determined. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to a preset number, each target data block is aggregated to obtain at least one aggregated object of the object data, which can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, a plurality of objects to be stored are obtained, so that each object to be stored is an object more suitable for storage, and each object to be stored is stored in a preset data pool, thereby making the distribution of the data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, and thus improving the storage efficiency of the object data.

[0010] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to explain the technical solutions of the present application.

[0012] Figure 1 is a flowchart showing an embodiment of the object data storage method of the present application Figure 1 ;

[0013] Figure 2Schematic flowchart of an embodiment of the storage method of the object data of the present application Figure 2 ;

[0014] Figure 3 Schematic flowchart of an embodiment of the storage method of the object data of the present application Figure 3 ;

[0015] Figure 4 Schematic structural diagram of an embodiment of the storage device of the object data of the present application;

[0016] Figure 5 Schematic structural diagram of an embodiment of the electronic device of the present application;

[0017] Figure 6 Schematic structural diagram of an embodiment of the computer-readable storage medium of the present application. Detailed implementation manners

[0018] The solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings of the specification.

[0019] In the following description, specific details such as specific system architectures, interfaces, and technologies are set forth for the purpose of illustration and not limitation, so as to thoroughly understand the present application.

[0020] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. In addition, "multiple" in this article means two or more than two. In addition, the term "at least one" in this article represents any one of multiple or any combination of at least two of multiple. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set composed of A, B, and C.

[0021] The present application provides a method for storing some object data and a storage device for object data. The application scenarios of the method for storing object data include, but are not limited to, the scenario of storing object data. The execution subject of the method for storing object data can be a storage device for object data or a server capable of implementing the storage of object data. For example, the storage device for object data can be set in a terminal device, a server, or other processing devices. Among them, the terminal device can be a device for storing object data, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementation manners, the method for storing object data can be implemented by a processor calling computer-readable instructions stored in a memory.

[0022] Object storage is a data storage architecture. During the process of object storage, data is stored as "objects", which is different from the traditional file system or block storage method. In object storage, each object data includes the data itself, metadata, and a unique identifier. Among them, the metadata included in each object data is used to describe the attributes and information of the object data. Ceph object storage is based on its own RADOS distributed object storage architecture, and RGW (Ceph object gateway) is developed on it to enable Amazon S3 and OpenStack Swift objects to be easily migrated to the distributed file storage system Ceph. It can be understood that object data includes, but is not limited to, two types: Amazon S3 objects and OpenStack Swift objects.

[0023] Exemplarily, during the process of uploading object data, the object storage gateway RGW can be used to split the object data, and then upload the split object data to the distributed file storage system. Among them, according to the different ways of splitting object data by the object storage gateway, the ways of uploading object data are different. For any object data, the object data is split according to the split parameters. The split parameters include the cut block parameters and the cut stripe parameters. The split parameters of different object data can be the same or different. For different splitting methods, the settings of the split parameters can be the same or different. The cut block parameter can be chunk_size, which can represent the RGW cut block size. The cut stripe parameter can be stripe_size, which can represent the RGW cut stripe size. The object data that needs to be split can be an S3 object.

[0024] In some application scenarios, when the object data is sliced in a holistic manner, the upload method is holistic upload. Specifically, if the size of the object data is greater than the slice block parameter, according to the slice block parameter and the slice strip parameter in the slice parameters of holistic slicing, first slice out a head object from the object data according to the size of the slice block parameter, and the head object is also the head object. Then, evenly slice the remaining object according to the slice strip parameter to obtain at least one strip object. It can be understood that when evenly slicing the remaining object according to the slice strip parameter to obtain at least one strip object, the size of the last strip object may be less than or equal to the slice strip parameter. When the size of the last strip object is less than the slice strip parameter, the last strip object can be a small object. At this time, for any object data in holistic slicing, the number of small objects is 1. In some other application scenarios, if the size of the object data is less than or equal to the minimum value of the slice block parameter and the slice strip parameter, directly use the entire object data as a small object. At this time, for any object data in holistic slicing, the number of small objects is 1.

[0025] In some other application scenarios, when the object data is sliced in a segmented manner, the upload method is segmented upload. Specifically, first generate a head object with a size of 0 for the object data, and the head object is also the head object. Among them, the metadata of the object data is recorded in the head object. The slice parameters also include slice part parameters, and the slice part parameter can be part_size, which is used to represent the user's setting of the size of each part when slicing the object data in segmented slicing. The slice part parameter can be a fixed value or a range of sizes, including the minimum part parameter to the maximum part parameter. Then, slice the object data into at least one part object according to the slice part parameter. Among them, the size of the part object can be expressed as part_size. For each part object, when the size of the part object is less than the slice strip parameter, directly use the part object as a small object. At this time, for any object data in segmented slicing, the number of small objects obtained by slicing the object data is equal to the number of each part object, and the small objects are at least one.

[0026] In some other application scenarios, for each part of the object, when the size of the part of the object is greater than or equal to the cutting stripe parameter, for each part of the object, it is evenly divided according to the cutting stripe parameter stripe_size. The first piece obtained by dividing each part of the object is called the multipart object, and at least one divided data block obtained by dividing the remaining part of the part of the object is called each shadow object. It can be understood that the size of each shadow object is less than or equal to the cutting stripe parameter. For each part of the object, the size of the last shadow object obtained by dividing the part of the object may be less than or equal to the cutting stripe parameter. When the size of the last shadow object is less than the cutting stripe parameter, the last shadow object can be a small object. At this time, for any object data in segmented division, the number of small objects is less than or equal to the number of each part of the object, and the small objects are at least one.

[0027] It can be understood that the storage method of storing data in units of object storage is usually used to store unstructured data, that is, the type of object data is unstructured data. For example, the form of object data can be documents, images, audio, videos, etc. The object data is the data corresponding to the target server in the target cluster, where the target cluster includes at least one server. The target cluster corresponds to at least one preset data pool. Each preset data pool is used to store data blocks related to object data. The importance of the data blocks stored in each preset data pool decreases in turn. Each preset data pool includes a cache pool and at least one other data pool. The object data corresponding to the data block stored in the cache pool has the highest importance, that is, the object data stored in the cache pool is hot data. Each preset data pool is a common data pool in the target cluster, and each server in the target cluster can store object data in each preset data pool. At least one initial data block obtained by dividing the object data through the above division method by using the RGW gateway can be first stored in the cache pool of the target cluster.

[0028] Based on this, during the storage process of object data, in the scenario where S3 object data is docked to RADOS, the RGW gateway first splits the S3 object into multiple initial data blocks. The initial data blocks can be represented as RADOS objects. For any object data, in addition to the large data blocks that meet the cutting block parameters, cutting stripe parameters, and cutting part parameter sizes in at least one initial data block obtained through overall splitting or segmented splitting, the above-mentioned small objects are also included. At this time, when the RGW gateway splits the object data, it may form a natural mixture of uppercase and lowercase, that is, at least one initial data block is a mixture of large data blocks and small objects. When storing each object data in each preset data pool, usually the initial data blocks are randomly aggregated into a fixed size, and the small objects cut from the same S3 object may be aggregated into different large objects, resulting in additional costs for storing and / or reading the same object data, that is, additional costs are required when operating the entire S3 object, increasing the pressure on the storage system.

[0029] To this end, the present application provides a method for storing object data. Based on the positions of the data blocks in each object data obtained, the categories of the data blocks are determined. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to the preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data, which can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on the at least one aggregated object corresponding to each object data, several objects to be stored are obtained, so that each object to be stored is an object more suitable for storage. The objects to be stored are stored in the preset data pool, thereby making the distribution of the data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0030] For example, for any object data, the number of initial data blocks is 10, among which 5 initial data blocks belong to the above-mentioned small objects. In the case of random aggregation, the above-mentioned small objects may be aggregated in 5 aggregated objects respectively. In this way, when object data with a larger size is segmented and stored, it may be aggregated in multiple unpredictable aggregated objects. In response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to the preset number, the present application aggregates each target data block to obtain at least one aggregated object of the object data, which can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. The number of aggregated objects obtained by aggregating the above-mentioned small objects may be less than 5. For the S3 object size mixed write scenario, the S3 object can represent object data. After obtaining at least one aggregated object by managing at least one initial data block with a mixed size generated after cutting, several storage objects to be stored are determined, which can alleviate the problem that object data with a larger size may be aggregated in multiple unpredictable aggregated objects when segmented and stored, and at the same time take into account the aggregation of small objects, improving the performance of mixed writing of large and small objects obtained by segmenting object data. When operating on the storage objects to be stored subsequently, the pressure on the storage system during operation can be reduced.

[0031] Please refer to Figure 1 , Figure 1 which is a flowchart of an embodiment of the storage method of object data in the present application. Figure 1 Specifically, the storage method of object data may include the following steps:

[0032] Step S11: Obtain several data blocks of at least one object data.

[0033] The at least one object data may be one object data or multiple object data. The RGW gateway is used to segment each object data to obtain several initial data blocks of each object data. The method of segmenting each object data includes, but is not limited to, the above-mentioned overall segmentation or the above-mentioned segmented segmentation. In some application scenarios, the several data blocks may be the several initial data blocks of each object data. Specifically, the several data blocks include, but are not limited to, the head object during overall segmentation, at least one stripe object, the small object corresponding to the entire object data obtained due to the smaller size of the object data during overall segmentation, the head object during segmented segmentation, multiple partial objects in each partial object, at least one shadow object in each partial object, and the small object directly corresponding to the whole due to the smaller size of some partial objects during segmented segmentation, etc.

[0034] In some other application scenarios, the above-mentioned several data blocks may be data blocks obtained by preset filtering of several initial data blocks of each object data. Among them, the preset filtering method may be filtering based on the relationship between the size of each initial data block and the preset size. The preset size may be a fixed value set according to the storage requirements of the object data, or may be determined according to the splitting parameters of the RGW gateway. Specifically, the several data blocks include but are not limited to the last stripe object during overall splitting, the small object corresponding to the entire object data obtained due to the small size of the object data during overall splitting, the head object during segmented splitting, the last shadow object in each partial object, etc.

[0035] The above step S11 may be directly using the several initial data blocks obtained by the RGW gateway splitting each object data as the above-mentioned several data blocks. The above step S11 may also be filtering the several initial data blocks of each object data according to the preset filtering method to obtain the above-mentioned several data blocks. The several data blocks are at least part of the several initial data blocks.

[0036] Step S12: Determine the category of each data block based on the position of each data block in each object data.

[0037] In the case where the several data blocks are several initial data blocks, step S12 only determines the category of at least part of the several initial data blocks that meet the size condition. In this application, the several data blocks in the above step S11 are the several initial data blocks after the preset filtering method. The several data blocks include but are not limited to the last stripe object during overall splitting, the small object corresponding to the entire object data obtained due to the small size of the object data during overall splitting, the head object during segmented splitting, the last shadow object in each partial object as examples.

[0038] The position in each object data may be the relative position relationship between the data block obtained by overall splitting or segmented splitting and the object data to which the data block belongs. The relative position relationship includes the first type of relative position, the second type of relative position, the third type of relative position, and the fourth type of relative position. The first type of relative position is used to represent that during segmented splitting, the data block is at the end position corresponding to the last shadow object in each partial object of the object data to which the data block belongs. The second type of relative position is used to represent that during segmented splitting, the data block is at the starting position corresponding to the head object in the object data to which the data block belongs. The third type of relative position is used to represent that during overall splitting, the data block is at all positions of the object data to which the data block belongs, that is, the data block in the third type of relative position is the small object corresponding to the entire object data obtained due to the small size of the object data during overall splitting. The fourth type of relative position is used to represent that during overall splitting, the data block is at the end position corresponding to the last stripe object in the object data to which the data block belongs.

[0039] The categories of each data block include a first category corresponding to the relative position of the first type, a second category corresponding to the relative position of the second type, a third category corresponding to the relative position of the third type, and a fourth category corresponding to the relative position of the fourth type.

[0040] In some application scenarios, the above step S12 may be, for each data block, according to the relative position relationship of the data block in the object data to which the data block belongs, taking the category corresponding to the relative position relationship of the data block as the category of the data block. In some other application scenarios, the above step S12 may be, for each data block, determining the category of the data block according to the segmentation method of the data block and the relationship between the data block and the target object in the object data to which the data block belongs. The target object may be one of the last shadow object, the head object, the overall object data, and the last stripe object. Specifically, in response to the segmentation method of the data block being segmented segmentation and the data block being the last shadow object among the partial objects of the object data to which the data block belongs, the category of the data block is determined as the first category. In response to the segmentation method of the data block being segmented segmentation and the data block being the head object in the object data to which the data block belongs, the category of the data block is determined as the second category. In response to the segmentation method of the data block being overall segmentation and the data block being the part corresponding to the overall object data to which the data block belongs, the category of the data block is determined as the third category. In response to the segmentation method of the data block being overall segmentation and the data block being the last stripe object in the object data to which the data block belongs, the category of the data block is determined as the fourth category.

[0041] Before step S13, based on the categories of each data block, the aggregation priorities of each data block are sorted in sequence according to a preset sorting rule. The preset sorting rule may be determined according to the segmentation method, or may be determined according to the segmentation method and the target object, or may also be preset according to the configuration instructions of the configuration interface. In some application scenarios, the aggregation priority of the data blocks belonging to the segmented segmentation is higher than that of the data blocks belonging to the overall segmentation. In some other application scenarios, the aggregation priority of the last shadow object in the target object is higher than that of other objects in the target object. In some other application scenarios, the aggregation priority of the data blocks of the third category may be equal to the aggregation priority of the data blocks of the fourth category. Exemplarily, in this application, taking the aggregation priorities of each data block from high to low as the data blocks of the first category, the data blocks of the second category, the data blocks of the third category, and the data blocks of the fourth category as an example.

[0042] Judge whether the number of data blocks of the first category among the data blocks belonging to the same object data is greater than or equal to the preset number.

[0043] Step S13: For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to a preset number, aggregate the target data blocks to obtain at least one aggregated object of the object data.

[0044] Each target data block is a data block of the first category among the data blocks belonging to the same object data. Each target data block is a data block with a size smaller than a preset size in the last shadow object obtained by segmenting the same object data. The preset number can be dynamically set according to the storage precision of the object data. The value of the preset number can be greater than or equal to 2. For example, the preset number is 2.

[0045] For each object data, when the number of target data blocks of the object data is greater than or equal to the preset number, aggregate the target data blocks to obtain at least one aggregated object of the object data. Specifically, at least part of the target data blocks are sequentially aggregated into aggregated objects of a fixed size. First, each target data block is used as each data block to be aggregated. At least part of the data blocks to be aggregated are aggregated into at least one aggregated object. Among them, aggregating at least part of the data blocks to be aggregated into at least one aggregated object includes:

[0046] In response to at least part of the data blocks to be aggregated being aggregated into an aggregated object of a fixed size, the remaining data blocks to be aggregated among the data blocks to be aggregated are used as new data blocks to be aggregated. Specifically, determine whether the first N objects to be aggregated are just aggregated into an aggregated object of a fixed size. If so, use the successfully aggregated aggregated object as the first candidate object, and use the (N + 1)-th object to be aggregated and the subsequent objects to be aggregated as the new objects to be aggregated. Aggregate the new objects to be aggregated according to a fixed size until all the target data blocks belonging to the same object data are aggregated. During the aggregation process, the aggregation method can be to aggregate according to the distance of each data block to be aggregated from the head object of the object data, or to aggregate according to the size order. During the aggregation according to the size order, first aggregate the target data block with the largest size among the data blocks to be aggregated, and then search for at least one data block to be aggregated whose size sum with the target data block to be aggregated is equal to the fixed size in descending order of size.

[0047] Specifically, if the aggregated object obtained by aggregating the first N objects to be aggregated is smaller than the fixed size, and the aggregated object obtained by aggregating the first N + 1 objects to be aggregated is larger than the fixed size, the aggregated object obtained by aggregating the first N objects to be aggregated is used as a candidate object. The (N + 1)-th object to be aggregated and the subsequent objects to be aggregated are used as the new objects to be aggregated respectively. Aggregate each new object to be aggregated according to the fixed size until all target data blocks belonging to the same object data are aggregated, obtaining all candidate objects, and using each candidate object as each aggregated object. Among them, there may be candidate objects with sizes smaller than the fixed size among the aggregated candidate objects. It can be understood that the present application does not limit the specific aggregation method during the aggregation process, that is, the sorting of each data block to be aggregated is not limited.

[0048] Exemplarily, the splitting method of object A data is segment splitting. According to the above splitting parameters, 1 head object and 4 partial objects are obtained. If the splitting strip parameter of the splitting parameter is 4M, and the splitting partial parameter in the splitting parameter is greater than 4M, and each splitting partial parameter is at least partially different. The sizes of the last shadow objects in each partial object are 1M, 3M, 3M, and 4M respectively. The preset size is 2M. At this time, there are 3 data blocks belonging to the first category in object A data, which are the last shadow objects with sizes of 1M, 3M, and 3M respectively. That is, the target data blocks with the highest aggregation priority belonging to object A data are the last shadow objects with sizes of 1M, 3M, and 3M here. The preset quantity is 2. The fixed size in the above aggregation process can be set to 4M. The last shadow objects with sizes of 1M, 3M, and 3M are used as the objects to be aggregated respectively. Aggregate each object to be aggregated belonging to object A data according to the above aggregation method to obtain 2 candidate objects with sizes of 4M and 3M respectively. And use these 2 candidate objects as each aggregated object belonging to object A data.

[0049] Step S14: Determine a plurality of objects to be stored based on at least one aggregated object corresponding to each object data, and store each object to be stored in a preset data pool.

[0050] The object to be stored is used to represent an object that can be stored in the preset data pool. Among them, the object to be stored may be the aggregated object in step S13 above, or an aggregated object filled with the aggregated object in step S13 above. The method of filling the aggregated object in step S13 above with an aggregated object may be to fill the data block with an aggregation priority lower than the highest aggregation priority in the aggregated object of any object data obtained in step S13 above to obtain a new aggregated object.

[0051] The above step S14 may be to first determine the aggregation state of each aggregation object according to the relationship between the size of each aggregation object and the threshold size. The aggregation state may be a full aggregation state or a non-full aggregation state. Reaching the full aggregation state may indicate that the size of the aggregation object is equal to the threshold size. Not reaching the full aggregation state may indicate that the size of the aggregation object is less than the threshold size. The threshold size may be greater than or equal to the above fixed size. In this application, the case where the threshold size is equal to the above fixed size is taken as an example. For each aggregation object, in response to the aggregation state of the aggregation object reaching the full aggregation state, the aggregation object is used as an object to be stored. Alternatively, for each aggregation object, in response to the aggregation state of the aggregation object not reaching the full aggregation state, the aggregation object is used as a target aggregation object. The target aggregation object is filled with data blocks based on the target aggregation priority to obtain a new aggregation object, and the new aggregation object is used as an object to be stored. The target aggregation priority is an aggregation priority lower than the highest aggregation priority. In some application scenarios, the objects to be stored are directly stored in a preset data pool. In other application scenarios, for each object to be stored, in response to the size of the object to be stored being equal to the storage size, the objects to be stored are stored in the preset data pool. The storage size may be the above threshold size. In other application scenarios, for each object to be stored, in response to the survival time of the object to be stored being equal to the threshold time, the objects to be stored are stored in the preset data pool. The survival time of the object to be stored is used to represent the time period from the start time to the end time. The start time of the survival time of the object to be stored may be the aggregation completion time of the data block belonging to the highest aggregation priority in the object to be stored. The end time of the survival time of the object to be stored may be the time when it is determined whether the survival time of the object to be stored is equal to the threshold time. It can be understood that the threshold size, the storage size, and the threshold time can be dynamically set according to the storage requirements of the object data.

[0052] It can be understood that the RGW gateway can adapt object data to be accessed into the Ceph storage cluster. The RGW gateway accesses object data according to the user's writing method, i.e., overall upload or segmented upload, combines the Ceph storage cluster settings to set the split block parameter chunk_size and the split stripe parameter stripe_size, and performs split processing related to overall splitting or segmented splitting of the object data to obtain a number of initial data blocks. After splitting, a scenario of writing the initial data blocks with a mixture of large and small objects is generated. Especially when the object data is written as a mixture of the initial data block sizes, the Ceph storage cluster needs to face a more complex scenario of writing the initial data blocks in a mixture. This application processes the initial data blocks with a mixture of large and small sizes generated after splitting, and can write the objects to be aggregated into a high-speed device, that is, a cache pool. From the perspective of the same object data, the first type of target data blocks generated by splitting are preferentially aggregated to improve the performance of the mixture of large and small object writes after the object data splitting is converted into initial data blocks. It can be understood that the Ceph storage cluster can be the above-mentioned target cluster.

[0053] In some other application scenarios, after the above step S13, this application can further include the following steps: for each object data, record the storage location of at least one aggregated object of the object data and the distribution information of the data blocks in each aggregated object into a preset information set so as to operate on the object data according to the preset information set. The preset information set can be the index set corresponding to the head object in the object data. Operating on the object data includes operations such as reading and deleting the object data as a whole. Exemplarily, when creating a storage pool for object data, create a cache pool on a high-speed device. The user can select a high-speed device as the cache pool according to their own resource conditions. The high-speed device can be a solid-state drive or a new type of hard disk such as PMEM.

[0054] Based on the positions of the obtained data blocks in each object data, the categories of the data blocks are determined. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data, which can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, a number of objects to be stored are obtained, so that each object to be stored is a more suitable object for storage. The objects to be stored are stored in a preset data pool, thereby making the distribution of the data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0055] In some embodiments, step S11 may include the following steps: First, obtain a number of initial data blocks of at least one object data and the segmentation parameters of each object data. The segmentation parameters include the cutting block parameters and the cutting strip parameters when segmenting each object data. Subsequently, use the minimum value among the segmentation parameters of each object data as the division threshold. Then, determine the number of data blocks from the number of initial data blocks that are less than or equal to the division threshold.

[0056] The at least one object data can be one object data or multiple object data. In the case where the object data is one, the segmentation method of the object data can be segmental segmentation. The segmentation parameters may further include the cutting part parameters for segmental segmentation of each object data.

[0057] In the case where the number of data blocks can be the data blocks obtained by preset filtering of the number of initial data blocks of each object data, determine the preset size in the preset filtering method based on the segmentation parameters. Specifically, the division threshold is the preset size. In some application scenarios, the preset filtering method may be to determine the number of data blocks from the number of initial data blocks that are less than or equal to the division threshold. In other application scenarios, determine the filtered data blocks from the number of initial data blocks that are greater than the division threshold. Directly store the filtered data blocks in the preset data pool according to the preset filtering period. The minimum value in the segmentation parameters can be the value corresponding to the cutting block parameter or the value corresponding to the cutting strip parameter. In other application scenarios, when the cutting block parameter and the cutting strip parameter in the segmentation parameters are equal, the division threshold can be the value corresponding to the cutting block parameter and the value corresponding to the cutting strip parameter.

[0058] Exemplarily, at least one object data includes A object data, B object data, C object data, and D object data. Each object data is segmented according to the segmentation parameters to obtain at least one initial data block corresponding to each object data. Among them, A object data and B object data are segmented by parts. C object data and D object data are segmented as a whole. Among them, the initial data blocks of A object data can be 1 head object head, 4 multi-part objects multipart, and 8 shadow objects shadow respectively. The initial data blocks of A object data can be 1 head object head, 3 multi-part objects multipart, and 3 shadow objects shadow respectively. In the case where the size of C object data is less than or equal to the minimum value of the cut block parameter and the cut stripe parameter, the C object data as a whole is regarded as a small object, that is, the initial data block of C object data is the data block corresponding to the whole C object data. The initial data blocks of D object data can be 1 head object head and 4 stripe objects stripe respectively. According to the division threshold, several initial data blocks are filtered to obtain several data blocks. The division threshold is the minimum value between the cut block parameter and the cut stripe parameter in the segmentation parameters. Exemplarily, several data blocks include the head object of A object data and 3 tail shadow objects, the head object of B object data and 3 tail shadow objects, the data block corresponding to the whole C object data, and the tail stripe object corresponding to D object data. Exemplarily, after the above step S12, the data blocks belonging to the first category are 3 tail shadow objects corresponding to A object data and 3 tail shadow objects corresponding to B object data. The data blocks belonging to the second category are the head object of A object data and the head object of B object data. The data blocks belonging to the third category are the data blocks corresponding to the whole C object data. The data blocks belonging to the fourth category are the tail stripe objects corresponding to D object data.

[0059] It can be understood that a small file threshold is set for the RGW gateway. In actual implementation, the RGW segments the object data according to the cut block parameter chunk_size or the cut stripe parameter stripe_size, and the sizes of these two parameters can be dynamically configured. In this application, the small file threshold, that is, the division threshold, is default set to the minimum value in the above segmentation parameters. The division threshold (Small Object Threshold, SOT) can be expressed as min(chunk_size, stripe_size). Set the fixed size in the aggregation process, that is, the size used to represent the expected aggregated object. For example, the above fixed size can be default set to 4M, and this value can be configured according to the actual situation. It can be understood that this application takes 4M as an example and does not limit the specific value of the above fixed size.

[0060] After the object data is written to the RGW gateway, the RGW gateway performs object data splitting. The splitting rules remain the same as the original ones. In some application scenarios, for overall uploads, the RGW gateway will perform overall splitting. The RGW gateway first splits out a header object (i.e., the head object) according to the chunk size parameter of the cut block. This header object contains the metadata of the entire object data and some actual data in the object data. Then, the other data in the object data except the header object is evenly split according to the stripe size to obtain each stripe object in turn until the last stripe object of the object data is obtained and then the uniform splitting ends. And the header object and each stripe object are used as the initial data blocks. In other application scenarios, for segmented uploads, the RGW gateway will perform segmented splitting. The RGW gateway first generates a header object with a size of 0 (i.e., the head object). The metadata of the object data is recorded in this header object. Then, the part size parameter of the upload segment cut selected by the user is split into at least one partial object. For each partial object, it is evenly split according to the stripe size parameter of the cut stripe to obtain at least one initial data block. Among them, the first initial data block evenly split in each partial object is called a multipart object, and the initial data blocks obtained by splitting the data other than the multipart object in this partial object are called shadow objects. According to the preset splitting rules, the sizes of all the initial data blocks are compared with the division threshold SOT. The initial data blocks with sizes less than or equal to the division threshold SOT among several initial data blocks are used as data blocks and written into the cache pool. The initial data blocks with sizes greater than the division threshold SOT among several initial data blocks are directly written into the preset data pool. It can be understood that all the initial data blocks can be RADOS objects.

[0061] It can be considered that this application can optimize the scenario of object data access to the Ceph storage cluster. From the perspective of object data, the data blocks in the object data processed by the RGW gateway are aggregated. Compared with directly storing each overly scattered initial data block, this application can make the object data access to the Ceph storage cluster more efficient. In addition, for the S3 size mixed write scenario, the data blocks in the object data processed by the RGW gateway are preferentially aggregated according to the highest aggregation priority, improving the efficiency of operating object data in the size object mixed write scenario of object data, thereby improving the efficiency of storing object data.

[0062] In some embodiments, the step of determining a plurality of objects to be stored based on at least one aggregated object corresponding to each object data may include the following steps: First, according to the aggregation priorities of each category, the data blocks of each category are sequentially taken as a batch of data blocks to be aggregated. Subsequently, based on at least one aggregated object corresponding to each object data, the first aggregated objects corresponding to each first batch of data blocks to be aggregated are determined, and the aggregation priority of the category to which each first batch of data blocks to be aggregated belongs is the highest. Then, for the non-first-batch data blocks to be aggregated, in response to the existence of a target aggregated object with free space among the aggregated objects obtained by aggregating the previous batch of data blocks to be aggregated, the target aggregated object is filled with each data block to be aggregated in this batch to obtain the aggregated object corresponding to this batch. Or, in response to the non-existence of a target aggregated object among the aggregated objects obtained by aggregating the previous batch of data blocks to be aggregated, the data blocks to be aggregated in this batch are aggregated to obtain the aggregated object corresponding to this batch. Finally, the aggregated object corresponding to the last batch of data blocks to be aggregated is used as a plurality of objects to be stored.

[0063] The aggregation priorities of each category can be dynamically set according to the priority configuration instructions generated by the configuration operation of the configuration interface. In this application, taking the aggregation priorities of each data block from high to low as the data blocks of the first category, the data blocks of the second category, the data blocks of the third category, and the data blocks of the fourth category as an example.

[0064] Each batch of data blocks to be aggregated can be, from high to low in terms of aggregation priority, the first batch of data blocks to be aggregated corresponding to the data blocks of the first category, the second batch of data blocks to be aggregated corresponding to the data blocks of the second category, the third batch of data blocks to be aggregated corresponding to the data blocks of the third category, and the fourth batch of data blocks to be aggregated corresponding to the data blocks of the fourth category. Among them, the first batch of data blocks to be aggregated is the first batch of data blocks to be aggregated. The non-first-batch data blocks to be aggregated are the second batch of data blocks to be aggregated, the third batch of data blocks to be aggregated, and the fourth batch of data blocks to be aggregated respectively.

[0065] Each of the above-mentioned first batch of data blocks to be aggregated is a data block belonging to the first category. The step of determining the first aggregated objects corresponding to each first batch of data blocks to be aggregated based on at least one aggregated object corresponding to each object data may be to directly use each aggregated object in at least one aggregated object corresponding to each object data as the first aggregated objects corresponding to each first batch of data blocks to be aggregated.

[0066] The target aggregated object with free space may be an aggregated object with a size smaller than the aggregation size among the previous batch of data blocks to be aggregated. That is to say, the target aggregated object is an aggregated object with a cavity. It can be understood that the size of the data block in this application is used to represent the size of the memory space occupied by the data block.

[0067] In some application scenarios, for non-first-batch data blocks to be aggregated, when there are target aggregated objects with free space in the aggregated objects obtained by aggregating the previous batch of data blocks to be aggregated, each data block to be aggregated in this batch is used to fill the target aggregated objects to obtain the aggregated objects corresponding to this batch. Specifically, the number of target aggregated objects with free space can be one or more. For each target aggregated object, each data block to be aggregated in this batch is used to fill the target aggregated object until the size of the new aggregated object obtained reaches the aggregation size, or until each data block to be aggregated in this batch cannot fill the target aggregated object and the size of the new aggregated object obtained by filling the target aggregated object is less than the aggregation size. Then, the remaining data blocks to be aggregated in this batch are aggregated to obtain an aggregated object. The aggregated objects other than the target aggregated objects in the aggregated objects obtained from the previous batch of data blocks to be aggregated, the new aggregated objects corresponding to the target aggregated objects obtained in this batch, and the aggregated objects obtained by aggregating the remaining data blocks to be aggregated in this batch are used as the aggregated objects corresponding to this batch. Among them, the remaining data blocks to be aggregated in this batch are the data blocks to be aggregated in this batch that have not been used to fill the target aggregated objects. Specifically, the way to aggregate the remaining data blocks to be aggregated in this batch can be to separately aggregate each remaining data block into different aggregated objects, or to aggregate each remaining data block into at least one aggregated object according to the aggregation size. Exemplarily, the aggregated objects corresponding to this batch include the aggregated objects in the previous batch that do not have free space, the new aggregated objects obtained after filling the target aggregated objects with free space in the previous batch, the aggregated objects in the previous batch that cannot be filled by the target aggregated objects with free space, and the aggregated objects obtained by aggregating the remaining data blocks to be aggregated in this batch.

[0068] In other application scenarios, for non-first-batch data blocks to be aggregated, in response to the absence of target aggregated objects in the aggregated objects obtained by aggregating the previous batch of data blocks to be aggregated, the data blocks to be aggregated in this batch are aggregated to obtain the aggregated objects corresponding to this batch. The way to aggregate the data blocks to be aggregated in this batch can be to separately aggregate each data block to be aggregated into different aggregated objects, or to aggregate each data block to be aggregated into at least one aggregated object according to the aggregation size. Exemplarily, the aggregated objects corresponding to this batch include the aggregated objects in the previous batch that do not have free space and the aggregated objects obtained by aggregating the data blocks to be aggregated in this batch.

[0069] It can be understood that for the non-first batch of data blocks to be aggregated, the aggregation sequence of the data blocks to be aggregated in each batch is to first aggregate the second batch of data blocks to be aggregated corresponding to the data blocks of the second category, then aggregate the third batch of data blocks to be aggregated corresponding to the data blocks of the third category, and then aggregate the fourth batch of data blocks to be aggregated corresponding to the data blocks of the fourth category.

[0070] Since the size of the data blocks belonging to the second category is 0, the batch of data blocks to be aggregated corresponding to the data blocks of the second category do not need to participate in the filling or aggregation of the non-first batch of data blocks to be aggregated. For the non-first batch of data blocks to be aggregated, the aggregation sequence of the data blocks to be aggregated in each batch is to first aggregate the third batch of data blocks to be aggregated corresponding to the data blocks of the third category, and then aggregate the fourth batch of data blocks to be aggregated corresponding to the data blocks of the fourth category. Among them, the second batch of data blocks to be aggregated corresponding to the data blocks of the second category can record relevant information in the data blocks of the second category belonging to the same object data after determining the first batch of aggregation objects. The relevant information can be the pre-aggregation information of the first batch of aggregation objects, the storage location where they are located, and the distribution information of the data blocks. Record the relevant information into the preset information set linked by the data blocks of the second category belonging to the same object data.

[0071] It can be understood that aggregating the data blocks to be aggregated in batches can improve the aggregation efficiency, thereby enhancing the efficiency of operating on object data in the scenario of mixed writing of large and small object data, and thus improving the efficiency of storing object data. In addition, filling the target aggregation object with available space can make the sizes of each aggregation object tend to the aggregation size, resulting in fewer subsequent operations on the aggregation objects. For example, in the process of downgrading the storage of aggregation objects, fewer operations on the aggregation objects can achieve the downgrading storage of more object data.

[0072] In some embodiments, the step of determining the first batch of aggregation objects corresponding to the first batch of data blocks to be aggregated based on at least one aggregation object corresponding to each object data may include the following steps: Determine the aggregation status of each aggregation object based on the relationship between the size of at least one aggregation object corresponding to each object data and the aggregation size. The aggregation status includes reaching the full aggregation status or not reaching the full aggregation status. For each aggregation object, in response to the aggregation status of the aggregation object reaching the full aggregation status, use the aggregation object as the first batch of aggregation objects corresponding to the first batch of data blocks to be aggregated. Or, in response to the aggregation status of at least one candidate aggregation object not reaching the full aggregation status, determine the first batch of aggregation objects corresponding to the first batch of data blocks to be aggregated based on each candidate aggregation object. The candidate aggregation object is at least one of the aggregation objects.

[0073] At least one aggregated object of each object data includes at least one first aggregated object and / or at least one second aggregated object. The above step S13 may be to aggregate data blocks that belong to the same object data and are of the first category to obtain at least one first aggregated object and / or at least one second aggregated object. The size of each first aggregated object is equal to the aggregation size, and the size of each second aggregated object is smaller than the aggregation size. Among them, the aggregation state of the first aggregated object reaches the full aggregation state. The aggregation state of the second aggregated object does not reach the full aggregation state.

[0074] For each aggregated object above, in response to the aggregation state of the aggregated object reaching the full aggregation state, the step of taking the aggregated object as the first batch of aggregated objects corresponding to the first batch of data blocks to be aggregated may be to take each first aggregated object as the first batch of aggregated objects corresponding to the first batch of data blocks to be aggregated.

[0075] The above step of determining the first batch of aggregated objects corresponding to the first batch of data blocks to be aggregated based on each candidate aggregated object may be that for each second aggregated object, other second aggregated objects are used to aggregate the second aggregated object until the size of the new aggregated object obtained reaches the aggregation size, or until the second aggregated object cannot fill other second aggregated objects and the size of the filled new aggregated object is smaller than the aggregation size, and each new aggregated object is taken as each first batch of aggregated objects. It can be understood that, compared with directly taking all second aggregated objects as each first batch of aggregated objects, the number of each first batch of aggregated objects obtained by performing secondary aggregation among the second aggregated objects is smaller, making the subsequent efficiency higher when filling or aggregating non-first-batch data blocks to be aggregated.

[0076] Exemplarily, each initial data block passes through a preset filter to obtain each data block in the cache pool. There may be 4 types of data blocks in the cache pool: In some application scenarios, when segmenting, the last shadow object cut from the part_size of the partial object obtained from the object data and with a size smaller than the above division threshold SOT is recorded as a data block of the first category. In some application scenarios, the head object with a size of 0 when segmenting is recorded as a data block of the second category. In some application scenarios, the object data that is not segmented during overall segmentation (including the head part of the object data) and with a size smaller than the above division threshold SOT is recorded as a data block of the third category. In some application scenarios, the last strip object during overall segmentation and with a size smaller than the above division threshold SOT is recorded as a data block of the fourth category.

[0077] Exemplarily, in the cache pool, the aggregation thread aggregates the data blocks of the above first category, second category, third category, and fourth category. There can be multiple aggregation threads. The number of aggregation threads can be equal to the value of the batch of data to be aggregated. The aggregation method is as follows: In some application scenarios, the aggregation priority of the data blocks of the first category is the highest. First, process the data blocks belonging to the first category, and aggregate the data blocks belonging to the same object data into the same aggregation object. For the same object data, during the aggregation process, it is not allowed to write data blocks of other object data in the middle. According to the size of the aggregation size of the set aggregation object, there may be three situations. The first is that the aggregation object is less than the aggregation size of 4M after aggregating the data blocks of the first category. The second situation is that it is exactly equal to the aggregation size of 4M after aggregating the data blocks of the first category. The third situation is that it is greater than the aggregation size of 4M after aggregating the data blocks of the first category. Among them, for the third situation, on the premise of not splitting the data blocks of the first category, judge whether the first N data blocks of the first category are just aggregated into a size of the aggregation size of 4M. If the first N data blocks of the first category are just aggregated into a size of the aggregation size of 4M, then the (N + 1)-th data block of the first category creates a new aggregation object for aggregation until all the data blocks of the first category of this object data are aggregated. If the first N data blocks of the first category are aggregated into a size less than the aggregation size of 4M, then the (N + 1)-th data block of the first category directly creates a new aggregation object, and the previous aggregation object leaves a hole until all the data blocks of the first category of this object data are aggregated. For each aggregated aggregation object, a distributed information of the aggregated data block is generated. Each distributed information of the aggregated data block includes the identification information (Aggregation Object ID, AOID) of each data block.

[0078] In some other application scenarios, in the aggregation process of the data blocks of the first category, according to the aggregation situation of the data blocks of the first category, relevant information in the header object corresponding to the object data in the data blocks of the second category belonging to the same object data is increased, and the distributed information AOID of each aggregated data block and the offset and length situation of each data block in the aggregation object in the aggregation object are recorded in the corresponding header object.

[0079] In some other application scenarios, for the data blocks of the second category, since the size of this type of data block is zero, that is, the occupied space is 0, and it only serves as an index to enable the storage bucket to link to the object data corresponding to the data blocks of the second category, this type of data block of the second category is not clustered according to the aggregation priority.

[0080] In some other application scenarios, the aggregation priority of the third category of data blocks is second only to that of the first category of data blocks. For the third category of data blocks, first check whether the third category of data blocks can be stored in the distribution information AOID of the generated aggregated data blocks. If it can be stored, it means there are holes in the aggregation object obtained by aggregating the first category of data blocks. Then, fill the third category of data blocks into the target aggregation object with free space to obtain a new aggregation object. Among them, use the distribution information AOID of the aggregated data blocks corresponding to the target aggregation object with free space as the distribution information of the data blocks before aggregation corresponding to the new aggregation object. Update the distribution information AOID of the data blocks before aggregation corresponding to the new aggregation object and its offset record in the exclusive distribution information AOID of the data blocks before aggregation to the header object of the object data corresponding to the third category of data blocks. In some other application scenarios, if the aggregation object obtained in the above step S13 does not have enough space to store the third category of data blocks, create a new aggregation object, aggregate the third category of data blocks therein, generate the distribution information AOID of the aggregated data blocks corresponding to the new aggregation object, and then modify the metadata of the header object of the object data to which the third category of data blocks belong, recording the aggregation object OID corresponding to the information before aggregation, the distribution information AOID of the aggregated data blocks corresponding to the new aggregation object, and the offset and length in the distribution information of the aggregated data blocks.

[0081] In some other application scenarios, the aggregation priority of the fourth category of data blocks is second only to that of the third category of data blocks. Finally, aggregate the fourth category of data blocks. When aggregating the fourth category of data blocks, first check whether there is a target aggregation object with free space in the aggregation object obtained by aggregating the previous batch of data blocks to be aggregated. If it can be stored, fill the fourth category of data blocks into the target aggregation object with free space to obtain a new aggregation object. Among them, use the distribution information AOID of the aggregated data blocks corresponding to the target aggregation object with free space as the distribution information of the data blocks before aggregation corresponding to the new aggregation object. Update the distribution information AOID of the data blocks before aggregation corresponding to the new aggregation object and its offset record in the exclusive distribution information AOID of the data blocks before aggregation to the header object of the object data corresponding to the fourth category of data blocks. In some other application scenarios, if the aggregation object obtained in the above step S13 does not have enough space to store the fourth category of data blocks, create a new aggregation object, aggregate the fourth category of data blocks therein, generate the distribution information AOID of the aggregated data blocks corresponding to the new aggregation object, and then modify the metadata of the header object of the object data to which the fourth category of data blocks belong, recording the aggregation object OID corresponding to the information before aggregation, the distribution information AOID of the aggregated data blocks corresponding to the new aggregation object, and the offset and length in the distribution information of the aggregated data blocks.

[0082] It can be considered that the present application classifies the data blocks generated after docking the object data with the Ceph storage cluster, and aggregates the data blocks of different categories in the cache pool in sequence according to the aggregation priority, so that the aggregated objects after aggregation are stored subsequently, which can improve the disk space utilization rate.

[0083] In some embodiments, after the step of determining several objects to be stored based on at least one aggregated object corresponding to each object data, the storage method of the object data may further include the following steps: recording the pre-aggregation information, the storage location, and the distribution information of the data blocks of each object to be stored into a preset information set, so as to read each object to be stored based on the preset information set.

[0084] The pre-aggregation information may be at least one object data corresponding to the object to be stored, the splitting method of the corresponding object data, and the initial data blocks included in the corresponding object data. The storage location may be the location of the storage pool where the object to be stored is located. The distribution information of the data blocks may refer to the data blocks included in the object to be stored and the categories of the data blocks.

[0085] In some application scenarios, in response to the distribution information of the data blocks in the object to be stored including data blocks of the first category, record the pre-aggregation information, the storage location, and the distribution information of the data blocks of the object to be stored into the preset information set linked to the head object of the object data to which the data blocks of the first category belong, so as to read each object to be stored based on the preset information set. In other application scenarios, in response to the distribution information of the data blocks in the object to be stored not including data blocks of the first category, record the pre-aggregation information, the storage location, and the distribution information of the data blocks of the object to be stored into the preset information set at the preset location, so as to read each object to be stored based on the preset information set.

[0086] In some application scenarios, the relevant information of each object data is stored in a storage bucket. Each index in the storage bucket may correspond to an object data. The preset information set may correspond to the storage bucket.

[0087] Recording the storage locations of the objects to be stored in a preset information set may involve updating the information of the head object of the object data to which each object to be stored belongs in the index of the storage bucket, and adding the location identification information of the head object of the object data to which it belongs in the cache pool or the preset data pool to the index information. When reading object data, through the index in the storage bucket, the storage pool where the head object of the object data is located is obtained, and through the head object, the distribution information of the data blocks after the object data is sliced is obtained and read. The specific steps for reading object data are as follows: For the unaggregated part after the object data is sliced, the reading rule is the same as the original reading rule, and it can be directly read. For the aggregated part after the object data is sliced, the distribution information of the aggregated data blocks including the identification information of each data block and the offset and length of each data block is obtained from the head object, and the information before aggregation corresponding to the aggregated data blocks is read according to the offset and length of each data block. The distribution information of the aggregated data blocks including the identification information of each data block can be represented as AOID. The information before aggregation corresponding to the aggregated data blocks can be represented as OID. The name of the information before aggregation corresponding to the read aggregated data blocks naturally has the composition order of several initial data blocks of the object data, which is the same as the original rule, and the object data can be spliced.

[0088] It can be considered that the present application manages the stored objects after aggregation through the index of the aggregated objects, that is, the preset information set, which involves the indexing process from the storage bucket index to the head object, then to the aggregated object, and finally to the data blocks obtained by slicing the object data. Among them, the storage bucket index to the head object remains unchanged, and a new index from the aggregated object to the data blocks obtained by slicing the object data is added, which improves the number of objects that can be managed by a single storage bucket, improves the storage capacity, and improves the storage efficiency. In addition, the metadata saved in the head object in the present application only adds an aggregated object to the object data to which the aggregated data blocks in the storage bucket belong, and the rest of the process remains unchanged, which has little impact on the reading performance and is simple to implement.

[0089] Please refer to Figure 2 , Figure 2 which is the flowchart of an embodiment of the storage method of the object data of the present application. Figure 2 .

[0090] In some embodiments, the above step of storing each object to be stored in the preset data pool may include the following steps: For each object to be stored, perform the following steps as Figure 2 shown: Step S21: Based on the size of the object to be stored, determine the waiting storage state of the object to be stored. The waiting storage state of the object to be stored includes entering the waiting storage state or not entering the waiting storage state. Step S22: In response to the waiting storage state of the object to be stored being entering the waiting storage state, store the object to be stored in the preset data pool.

[0091] The waiting storage state being the state of entering the waiting storage state may indicate that the object to be stored can be directly stored in the preset data pool. The waiting storage state being the state of not entering the waiting storage state may indicate that the object to be stored cannot be directly stored in the preset data pool. Among them, the object to be stored with the waiting storage state being the state of entering the waiting storage state can be immediately stored in the preset data pool, or can be stored in the preset data pool sequentially according to the storage batches. The storage batches can be a batch of a preset number of objects to be stored. When a storage batch is full, it is stored in the preset data pool as a unit of a storage batch.

[0092] The above step S21 may be to determine the waiting storage state of the object to be stored based on the relationship between the size of the object to be stored and the threshold size. The threshold size may be equal to the value of the above aggregation size. For example, the threshold size may be 4M. In response to the size of the object to be stored being equal to the threshold size, determine that the waiting storage state of the object to be stored is the state of entering the waiting storage state. In response to the size of the object to be stored being less than the threshold size, determine that the waiting storage state of the object to be stored is the state of not entering the waiting storage state.

[0093] It can be considered that by determining whether the object to be stored can be directly stored in the preset data pool through the storage state of the object to be stored, the size of the object to be stored during the process of storing data in the preset data pool is the size that meets the storage conditions, thereby improving the storage efficiency.

[0094] In some embodiments, the above step S21 may include the following steps: Determine whether the size of the object to be stored is equal to the threshold size. In response to the size of the object to be stored being equal to the threshold size, determine the waiting storage state of the object to be stored as the state of entering the waiting storage state. Or, in response to the size of the object to be stored not being equal to the threshold size, determine whether the aggregation time of the object to be stored reaches the preset time. In response to the aggregation time of the object to be stored reaching the preset time, determine the waiting storage state of the object to be stored as the state of entering the waiting storage state.

[0095] In some application scenarios, when the size of the object to be stored is equal to the threshold size, the waiting storage state of the object to be stored is determined to enter the waiting storage state. In other application scenarios, when the size of the object to be stored is not equal to the threshold size, it is determined whether the aggregation time of the object to be stored reaches the preset time. The aggregation time of the object to be stored may be the above-mentioned survival time. The preset time may be the above-mentioned threshold time. Among them, the specific value of the preset time can be dynamically set according to the storage requirements of the object data. For example, the preset time may be 24 hours. The aggregation time of the object to be stored is used to represent the time period from the start time to the end time. If the object to be stored contains a data block with the highest aggregation priority, the start time of the aggregation time of the object to be stored may be the aggregation completion time of the data block with the highest aggregation priority in the object to be stored. If the object to be stored does not contain a data block with the highest aggregation priority, the start time of the aggregation time of the object to be stored may be the aggregation completion time of the whole object to be stored. The end time of the aggregation time of the object to be stored may be the time when the step of determining whether the aggregation time of the object to be stored is equal to the preset time is located. In some application scenarios, when the aggregation time of the object to be stored reaches the preset time, the waiting storage state of the object to be stored is determined to enter the waiting storage state. In other application scenarios, when the aggregation time of the object to be stored does not reach the preset time, the aggregation time of the waiting storage object reaches the preset time, and the waiting storage state of the object to be stored is determined to enter the waiting storage state.

[0096] It can be considered that determining whether the object to be stored meets the storage conditions by combining the size of the object to be stored and the aggregation time makes the object to be stored in the preset data pool a more storage-condition-satisfied object, thereby improving the storage efficiency of the object data.

[0097] In some embodiments, the above step S22 may include the following steps: in response to the waiting storage state of the object to be stored being in the waiting storage state, obtaining the number of reads of the object to be stored in a preset time period. In response to the number of reads of the object to be stored in the preset time period not reaching the threshold number of times, storing the object to be stored in the preset data pool.

[0098] The number of reads is used to represent the reading situation of any part of the data blocks in the object to be stored by the target cluster. Exemplarily, when the object to be stored contains at least some data blocks of multiple object data, for any one of the contained object data being read, the number of reads of the object to be stored will be counted once. The preset time period is the time period corresponding to determining the usage situation of the object to be stored. The preset time period can be dynamically set according to the storage requirements of the object data. For example, the preset time period may be the time period in which the above-mentioned aggregation time is located.

[0099] Determine whether the read count of the object to be stored within a preset time period reaches the count threshold. In the case where the read count of the object to be stored within the preset time period reaches the count threshold, mark the data status of the object to be stored as hot data.

[0100] In the case where the read count of the object to be stored within the preset time period does not reach the count threshold, directly store the object to be stored into the preset data pool. Specifically, in the case where the read count of the object to be stored within the preset time period does not reach the count threshold, mark the data status of the object to be stored as non-hot data, and store the object to be stored into the preset data pool.

[0101] It can be considered that by using the read count of the object to be stored within the preset time period to determine whether the data status is hot data, keeping the object to be stored corresponding to the hot data in the cache pool, and storing the object to be stored with the data status of non-hot data into the preset data pool, it can make the object to be stored with a higher usage frequency in the cache pool and the object to be stored with a lower usage frequency in the preset data pool, realizing data status partitioning, thereby achieving downgraded storage between the cache pool and the preset data pool.

[0102] Please refer to Figure 3 , Figure 3 which is the flowchart of an embodiment of the method for storing object data in this application. Figure 3 .

[0103] The above steps of determining whether the aggregation time of the object to be stored reaches the preset time in response to the size of the object to be stored not being equal to the threshold size include the following steps S31 and S32. Step S31: In response to the size of the object to be stored not being equal to the threshold size, determine whether the size of the object to be stored exceeds the aggregation threshold. The aggregation threshold is the product of the threshold size and the preset ratio. Step S32: In response to the size of the object to be stored not exceeding the aggregation threshold, determine whether the aggregation time of the object to be stored reaches the preset time.

[0104] Exemplarily, before the object to be stored is stored into the preset data pool, the object to be stored is in the cache pool. Among them, there is a flush thread in the cache pool to flush the object to be stored in the high-speed cache pool into the preset data pool. Flushing is also storing. For each object to be stored, the steps of storing the object to be stored into the preset data pool can refer to the following content:

[0105] The above threshold size is 4M. Determine whether the size of the object to be stored reaches 4M. When the size of the object to be stored reaches 4M, determine the waiting storage state of the object to be stored as entering the waiting storage state. When the waiting storage state of the object to be stored is in the waiting storage state, it indicates that the object to be stored meets the flushing condition. When the size of the object to be stored does not reach 4M, determine whether the size of the object to be stored exceeds the aggregation threshold. The aggregation threshold can be less than the above threshold size. The aggregation threshold can be dynamically set according to the storage requirements of the object data. Specifically, the aggregation threshold can be the product of the threshold size and the preset ratio. The preset ratio can be set to 80%. Among them, the specific value of the preset ratio can be adjusted according to the actual situation. When the size of the object to be stored does not exceed the aggregation threshold, determine whether the aggregation time of the object to be stored reaches the preset time. Specifically, when the size of the object to be stored is equal to or less than the aggregation threshold, determine whether the aggregation time of the object to be stored reaches the preset time. When the aggregation time of the object to be stored reaches the preset time, determine the waiting storage state of the object to be stored as entering the waiting storage state.

[0106] For the aggregated objects that meet the flushing condition, that is, for the objects to be stored whose waiting storage state is in the waiting storage state, determine whether the number of reads of the object to be stored within the preset time reaches the threshold number of times. If the number of reads reaches the threshold number of times, it is considered that the object to be stored belongs to hot data and is not temporarily stored in the preset data pool. If the number of reads does not reach the threshold number of times, it is considered that the object to be stored belongs to non-hot data, and the object to be stored data is stored from the cache pool to the preset data pool. Among them, during the process of storing the object to be stored data into the preset data pool, for the object to be stored whose waiting storage state is in the waiting storage state, non-hot data, but the size does not reach the threshold size, use the preset data to fill the object to be stored so that the size of the filled object to be stored is equal to the threshold size, and store the filled object to be stored into the preset data pool.

[0107] In some other application scenarios, for at least one aggregated object obtained by aggregating data blocks belonging to the second category, for each object to be stored corresponding to the aggregated object, only when the aggregation time of the object to be stored reaches the preset time and the number of reads of the object to be stored does not reach the threshold number of times, store such an object to be stored into the preset data pool. For such an object to be stored, there is no need to fill it to the threshold size, and directly store such an object to be stored into the preset data pool.

[0108] It can be considered that the present application can implement the process of storing the object to be stored into the preset data pool according to the storage requirements of the object data. By detecting the size of the object to be stored and the aggregation time in the cache pool, it is determined whether the waiting storage state of the object to be stored is the state of entering the waiting storage state. Among them, the object to be stored in the waiting storage state that meets the flushing condition, combined with the hot data judgment mechanism, performs delayed storage processing on the high-hot data, thereby improving the throughput capacity of the target cluster for the object to be stored.

[0109] In the above solution, based on the positions of the respective data blocks in each object data, the categories of the respective data blocks are determined. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to the preset number, the respective target data blocks are aggregated to obtain at least one aggregated object of the object data, which can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on the at least one aggregated object corresponding to each object data, a number of objects to be stored are obtained, so that each object to be stored is an object more suitable for storage, and the respective objects to be stored are stored in the preset data pool, thereby making the distribution of the data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0110] Please refer to Figure 4 , Figure 4 is a schematic structural diagram of an embodiment of the storage device for object data of the present application. The storage device 40 for object data includes an acquisition module 41, a first determination module 42, an aggregation module 43, and a second determination module 44. The acquisition module 41 is configured to acquire a number of data blocks of at least one object data; the first determination module 42 is configured to determine the categories of the respective data blocks based on the positions of the respective data blocks in each object data; the aggregation module 43 is configured to, for each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to the preset number, aggregate the respective target data blocks to obtain at least one aggregated object of the object data; the second determination module 44 is configured to determine a number of objects to be stored based on the at least one aggregated object corresponding to each object data, and store the respective objects to be stored in the preset data pool.

[0111] Based on the positions of the obtained data blocks in each object data, the categories of the data blocks are determined. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to the preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data, which can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on the at least one aggregated object corresponding to each object data, several objects to be stored are obtained, so that each object to be stored is an object more suitable for storage. The objects to be stored are stored in a preset data pool, thereby making the distribution of the data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0112] For the functions executed by each module, please refer to the storage method of object data, which will not be elaborated here.

[0113] Please refer to Figure 5 , Figure 5 FIG. is a schematic structural diagram of an embodiment of an electronic device of the present application. The electronic device 50 includes a memory 51 and a processor 52. The processor 52 is configured to execute program instructions stored in the memory 51 to implement the steps in the embodiment of the above object data storage method. In a specific implementation scenario, the electronic device 50 may include, but is not limited to: a multi-camera device, a microcomputer, a server. In addition, the electronic device 50 may also include mobile devices such as a laptop computer and a tablet computer, which are not limited here.

[0114] Specifically, the processor 52 is configured to control itself and the memory 51 to implement the steps in the embodiment of the above object data storage method. The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Additionally, the processor 52 may be implemented jointly by integrated circuit chips.

[0115] Based on the positions of the obtained data blocks in each object data, the categories of the data blocks are determined. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to the preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data, which can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on the at least one aggregated object corresponding to each object data, several objects to be stored are obtained, such that each object to be stored is an object more suitable for storage. The objects to be stored are stored in a preset data pool, thereby making the distribution of the data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0116] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present application. On the computer-readable storage medium 60, there is a program instruction 601 stored, and when the program instruction 601 is executed by a processor, the steps in the above-described method embodiment for storing any object data are implemented.

[0117] Based on the positions of the obtained data blocks in each object data, the categories of the data blocks are determined. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to the preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data, which can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on the at least one aggregated object corresponding to each object data, several objects to be stored are obtained, such that each object to be stored is an object more suitable for storage. The objects to be stored are stored in a preset data pool, thereby making the distribution of the data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0118] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0119] The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments. The same or similar parts can be referred to each other. For the sake of brevity, they will not be repeated herein.

[0120] In several embodiments provided in this application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0121] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0122] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of this application. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs and other various media that can store program codes.

Claims

1. A storage method for object data, characterized in that, The method includes: Obtaining a plurality of data blocks of at least one object data; Determining the category of each data block based on the position of each data block in each object data; For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to a preset number, aggregating each target data block to obtain at least one aggregated object of the object data; Determining a plurality of objects to be stored based on at least one aggregated object corresponding to each object data, and storing each object to be stored in a preset data pool.

2. The method according to claim 1, characterized in that, The determining a plurality of objects to be stored based on at least one aggregated object corresponding to each object data includes: Sequentially taking the data blocks of each category as a batch of data blocks to be aggregated according to the aggregation priority of each category; Determining the first batch of aggregated objects corresponding to each first batch of data blocks to be aggregated based on at least one aggregated object corresponding to each object data, and the aggregation priority of the category corresponding to each first batch of data blocks to be aggregated is the highest; For non-first-batch data blocks to be aggregated, in response to there being a target aggregated object with free space among the aggregated objects obtained by aggregating the previous batch of data blocks to be aggregated, filling the target aggregated object with each data block to be aggregated in this batch to obtain the aggregated object corresponding to this batch; or, in response to there being no target aggregated object among the aggregated objects obtained by aggregating the previous batch of data blocks to be aggregated, performing an aggregation process on the data blocks to be aggregated in this batch to obtain the aggregated object corresponding to this batch; Taking the aggregated object corresponding to the last batch of data blocks to be aggregated as the plurality of objects to be stored.

3. The method according to claim 2, wherein The determining the first batch of aggregated objects corresponding to each first batch of data blocks to be aggregated based on at least one aggregated object corresponding to each object data includes: Determining the aggregation state of each aggregated object based on the relationship between the size of at least one aggregated object corresponding to each object data and the aggregation size, where the aggregation state includes reaching the full aggregation state or not reaching the full aggregation state; For each aggregated object, in response to the aggregation state of the aggregated object being the reaching the full aggregation state, taking the aggregated object as the first batch of aggregated objects corresponding to the first batch of data blocks to be aggregated; or, In response to the aggregation state of at least one candidate aggregated object being not reaching the full aggregation state, determining the first batch of aggregated objects corresponding to the first batch of data blocks to be aggregated based on each candidate aggregated object, where the candidate aggregated object is at least one of the aggregated objects.

4. The method according to any one of claims 1 to 3, characterized in that The storing each object to be stored in a preset data pool includes: Performing the following steps for each object to be stored: Determining the waiting storage state of the object to be stored based on the size of the object to be stored, where the waiting storage state of the object to be stored includes entering the waiting storage state or not entering the waiting storage state; In response to the waiting storage state of the object to be stored being the entering the waiting storage state, storing the object to be stored in the preset data pool.

5. The method according to claim 4, wherein The determining the waiting storage state of the object to be stored based on the size of the object to be stored includes: Determine whether the size of the object to be stored is equal to the threshold size; In response to the size of the object to be stored being equal to the threshold size, determine the waiting storage state of the object to be stored as the entering waiting storage state; or, In response to the size of the object to be stored not being equal to the threshold size, determine whether the aggregation time of the object to be stored reaches the preset time; In response to the aggregation time of the object to be stored reaching the preset time, determine the waiting storage state of the object to be stored as the entering waiting storage state.

6. The method according to claim 4, characterized in that The step of storing the object to be stored in the preset data pool in response to the waiting storage state of the object to be stored being the entering waiting storage state includes: In response to the waiting storage state of the object to be stored being the entering waiting storage state, obtain the number of read times of the object to be stored in a preset time period; In response to the number of read times of the object to be stored in the preset time period not reaching the number threshold, store the object to be stored in the preset data pool.

7. The method according to any one of claims 1 to 3, characterized in that, After determining a number of objects to be stored based on at least one aggregated object corresponding to each object data, the method further includes: Record the pre-aggregation information, the storage location, and the distribution information of data blocks of each object to be stored in a preset information set, so as to read each object to be stored based on the preset information set.

8. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining a number of data blocks of at least one object data includes: Obtain a number of initial data blocks of at least one object data and the segmentation parameters of each object data, where the segmentation parameters include the cutting block parameter and the cutting strip parameter when each object data is segmented; Take the minimum value in the segmentation parameters of each object data as the division threshold; Determine the initial data blocks that are less than or equal to the division threshold among the number of initial data blocks as the number of data blocks.

9. An electronic device, characterized in that, including: A memory and a processor, where the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1-8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they are used to implement the method according to any one of claims 1-8.

Images