Storage method of object data, electronic device, and storage medium

Abstract

The application discloses a kind of object data storage method, electronic equipment and storage medium.The object data storage method includes: obtaining several data blocks of at least one object data;Determine the category of each data block based on the position of each data block in each object data;For each object data, in response to the number of target data blocks belonging to the highest priority category in the object data is greater than or equal to the preset number, aggregate each target data block to obtain at least one aggregated object of the object data;Determine several to be stored objects based on at least one aggregated object corresponding to each object data, and store each to be stored object to the preset data pool.The above scheme can ensure that the data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects, thereby improving the storage efficiency of object data.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a method for storing object data, an electronic device, and a storage medium. Background Technology

[0002] Ceph, a widely adopted distributed file storage system, is an automatically rebalancing and recoverable distributed file storage system that provides block device storage, file system storage, and object storage capabilities. Ceph provides an object storage interface through its Rados Gateway (RGW), offering users convenient and fast object data operations. Users can write and delete object data to and from the Object Storage Device (OSD) via the RGW. In scenarios where object data is stored on a Rados device, the RGW gateway first splits the object data into multiple Rados objects of varying sizes, creating a natural case-insensitive situation. Typically, these smaller Rados objects are directly aggregated, potentially aggregating Rados objects from the same object data into different larger Rados objects. This results in overly scattered and unpredictable large Rados objects, leading to excessive computational resources consumed during subsequent operations or retrieval of the entire object data, increasing the pressure on the storage system.

[0003] Given the shortcomings of existing technologies, how to provide a solution to improve the storage efficiency of object data is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides at least one method for storing object data, an electronic device, and a storage medium.

[0005] This application provides a method for storing object data, including: obtaining a plurality of data blocks of at least one object data; determining the category of each data block based on its position in each object data; for each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the object data being greater than or equal to a preset number, aggregating each target data block to obtain at least one aggregated object of the object data; determining a plurality of objects to be stored based on the at least one aggregated object corresponding to each object data, and storing each object to be stored in a preset data pool.

[0006] This application provides a storage device for object data, including: an acquisition module, a first determination module, an aggregation module, and a second determination module; the acquisition module is used to acquire a plurality of data blocks of at least one object data; the first determination module is used to determine the category of each data block based on the position of each data block in each object data; the aggregation module is used to aggregate each target data block for each object data in response to the number of target data blocks in the object data belonging to the category with the highest aggregation priority being greater than or equal to a preset number, to obtain at least one aggregated object of the object data; the second determination module is used to determine a plurality of objects to be stored based on the at least one aggregated object corresponding to each object data, and to store each object to be stored in a preset data pool.

[0007] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described method for storing object data.

[0008] This application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described method for storing object data.

[0009] The above scheme determines the category of each data block based on its position within each object data. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority being greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data. This ensures that data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, several objects to be stored are obtained, making each object to be stored a more suitable object for storage. Each object to be stored is stored in a preset data pool, thereby making the distribution of data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0012] Figure 1 This is a flowchart illustrating an embodiment of the data storage method for the object of this application. Figure 1 ;

[0013] Figure 2This is a flowchart illustrating an embodiment of the data storage method for the object of this application. Figure 2 ;

[0014] Figure 3 This is a flowchart illustrating an embodiment of the data storage method for the object of this application. Figure 3 ;

[0015] Figure 4 This is a schematic diagram of the structure of an embodiment of a storage device for the object data of this application;

[0016] Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0017] Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0020] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0021] This application provides methods and devices for storing object data. The application scenarios for these object data storage methods include, but are not limited to, object data storage scenarios. The execution entity of the object data storage method can be an object data storage device or a server capable of storing object data. For example, the object data storage device can be located within a terminal device, server, or other processing device. The terminal device can be a device for storing object data, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, etc. In some possible implementations, the object data storage method can be implemented by a processor calling computer-readable instructions stored in memory.

[0022] Object storage is a data storage architecture that stores data as "objects," unlike traditional file systems or block storage. In object storage, each object contains the data itself, metadata, and a unique identifier. The metadata within each object describes its attributes and information. Ceph object storage is based on its own RADOS distributed object storage architecture, and the RGW (Ceph Object Gateway) built on top of it allows for easy migration of Amazon S3 and OpenStack Swift objects to the distributed file storage system Ceph. It's understood that object data includes, but is not limited to, Amazon S3 objects and OpenStack Swift objects.

[0023] For example, during the upload of object data, the object storage gateway RGW can be used to segment the object data, and then the segmented object data can be uploaded to a distributed file storage system. The method of uploading object data varies depending on how the object storage gateway segments the object data. For any given object data, the object data is segmented according to segmentation parameters. Segmentation parameters include chunk parameters and stripe parameters. The segmentation parameters for different object data can be the same or different. For different segmentation methods, the settings of the segmentation parameters can be the same or different. The chunk parameter can be `chunk_size`, which represents the RGW chunk size. The stripe parameter can be `stripe_size`, which represents the RGW stripe size. The object data to be segmented can be an S3 object.

[0024] In some application scenarios, when the object data is split as a whole, the upload method is also a whole upload. Specifically, if the size of the object data is larger than the cutting block parameter, based on the cutting block parameter and the cutting strip parameter in the overall splitting parameters, a header object is first split from the object data according to the size of the cutting block parameter. Then, the remaining objects are evenly split according to the cutting strip parameter to obtain at least one strip object. It can be understood that the size of the last strip object in the above-mentioned even splitting of the remaining objects according to the cutting strip parameter may be less than or equal to the cutting strip parameter. If the size of the last strip object is less than the cutting strip parameter, it can be a small object. In this case, for any object data in the overall splitting, the number of small objects is 1. In other application scenarios, if the size of the object data is less than or equal to the minimum of the cutting block parameter and the cutting strip parameter, the entire object data is directly treated as a small object. In this case, for any object data in the overall splitting, the number of small objects is 1.

[0025] In other application scenarios, when the object data is segmented, the upload method is segmented upload. Specifically, a header object of size 0 is first generated for the object data; this header object contains the metadata of the object data. The segmentation parameters also include a part parameter, which can be `part_size`, representing the user's setting of the size of each part of the object data during segmentation. The part parameter can be a fixed value or a range of sizes, including the smallest to the largest part. Then, the object data is segmented into at least one part object according to the part parameter. The size of each part object can be represented as `part_size`. For each part object, if the size of the part object is smaller than the segmentation strip parameter, the part object is directly treated as a small object. In this case, for any object data in the segmentation process, the number of small objects obtained from the segmentation is equal to the number of part objects, and there is at least one small object.

[0026] In other application scenarios, for each part of the object, if the size of the part is greater than or equal to the stripe parameter, the object is uniformly divided according to the stripe_size parameter. The first block obtained from each part is called a multipart object, and the remaining parts of the part are divided into at least one shadow object. It is understood that the size of each shadow object is less than or equal to the stripe parameter. For each part of the object, the size of the last shadow object obtained from that part may be less than or equal to the stripe parameter. If the size of the last shadow object is less than the stripe parameter, it can be a small object. In this case, for any object data in the segmented partitioning, the number of small objects is less than or equal to the number of parts, and there is at least one small object.

[0027] It is understandable that object storage is a storage method that stores data in units of object storage, typically used to store unstructured data, meaning the object data type is unstructured, such as documents, images, audio, and video. The object data corresponds to the data of the target server in the target cluster, which includes at least one server. The target cluster corresponds to at least one preset data pool. Each preset data pool is used to store data blocks related to the object data. The importance of the data blocks stored in each preset data pool decreases sequentially. Each preset data pool includes a cache pool and at least one other data pool. The object data corresponding to the data blocks stored in the cache pool has the highest importance; that is, the object data stored in the cache pool is hot data. Each preset data pool is a shared data pool in the target cluster, and each server in the target cluster can store object data in its respective preset data pool. At least one initial data block obtained by using the RGW gateway to segment the object data using the above segmentation method can first be stored in the cache pool of the target cluster.

[0028] Based on this, during the storage of object data, in a scenario where S3 object data is connected to RADOS, the RGW gateway first divides the S3 object into multiple initial data blocks. These initial data blocks can be represented as RADOS objects. For any given object data, at least one initial data block obtained through overall or segmented partitioning includes not only large data blocks satisfying the partitioning parameters (block size, strip size, and portion size), but also the aforementioned smaller objects. In this case, the RGW gateway may create a natural mixed-case scenario when partitioning the object data, meaning at least one initial data block contains a mixture of large and small objects. When storing object data in preset data pools, the initial data blocks are typically randomly aggregated into fixed-size blocks. This may result in smaller objects from the same S3 object being aggregated into different larger objects, leading to additional overhead when storing and / or retrieving the same object data, thus increasing the pressure on the storage system.

[0029] To address this, this application provides a method for storing object data. Based on the position of each data block within each object data, the category of each data block is determined. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority being greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data. This ensures that data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, several objects to be stored are obtained, making each object to be stored a more suitable object for storage. Each object to be stored is stored in a preset data pool, thereby making the distribution of data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of object data.

[0030] For example, for any object data, the initial number of data blocks is 10, of which 5 initial data blocks belong to the aforementioned small objects. In the case of random aggregation, the aforementioned small objects may be aggregated into 5 aggregate objects respectively. In this way, when large object data is split and stored, it may be aggregated into multiple unpredictable aggregate objects. However, this application responds to the fact that the number of target data blocks in the object data belonging to the category with the highest aggregation priority is greater than or equal to a preset number, and aggregates each target data block to obtain at least one aggregate object of the object data. This can ensure that data blocks belonging to the same object data are more concentratedly aggregated into fewer aggregate objects, and the number of aggregate objects obtained from the aforementioned small object aggregation may be less than 5. This application is aimed at the scenario of mixed size writing of S3 objects. S3 objects can represent object data. By managing at least one initial data block of mixed size generated after splitting to obtain at least one aggregate object, several storage objects are determined. This can alleviate the problem that large object data may be aggregated into multiple unpredictable aggregate objects when split and stored, while also taking into account the aggregation of small objects, thus improving the performance of mixed size writing of object data splitting. This can reduce the pressure on the storage system when performing subsequent operations on the stored objects.

[0031] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the data storage method for the object of this application. Figure 1 Specifically, methods for storing object data may include the following steps:

[0032] Step S11: Obtain several data blocks of at least one object data.

[0033] At least one object data can be a single object data or multiple object data. The RGW gateway is used to segment each object data to obtain several initial data blocks. The methods for segmenting the object data include, but are not limited to, the overall segmentation or the segmented segmentation described above. In some application scenarios, several data blocks can be several initial data blocks of each object data. Specifically, several data blocks include, but are not limited to, the header object and at least one stripe object in overall segmentation, small objects corresponding to the entire object data obtained due to the small size of the object data in overall segmentation, the header object, multiple parts of each part object, at least one shadow object in each part object in segmented segmentation, and small objects directly corresponding to the overall small size of the part objects in segmented segmentation, etc.

[0034] In other application scenarios, the aforementioned data blocks can be data blocks obtained by filtering several initial data blocks of each object data using a preset filter. The preset filtering method can be based on the relationship between the size of each initial data block and a preset size. The preset size can be a fixed value set according to the storage requirements of the object data, or it can be determined according to the segmentation parameters of the RGW gateway. Specifically, the data blocks include, but are not limited to, the last stripe object during overall segmentation, small objects corresponding to the entire object data obtained due to the small size of the object data during overall segmentation, the head object during segmentation, and the last shadow object in each part of the object, etc.

[0035] Step S11 above can be achieved by dividing the object data of the RGW gateway into several initial data blocks, which are then directly used as the aforementioned several data blocks. Alternatively, step S11 can be achieved by filtering the several initial data blocks of the object data according to a preset filtering method to obtain the aforementioned several data blocks. The several data blocks are at least a portion of the several initial data blocks.

[0036] Step S12: Determine the category of each data block based on its position in the object data.

[0037] When several data blocks are several initial data blocks, step S12 only determines the category of at least a portion of the several initial data blocks that meet the size conditions. In this application, the several data blocks in step S11 above are several initial data blocks after being filtered by a preset method. The several data blocks include, but are not limited to, the last strip object when the whole is split, the small object corresponding to the whole object data obtained when the object data is small due to its small size, the head object and the last shadow object in each part of the object when the segment is split.

[0038] The position of a data block within an object data set can be considered as a relative position between the data block obtained from a complete or segmented partition and the object data to which it belongs. Relative positional relationships include four types: Type 1, Type 2, Type 3, and Type 4. Type 1 relative position indicates that, during segmented partitioning, the data block is located at the end position corresponding to the last shadow object within the object data to which it belongs. Type 2 relative position indicates that, during segmented partitioning, the data block is located at the beginning position corresponding to the head object within the object data to which it belongs. Type 3 relative position indicates that, during complete partitioning, the data block is located in all positions within the object data to which it belongs; that is, a data block in Type 3 relative position is a small object corresponding to the entire object data obtained during complete partitioning due to its smaller size. Type 4 relative position indicates that, during complete partitioning, the data block is located at the end position corresponding to the last stripe object within the object data to which it belongs.

[0039] Each data block is categorized into four categories: the first category corresponding to the first category, the second category corresponding to the second category, the third category corresponding to the third category, and the fourth category corresponding to the fourth category.

[0040] In some application scenarios, step S12 above can be used to determine the category of each data block based on its relative position to the object data to which it belongs. In other application scenarios, step S12 above can be used to determine the category of each data block based on its segmentation method and the relationship between the data block and the target object in the object data to which it belongs. The target object can be one of the following: the last shadow object, the head object, the overall object data, or the last strip object. Specifically, in response to the data block being segmented and being the last shadow object among the various parts of the object data to which it belongs, the data block is determined to be in the first category. In response to the data block being segmented and being the head object in the object data to which it belongs, the data block is determined to be in the second category. In response to the data block being segmented as a whole and being a part corresponding to the overall object data to which it belongs, the data block is determined to be in the third category. In response to the data block being split as a whole and the data block being the last stripe object in the object data to which the data block belongs, the data block is classified as the fourth category.

[0041] Before step S13, the aggregation priority of each data block is sorted according to a preset sorting rule based on its category. The preset sorting rule can be determined by the partitioning method, the partitioning method and the target object, or it can be pre-set according to configuration instructions in the configuration interface. In some application scenarios, the aggregation priority of data blocks belonging to segmented partitions is higher than that of data blocks belonging to overall partitions. In other application scenarios, the aggregation priority of the last shadow object in the target object is higher than that of other objects in the target object. In still other application scenarios, the aggregation priority of data blocks in the third category can be equal to that of data blocks in the fourth category. For example, this application uses the example where the aggregation priorities of each data block, from highest to lowest, are data blocks in the first category, the second category, the third category, and the fourth category.

[0042] Determine whether the number of data blocks belonging to the same object and of the first category is greater than or equal to a preset number.

[0043] Step S13: For each object data, in response to the number of target data blocks in the object data belonging to the category with the highest aggregation priority being greater than or equal to a preset number, aggregate each target data block to obtain at least one aggregated object of the object data.

[0044] Each target data block is a data block belonging to the same object data that falls into the first category. Each target data block is also a data block whose size is smaller than a preset size, located in the last shadow object obtained after segmenting the same object data. The preset number can be dynamically set according to the storage precision of the object data. The preset number can be greater than or equal to 2. For example, the preset number is 2.

[0045] For each object data, if the number of target data blocks is greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data. Specifically, at least a portion of each target data block is sequentially aggregated into an aggregated object of a fixed size. First, each target data block is treated as a data block to be aggregated. At least a portion of each data block to be aggregated is aggregated into at least one aggregated object. The above-mentioned aggregation of at least a portion of each data block to be aggregated into at least one aggregated object includes:

[0046] In response to the aggregation of at least a portion of the data in each data block to be aggregated into a fixed-size aggregate object, the remaining data blocks to be aggregated in each data block are used as new data blocks to be aggregated. Specifically, it is determined whether the first N data blocks to be aggregated can be aggregated into a fixed-size aggregate object. If so, the successfully aggregated aggregate object is used as the first candidate object, and the (N+1)th data block to be aggregated and subsequent data blocks to be aggregated are used as new data blocks to be aggregated. Each new data block to be aggregated is aggregated according to a fixed size until all target data blocks belonging to the same object data are aggregated. During the aggregation process, the aggregation method can be based on the distance of each data block to be aggregated from the head object of the object data, or it can be based on the size order. In the process of aggregating according to the size order, the target data block to be aggregated with the largest size among the data blocks to be aggregated is aggregated first, and then at least one data block to be aggregated whose size sums with the target data block to be aggregated is equal to the fixed size is searched in descending order of size.

[0047] Specifically, if the aggregated object obtained by aggregating the first N objects to be aggregated is smaller than a fixed size, and the aggregated object obtained by aggregating the first N+1 objects to be aggregated is larger than a fixed size, the aggregated object obtained by aggregating the first N objects to be aggregated is considered as a candidate object. The N+1th object to be aggregated and subsequent objects to be aggregated are considered as new objects to be aggregated. Each new object to be aggregated is aggregated according to a fixed size until all target data blocks belonging to the same object data are aggregated, resulting in all candidate objects, which are then used as aggregated objects. It is possible that some of the candidate objects obtained from the aggregation process have a size smaller than the fixed size. It is understood that this application does not limit the specific aggregation method during the aggregation process, that is, the order of the data blocks to be aggregated is not limited.

[0048] For example, the A object data is segmented, resulting in one header object and four part objects based on the aforementioned segmentation parameters. If the segmentation parameter for the cutting strip is 4M, the segmentation parameter for each part is greater than 4M, and the parameters for each segment are at least partially different. The sizes of the last shadow objects in each part object are 1M, 3M, 3M, and 4M, respectively. The preset size is 2M. At this time, there are three data blocks belonging to the first category in the A object data: the last shadow objects of sizes 1M, 3M, and 3M. These are the target data blocks with the highest aggregation priority belonging to the A object data, namely the last shadow objects of sizes 1M, 3M, and 3M. The preset number is 2. The fixed size in the above aggregation process can be set to 4M. The last shadow objects of sizes 1M, 3M, and 3M are respectively taken as objects to be aggregated. Aggregating each object to be aggregated belonging to the A object data according to the above aggregation method yields two candidate objects with sizes of 4M and 3M, respectively. These two candidate objects are then treated as aggregate objects belonging to the data of object A.

[0049] Step S14: Based on at least one aggregated object corresponding to each object data, determine several objects to be stored, and store each object to be stored in a preset data pool.

[0050] The object to be stored represents an object that can be stored in a preset data pool. The object to be stored can be the aggregate object from step S13 above, or an aggregate object populated with the aggregate object from step S13 above. The method for populating the aggregate object from step S13 above can be to fill any aggregate object of object data obtained in step S13 with data blocks whose aggregation priority is lower than the highest aggregation priority, resulting in a new aggregate object.

[0051] Step S14 above can be used to first determine the aggregation state of each aggregation object based on the relationship between the size of each aggregation object and the threshold size. The aggregation state can be either full aggregation state or not full aggregation state. Reaching full aggregation state means that the size of the aggregation object is equal to the threshold size. Not reaching full aggregation state means that the size of the aggregation object is less than the threshold size. The threshold size can be greater than or equal to the aforementioned fixed size. This application takes the threshold size being equal to the aforementioned fixed size as an example. For each aggregation object, in response to the aggregation state of the aggregation object being full aggregation state, the aggregation object is treated as a storage object. Or, for each aggregation object, in response to the aggregation state of the aggregation object not reaching full aggregation state, the aggregation object is treated as a target aggregation object. Data blocks based on the target aggregation priority are used to fill the target aggregation object to obtain a new aggregation object, and the new aggregation object is treated as a storage object. The target aggregation priority is an aggregation priority lower than the highest aggregation priority. In some application scenarios, each storage object is directly stored in a preset data pool. In other application scenarios, for each storage object, in response to the size of the storage object being equal to the storage size, each storage object is stored in a preset data pool. The storage size can be the aforementioned threshold size. In other application scenarios, for each object to be stored, in response to the object's lifetime equaling a threshold time, each object is stored in a preset data pool. The lifetime of an object to be stored represents the time period from the start time to the end time. The start time of an object's lifetime can be the time when the aggregation of the highest-priority data block within that object is completed. The end time of an object's lifetime can be the time at which the step determining whether the object's lifetime equals the threshold time occurs. It is understood that the threshold size, storage size, and threshold time can be dynamically set according to the storage needs of the object data.

[0052] It is understandable that the RGW gateway can adapt to object data accessing a Ceph storage cluster. The RGW gateway, based on the user's write method (whether it's a whole upload or a segmented upload), and in conjunction with the Ceph storage cluster's settings for chunk_size and stripe_size, performs overall or segmented splitting of the accessed object data to obtain several initial data blocks. This results in scenarios where initial data blocks of mixed sizes are generated and written to the Ceph storage cluster. Especially when object data is written with mixed initial data block sizes, the Ceph storage cluster faces a more complex scenario of mixed initial data block writes. This application processes the mixed-size initial data blocks generated after splitting, allowing the objects to be aggregated to be written to a high-speed device, i.e., a cache pool. From the perspective of belonging to the same object data, the first category of target data blocks generated by splitting are preferentially aggregated, improving the performance of mixed-size writes after object data is split and converted into initial data blocks. It is understood that the Ceph storage cluster can be the aforementioned target cluster.

[0053] In other application scenarios, after step S13 above, this application may further include the following steps: For each object data, the storage location of at least one aggregate object of the object data and the distribution information of data blocks in each aggregate object are recorded in a preset information set so as to operate the object data according to the preset information set. The preset information set may be the index set corresponding to the header object in the object data. Operating the object data includes operations such as reading and deleting the object data as a whole. For example, when creating a storage pool for object data, a cache pool is created on a high-speed device. Users can select a high-speed device as the cache pool according to their own resource conditions. The high-speed device may be a solid-state drive or a new type of hard drive such as PMEM.

[0054] The above scheme determines the category of each data block based on its position within each object data. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority being greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data. This ensures that data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, several objects to be stored are obtained, making each object to be stored a more suitable object for storage. Each object to be stored is stored in a preset data pool, thereby making the distribution of data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0055] In some embodiments, step S11 may include the following steps: First, obtain several initial data blocks of at least one object data and the segmentation parameters of each object data. The segmentation parameters include the cutting block parameters and cutting strip parameters when segmenting each object data. Then, use the minimum value among the segmentation parameters of each object data as the partitioning threshold. Next, determine the initial data blocks that are less than or equal to the partitioning threshold as several data blocks.

[0056] At least one object data can be a single object data or multiple object data. If there is only one object data, it can be segmented. The segmentation parameters can also include parameters specifying the cutting portion for segmenting each object data.

[0057] When the aforementioned data blocks can be data blocks obtained by filtering several initial data blocks of each object's data, the preset size in the preset filtering method is determined based on the segmentation parameters. Specifically, the segmentation threshold is the preset size. In some application scenarios, the preset filtering method may determine several initial data blocks that are less than or equal to the segmentation threshold as several data blocks. In other application scenarios, several initial data blocks that are greater than the segmentation threshold are determined as filtered data blocks. The filtered data blocks are directly stored in a preset data pool according to a preset filtering period. The minimum value in the segmentation parameters can be the value corresponding to the block cutting parameter or the value corresponding to the strip cutting parameter. In other application scenarios, when the block cutting parameter and the strip cutting parameter are equal, the segmentation threshold can be the value corresponding to the block cutting parameter and the value corresponding to the strip cutting parameter.

[0058] For example, at least one object data includes object data A, object data B, object data C, and object data D. Each object data is segmented according to segmentation parameters to obtain at least one initial data block corresponding to each object data. Object data A and object data are segmented in segments. Object data C and object data are segmented as a whole. The initial data block of object data A can be 1 head object, 4 multipart objects, and 8 shadow objects. The initial data block of object data D can be 1 head object, 3 multipart objects, and 3 shadow objects. If the size of object data C is less than or equal to the minimum value between the segmentation block parameter and the segmentation strip parameter, object data C is treated as a single small object; that is, the initial data block of object data C is the data block corresponding to the entire object data C. The initial data block of object data D can be 1 head object and 4 strip objects. Several initial data blocks are filtered according to a segmentation threshold to obtain several data blocks. The segmentation threshold is the minimum value between the segmentation block parameter and the segmentation strip parameter. For example, several data blocks include the header object and three tail shadow objects of object A, the header object and three tail shadow objects of object B, the data block corresponding to the entire data of object C, and the tail stripe object corresponding to the data of object D. For example, after step S12 above, the data blocks belonging to the first category are the three tail shadow objects corresponding to object A and the three tail shadow objects corresponding to object B. The data blocks belonging to the second category are the header objects of object A and the header objects of object B. The data blocks belonging to the third category are the data blocks corresponding to the entire data of object C. The data blocks belonging to the fourth category are the tail stripe objects corresponding to object D.

[0059] It is understandable that a small file threshold is set for the RGW gateway. In actual implementation, the RGW splits object data according to the chunk_size or stripe_size parameter, and the sizes of these two parameters can be dynamically configured. In this application, the small file threshold, i.e., the partitioning threshold, is set to the minimum value among the above partitioning parameters by default. The partitioning threshold (SOT) can be represented as min(chunk_size, stripe_size). A fixed size is set during the aggregation process, which represents the expected size of the aggregated object. For example, the above fixed size can be set to 4M by default, and this value can be configured according to the actual situation. It is understood that this application uses 4M as an example and does not limit the specific value of the above fixed size.

[0060] After object data is written to the RGW gateway, the RGW gateway performs object data segmentation. The segmentation rules remain consistent with the original. In some application scenarios, for whole uploads, the RGW gateway performs overall segmentation. The RGW gateway first segments a header object (i.e., a head object) according to the chunk_size parameter. This header object contains the metadata of the entire object data and some of the actual data in the object data. Then, the other data in the object data, excluding the header object, is evenly segmented according to stripe_size to obtain individual stripe objects, until the last stripe object of the object data is obtained, at which point the even segmentation ends. The header object and each stripe object are used as the initial data block. In other application scenarios, for segmented uploads, the RGW gateway performs segmented segmentation. The RGW gateway first generates a header object of size 0 (i.e., a head object), which records the metadata of the object data. Then, according to the user-selected upload segmentation parameter part_size, it segments the data into at least one part object. For each part of the object, it is uniformly divided according to the stripe_size parameter to obtain at least one initial data block. The first initial data block uniformly divided within each part of the object is called a multipart object, and the initial data blocks obtained by splitting the data outside the multipart object are called shadow objects. According to a preset splitting rule, the size of all initial data blocks is compared with the partitioning threshold SOT. Initial data blocks with a size less than or equal to the partitioning threshold SOT are considered data blocks and written to the cache pool. Initial data blocks with a size greater than the partitioning threshold SOT are directly written to the preset data pool. It is understood that all initial data blocks can be RADOS objects.

[0061] This application can be considered to optimize scenarios where object data is accessed to a Ceph storage cluster. From the perspective of object data, it aggregates data blocks in the object data processed by the RGW gateway. Compared to directly storing the overly scattered initial data blocks, this application makes accessing object data to the Ceph storage cluster more efficient. Furthermore, for S3 mixed-size write scenarios, it prioritizes the aggregation of data blocks in the object data processed by the RGW gateway according to the highest aggregation priority, improving the efficiency of object data operations in mixed-size write scenarios, thereby increasing the efficiency of storing object data.

[0062] In some embodiments, the step of determining a plurality of objects to be stored based on at least one aggregated object corresponding to each object data may include the following steps: First, according to the aggregation priority of each category, the data blocks of each category are sequentially treated as a batch of data blocks to be aggregated. Then, based on at least one aggregated object corresponding to each object data, the first batch of aggregated objects corresponding to each batch of data blocks to be aggregated are determined, with the category corresponding to the first batch of data blocks to be aggregated having the highest aggregation priority. Next, for data blocks not in the first batch of data blocks to be aggregated, in response to the existence of a target aggregated object with free space in the aggregated objects obtained from the aggregation of the previous batch of data blocks to be aggregated, the target aggregated object is filled using each data block to be aggregated in this batch to obtain the aggregated object corresponding to this batch. Alternatively, in response to the absence of a target aggregated object in the aggregated objects obtained from the aggregation of the previous batch of data blocks to be aggregated, the data blocks to be aggregated in this batch are aggregated to obtain the aggregated object corresponding to this batch. Finally, the aggregated objects corresponding to the last batch of data blocks to be aggregated are treated as a plurality of objects to be stored.

[0063] The aggregation priorities of each category can be dynamically set according to the priority configuration instructions generated by the configuration interface. This application takes as an example that the aggregation priorities of each data block can be divided into four categories, from high to low: the first category, the second category, the third category, and the fourth category.

[0064] The data blocks to be aggregated are categorized by aggregation priority from highest to lowest as follows: the first batch of data blocks corresponding to the first category, the second batch corresponding to the second category, the third batch corresponding to the third category, and the fourth batch corresponding to the fourth category. The first batch of data blocks to be aggregated is the first batch of aggregated data blocks. The data blocks not in the first batch are designated as the second, third, and fourth batches, respectively.

[0065] The aforementioned initial batch of data blocks to be aggregated are data blocks belonging to the first category. The step of determining the first batch of aggregation objects corresponding to each initial batch of data blocks to be aggregated based on at least one aggregation object corresponding to each object data can be to directly use each aggregation object among the at least one aggregation object corresponding to each object data as the first batch of aggregation objects corresponding to each initial batch of data blocks to be aggregated.

[0066] The target aggregation object with free space can be an aggregation object from the previous batch of data blocks to be aggregated whose size is smaller than the aggregation size. That is, the target aggregation object is an aggregation object with empty spaces. It is understood that in this application, the size of the data block is used to represent the amount of memory space occupied by the data block.

[0067] In some application scenarios, for non-initial batches of data blocks to be aggregated, if there are target aggregation objects with free space in the aggregation objects obtained from the previous batch of data blocks to be aggregated, the target aggregation objects are filled using the data blocks to be aggregated in the current batch to obtain the corresponding aggregation objects for the current batch. Specifically, the number of target aggregation objects with free space can be one or more. For each target aggregation object, the target aggregation object is filled using the data blocks to be aggregated in the current batch until the size of the new aggregation object reaches the aggregation size, or until the data blocks to be aggregated in the current batch cannot fill the target aggregation object and the size of the new aggregation object corresponding to the filled target aggregation object is smaller than the aggregation size. Then, the remaining data blocks to be aggregated in the current batch are aggregated to obtain aggregation objects. The aggregation objects obtained by aggregating the aggregation objects other than the target aggregation object from the aggregation objects obtained from the previous batch of data blocks to be aggregated, the new aggregation object corresponding to the target aggregation object obtained in the current batch, and the remaining data blocks to be aggregated in the current batch are taken as the aggregation objects corresponding to the current batch. Among them, the remaining data blocks to be aggregated in the current batch are the data blocks to be aggregated in the current batch that have not been filled with target aggregation objects. Specifically, the remaining data blocks to be aggregated in this batch can be aggregated separately into different aggregation objects, or the remaining data blocks to be aggregated can be aggregated into at least one aggregation object according to the aggregation size. For example, the aggregation objects corresponding to this batch include aggregation objects from the previous batch that do not have free space, new aggregation objects obtained by filling target aggregation objects from the previous batch that have free space, aggregation objects from the previous batch that have free space but cannot be filled, and aggregation objects obtained by aggregating the remaining data blocks to be aggregated in this batch.

[0068] In other application scenarios, for data blocks not in the first batch to be aggregated, in response to the absence of a target aggregation object in the aggregation object obtained from the aggregation of the previous batch of data blocks to be aggregated, aggregation processing is performed on the data blocks to be aggregated in the current batch to obtain the corresponding aggregation object for the current batch. The aggregation of the data blocks to be aggregated in the current batch can be performed by aggregating each data block to be aggregated into a separate aggregation object, or by aggregating each data block to be aggregated into at least one aggregation object according to the aggregation size. For example, the aggregation object corresponding to the current batch includes the aggregation object from the previous batch that does not have free space, and the aggregation object obtained by aggregating the data blocks to be aggregated in the current batch.

[0069] It is understandable that, for data blocks that are not in the first batch to be aggregated, the aggregation order of each batch of data blocks to be aggregated is as follows: first, aggregate the second batch of data blocks corresponding to the second category, then aggregate the third batch of data blocks corresponding to the third category, and finally aggregate the fourth batch of data blocks corresponding to the fourth category.

[0070] Because the size of data blocks belonging to the second category is 0, the batch of data blocks to be aggregated corresponding to the data blocks of the second category do not need to participate in the filling or aggregation of non-first batch data blocks to be aggregated. For non-first batch data blocks to be aggregated, the aggregation order of each batch of data blocks to be aggregated is as follows: first aggregate the third batch of data blocks to be aggregated corresponding to the data blocks of the third category, and then aggregate the fourth batch of data blocks to be aggregated corresponding to the data blocks of the fourth category. Among them, the second batch of data blocks to be aggregated corresponding to the data blocks of the second category can record relevant information in the data blocks of the second category belonging to the same object data after the first batch of aggregation objects is determined. The relevant information can be the pre-aggregation information of the first batch of aggregation objects, their storage location, and the distribution information of the data blocks. The relevant information is recorded in the preset information set linked to the data blocks of the second category belonging to the same object data.

[0071] It's understandable that batching the data blocks to be aggregated improves aggregation efficiency, thereby increasing the efficiency of manipulating object data in scenarios where large and small objects are mixed, and ultimately improving the efficiency of storing object data. Furthermore, filling target aggregation objects with available free space allows the size of each aggregation object to approach the aggregate size, resulting in fewer subsequent operations on the aggregation objects. For example, during the degradation storage process of aggregation objects, fewer operations on the aggregation objects can achieve degradation storage of a larger amount of object data.

[0072] In some embodiments, the step of determining the first batch of aggregate objects corresponding to each first batch of data blocks to be aggregated based on at least one aggregate object corresponding to each object data may include the following steps: determining the aggregation state of each aggregate object based on the relationship between the size of at least one aggregate object corresponding to each object data and the aggregation size. The aggregation state includes reaching a full aggregation state or not reaching a full aggregation state. For each aggregate object, in response to the aggregation state of the aggregate object reaching a full aggregation state, the aggregate object is designated as the first batch of aggregate objects corresponding to the first batch of data blocks to be aggregated. Alternatively, in response to the aggregation state of at least one candidate aggregate object not reaching a full aggregation state, the first batch of aggregate objects corresponding to the first batch of data blocks to be aggregated is determined based on each candidate aggregate object. The candidate aggregate objects are at least one of the aggregate objects.

[0073] Each aggregate object of the object data includes at least one first aggregate object and / or at least one second aggregate object. Step S13 above may involve aggregating data blocks belonging to the same object data and of the first category to obtain at least one first aggregate object and / or at least one second aggregate object. The size of each first aggregate object is equal to the aggregate size, and the size of each second aggregate object is smaller than the aggregate size. The first aggregate object is in a fully aggregated state, while the second aggregate object is in a partially aggregated state.

[0074] For each aggregation object, in response to the aggregation state of the aggregation object reaching the full aggregation state, the step of using the aggregation object as the first batch of aggregation objects corresponding to the first batch of data blocks to be aggregated can be to use each first aggregation object as the first batch of aggregation objects corresponding to the first batch of data blocks to be aggregated.

[0075] The above-described step of determining the first batch of aggregation objects corresponding to the first batch of data blocks to be aggregated based on each candidate aggregation object can be as follows: for each second aggregation object, aggregate it using other second aggregation objects until the size of the resulting new aggregation object reaches the aggregation size, or until the second aggregation object can no longer fill other second aggregation objects and the size of the new aggregation object obtained by filling is smaller than the aggregation size, and then use each new aggregation object as the first batch of aggregation objects. It is understandable that compared to directly using all second aggregation objects as the first batch of aggregation objects, performing secondary aggregation among the second aggregation objects results in fewer first batch of aggregation objects, making subsequent filling or aggregation of non-first batch data blocks to be aggregated more efficient.

[0076] For example, each initial data block is filtered before being placed in the cache pool. The cache pool may contain four types of data blocks: In some application scenarios, during segmentation, the last shadow object segmented from the partial object `part_size` of the object data, with a size smaller than the aforementioned segmentation threshold `SOT`, is categorized as a first-class data block. In some application scenarios, the header object with a size of 0 during segmentation is categorized as a second-class data block. In some application scenarios, the object data (including the head portion of the object data) that was not segmented during overall segmentation, with a size smaller than the aforementioned segmentation threshold `SOT`, is categorized as a third-class data block. In some application scenarios, the last stripe object during overall segmentation, with a size smaller than the aforementioned segmentation threshold `SOT`, is categorized as a fourth-class data block.

[0077] For example, in the cache pool, aggregation threads aggregate data blocks of the first, second, third, and fourth categories mentioned above. There can be multiple aggregation threads. The number of aggregation threads can be equal to the number of batches of data to be aggregated. The aggregation method is as follows: In some application scenarios, the aggregation priority of data blocks of the first category is the highest. Data blocks belonging to the first category are processed first, and data belonging to the same object are aggregated into the same aggregation object. For the same object data, during the aggregation process, data blocks of other object data are not allowed to be written in between. Depending on the set aggregation size of the aggregation object, three situations may occur. The first situation is that the aggregation object is smaller than the aggregation size of 4M after aggregating the data blocks of the first category. The second situation is that the aggregation size is exactly equal to 4M after aggregating the data blocks of the first category. The third situation is that the aggregation size is larger than 4M after aggregating the data blocks of the first category. Specifically, for the third situation, assuming that the data blocks of the first category are not split, it is determined whether the first N data blocks of the first category are aggregated to a size of 4M. If the first N data blocks of category 1 are aggregated to a size of 4M, then the (N+1)th data block of category 1 is aggregated into a new aggregation object, and this process continues until all data blocks of category 1 in that object are aggregated. If the first N data blocks of category 1 are aggregated to a size smaller than 4M, then the (N+1)th data block of category 1 is aggregated into a new aggregation object, leaving the previous aggregation object empty, until all data blocks of category 1 in that object are aggregated. Each aggregation object generates aggregated data block distribution information. Each aggregated data block distribution information includes the identifier information of each data block (Aggregation Object ID, AOID).

[0078] In other application scenarios, for the aggregation process of the first category of data blocks, relevant information is added to the header object of the corresponding object data in the second category of data blocks that belong to the same object data, based on the aggregation situation of the first category of data blocks. The distribution information AOID of each aggregated data block and the offset and length of each data block in the aggregation object are recorded in the corresponding header object.

[0079] In other application scenarios, for the second category of data blocks, since this type of data block has no size, that is, it occupies 0 space, and is only used as an index so that the storage bucket can link to the object data corresponding to the second category of data blocks, the second category of data blocks are not clustered according to the aggregation priority.

[0080] In other application scenarios, the aggregation priority of third-category data blocks is the highest, second only to first-category data blocks. For third-category data blocks, firstly, check if the distribution information AOID of the generated aggregated data blocks can accommodate them. If it can, it indicates that the aggregated object obtained from the aggregation of first-category data blocks has gaps. Then, the third-category data blocks are filled into the target aggregated object with available space, resulting in a new aggregated object. The distribution information AOID of the aggregated data blocks corresponding to the target aggregated object with available space is used as the distribution information of the data blocks before aggregation for the new aggregated object. The distribution information AOID of the data blocks before aggregation for the new aggregated object, along with its offset in the distribution information AOID of the data blocks before aggregation, is updated in the header object of the object data corresponding to the third-category data blocks. In other application scenarios, if the aggregated object obtained in step S13 above does not have enough space to store the third category of data blocks, a new aggregated object is created, the third category of data blocks are aggregated into it, and the distribution information AOID of the aggregated data blocks corresponding to the new aggregated object is generated. Then, the metadata of the header object of the object data to which the third category of data blocks belong is modified to record the corresponding aggregated object OID in the information before aggregation, the distribution information AOID of the aggregated data blocks corresponding to the new aggregated object, and the offset and length in the distribution information of the aggregated data blocks.

[0081] In other application scenarios, the aggregation priority of the fourth category of data blocks is the highest, second only to the third category. Finally, the fourth category of data blocks are aggregated. When aggregating this type of fourth category data block, it first checks if there is a target aggregation object with free space in the aggregation object obtained from the previous batch of data blocks to be aggregated. If so, the fourth category data blocks are filled into the target aggregation object with free space, resulting in a new aggregation object. The distribution information AOID of the aggregated data blocks corresponding to the target aggregation object with free space is used as the distribution information of the data blocks before aggregation corresponding to the new aggregation object. The distribution information AOID of the data blocks before aggregation corresponding to the new aggregation object and its offset record in the exclusive AOID of the distribution information of the data blocks before aggregation are updated in the header object of the object data corresponding to the fourth category of data blocks. In other application scenarios, if the aggregated object obtained in step S13 above does not have enough space to store the fourth category of data blocks, a new aggregated object is created, the fourth category of data blocks are aggregated into it, and the distribution information AOID of the aggregated data blocks corresponding to the new aggregated object is generated. Then, the metadata of the header object of the object data to which the fourth category of data blocks belong is modified to record the corresponding aggregated object OID in the information before aggregation, the distribution information AOID of the aggregated data blocks corresponding to the new aggregated object, and the offset and length in the distribution information of the aggregated data blocks.

[0082] It can be considered that this application classifies the data blocks generated after the object data is connected to the Ceph storage cluster, and aggregates the data blocks of different categories in the cache pool according to the aggregation priority, so that the aggregated objects are stored later, which can improve the disk space utilization.

[0083] In some embodiments, after the steps of determining a plurality of objects to be stored based on at least one aggregated object corresponding to each object data, the object data storage method may further include the following steps: recording the pre-aggregation information, storage location, and data block distribution information of each object to be stored into a preset information set, so as to read each object to be stored based on the preset information set.

[0084] The information prior to aggregation may include at least one object data item corresponding to the object to be stored, the method of splitting the corresponding object data, and the initial data blocks included in the corresponding object data. The storage location may be the location of the storage pool where the object to be stored resides. The data block distribution information may refer to the data blocks contained in the object to be stored and the categories of the data blocks.

[0085] In some application scenarios, in response to the distribution information of data blocks in the object to be stored including data blocks of the first category, the pre-aggregation information, storage location, and data block distribution information of the object to be stored are recorded in a preset information set of the header object links of the object data to which the data blocks of the first category belong, so that each object to be stored can be read based on the preset information set. In other application scenarios, in response to the distribution information of data blocks in the object to be stored not including data blocks of the first category, the pre-aggregation information, storage location, and data block distribution information of the object to be stored are recorded in a preset information set at a preset location, so that each object to be stored can be read based on the preset information set.

[0086] In some application scenarios, information related to each object's data is stored in buckets. Each index in a bucket can correspond to a piece of object data. A pre-defined information set can correspond to a bucket.

[0087] Recording the storage location of each object to be stored in a preset information set can be done by updating the header object information of the object data to which each object belongs in the bucket's index, and adding the location identifier information of the header object of the object data in the cache pool or preset data pool to the index information. When reading object data, the storage pool where the header object of the object data is located is obtained through the index in the bucket. The distribution information of the data blocks after the object data is split is obtained and read from the header object. The specific steps for reading object data are as follows: For the part of the object data that has been split but not aggregated, the reading rules remain consistent with the original reading rules, and reading can be performed directly. For the part of the object data that has been split and aggregated, the distribution information of the aggregated data blocks, including the identifier information of each data block and the offset and length of each data block, is obtained from the header object. Based on the offset and length of each data block, the pre-aggregation information corresponding to the aggregated data blocks is read. The distribution information of the aggregated data blocks, including the identifier information of each data block, can be represented as AOID. The pre-aggregation information corresponding to the aggregated data blocks can be represented as OID. The order of several initial data blocks whose names naturally contain object data is read from the names of the aggregated data blocks corresponding to the information before aggregation. This order is consistent with the original rules, and then the object data can be concatenated.

[0088] This application manages aggregated objects to be stored through the index of the aggregated object, which is a preset information set. This involves an indexing process from the bucket index to the header object, then to the aggregated object, and finally to the data blocks obtained by object data partitioning. The bucket index to the header object remains unchanged, while the index from the aggregated object to the data blocks obtained by object data partitioning is added. This increases the number of objects that a single bucket can manage, improving storage capacity and efficiency. Furthermore, while the header object stores metadata, only the aggregated object entry is added to the object data belonging to the aggregated data blocks within the bucket; the rest of the process remains unchanged, resulting in minimal impact on read performance and simple implementation.

[0089] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the data storage method for the object of this application. Figure 2 .

[0090] In some embodiments, the step of storing each object to be stored to a preset data pool may include the following steps: performing the following for each object to be stored: Figure 2 The following steps are shown: Step S21: Determine the waiting storage state of the object to be stored based on its size. The waiting storage state of the object to be stored includes entering the waiting storage state or not entering the waiting storage state. Step S22: In response to the waiting storage state of the object to be stored being entered, store the object to be stored in the preset data pool.

[0091] A pending storage status indicates that an object can be directly stored in the preset data pool. A pending storage status that has not been entered indicates that an object cannot be directly stored in the preset data pool. Objects in the pending storage status can be stored immediately or sequentially in storage batches. Storage batches can consist of a preset number of objects; once a batch is full, objects are stored in the preset data pool as a single batch.

[0092] Step S21 above can determine the waiting storage state of the object to be stored based on the relationship between the object to be stored and the threshold size. The threshold size can be equal to the value of the aggregated size mentioned above. For example, the threshold size can be 4M. In response to the size of the object to be stored being equal to the threshold size, the waiting storage state of the object to be stored is determined to be "entering the waiting storage state". In response to the size of the object to be stored being less than the threshold size, the waiting storage state of the object to be stored is determined to be "not entering the waiting storage state".

[0093] It can be assumed that the storage status of the object to be stored determines whether the object can be directly stored in the preset data pool. During the process of storing data into the preset data pool, the size of the object to be stored is the size that meets the storage conditions, thereby improving storage efficiency.

[0094] In some embodiments, step S21 may include the following steps: determining whether the size of the object to be stored is equal to a threshold size. In response to the size of the object to be stored being equal to the threshold size, determining the waiting storage state of the object to be stored as entering a waiting storage state. Alternatively, in response to the size of the object to be stored not being equal to the threshold size, determining whether the aggregation time of the object to be stored has reached a preset time. In response to the aggregation time of the object to be stored reaching the preset time, determining the waiting storage state of the object to be stored as entering a waiting storage state.

[0095] In some application scenarios, when the size of the object to be stored equals a threshold, the object's waiting-to-store state is determined as entering the waiting-to-store state. In other application scenarios, when the size of the object to be stored does not equal the threshold, it is determined whether the aggregation time of the object to be stored has reached a preset time. The aggregation time of the object to be stored can be the aforementioned lifetime. The preset time can be the aforementioned threshold time. The specific value of the preset time can be dynamically set according to the storage requirements of the object data. For example, the preset time can be 24 hours. The aggregation time of the object to be stored represents the time period from the start time to the end time. If the object to be stored contains a data block with the highest aggregation priority, the start time of the aggregation time of the object to be stored can be the aggregation completion time of the data block with the highest aggregation priority in the object to be stored. If the object to be stored does not contain a data block with the highest aggregation priority, the start time of the aggregation time of the object to be stored can be the aggregation completion time of the entire object to be stored. The end time of the aggregation time of the object to be stored can be the time at which the step of determining whether the aggregation time of the object to be stored equals the preset time occurs. In some application scenarios, when the aggregation time of the object to be stored reaches the preset time, the object's waiting-to-store state is determined as entering the waiting-to-store state. In other application scenarios, if the aggregation time of the object to be stored has not reached the preset time, the object to be stored will wait for the aggregation time to reach the preset time, and the waiting storage state of the object to be stored will be determined as entering the waiting storage state.

[0096] It can be argued that by combining the size of the object to be stored with the aggregation time to determine whether the object meets the storage conditions, the objects to be stored in the preset data pool are more likely to meet the storage conditions, thereby improving the storage efficiency of object data.

[0097] In some embodiments, step S22 may include the following steps: In response to the object to be stored being in a waiting-to-store state, obtaining the number of times the object to be stored has been read within a preset time period. In response to the object to be stored not reaching a threshold number of reads within the preset time period, storing the object to be stored in a preset data pool.

[0098] The read count represents the number of times the target cluster reads any part of the data blocks in the object to be stored. For example, if the object to be stored contains at least some data blocks of multiple object data, reading any one of the contained object data will count as one read of the object to be stored. The preset time period is the time period corresponding to the determination of the usage of the object to be stored. The preset time period can be dynamically set according to the storage requirements of the object data. For example, the preset time period can be the time period in which the above-mentioned aggregation time falls.

[0099] Determine whether the number of reads of the object to be stored within a preset time period has reached a threshold. If the number of reads of the object to be stored within the preset time period reaches the threshold, mark the data status of the object to be stored as hot data.

[0100] If the number of reads of an object to be stored within a preset time period does not reach a threshold, the object will be directly stored in a preset data pool. Specifically, if the number of reads of an object to be stored within a preset time period does not reach a threshold, the data status of the object to be stored will be marked as non-hot data, and the object will be stored in the preset data pool.

[0101] It can be assumed that the number of times the object to be stored is read within a preset time period determines whether the data is hot data. The objects to be stored corresponding to hot data are kept in the cache pool, while the objects to be stored that are not hot data are stored in the preset data pool. This allows the cache pool to contain objects to be stored that are used more frequently, while the preset data pool contains objects to be stored that are used less frequently, thus achieving data state partitioning and thus realizing degraded storage between the cache pool and the preset data pool.

[0102] Please see Figure 3 , Figure 3 This is a flowchart illustrating an embodiment of the data storage method for the object of this application. Figure 3 .

[0103] The above-mentioned step of determining whether the aggregation time of the object to be stored has reached a preset time in response to the size of the object not being equal to the threshold size includes the following steps S31 and S32. Step S31: In response to the size of the object to be stored not being equal to the threshold size, determine whether the size of the object to be stored exceeds the aggregation threshold. The aggregation threshold is the product of the threshold size and a preset ratio. Step S32: In response to the size of the object to be stored not exceeding the aggregation threshold, determine whether the aggregation time of the object to be stored has reached the preset time.

[0104] For example, before an object is stored in the preset data pool, it resides in a cache pool. A flushing thread in the cache pool flushes the objects to be stored from the cache pool to the preset data pool. This flushing is equivalent to storage. The steps for storing each object to the preset data pool can be found below:

[0105] The threshold value is 4M. It is determined whether the size of the object to be stored reaches 4M. If the size of the object reaches 4M, the object's waiting storage state is set to "entering the waiting storage state." If the object's waiting storage state is "entering the waiting storage state," it indicates that the object meets the flushing condition. If the size of the object does not reach 4M, it is determined whether the size of the object exceeds the aggregation threshold, which can be less than the threshold value mentioned above. The aggregation threshold can be dynamically set according to the storage requirements of the object data. Specifically, the aggregation threshold can be the product of the threshold value and a preset ratio. The preset ratio can be set to 80%. The specific value of the preset ratio can be adjusted according to the actual situation. If the size of the object does not exceed the aggregation threshold, it is determined whether the aggregation time of the object has reached a preset time. Specifically, if the size of the object is equal to or less than the aggregation threshold, it is determined whether the aggregation time of the object has reached a preset time. If the aggregation time of the object reaches the preset time, the object's waiting storage state is set to "entering the waiting storage state."

[0106] For aggregated objects that meet the flushing conditions, i.e., objects awaiting storage in the "entering storage" state, it is determined whether the number of reads of the object to be stored within a preset time reaches a threshold. If the number of reads reaches the threshold, the object to be stored is considered hot data and is not stored in the preset data pool. If the number of reads does not reach the threshold, the object to be stored is considered non-hot data and is stored from the cache pool to the preset data pool. During the process of storing data to the preset data pool, for objects awaiting storage in the "entering storage" state, non-hot data, but whose size has not reached the threshold, the objects are populated with preset data to make their size equal to the threshold size, and then stored in the preset data pool.

[0107] In other application scenarios, for at least one aggregated object obtained by aggregating data blocks belonging to the second category, the corresponding object to be stored is only stored in a preset data pool if the aggregation time of the object to be stored reaches a preset time and the number of reads of the object to be stored does not reach a threshold. For this type of object to be stored, there is no need to fill it to the threshold size; it can be directly stored in the preset data pool.

[0108] It can be considered that this application can realize the storage of objects to be stored to a preset data pool according to the storage requirements of object data. By detecting the size of the objects to be stored and the aggregation time in the cache pool, it can determine whether the waiting storage state of the objects to be stored is the waiting storage state. Among them, the objects to be stored that are in the waiting storage state meet the flushing conditions. Combined with the hot data judgment mechanism, the storage processing of high hot data is delayed, thereby improving the throughput of the target cluster for objects to be stored.

[0109] The above scheme determines the category of each data block based on its position within each object data. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority being greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data. This ensures that data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, several objects to be stored are obtained, making each object to be stored a more suitable object for storage. Each object to be stored is stored in a preset data pool, thereby making the distribution of data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0110] Please see Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the object data storage device of this application. The object data storage device 40 includes an acquisition module 41, a first determination module 42, an aggregation module 43, and a second determination module 44. The acquisition module 41 is used to acquire a plurality of data blocks of at least one object data; the first determination module 42 is used to determine the category of each data block based on the position of each data block in each object data; the aggregation module 43 is used to aggregate each target data block for each object data in response to the number of target data blocks in the object data belonging to the category with the highest aggregation priority being greater than or equal to a preset number, to obtain at least one aggregated object of the object data; the second determination module 44 is used to determine a plurality of objects to be stored based on the at least one aggregated object corresponding to each object data, and to store each object to be stored in a preset data pool.

[0111] The above scheme determines the category of each data block based on its position within each object data. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority being greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data. This ensures that data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, several objects to be stored are obtained, making each object to be stored a more suitable object for storage. Each object to be stored is stored in a preset data pool, thereby making the distribution of data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0112] For details on the functions performed by each module, please refer to the object data storage method; these details will not be elaborated upon here.

[0113] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 50 includes a memory 51 and a processor 52. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in the above-described embodiment of the object data storage method. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a multi-camera device, a microcomputer, or a server. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.

[0114] Specifically, processor 52 controls itself and memory 51 to implement the steps in the above-described method embodiment for storing object data. Processor 52 can also be called a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0115] The above scheme determines the category of each data block based on its position within each object data. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority being greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data. This ensures that data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, several objects to be stored are obtained, making each object to be stored a more suitable object for storage. Each object to be stored is stored in a preset data pool, thereby making the distribution of data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0116] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an embodiment of a computer-readable storage medium according to this application. The computer-readable storage medium 60 stores program instructions 601 thereon, which, when executed by a processor, implement the steps in any of the above-described embodiments of the object data storage method.

[0117] The above scheme determines the category of each data block based on its position within each object data. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority being greater than or equal to a preset number, the target data blocks are aggregated to obtain at least one aggregated object of the object data. This ensures that data blocks belonging to the same object data are more concentratedly aggregated in fewer aggregated objects. Based on at least one aggregated object corresponding to each object data, several objects to be stored are obtained, making each object to be stored a more suitable object for storage. Each object to be stored is stored in a preset data pool, thereby making the distribution of data blocks belonging to the same object data in the objects stored in the preset data pool more concentrated, thus improving the storage efficiency of the object data.

[0118] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0119] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for storing object data, characterized in that, The method includes: Acquiring at least one object data into several data blocks includes: acquiring at least one object data into several initial data blocks and segmentation parameters for each object data, wherein the segmentation parameters include cutting block parameters and cutting strip parameters when segmenting each object data; using the minimum value among the segmentation parameters of each object data as a partitioning threshold; and determining the initial data blocks that are less than or equal to the partitioning threshold as the several data blocks. The category of each data block is determined based on its position in the object data. For each object data, in response to the number of target data blocks belonging to the category with the highest aggregation priority in the same object data being greater than or equal to a preset number, each target data block is aggregated to obtain at least one aggregated object of the object data. Based on at least one aggregated object corresponding to each of the object data, several objects to be stored are determined, and each object to be stored is stored in a preset data pool.

2. The method according to claim 1, characterized in that, The step of determining a plurality of objects to be stored based on at least one aggregate object corresponding to each of the object data includes: Based on the aggregation priority of each category, the data blocks of each category are sequentially treated as a batch of data blocks to be aggregated. Based on at least one aggregation object corresponding to each of the object data, determine the first batch of aggregation objects corresponding to each first batch of data blocks to be aggregated, and the aggregation priority corresponding to the category of each first batch of data blocks to be aggregated is the highest. For non-first batch of data blocks to be aggregated, if there is a target aggregation object with free space in the aggregation object obtained from the aggregation of the previous batch of data blocks to be aggregated, the target aggregation object is filled using each of the data blocks to be aggregated in this batch to obtain the corresponding aggregation object for this batch; or, if there is no target aggregation object in the aggregation object obtained from the aggregation of the previous batch of data blocks to be aggregated, the data blocks to be aggregated in this batch are aggregated to obtain the corresponding aggregation object for this batch. The aggregation objects corresponding to the last batch of data blocks to be aggregated are taken as the several objects to be stored.

3. The method according to claim 2, characterized in that, The step of determining the first batch of aggregation objects corresponding to each first batch of data blocks to be aggregated based on at least one aggregation object corresponding to each of the object data includes: Based on the relationship between the size of at least one aggregated object corresponding to each of the object data and the aggregate size, the aggregation state of each of the aggregated objects is determined, and the aggregation state includes reaching the full aggregation state or not reaching the full aggregation state. For each of the aggregate objects, in response to the aggregate object's aggregation state reaching the full aggregation state, the aggregate object is designated as the first batch of aggregate objects corresponding to the first batch of data blocks to be aggregated; or... In response to the aggregation state of at least one candidate aggregation object being the state of not reaching full aggregation, the first batch of aggregation objects corresponding to the first batch of data blocks to be aggregated are determined based on each of the candidate aggregation objects, wherein the candidate aggregation objects are at least one of the aggregation objects.

4. The method according to any one of claims 1 to 3, characterized in that, The step of storing each of the objects to be stored into a preset data pool includes: For each object to be stored, perform the following steps: Based on the size of the object to be stored, the waiting storage state of the object to be stored is determined, and the waiting storage state of the object to be stored includes entering the waiting storage state or not entering the waiting storage state; In response to the object to be stored being in the waiting storage state, the object to be stored is stored in the preset data pool.

5. The method according to claim 4, characterized in that, Determining the waiting-to-store status of the object based on its size includes: Determine whether the size of the object to be stored is equal to the threshold size; In response to the size of the object to be stored being equal to a threshold size, the waiting-to-store state of the object to be stored is determined as entering the waiting-to-store state; or, In response to the fact that the size of the object to be stored is not equal to the threshold size, determine whether the aggregation time of the object to be stored has reached the preset time; In response to the aggregation time of the object to be stored reaching the preset time, the waiting storage state of the object to be stored is determined as entering the waiting storage state.

6. The method according to claim 4, characterized in that, The step of responding to the waiting storage state of the object to be stored as entering the waiting storage state and storing the object to be stored in the preset data pool includes: In response to the waiting storage state of the object to be stored being "entering the waiting storage state", the number of times the object to be stored is read within a preset time period is obtained; If the number of reads of the object to be stored within a preset time period does not reach a threshold, the object to be stored is stored in the preset data pool.

7. The method according to any one of claims 1 to 3, characterized in that, After determining a plurality of objects to be stored based on at least one aggregate object corresponding to each of the object data, the method further includes: The information before aggregation, storage location, and data block distribution information of each object to be stored are recorded in a preset information set so that each object to be stored can be read based on the preset information set.

8. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to perform the method as claimed in any one of claims 1-7.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they are used to implement the method as described in any one of claims 1-7.

Images