A metadata pre-deletion analysis-based mass object deletion method and system

Abstract

The application provides a mass object deletion method and system based on metadata pre-deletion analysis, and belongs to the technical field of big data processing. The method comprises the following steps: determining objects to be deleted according to a preset filtering condition, classifying and grading the objects, and storing the objects in a to-be-deleted object cluster; dynamically scheduling a corresponding deletion instruction according to the data state of the to-be-deleted object cluster; locating the storage position of the to-be-deleted object by calculating the storage position of the to-be-deleted object according to the headobject file of the to-be-deleted object in the to-be-deleted object cluster through a splicing rule; and deleting the to-be-deleted object at the storage position of the to-be-deleted object by using the deletion instruction. The method and system provided by the application delete the to-be-deleted object at the storage position of the to-be-deleted object by using the deletion instruction, compared with the query positioning method in the prior art, the process of screening from hundreds of millions of data is avoided, and the efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and particularly to the field of metadata deletion technology, specifically to a method and system for deleting massive objects based on metadata pre-deletion analysis. Background Technology

[0002] With the rise of cloud computing and big data, home video cameras are now commonplace in almost every household. The maturity of these technologies has led to a surge in data, making object storage increasingly important. Object storage is particularly well-suited for unstructured text, audio, and video formats, and its access is relatively convenient. Ensuring the stability of object storage clusters and managing the lifecycle of objects is crucial, especially in Ceph clusters. Deleting objects from very large buckets is a key technical issue, typically addressed by three methods: First, setting the object's lifecycle through RGW, allowing the object gateway to automatically delete objects. The core bottleneck is retrieving the list from the very large bucket, which is extremely time-consuming. Second, actively initiating deletion operations by calling the S3 interface, with a maximum of 1000 objects per request. Third, deleting the pool and rebuilding it.

[0003] For very large buckets, the deletion lifecycle is slow. According to tests, the maximum number of 4M objects deleted per second in an 8P cluster is 3000. For scenarios with a read-to-write ratio of 1:1, it will be the bottleneck of cluster service capacity. It is necessary to ensure that deletion is faster than writing. Otherwise, the speed will be jagged, eventually causing the cluster to be full and resulting in cluster instability.

[0004] Therefore, it is necessary to provide a method and system for deleting massive objects based on metadata pre-deletion analysis to solve the above problems. Summary of the Invention

[0005] This invention addresses the problems of existing technologies by providing a method and system for deleting massive objects based on metadata pre-deletion analysis. The technical solution adopted is as follows:

[0006] Firstly, a method for deleting massive objects based on metadata pre-deletion analysis, the method comprising:

[0007] Based on preset filtering conditions, the objects to be deleted are determined, the objects are classified and graded, and stored in the object cluster to be deleted.

[0008] Based on the data status of the cluster of objects to be deleted, the corresponding deletion command is dynamically scheduled through load balancing.

[0009] Based on the headobject file of the object to be deleted in the cluster of objects to be deleted, the storage location of the object to be deleted is calculated and located by concatenation rules, and the deletion command is used to delete the object to be deleted at the storage location of the object to be deleted.

[0010] According to an optional implementation, determining the objects to be deleted based on preset filtering conditions, classifying and categorizing the objects, and storing them in a cluster of objects to be deleted includes:

[0011] Develop a task plan based on preset filtering conditions;

[0012] Based on the established task plan, the index data of objects in the index pool and the shard data of objects in the data pool are filtered, and the filtered index data of objects in the index pool and the shard data of objects in the data pool are classified and graded.

[0013] The index data of objects in the categorized and graded index pool and the sharded data of objects in the data pool are stored in the cluster of objects to be deleted;

[0014] The pre-defined expiration rules are used to query and record the storage location of the index data and object shard data of the objects to be deleted in the cluster.

[0015] According to an optional implementation, based on the data status of the cluster of objects to be deleted, the corresponding deletion command is dynamically scheduled through load balancing, including:

[0016] Based on the load status of the hosts in the cluster of objects to be deleted and the load status of the Ceph cluster, the data evaluation corresponding to the load status is calculated to generate the guidance result for the dynamic load scheduling.

[0017] Based on the guidance results, each deletion method is assigned a weight, and the deletion method with the highest weight is selected as the deletion instruction.

[0018] According to optional implementations, the deletion method includes: deletion via the S3 interface, deletion via the data pool, and deletion via an index.

[0019] According to an optional implementation, based on the data status of the cluster of objects to be deleted, the corresponding deletion instruction is further dynamically scheduled by load balancing:

[0020] The load status is collected and calculated to determine idle time scheduling and control the impact of deletion traffic on read and write operations;

[0021] Monitor the Ceph cluster and control the rados project of the Ceph cluster according to bandwidth.

[0022] The system queries the list of objects to be deleted based on metadata and performs lifecycle-based minimum deletion based on the amount of data in each object.

[0023] According to an optional implementation, the splicing rules include:

[0024] Assemble markid_ + object name + shadow / multipart_filename_prefix_order.

[0025] According to an optional implementation, based on the headobject file of the object to be deleted in the object cluster, the storage location of the object to be deleted is calculated and located according to the concatenation rules, and the object to be deleted is deleted at the storage location using a deletion command, including:

[0026] When the size of the headobject file of the object to be deleted is 4M, the storage location of the object to be deleted is calculated and located by concatenation rules. The deletion command is used to delete the headobject file at the storage location of the object to be deleted. After deletion, the remaining shadow data is deleted.

[0027] When the size of the headobject file of the object to be deleted is 0kb, the concatenation rule is used to calculate and locate the storage location of the object to be deleted. The deletion command is used to delete the headobject file at the storage location of the object to be deleted, avoiding the process of filtering from hundreds of millions of data. After deletion, the data nosuchkey is accessed, and the remaining multipart and shadow data are deleted asynchronously through metadata.

[0028] Secondly, embodiments of the present invention provide a massive object deletion system based on metadata pre-deletion analysis, the system comprising:

[0029] The preset classification module is used to determine the objects to be deleted according to preset filtering conditions, classify and classify the objects, and store them in the object cluster to be deleted.

[0030] The program selects a module to dynamically schedule the corresponding deletion command based on the data status of the cluster of objects to be deleted through load balancing.

[0031] The location deletion module is used to calculate and locate the storage location of the object to be deleted based on the headobject file of the object to be deleted in the object cluster, according to the concatenation rules, and then delete the object to be deleted at the storage location using a deletion command.

[0032] According to an optional implementation, the preset hierarchical module is used to filter the index and data.

[0033] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, they implement the method described in the first aspect above.

[0034] Fourthly, embodiments of the present invention provide a computer storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it implements the method described in the first aspect.

[0035] The embodiments of the present invention have the following advantages:

[0036] Compared to existing technologies, object list processing is very time-consuming, especially for single-bucket databases with hundreds of millions of objects. Retrieving 1,000 objects at a time is a performance bottleneck, and how to quickly select the objects to be deleted is a major problem. The method of this invention classifies and grades the objects and stores them in a cluster of objects to be deleted. Based on the load status of the hosts in the cluster and the load status of the Ceph cluster, the data evaluation corresponding to the load status is calculated to generate guidance results for dynamic load scheduling. According to the guidance results, each deletion method is assigned a weight, and the deletion method with the highest weight is selected as the deletion instruction. The storage location of the object to be deleted is calculated and located through concatenation rules, and the deletion instruction is used to delete the object at the storage location of the object to be deleted. Compared with the existing technology of locating by query, calculating and locating the storage location of the object to be deleted can delete the object fragment data at the specified location, avoiding the process of filtering from hundreds of millions of data, thus improving efficiency. Attached Figure Description

[0037] Figure 1 This is a flowchart of a method for deleting massive objects based on metadata pre-deletion analysis provided by an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram illustrating the principle of the deletion instruction provided in this embodiment of the invention;

[0039] Figure 3 This is a schematic diagram illustrating the deletion of the index of the object to be deleted using index deletion provided in an embodiment of the present invention. Detailed Implementation

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] The following reference Figures 1 to 3 This paper will provide a detailed description of the massive object deletion method and system based on metadata pre-deletion analysis of the present invention.

[0042] Example 1:

[0043] Figure 1 A flowchart illustrating a method for deleting massive objects based on metadata pre-deletion analysis is shown, such as... Figure 1As shown in the figure, this embodiment provides a method for deleting massive objects based on metadata pre-deletion analysis, including:

[0044] Step S1: Determine the objects to be deleted according to the preset filtering conditions, classify and classify the objects, and store them in the object cluster to be deleted;

[0045] Step S2: Based on the data status of the cluster of objects to be deleted, dynamically schedule the corresponding deletion instructions through load balancing;

[0046] Step S3: Based on the headobject file of the object to be deleted in the cluster of objects to be deleted, calculate and locate the storage location of the object to be deleted through the concatenation rules, and use the deletion command to delete the object to be deleted at the storage location of the object to be deleted.

[0047] Furthermore, step S1 includes:

[0048] Staff members use date, user ID, and other preset filtering conditions to create task plans;

[0049] Based on the established task plan, the index data of objects in the index pool and the object shard data in the data pool are filtered. The filtered index data and object shard data are then categorized and graded. The object index data is stored on a specific shard in the index pool. The categorization of index data and object shard data in the index pool and data pool can be achieved through the encoding of file path names. For example:

[0050] The file is named / key1 / key2 / key3 / DATE / filename.txt, where key1 corresponds to category 1; key2 and key3 correspond to subcategories, categorizing the data in filename.txt according to date.

[0051] The index data of objects in the categorized and graded index pool and the sharded data of objects in the data pool are stored in the cluster of objects to be deleted;

[0052] The pre-defined expiration rules are used to query and record the storage location of the index data and object shard data of the objects in the cluster to be deleted, so that the object data can be directly deleted based on the location during subsequent deletion steps.

[0053] In this embodiment, the specific steps for filtering the index data of objects in the index pool based on the established task plan are as follows:

[0054] Step S101: Filter the shard 9b6c6ba32e1.24630.1.8 in the index pool using test2, and activate the omapkey. The specific code is as follows:

[0055] rados listomapkeys-p

[0056] default.rgw.buckets.index.dir.87671243-04e2-4dfe-accb-7

[0057] 9b6c6ba32e1.24630.1.8|grep test2

[0058] Step S102: Destroy the object's index data using the rmomapkey command. The specific code is as follows:

[0059] rados rmomapkey-p

[0060] default.rgw.buckets.index.dir.87671243-04e2-4dfe-accb-79b6

[0061] c6ba32e1.24630.1.8userid / deviceid / date / rf / test2.pptx

[0062] In this embodiment, the specific steps for filtering object shard data in the data pool based on the established task plan are as follows:

[0063] Step S111: Use test2.pptx to filter the list in the data pool and obtain the filenames of the object sharded data stored on disk. The specific code is as follows:

[0064] [root@node6~]#rados-p default.rgw.buckets.data ls|grep test2.pptx

[0065] 87671243-04e2-4dfe-accb-79b6c6ba32e1.24630.1__multipart_userid / deviceid / date / rf / test2.pptx.2~ak

[0066] z1dDWFkdnAN-gQmX15iqF9DY4kTkM.1

[0067] 87671243-04e2-4dfe-accb-79b6c6ba32e1.24630.1__shadow_userid / deviceid / date / rf / test2.pptx.2~akz1d

[0068] DWFkdnAN-gQmX15iqF9DY4kTkM.2_1

[0069] Step S112: Use rados rm object name to physically delete object fragment data. The specific code is as follows:

[0070] rados -p default.rgw.buckets.data rm filename

[0071] Further, in step S2, based on the data status of the cluster of objects to be deleted, the corresponding deletion command is dynamically scheduled through load balancing, including:

[0072] Step S201: Based on the load status of the hosts in the cluster of objects to be deleted and the load status of the Ceph cluster, calculate the data evaluation corresponding to the load status and generate guidance results for dynamic load scheduling.

[0073] Step S202: Assign weights to each deletion method according to the guidance results, and select the deletion method with the highest weight as the deletion instruction. In this embodiment, the deletion methods include: deletion via S3 interface, deletion via data pool, and deletion via index. The weight assigned to the deletion via S3 interface method is 0.2, the weight assigned to the deletion via data pool method is 0.2, and the weight assigned to the deletion via index method is 0.5. Therefore, the deletion via index method is selected as the deletion instruction.

[0074] like Figure 2 As shown, in step S201, the host load status includes network traffic, CPU utilization, memory utilization, disk utilization, IO pressure, etc., and the Ceph cluster load status includes pool, pg, osd, and rgw load status. The calculation of the data evaluation corresponding to the load status includes comprehensive calculation of the load status data according to the corresponding evaluation rules. Specifically, the evaluation rules include: CPU utilization is less than 50%, memory utilization is less than 50%, and network traffic is less than 1G, etc.

[0075] Meanwhile, step S2 also includes:

[0076] Step S211: Collect and calculate the load status, determine idle time scheduling, and control the impact of deletion traffic on read and write operations, thereby avoiding competition between deletion requests and read / write requests.

[0077] Step S212: Monitor the Ceph cluster and control the bandwidth of the rados (Reliable, Autonomic Distributed Object Store, is one of the core components of Ceph. As a sub-project of Ceph Distributed File System, it is specifically designed for Ceph's needs and can provide a stable, scalable, and high-performance single logical object (Object) storage interface and a storage system that enables nodes to adapt and self-manage on dynamically changing and heterogeneous storage device clusters) project of the Ceph cluster according to bandwidth control.

[0078] Step S213: Query the list of objects to be deleted based on the metadata, and perform lifecycle-based minimum deletion based on the amount of data of the objects to be deleted.

[0079] Specifically, this embodiment uses IOPS to monitor the load status of OSD, the most important component of the Ceph cluster. IOPS (Input / Output Operations Per Second) is a measurement method used for performance testing of computer storage devices (such as hard disks (HDDs), solid-state drives (SSDs), or storage area networks (SANs). It can be regarded as the number of read and write operations per second. Like other performance tests, the IOPS proposed by storage device manufacturers does not guarantee the performance under actual application. The bandwidth level is used to control and schedule the rados of the Ceph cluster. The manifest is obtained by querying the list of deleted objects based on metadata. The manifest is a software file, belonging to AndroidManifest.xml. It provides important information in simple Ceph applications. It can run the code of any application. The manifest directly deletes the corresponding objects based on the rm command. rm -rf is a file deletion command under UNIX system. Its function is to silently force recursive deletion of files. You only need to type "rm -rf" (without double quotes) in the UNIX command line interface to call this command, delete all files in the current directory, and they cannot be recovered.

[0080] Furthermore, the splicing rules in step S3 are as follows:

[0081] Assemble markid_ + object name + shadow / multipart_filename_prefix_order, where multipart is the fragment corresponding to the object to be deleted, and shadow is the split of the remaining fragments after dividing the object to be deleted into multiple fragments. For example, if a 5M object to be deleted, radogw, is split into 4M units to form a fragment, then a 4M multipart will be obtained, and the remaining 1M will be used as shadow.

[0082] Specifically, in this embodiment, the size of the headobject file of the object to be deleted in step S3 is 4M or 0kb. When the object to be deleted is uploaded by the user in a whole upload manner, the size of the headobject file of the object to be deleted is 4M. When the object to be deleted is uploaded by the user in a segmented upload manner, the size of the headobject file of the object to be deleted is 0kb.

[0083] Specifically, when a user uploads an object to be deleted as a whole, the object size must be less than or equal to the fragment size or greater than the fragment size. When the object size is less than or equal to the fragment size, each uploaded object corresponds to only one RADOS object, named after the object, and its metadata is stored in the extended attributes of that RADOS object. When the object size is greater than the fragment size, it is broken down into a first object equal to the fragment size, multiple intermediate objects equal to the stripe size, and a third object less than or equal to the stripe size. The small tail object and the head object are named after the object to be deleted. In RGW, this object is called the headobject. The data part of the headobject stores the data of the first rgw_max_chunk_size bytes of the object to be deleted. Therefore, the size of the headobject file is 4M. The extended attributes part stores the metadata information and manifest information of the headobject. The middle and tail objects store the remaining data of the application object. The object name is "shadow_" + "." + "32-bit random string" + "_" + "strip number", where the strip number starts from 1.

[0084] Specifically, when a user uploads an object to be deleted using the chunked upload method, RGW divides each chunk of the object to be deleted into multiple RADOS objects according to the stripe size. The first RADOS object of each chunk is named as follows: "multipart" + "user uploaded object name" + "segment upload ID" + "segment number". The names of the remaining objects are as follows: "_shadow_" + "user uploaded object name" + "segment upload ID" + "segment number" + "_" + "strip number". After all chunks have been uploaded, RGW reads the information of each chunk from the chunked upload temporary object in the data_extra_pool, mainly the manifest information of each chunk, and forms a manifest. Then, it generates a new RADOS object, namely headobject, which is used to store the metadata information of the object to be deleted uploaded using the chunked upload method and the manifest of each segment. Therefore, headobject only contains metadata and its own size is 0kb.

[0085] Furthermore, in step S3, based on the headobject file of the object to be deleted in the cluster of objects to be deleted, the storage location of the object to be deleted is calculated and located according to the concatenation rules, and the deletion command is used to delete the object to be deleted at the storage location of the object to be deleted, including:

[0086] When the size of the headobject file of the object to be deleted is 4M, the storage location of the object to be deleted is calculated and located by concatenation rules. The deletion command is used to delete the headobject file at the storage location of the object to be deleted. After deletion, the remaining shadow data is deleted.

[0087] When the size of the headobject file of the object to be deleted is 0kb, the concatenation rule is used to calculate and locate the storage location of the object to be deleted. The deletion command is used to delete the headobject file at the storage location of the object to be deleted, avoiding the process of filtering from hundreds of millions of data. After deletion, the data nosuchkey is accessed, and the remaining multipart and shadow data are deleted asynchronously through metadata.

[0088] like Figure 3 As shown, in this embodiment, the index deletion method is used as the deletion command. The storage location of the object to be deleted is calculated and located through the concatenation rules. The index of the object to be deleted, i.e. the headobject file of the object to be deleted, is deleted, and the deletion result is fed back to the user terminal.

[0089] Example 2:

[0090] This invention provides a massive object deletion system based on metadata pre-deletion analysis, the system comprising:

[0091] The preset classification module is used to determine the objects to be deleted according to preset filtering conditions, classify and classify the objects, and store them in the object cluster to be deleted.

[0092] The program selects a module to dynamically schedule the corresponding deletion command based on the data status of the cluster of objects to be deleted through load balancing.

[0093] The location deletion module is used to calculate and locate the storage location of the object to be deleted based on the headobject file of the object to be deleted in the object cluster, according to the concatenation rules, and then delete the object to be deleted at the storage location using a deletion command.

[0094] Example 3:

[0095] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method of Embodiment 1;

[0096] In practical applications, the processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller unit (MCU), microprocessor, or other electronic components to execute the methods described in the above embodiments.

[0097] The method implemented in this embodiment includes:

[0098] The implementation scheme described in Example 1.

[0099] Example 4:

[0100] This embodiment also provides a computer storage medium, in which a computer program is stored, and when the computer program is executed by one or more processors, it implements the method of embodiment one.

[0101] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0102] The method implemented in this embodiment includes:

[0103] The implementation scheme described in Example 1.

[0104] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0105] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0106] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0107] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0108] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatial relative descriptions used herein will be interpreted accordingly.

[0109] In the detailed description above, reference has been made to the accompanying drawings, which form part of this document. In the drawings, similar symbols typically identify similar parts unless the context otherwise indicates otherwise. The illustrated embodiments described in the detailed specification, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for deleting massive objects based on metadata pre-deletion analysis, characterized in that, The method includes: Based on preset filtering conditions, the objects to be deleted are determined, the objects are classified and graded, and stored in the object cluster to be deleted. Based on the data status of the cluster of objects to be deleted, the corresponding deletion command is dynamically scheduled through load balancing. Based on the headobject file of the object to be deleted in the cluster of objects to be deleted, the storage location of the object to be deleted is calculated and located according to the concatenation rules, and the deletion command is used to delete the object to be deleted at the storage location; specifically including: When the size of the headobject file of the object to be deleted is 4M, the storage location of the object to be deleted is calculated and located by concatenation rules. The deletion command is used to delete the headobject file at the storage location of the object to be deleted. After deletion, the remaining shadow data is deleted. When the size of the headobject file of the object to be deleted is 0kb, the concatenation rule is calculated to locate the storage location of the object to be deleted. The deletion command is used to delete the headobject file at the storage location of the object to be deleted, avoiding the process of filtering from hundreds of millions of data. After deletion, the data nosuchkey is accessed, and the remaining multipart and shadow data are deleted asynchronously through metadata. The splicing rules include: Assemble markid_ + object name + shadow / multipart_filename_prefix_order.

2. The method for deleting massive objects based on metadata pre-deletion analysis according to claim 1, characterized in that, Based on preset filtering conditions, objects to be deleted are identified, categorized and classified, and stored in a cluster of objects to be deleted, including: Develop a task plan based on preset filtering conditions; Based on the established task plan, the index data of objects in the index pool and the shard data of objects in the data pool are filtered, and the filtered index data of objects in the index pool and the shard data of objects in the data pool are classified and graded. The index data of objects in the categorized and graded index pool and the sharded data of objects in the data pool are stored in the cluster of objects to be deleted; The pre-defined expiration rules are used to query and record the storage location of the index data and object shard data of the objects to be deleted in the cluster.

3. The method for deleting massive objects based on metadata pre-deletion analysis according to claim 1, characterized in that, Based on the data status of the cluster of objects to be deleted, the corresponding deletion instructions are dynamically scheduled through load balancing, including: Based on the load status of the hosts in the cluster of objects to be deleted and the load status of the Ceph cluster, the data evaluation corresponding to the load status is calculated to generate the guidance result for the dynamic load scheduling. Based on the guidance results, each deletion method is assigned a weight, and the deletion method with the highest weight is selected as the deletion instruction.

4. The method for deleting massive objects based on metadata pre-deletion analysis according to claim 3, characterized in that, The deletion methods include: deletion via the S3 interface, deletion via the data pool, and deletion via the index.

5. The method for deleting massive objects based on metadata pre-deletion analysis according to claim 3, characterized in that, Based on the data status of the cluster of objects to be deleted, the corresponding deletion instructions, through load balancing and dynamic scheduling, also include: The load status is collected and calculated to determine idle time scheduling and control the impact of deletion traffic on read and write operations; Monitor the Ceph cluster and control the rados project of the Ceph cluster according to bandwidth. The system queries the list of objects to be deleted based on metadata and performs lifecycle-based minimum deletion based on the amount of data in each object.

6. A massive object deletion system based on metadata pre-deletion analysis, comprising deleting massive objects using the massive object deletion method based on metadata pre-deletion analysis as described in any one of claims 1-5, characterized in that, The system includes: The preset classification module is used to determine the objects to be deleted according to preset filtering conditions, classify and classify the objects, and store them in the object cluster to be deleted. The program selects a module to dynamically schedule the corresponding deletion command based on the data status of the cluster of objects to be deleted through load balancing. The location deletion module is used to calculate and locate the storage location of the object to be deleted based on the headobject file of the object to be deleted in the object cluster, according to the concatenation rules, and then delete the object to be deleted at the storage location using a deletion command.

7. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the massive object deletion method based on metadata pre-deletion analysis as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, is used to implement the massive object deletion method based on metadata pre-deletion analysis as described in any one of claims 1-5.

Images