Data deletion method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides a data deletion method, device, storage medium and electronic device, wherein the method includes: storing the value of the specified field in each data in the target data set in each shard of the target shard set according to the first storage method, responding to the target deletion instruction to request to delete the target data subset corresponding to the target index in the target data set, obtaining the target shard subset corresponding to the target index in the target shard set, the target shard subset stores the target data subset to be deleted corresponding to the target index, the first deletion speed of the value of the specified field stored in the first storage method is greater than the second deletion speed corresponding to the second storage method; determining the processing node set corresponding to the target shard subset; deleting the target data subset in parallel through the processing node set. The embodiment of the present invention solves the problem of slow deletion speed of indexed data existing in the related art.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data technology, and in particular, to a data deletion method, device, storage medium and electronic device. Background Art

[0002] Elasticsearch (or ES for short) is a search engine based on the Lucene library. It provides a distributed, multi-user full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and released as open source software under the Apache license. Lucene is an open source library for full-text retrieval and search, supported and provided by the Apache Software Foundation. Lucene provides a simple but powerful application program interface that can do full-text indexing and searching. Lucene is widely used as a standard base library for search applications.

[0003] As a search engine, Elasticsearch needs to store a large amount of data. While constantly storing new data, there is also a need to delete expired data. Improving the performance and stability of Elasticsearch's data deletion is one of the long-term goals of optimizing Elasticsearch. In the related technology, the data corresponding to an index is deleted through the DeleteByQuery method. However, the deletion method in the related technology can only delete the entire index, but cannot delete only part of the data in the index. Or, because the data of an index is stored in different shards, and different shards often use different storage methods because they store different data, this leads to differences in the speed of reading related field values ​​in the deletion process of different shards, thereby affecting the deletion performance of Elasticsearch. Therefore, there is a problem of slow deletion speed for indexed data in the related technology.

[0004] With respect to the problem of slow deletion speed of indexed data existing in the related art, no effective solution has been proposed yet. Summary of the invention

[0005] Embodiments of the present invention provide a data deletion method, device, storage medium and electronic device to at least solve the problem of slow deletion speed of indexed data existing in the related art.

[0006] According to an embodiment of the present invention, a data deletion method is provided, comprising: storing a target data set in each shard in a target shard set, wherein the value of a specified field in each data in the target data set is stored in accordance with a first storage method, the value of the specified field in each data in the target data set is allowed to be stored in a manner including the first storage method and a second storage method different from the first storage method, a first deletion speed of the value of the specified field stored in the first storage method is greater than a second deletion speed of the value of the specified field stored in the second storage method; in response to a target deletion instruction, obtaining a target shard sub-index corresponding to a target index in the target shard set. set, wherein the target deletion instruction is used to request deletion of a target data subset corresponding to the target index in the target data set, the target shard subset stores the target data subset to be deleted corresponding to the target index, and each shard in the target shard subset stores part of the data in the target data subset; determine a processing node set corresponding to the target shard subset, wherein the processing node set is used to delete the target data subset stored in the target shard subset in parallel; delete the target data subset stored in the target shard subset in parallel through the processing node set, wherein the value of the specified field of each data in the target data subset is deleted according to the first deletion speed.

[0007] In an exemplary embodiment, before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, the method further includes: establishing a corresponding relationship between the target shard subset and the processing node set, when each data in the target data subset is stored in the corresponding shard in the target shard subset, wherein each processing node in the processing node set has a corresponding relationship with at least one shard in the target shard subset, and the number of shards corresponding to each processing node is within a preset number range, or the difference between the number of shards corresponding to each processing node is less than or equal to a preset threshold.

[0008] In an exemplary embodiment, determining the set of processing nodes corresponding to the target shard subset includes: when the target deletion instruction indicates that the number of shards allowed to perform parallel deletion is the number of shards in the target shard subset, determining the processing nodes that have a corresponding relationship with each shard in the target shard subset to obtain the processing node set.

[0009] In an exemplary embodiment, the parallel deletion of the target data subset stored in the target shard subset by the processing node set includes: when the target deletion instruction indicates a target amount of data to be deleted in each batch, deleting the target data subset in parallel and in batches by each processing node in the processing node set, wherein the amount of data in the target data subset deleted in each batch by each processing node is the target data amount.

[0010] In an exemplary embodiment, before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, the method further includes: determining the target data volume according to the processing speed of the processing node in the processing node set, and setting a parameter for indicating the target data volume in the target deletion instruction, wherein the processing speed of the processing node is used to indicate the computing speed of the processing node itself, and / or the speed at which the processing node deletes the data stored in the corresponding shard.

[0011] In an exemplary embodiment, the deleting the target data subset in parallel and in batches through each processing node in the processing node set includes: when the number of processing nodes in the processing node set is N and the i-th processing node in the processing node set is set to delete the data belonging to the target data subset stored in the i-th group of shards in parallel and in batches, a batch of data belonging to the target data subset stored in each shard in the i-th group of shards is deleted in parallel through the following steps, wherein i is a positive integer greater than or equal to 1 and less than or equal to N, N is a positive integer greater than or equal to 1, and the i-th group of shards includes M i shards, M i is equal to 1 or is a positive integer greater than or equal to 2: i In the case where there is undeleted data in the data belonging to the target data subset stored in each of the shards, and the amount of the undeleted data is greater than the target data amount, in the M shards, i A batch of data to be deleted is determined from the undeleted data in each of the shards, and a total of M i Batch data, wherein the amount of data in the batch to be deleted is the target data amount; deleting the M batch data through the i-th processing node i Batch data.

[0012] In an exemplary embodiment, deleting the target data subset in parallel and in batches by each processing node in the processing node set includes: when the number of processing nodes in the processing node set is N and the i-th processing node in the processing node set is set to delete the data belonging to the target data subset stored in the i-th group of shards in parallel and in batches, determining whether the i-th processing node completes the deletion within a predetermined time period. i The current batch of data to be deleted in each of the shards, where i is a positive integer greater than or equal to 1 and less than or equal to N, N is a positive integer greater than or equal to 1, and the i-th group of shards includes M i shards, M i is equal to 1 or a positive integer greater than or equal to 2; when it is determined that the i-th processing node has not completed deleting the current batch of data to be deleted in the first part of the shards within the predetermined time period, stop deleting the data belonging to the target data subset stored in the first part of the shards; when it is determined that the i-th processing node has completed deleting the current batch of data to be deleted in the second part of the shards within the predetermined time period, continue to delete the next batch of data to be deleted in the second part of the shards in parallel.

[0013] In an exemplary embodiment, before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, the method further includes: determining the predetermined duration according to the processing speed of the processing node in the processing node set, and setting a parameter for indicating the predetermined duration in the target deletion instruction, wherein the processing speed of the processing node is used to indicate the computing speed of the processing node itself, and / or the speed at which the processing node deletes data stored in the corresponding shard.

[0014] According to another embodiment of the present invention, there is also provided a data deletion device, comprising: a storage module, used to store a target data set in each shard in a target shard set, wherein the value of a specified field in each data in the target data set is stored according to a first storage method, and the storage methods allowed for the value of the specified field in each data in the target data set include the first storage method and a second storage method different from the first storage method, and a first deletion speed of the value of the specified field stored in the first storage method is greater than a second deletion speed of the value of the specified field stored in the second storage method; a first acquisition module, used to respond to a target deletion instruction and acquire a target shard corresponding to a target index in the target shard set. Subset, wherein the target deletion instruction is used to request deletion of the target data subset corresponding to the target index in the target data set, the target shard subset stores the target data subset to be deleted corresponding to the target index, and each shard in the target shard subset stores part of the data in the target data subset; a first determination module is used to determine the processing node set corresponding to the target shard subset, wherein the processing node set is used to delete the target data subset stored in the target shard subset in parallel; a deletion module is used to delete the target data subset stored in the target shard subset in parallel through the processing node set, wherein the value of the specified field of each data in the target data subset is deleted according to the first deletion speed.

[0015] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0016] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0017] Through the present invention, a target data set is stored in each shard in a target shard set, wherein the value of a specified field in each data in the target data set is stored in accordance with a first storage method, and a first deletion speed corresponding to the value of the specified field stored in the first storage method is greater than a second deletion speed corresponding to the value of the specified field stored in the second storage method. In response to a target deletion instruction, a target shard subset corresponding to a target index is obtained in the above target shard set, wherein the target deletion instruction is used to request deletion of a target data subset corresponding to a target index in the target data set, and the target shard subset Part of the data in the target data subset is stored in each shard in the target data subset, and then the processing node set corresponding to the target shard subset is determined, and then the deletion operation is performed in parallel on the target data subset stored in the target shard subset through each node in the processing node set. Since the first deletion speed corresponding to the value of the specified field stored in the first storage method is greater than the second deletion speed corresponding to the value of the specified field stored in the second storage method, and the deletion operation is performed in parallel on the data on each shard in the target shard subset, the purpose of improving the deletion speed is achieved, thereby achieving the purpose of improving the overall deletion performance. It avoids the problem in the related art that different storage methods are used to store the values ​​of the specified field, resulting in a large difference in the deletion speed between different shards, thereby resulting in poor deletion performance for the data corresponding to the index. Therefore, the problem of slow deletion speed for indexed data existing in the related art is solved, and the effect of improving the deletion efficiency of indexed data is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a hardware structure block diagram of a mobile terminal of a data deletion method according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a data deletion method according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a data storage process according to an embodiment of the present invention;

[0021] Figure 4 is an overall flow chart of ES deleting data according to an embodiment of the present invention;

[0022] Figure 5 is a structural block diagram of a data deleting device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0025] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a block diagram of the hardware structure of a mobile terminal according to the data deletion method of an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0026] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data deletion method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0028] In this embodiment, a data deletion method is provided. Figure 2 is a flow chart of a data deletion method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0029] Step S202, storing the target data set into each shard in the target shard set, wherein the value of the specified field in each data in the target data set is stored in accordance with a first storage method, the storage methods allowed for the value of the specified field in each data in the target data set include the first storage method and a second storage method different from the first storage method, and a first deletion speed of the value of the specified field stored in the first storage method is greater than a second deletion speed of the value of the specified field stored in the second storage method;

[0030] Step S204, in response to the target deletion instruction, obtaining a target shard subset corresponding to the target index in the target shard set, wherein the target deletion instruction is used to request deletion of the target data subset corresponding to the target index in the target data set, the target shard subset stores the target data subset to be deleted corresponding to the target index, and each shard in the target shard subset stores part of the data in the target data subset;

[0031] Step S206, determining a processing node set corresponding to the target shard subset, wherein the processing node set is used to delete the target data subset stored in the target shard subset in parallel;

[0032] Step S208: deleting the target data subset stored in the target shard subset in parallel through the processing node set, wherein the value of the specified field of each data in the target data subset is deleted according to the first deletion speed.

[0033] Through the above steps, the target data set is stored in each shard in the target shard set, wherein the value of the specified field in each data in the target data set is stored according to the first storage method, and the first deletion speed corresponding to the value of the specified field stored in the first storage method is greater than the second deletion speed corresponding to the value of the specified field stored in the second storage method. In response to the target deletion instruction, a target shard subset corresponding to the target index is obtained in the above target shard set, wherein the target deletion instruction is used to request deletion of the target data subset corresponding to the target index in the target data set, and the target shard subset Each shard in the set stores part of the data in the target data subset, and then determines the processing node set corresponding to the target shard subset, and then performs the deletion operation on the target data subset stored in the target shard subset in parallel through each node in the processing node set. Since the first deletion speed corresponding to the value of the specified field stored in the first storage method is greater than the second deletion speed corresponding to the value of the specified field stored in the second storage method, and the deletion operation is performed in parallel on the data on each shard in the target shard subset, the purpose of improving the deletion speed is achieved, thereby achieving the purpose of improving the overall deletion performance. It avoids the problem in the related art that different storage methods are used to store the values ​​of the specified field, resulting in a large difference in the deletion speed between different shards, thereby resulting in poor deletion performance for the data corresponding to the index. Therefore, the problem of slow deletion speed for indexed data existing in the related art is solved, and the effect of improving the deletion efficiency of indexed data is achieved.

[0034] Among them, the executor of the above steps can be ES, or application, or client, or a processor with human-computer interaction capabilities configured on a storage device, or a processing device or processing unit with similar processing capabilities, etc., but not limited to these.

[0035] In the above embodiment, the target data set is stored in each shard in the target shard set, wherein the value of the specified field in each data in the target data set is stored according to the first storage method, for example, the specified field includes the _seq_no field and / or the _primary_term field. In actual applications, the _seq_no field and / or the _primary_term field can be stored in a variety of different ways. For example, the block storage method (corresponding to the above second storage method) can be used, and the ordinary storage method (corresponding to the above first storage method) can also be used to obtain the target deletion instruction, wherein the target deletion instruction is used to request the deletion of the target data set corresponding to the target index. The target data subset is a target data subset. For example, the target deletion instruction is an ES index deletion instruction to request the deletion of the target data subset corresponding to the ES index. In actual applications, the data corresponding to an index may be stored on different shards, that is, there may be data related to the ES index on multiple shards, and these data constitute the above target data subset. The target deletion instruction can be generated based on user triggers. For example, the user instructs the client to delete the data corresponding to the ES index (corresponding to the above target index). The target deletion instruction can also be generated periodically. For example, the application configures a predetermined period (such as one year, one month, or other durations). Every predetermined period, the application will generate a target deletion instruction to instruct ES to perform a deletion operation on the data corresponding to the index.In response to the target deletion instruction, a target shard subset corresponding to the target index is obtained in the target shard set, that is, it is determined on which shards the data related to the target index is stored, thereby obtaining the target shard subset, wherein the target data subset to be deleted corresponding to the target index is stored in the target shard subset, and each shard in the target shard subset stores part of the data in the target data subset. In this embodiment, the value of the specified field in each data in each shard in the target shard subset is stored in the same first storage method. For example, the above-mentioned specified field includes a _seq_no field and / or a _primary_term field. In actual applications, the _seq_no field and / or the _primary_term field can adopt a variety of different storage methods, for example, a block storage method (corresponding to the above-mentioned second When the target shard subset is selected, the ordinary storage method (corresponding to the first storage method mentioned above) can also be used. The block storage method can save a certain amount of space compared to the ordinary storage method, but the reading speed will be relatively slower. In this embodiment, the value of the specified field in each data in each shard in the target shard subset is stored in the same first storage method (such as the ordinary storage method mentioned above), or in other words, the data of the specified field is prohibited from being stored in the block storage method, which eliminates the situation that some shards are deleted slowly due to the block storage method, so that all shards can complete the deletion operation quickly, effectively improving the overall deletion performance of the index; then determine the processing node set corresponding to the target shard subset, and then delete the target data subset stored in the target shard subset in parallel through each processing node in the processing node set. Since the first deletion speed corresponding to the value of the specified field stored in the first storage method is greater than the second deletion speed corresponding to the value of the specified field stored in the second storage method, and the deletion operation is performed in parallel on the data on each shard in the target shard subset, the purpose of improving the deletion speed is achieved, thereby achieving the purpose of improving the overall deletion performance. This avoids the problem in the related art that different storage methods are used to store the values ​​of the specified field, resulting in large differences in the deletion speed between different shards, which leads to poor deletion performance for the data corresponding to the index. Therefore, the problem of slow deletion speed for indexed data in the related art is solved, achieving the effect of improving the deletion efficiency of indexed data. ;

[0036] Optionally, before obtaining the target deletion instruction, that is, before requesting to delete the target data subset corresponding to the target index, a query instruction requesting to query the target shard subset associated with the target index can be received, and in response to the query instruction, the target shard subset and the target number of shards are determined, wherein the target number of shards is the number of shards included in the target shard subset.

[0037] In an optional embodiment, before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, the method further includes: establishing a corresponding relationship between the target shard subset and the processing node set, when each data in the target data subset is stored in the corresponding shard in the target shard subset, wherein each processing node in the processing node set has a corresponding relationship with at least one shard in the target shard subset, and the number of shards corresponding to each processing node is within a preset number range, or the difference between the number of shards corresponding to each processing node is less than or equal to a preset threshold. In this embodiment, a correspondence between a target shard subset and a processing node set can be established, that is, there is a correspondence between each processing node and one or more shards. In practical applications, the number of shards corresponding to each processing node can be set within a preset number interval, for example, the number of shards corresponding to each processing node is x1 to x2, or the difference between the number of shards corresponding to each processing node is less than or equal to a preset threshold, that is, the number of shards corresponding to each processing node should not differ too much, and can be relatively balanced. Of course, this embodiment does not limit the number of shards corresponding to each node. The above-mentioned processing node can be a server. In practical applications, the corresponding number of shards can be set according to the capabilities of each processing node. Through this embodiment, the purpose of establishing a correspondence between each processing node and each shard is achieved.

[0038] In an optional embodiment, the determination of the processing node set corresponding to the target shard subset includes: when the target deletion instruction indicates that the number of shards allowed to be deleted in parallel is the number of shards in the target shard subset, determining the processing nodes corresponding to each shard in the target shard subset to obtain the processing node set. In this embodiment, the target deletion instruction may indicate the number of shards allowed to be deleted in parallel. When the target deletion instruction indicates that the number of shards allowed to be deleted in parallel is equal to the number of shards in the above target shard subset, determining the processing nodes corresponding to each shard in the target shard subset set to obtain the processing node set. For example, DeleteByQuery is the main method provided by Elasticsearch (or ES for short) to delete specific data in an index. The DeleteByQuery operation supports the configuration of the slices parameter. For example, when it is determined that the number of shards in the target shard subset is Q, the slices parameter can be set to Q, so that the deletion operation can be supported in parallel for the data corresponding to the target index on Q shards. Through this embodiment, the purpose of determining the processing node set corresponding to the target shard subset and the purpose of deleting the data on each shard in the target shard subset in parallel by using the processing node set are achieved.

[0039] In an optional embodiment, the target data subset stored in the target shard subset is deleted in parallel by the processing node set, including: when the target deletion instruction indicates the target data amount for each batch of deletion, the target data subset is deleted in parallel and in batches by each processing node in the processing node set, wherein the amount of data in the target data subset deleted by each processing node in each batch is the target data amount. In this embodiment, the target deletion instruction can also indicate the target data amount for each batch of deletion, so that each processing node can perform the deletion operation on the data on the corresponding shards in batches according to the target data amount in parallel. For example, DeleteByQuery can process the entire deletion process in batches, and obtain a batch of data for deletion each time. For example, by setting the scroll_size parameter, the size of the data amount deleted by DeleteByQuery can be specified. The size of the data amount cannot be too large or too small. If it is too small, the total batch of processing will increase, which will increase unnecessary network time consumption. If it is set too large, the amount of data processed in each batch is too much, which may cause excessive pressure on the cluster. Through this embodiment, the target data volume for each batch of deletion is set to achieve the purpose of deleting data on each shard in batches, thereby improving the deletion performance while avoiding excessive pressure on the cluster.

[0040] In an optional embodiment, before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, the method further includes: determining the target data volume according to the processing speed of the processing node in the processing node set, and setting a parameter for indicating the target data volume in the target deletion instruction, wherein the processing speed of the processing node is used to represent the computing speed of the processing node itself, and / or the speed at which the processing node deletes the data stored in the corresponding shard. In this embodiment, the target data volume for each batch of deletions can be determined according to the processing speed of the processing nodes in the processing node set, and then the parameter for indicating the target data volume is set in the target deletion instruction, for example, the scroll_size parameter is configured for the DeleteByQuery operation. In actual applications, the above-mentioned target data volume can also be set according to other performance parameters of the processing node, and the embodiment of the present application does not limit this. Through this embodiment, the purpose of setting the target data volume for each batch of deletions according to the performance of the processing node is achieved.

[0041] In an optional embodiment, the deleting the target data subset in parallel and in batches through each processing node in the processing node set includes: when the number of processing nodes in the processing node set is N and the i-th processing node in the processing node set is set to delete the data belonging to the target data subset stored in the i-th group of shards in parallel and in batches, a batch of data belonging to the target data subset stored in each shard in the i-th group of shards is deleted in parallel through the following steps, wherein i is a positive integer greater than or equal to 1 and less than or equal to N, N is a positive integer greater than or equal to 1, and the i-th group of shards includes M i shards, M i is equal to 1 or is a positive integer greater than or equal to 2: i In the case where there is undeleted data in the data belonging to the target data subset stored in each of the shards, and the amount of the undeleted data is greater than the target data amount, in the M shards, i A batch of data to be deleted is determined from the undeleted data in each of the shards, and a total of M i Batch data, wherein the amount of data in the batch to be deleted is the target data amount; deleting the M batch data through the i-th processing node i In this embodiment, when the number of processing nodes in the processing node set corresponding to the target shard subset is N, for any processing node in the processing node set (such as the i-th processing node), the i-th processing node corresponds to the i-th group of shards, where the i-th group of shards may include one or more shards, such as M i The i-th processing node can delete a batch of data belonging to the above target data subset stored in each shard in the i-th group of shards in parallel and in batches. For example, when M i When there is undeleted data in the target data subset stored in the shards, and the amount of undeleted data is greater than the target data amount, i A batch of data with a target data volume is determined from the undeleted data in each of the shards, and a batch of data in each shard is deleted through the i-th node; optionally, when M i When there is undeleted data in the data belonging to the above-mentioned target data subset stored in the shards, and the data amount of the undeleted data is less than or equal to the target data amount, the undeleted data can be determined as a batch of data to be deleted, and the batch of data to be deleted can be deleted through the i-th processing node; similarly, for other processing nodes except the i-th processing node, the same method as above is used to perform batch deletion operations, so that the deletion operation can be performed on the data belonging to the target data subset on each shard in parallel and in batches through the above-mentioned N processing nodes.

[0042] In an optional embodiment, deleting the target data subset in parallel and in batches by each processing node in the processing node set includes: when the number of processing nodes in the processing node set is N and the i-th processing node in the processing node set is set to delete the data belonging to the target data subset stored in the i-th group of shards in parallel and in batches, determining whether the i-th processing node completes deleting M within a predetermined time period. i The current batch of data to be deleted in each of the shards, where i is a positive integer greater than or equal to 1 and less than or equal to N, N is a positive integer greater than or equal to 1, and the i-th group of shards includes M i shards, M i is equal to 1 or is a positive integer greater than or equal to 2; when it is determined that the i-th processing node has not completed deleting the current batch of data to be deleted in the first part of the shards within the predetermined time length, stop deleting the data stored in the first part of the shards that belongs to the target data subset; when it is determined that the i-th processing node has completed deleting the current batch of data to be deleted in the second part of the shards within the predetermined time length, continue to delete the next batch of data to be deleted in the second part of the shards in parallel. In this embodiment, when the number of processing nodes in the processing node set corresponding to the above-mentioned target shard subset is N, for any processing node in the processing node set (such as the i-th processing node), the i-th processing node corresponds to the i-th group of shards, where the i-th group of shards may include one or more shards, such as M. i The i-th processing node can delete a batch of data belonging to the above target data subset stored in each shard in the i-th group of shards in parallel and in batches, and first determine whether the i-th processing node completes the deletion within a predetermined time (for example, 10 minutes, or 30 minutes, or other time). i The current batch of data to be deleted in each of the shards in the shards is determined to be not completed within the predetermined time period by the i-th processing node. i When the current batch of data to be deleted in the first part of the shards (part of the shards in the shards), the data belonging to the target data subset stored in the first part of the shards is stopped from being deleted. Optionally, in actual applications, after the current batch of data to be deleted in the first part of the shards is completely deleted, the subsequent batches of deletion operations on the first part of the shards are stopped. When it is determined that the i-th processing node has completed the deletion of the second part of the shards (such as the M above) within the predetermined time, the deletion operation is stopped. iWhen deleting the current batch of data to be deleted in the partial shards (except the first partial shards mentioned above) in the ith shards, the next batch of data to be deleted in the second partial shards can be deleted in parallel; that is, by setting a timeout for the deletion operation, if a batch of operations cannot be completed within the set timeout, the deletion operation will fail; similarly, for other processing nodes except the i-th processing node, the batch deletion operation is performed in the same way as mentioned above. Through this embodiment, by setting a predetermined time length, abnormal deletion operations can be discovered in time to avoid the problem of slow execution of the deletion operation on some shards and continuous occupation of resources, thereby achieving the purpose of improving the stability of the deletion operation.

[0043] In an optional embodiment, before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, the method further includes: determining the predetermined duration according to the processing speed of the processing node in the processing node set, and setting a parameter for indicating the predetermined duration in the target deletion instruction, wherein the processing speed of the processing node is used to represent the computing speed of the processing node itself, and / or the speed at which the processing node deletes the data stored in the corresponding shard. In this embodiment, the predetermined duration can be determined according to the processing speed of the processing node in the processing node set, and then a parameter for indicating the predetermined duration (or called timeout) is set in the target deletion instruction, for example, the scroll parameter is configured for the DeleteByQuery operation. In actual applications, the above-mentioned predetermined duration can also be set according to other performance parameters of the processing node, and the embodiment of the present application does not limit this. Through this embodiment, the purpose of setting the predetermined duration of each batch of deletion operations according to the performance of the processing node is achieved, and the purpose of timely discovering abnormal deletion operations can be achieved.

[0044] Obviously, the above described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The present invention will now be specifically described in conjunction with the embodiments, and the following takes deleting data corresponding to an index of ES as an example.

[0045] The embodiment of the present invention proposes a method for improving the deletion performance and stability of Elasticsearch, adaptively specifies the deletion parameters of DeleteByQuery for different indexes, and optimizes the deletion performance of DeleteByQuery from the bottom layer. The embodiment of the present invention is described in detail below.

[0046] 1. Optimize the underlying deletion performance of DeleteByQuery

[0047] An Elasticsearch index consists of many shards, each of which is a Lucene instance. When executing DeleteByQuery on an index, the actual operation is performed on the shard. When deleting data, the shard will obtain the values ​​of the _seq_no field and the _primary_term field (corresponding to the aforementioned specified fields) to compare the version number of the document when deleting to avoid the mistaken deletion of documents with newer version numbers. The _seq_no field and the _primary_term field are data of the NumericDocValues ​​type, which has multiple storage methods. Different storage methods have different space usage and reading speed. Generally, the storage method for NumericDocValues ​​type data in Lucene is selected as follows: If the block storage method (corresponding to the aforementioned second storage method) can save more than 10% of space, then the block storage method is used; otherwise, the ordinary storage method (corresponding to the aforementioned first storage method) is used. The block storage method can save a certain amount of space compared to the ordinary storage method, but the reading speed will be relatively slower.

[0048] In Elasticsearch, the data of an index is stored in many different shards. Therefore, different storage methods are often used between shards because different data is stored. This leads to different speeds of reading related field values ​​in different shards during the deletion process, which in turn leads to large differences in the time it takes to complete the deletion operation. Since a deletion process is not truly completed until the deletion operations on all shards are completed, the different deletion speeds between shards reduce the overall deletion performance of the index.

[0049] By prohibiting the data in the _seq_no field and the _primary_term field from being stored in block mode, the situation where some shards are slowly deleted due to the block storage method is eliminated, so that all shards can complete the deletion operation quickly, effectively improving the overall deletion performance of the index.

[0050] At the same time, this embodiment can optimize the performance of DeleteByQuery without increasing the extra storage space. The index file of Elasticsearch contains data in many different formats, and docvalue data only occupies a part of the index file; there are five types of docvalue data, namely SortedDocValues, SortedSetDocValues, NumericDocValues, SortedNumericDocValues ​​and BinaryDocValues, and the data of NumericDocValues ​​type is only one of them; the data of the _seq_no field and the _primary_term field is also only a small part of the data of the NumericDocValues ​​type; at the same time, only a part of the data of the _seq_no field and the _primary_term field is stored in block mode. Therefore, disabling the block storage mode for the _seq_no field and the _primary_term field can negligibly increase the extra storage space usage.

[0051] The embodiment of the present invention prohibits the use of block storage for the _seq_no field and the _primary_term field involved in DeleteByQuery in Elasticsearch, thereby increasing the speed of shard deletion and improving the overall deletion performance of the index without increasing additional storage space. DeleteByQuery is the main method provided by Elasticsearch to delete specific data in an index. Through DeleteByQuery, data within a specified time range or data with a specified field value can be deleted from the index, while retaining other useful data in the index.

[0052] The stored procedures for data of type NumericDocValues ​​are as follows Figure 3 As shown, including:

[0053] S302, NumericDocValues ​​type data writing, that is, requesting to write NumericDocValues ​​type data;

[0054] S304, determine whether it is the _seq_no field and the _primary_term field;

[0055] S306, when the judgment result of the above step S304 is yes, it is forbidden to use the block method to store data, that is, it is forbidden to use the block method to store the data of the _seq_no field and the _primary_term field;

[0056] S308, writing data, for example, writing data into the _seq_no field and the _primary_term field using a common storage method;

[0057] When the judgment result of the above step S304 is no, the NumericDocValues ​​type data in the above step S302 can be written. In practical applications, the data can be stored in a block manner or in a common storage manner.

[0058] S310, end.

[0059] 2. DeleteByQuery delete parameter adaptation

[0060] The overall process of deleting data in Elasticsearch is as follows Figure 4 As shown, the process includes:

[0061] S402, obtaining the number of shards of the target index (corresponding to the number of shards in the aforementioned target shard subset);

[0062] S404, configure the slices parameter using the number of slices (corresponding to the number of slices allowed to perform parallel deletion as described above);

[0063] S406, configuring the scroll_size parameter (corresponding to the target data volume for each batch of deletions);

[0064] S408, configuring scroll parameters (corresponding to the aforementioned predetermined duration);

[0065] S410, executing the DeleteByQuery operation, that is, corresponding to the aforementioned parallel and batch deletion of the target data set;

[0066] S412, end.

[0067] The above process is described in detail below.

[0068] The DeleteByQuery operation of Elasticsearch supports configuring many parameters. Different parameters have different effects. How to configure the parameters to optimize the performance and stability of DeleteByQeury is one of the methods to optimize DeleteByQuery.

[0069] DeleteByQuery supports the use of the slices parameter to parallelize the deletion process. This parallelization method can improve the performance of deletion by splitting the request into multiple smaller requests. If the slices parameter is configured too small, the server resources may not be fully utilized, resulting in a slow deletion process; if the slices parameter is configured too large, the server's computing resources may be occupied and other operations may be affected. In actual usage scenarios, the number of shards of an index is often carefully designed so that the index has a reasonable number of shards on each node. Therefore, making the slices size consistent with the number of shards of the target index can both reasonably use server resources and improve deletion performance.

[0070] DeleteByQuery processes the entire deletion process in batches, obtaining a batch of data each time for deletion. By setting the scroll_size parameter (corresponding to the aforementioned target data volume), the size of the data in each batch of DeleteByQuery can be specified. If the total amount of data to be processed remains unchanged, if the scroll_size parameter is configured too small, the data processed by DeleteByQuery in each batch will be small, and the total number of batches processed will increase, which will increase unnecessary network time consumption and increase the time consumption of DeleteByQuery; if the scroll_size parameter is configured too large, DeleteByQuery processes too much data in each batch, which may cause excessive pressure on the cluster. It has been verified in practice that setting the scroll_size value to 5000 is a reasonable value. Of course, the scroll_size parameter can be set to other values ​​according to the needs of the actual application scenario. It achieves the effect of improving the performance of DeleteByQuery without causing too much pressure on the cluster.

[0071] The DeleteByQuery operation has a timeout. If a batch of operations cannot be completed within the set timeout, the DeleteByQuery operation will fail. The timeout can be specified by setting the scroll parameter (corresponding to the aforementioned predetermined duration). If the scroll parameter is configured too small, DeleteByQuery deletion failures may occur frequently; if the scroll parameter is configured too large, some abnormal and slow-running DeleteByQuery operations will continue to occupy resources and cannot be discovered in time. It has been verified in practice that setting the scroll value to 30 minutes is a reasonable value. Of course, the scroll parameter can be set to other values ​​according to the needs of the actual application scenario. The purpose of both improving the stability of the DeleteByQuery operation and discovering abnormal DeleteByQuery operations in time is achieved.

[0072] In the above embodiment, the slices parameter of DeleteByQuery is adaptively specified for different indexes, so that when it is applied to different indexes, it can always use the most appropriate number of concurrency to perform operations, thereby achieving the best performance in terms of usage. At the same time, in order to take into account the performance and stability of DeleteByQeury, the value of the parameter scroll_size and the value of the parameter scroll are configured to avoid deletion failures.

[0073] The embodiment of the present invention provides a method for directly optimizing the underlying deletion performance of DeleteByQuery, and improves the deletion performance of ES shards by optimizing the storage method of specific fields. For the _seq_no field and the _primary_term field involved in DeleteByQuery in Elasticsearch, the possibility of using block storage is prohibited, which improves the speed of shard deletion and improves the overall deletion performance of the index without adding additional storage space. In addition, the embodiment of the present invention provides a method for adaptively specifying DeleteByQuery deletion parameters, which can effectively improve the deletion performance and the stability of deletion in combination with the configuration optimization of key parameters. The slices parameter of DeleteByQuery is adaptively specified for different indexes, so that when it is applied to different indexes, it can always use the most appropriate number of concurrent operations to achieve the best performance in terms of usage. At the same time, in order to take into account the performance and stability of DeleteByQeury, the value of the configuration parameter scroll_size is 5000, and the value of the parameter scroll is 30m to avoid the occurrence of deletion failures.

[0074] Compared with the related art, the embodiment of the present invention can effectively improve the deletion performance of DeleteByQuery and improve the stability of DeleteByQuery. For the _seq_no field and _primary_term field involved in DeleteByQuery in Elasticsearch, the possibility of using block storage is prohibited, which improves the speed of shard deletion and improves the overall deletion performance of the index without increasing additional storage space. . The key slices parameter of DeleteByQuery is adaptively specified for different indexes, so that when it is applied to different indexes, it can always use the most appropriate number of concurrency to operate, thereby achieving the best performance in terms of usage. At the same time, the value of the scroll_size parameter and the scroll parameter are configured to avoid deletion failures and improve the stability of DeleteByQeury.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0076] In this embodiment, a data deletion device is also provided. Figure 5 is a structural block diagram of a data deletion device according to an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0077] A storage module 502 is used to store the target data set into each shard in the target shard set, wherein the value of the specified field in each data in the target data set is stored according to a first storage method, the storage methods allowed for the value of the specified field in each data in the target data set include the first storage method and a second storage method different from the first storage method, and a first deletion speed of the value of the specified field stored in the first storage method is greater than a second deletion speed of the value of the specified field stored in the second storage method;

[0078] A first acquisition module 504 is configured to acquire, in response to a target deletion instruction, a target shard subset corresponding to a target index in the target shard set, wherein the target deletion instruction is used to request deletion of a target data subset corresponding to the target index in the target data set, the target data subset to be deleted corresponding to the target index is stored in the target shard subset, and each shard in the target shard subset stores part of the data in the target data subset;

[0079] A first determination module 506 is used to determine a processing node set corresponding to the target shard subset, wherein the processing node set is used to delete the target data subset stored in the target shard subset in parallel;

[0080] The deletion module 508 is used to delete the target data subset stored in the target shard subset in parallel through the processing node set, wherein the value of the specified field of each data in the target data subset is deleted according to the first deletion speed.

[0081] In an optional embodiment, the above-mentioned device also includes: an establishing module, which is used to establish a corresponding relationship between the target shard subset and the processing node set, before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, when each data in the target data subset is stored in the corresponding shard in the target shard subset, wherein each processing node in the processing node set has a corresponding relationship with at least one shard in the target shard subset, and the number of shards corresponding to each processing node is within a preset number range, or the difference between the number of shards corresponding to each processing node is less than or equal to a preset threshold.

[0082] In an optional embodiment, the above-mentioned first determination module 506 includes: a first determination unit, which is used to determine the processing nodes that have a corresponding relationship with each shard in the target shard subset when the target deletion instruction indicates that the number of shards allowed to perform parallel deletion is the number of shards in the target shard subset, and obtain the processing node set.

[0083] In an optional embodiment, the above-mentioned deletion module 508 includes: a first deletion unit, which is used to delete the target data subset in parallel and in batches through each processing node in the processing node set when the target deletion instruction indicates the target data amount for each batch of deletion, wherein the amount of data in the target data subset deleted by each processing node in each batch is the target data amount.

[0084] In an optional embodiment, the above-mentioned device also includes: a second determination module, which is used to determine the target data volume according to the processing speed of the processing nodes in the processing node set before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, and set a parameter for indicating the target data volume in the target deletion instruction, wherein the processing speed of the processing node is used to indicate the computing speed of the processing node itself, and / or the speed at which the processing node deletes the data stored in the corresponding shard.

[0085] In an optional embodiment, the above-mentioned deletion module 508 also includes: a second deletion unit, which is used to delete a batch of data belonging to the target data subset stored in each shard in the i-th group of shards in parallel through the following steps, when the number of processing nodes in the processing node set is N and the i-th processing node in the processing node set is set to delete the data belonging to the target data subset stored in the i-th group of shards in parallel and in batches, wherein i is a positive integer greater than or equal to 1 and less than or equal to N, N is a positive integer greater than or equal to 1, and the i-th group of shards includes M ishards, M i is equal to 1 or is a positive integer greater than or equal to 2: i In the case where there is undeleted data in the data belonging to the target data subset stored in each of the shards, and the amount of the undeleted data is greater than the target data amount, in the M shards, i A batch of data to be deleted is determined from the undeleted data in each of the shards, and a total of M i Batch data, wherein the amount of data in the batch to be deleted is the target data amount; deleting the M batch data through the i-th processing node i Batch data.

[0086] In an optional embodiment, the above-mentioned deletion module 508 further includes: a second determination unit, which is used to determine whether the i-th processing node completes the deletion of M within a predetermined time period when the number of processing nodes in the processing node set is N and the i-th processing node in the processing node set is set to delete the data belonging to the target data subset stored in the i-th group of shards in parallel and in batches. i The current batch of data to be deleted in each of the shards, where i is a positive integer greater than or equal to 1 and less than or equal to N, N is a positive integer greater than or equal to 1, and the i-th group of shards includes M i shards, M i is equal to 1 or is a positive integer greater than or equal to 2; a stopping unit, used to stop deleting the data belonging to the target data subset stored in the first part of the shards when it is determined that the i-th processing node has not completed deleting the current batch of data to be deleted in the first part of the shards within the predetermined time length; a third deleting unit, used to continue to delete the next batch of data to be deleted in the second part of the shards in parallel when it is determined that the i-th processing node has completed deleting the current batch of data to be deleted in the second part of the shards within the predetermined time length.

[0087] In an optional embodiment, the above-mentioned device also includes: a third determination module, which is used to determine the predetermined time length according to the processing speed of the processing nodes in the processing node set before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, and set a parameter for indicating the predetermined time length in the target deletion instruction, wherein the processing speed of the processing node is used to indicate the computing speed of the processing node itself, and / or the speed at which the processing node deletes the data stored in the corresponding shard.

[0088] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0089] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0090] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0091] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0092] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0093] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0094] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data deletion method, characterized in that: include: The target data set is stored in each shard in the target shard set, wherein the value of the specified field in each data in the target data set is stored according to a first storage method, the storage methods allowed for the value of the specified field in each data in the target data set include the first storage method and a second storage method different from the first storage method, and a first deletion speed of the value of the specified field stored in the first storage method is greater than a second deletion speed of the value of the specified field stored in the second storage method; In response to a target deletion instruction, a target shard subset corresponding to a target index is obtained in the target shard set, wherein the target deletion instruction is used to request deletion of a target data subset corresponding to the target index in the target data set, the target shard subset stores the target data subset to be deleted corresponding to the target index, and each shard in the target shard subset stores part of the data in the target data subset; Determine a processing node set corresponding to the target shard subset, wherein the processing node set is used to delete the target data subset stored in the target shard subset in parallel; Deleting the target data subset stored in the target shard subset in parallel by the processing node set, wherein the value of the specified field of each data in the target data subset is deleted according to the first deletion speed; The step of deleting the target data subset stored in the target shard subset in parallel by using the processing node set comprises: when the target deletion instruction indicates a target data amount to be deleted in each batch, deleting the target data subset in parallel and in batches by using each processing node in the processing node set, wherein the amount of data in the target data subset deleted in each batch by each processing node is the target data amount; Before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, the method further includes: determining the target data volume according to the processing speed of the processing node in the processing node set, and setting a parameter for indicating the target data volume in the target deletion instruction, wherein the processing speed of the processing node is used to indicate the computing speed of the processing node itself, and / or the speed at which the processing node deletes data stored in the corresponding shard.

2. The method according to claim 1, characterized in that: Before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, the method further includes: In the case where each data in the target data subset is stored in a corresponding shard in the target shard subset, a corresponding relationship is established between the target shard subset and the processing node set, wherein each processing node in the processing node set has a corresponding relationship with at least one shard in the target shard subset, and the number of shards corresponding to each processing node is within a preset number range, or the difference between the numbers of shards corresponding to each processing node is less than or equal to a preset threshold.

3. The method according to claim 2, characterized in that The determining of a set of processing nodes corresponding to the target shard subset includes: When the target deletion instruction indicates that the number of shards allowed to be deleted in parallel is the number of shards in the target shard subset, a processing node corresponding to each shard in the target shard subset is determined to obtain the processing node set.

4. The method according to claim 1, characterized in that The deleting the target data subset in batches in parallel through each processing node in the processing node set includes: In the case where the number of processing nodes in the processing node set is N and the i-th processing node in the processing node set is set to delete the data belonging to the target data subset stored in the i-th group of shards in parallel and in batches, a batch of data belonging to the target data subset stored in each shard in the i-th group of shards is deleted in parallel by the following steps, wherein i is a positive integer greater than or equal to 1 and less than or equal to N, N is a positive integer greater than or equal to 1, and the i-th group of shards includes M i shards, M i Equal to 1 or a positive integer greater than or equal to 2: In the M i In the case where there is undeleted data in the data belonging to the target data subset stored in each of the shards, and the amount of the undeleted data is greater than the target data amount, in the M shards, i A batch of data to be deleted is determined from the undeleted data in each of the shards, and a total of M i Batch data, wherein the amount of data in the batch of data to be deleted is the target data amount; deleting the M batch of data through the i-th processing node i Batch data.

5. The method according to claim 1, characterized in that The deleting the target data subset in batches in parallel through each processing node in the processing node set includes: In the case where the number of processing nodes in the processing node set is N and the i-th processing node in the processing node set is set to delete the data belonging to the target data subset stored in the i-th group of shards in parallel and in batches, determining whether the i-th processing node completes the deletion of M within a predetermined time period. i The current batch of data to be deleted in each of the shards, where i is a positive integer greater than or equal to 1 and less than or equal to N, N is a positive integer greater than or equal to 1, and the i-th group of shards includes M i shards, M i is equal to 1 or is a positive integer greater than or equal to 2; If it is determined that the i-th processing node has not completed deleting the current batch of data to be deleted in the first part of the shards within the predetermined time period, stop deleting the data belonging to the target data subset stored in the first part of the shards; When it is determined that the i-th processing node has completed deleting the current batch of data to be deleted in the second part of the shards within the predetermined time period, the next batch of data to be deleted in the second part of the shards continues to be deleted in parallel.

6. A data deletion device, characterized in that: include: A storage module, used for storing a target data set into each shard in a target shard set, wherein the value of a specified field in each data in the target data set is stored in accordance with a first storage method, the storage methods allowed for the value of the specified field in each data in the target data set include the first storage method and a second storage method different from the first storage method, and a first deletion speed of the value of the specified field stored in the first storage method is greater than a second deletion speed of the value of the specified field stored in the second storage method; A first acquisition module is configured to, in response to a target deletion instruction, acquire a target shard subset corresponding to a target index in the target shard set, wherein the target deletion instruction is used to request deletion of a target data subset corresponding to the target index in the target data set, the target shard subset stores the target data subset to be deleted corresponding to the target index, and each shard in the target shard subset stores part of the data in the target data subset; A first determination module is used to determine a processing node set corresponding to the target shard subset, wherein the processing node set is used to delete the target data subset stored in the target shard subset in parallel; a deletion module, configured to delete the target data subset stored in the target shard subset in parallel through the processing node set, wherein the value of the specified field of each data in the target data subset is deleted according to the first deletion speed; The deletion module includes: a first deletion unit, configured to delete the target data subset in parallel and in batches through each processing node in the processing node set when the target deletion instruction indicates a target data amount for each batch of deletions, wherein the amount of data in the target data subset deleted by each processing node in each batch is the target data amount; The device also includes: a second determination module, which is used to determine the target data volume according to the processing speed of the processing nodes in the processing node set before obtaining the target shard subset corresponding to the target index in the target shard set in response to the target deletion instruction, and set a parameter for indicating the target data volume in the target deletion instruction, wherein the processing speed of the processing node is used to represent the computing speed of the processing node itself, and / or the speed at which the processing node deletes the data stored in the corresponding shard.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 5 are implemented.

Images