Unlock instant, AI-driven research and patent intelligence for your innovation.

Distributed data deleting method, device and equipment based on deleting queue

A technology of distributed data and deletion method, applied in the computer field, can solve problems such as affecting the computing performance of distributed clusters, generating large temporary data, etc.

Inactive Publication Date: 2020-09-04
LANGCHAO ELECTRONIC INFORMATION IND CO LTD
View PDF3 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of this application is to provide a distributed data deletion method, device, device, and readable storage medium based on the deletion queue to solve the problem that the current distributed cluster will generate a large amount of temporary data in the big data scenario and affect the distributed cluster. Computational Performance Issues

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed data deleting method, device and equipment based on deleting queue
  • Distributed data deleting method, device and equipment based on deleting queue
  • Distributed data deleting method, device and equipment based on deleting queue

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0044] The following is an introduction to Embodiment 1 of a distributed data deletion method based on a deletion queue provided by this application, see figure 1 , embodiment one includes:

[0045] S101. Generate a deletion information item according to the file object to be deleted, wherein the file object to be deleted is an intermediate result in a big data computing scenario, and the deletion information item includes the storage information of the metadata object and the storage information of the data object to be deleted ;

[0046] S102. Add the deletion information item to a deletion queue;

[0047] S103. Place the deletion queue in the form of a log file, and send a deletion completion message to the business layer after the disk placement is completed;

[0048] S104. Perform an asynchronous deletion operation according to the deletion queue after placing the disk.

[0049] In this embodiment, the file object to be deleted refers to an intermediate result generate...

Embodiment 2

[0057] see figure 2 , embodiment two specifically includes:

[0058] S201, traversing the intermediate results in the big data computing scenario, and judging whether the number of links of the currently traversed file object is 0, wherein the number of links represents the number of inodes mounted to the file object; if so, jump to S202, Otherwise, traverse the next file object;

[0059] This embodiment can actively traverse the intermediate results, and evaluate whether the file object obtained through the traversal is a file object to be deleted. The specific evaluation method is: judge whether the link number of the file object is 0 (nlink=0) or whether the file object has other front-end business access, if the link number is 0 or there is no other front-end business access, the file object is considered to be deleted file object.

[0060] S202. Determine that the currently traversed file object is a file object to be deleted, and generate a deletion information item ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed data deletion method based on a deletion queue. The method comprises the steps of generating a deletion information item according to a to-be-deleted file object;adding the deletion information items to a deletion queue; dropping the deletion queue into a disk in the form of a log file, and sending a deletion completion message to a service layer after disk dropping is completed; and executing asynchronous deletion operation according to the deletion queue after disk falling. Visibly, the deletion queue is introduced into the method, the deletion queue isrecorded in a log file mode, asynchronous deletion operation is achieved, the influence of a deletion task of an intermediate result on the calculation process in a big data calculation scene is avoided, and the calculation performance of the distributed cluster is improved. In addition, the invention further provides a distributed data deleting device and equipment based on the deleting queue and a readable storage medium, and the technical effect of the distributed data deleting device and equipment corresponds to the technical effect of the method.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a method, device, device and readable storage medium for deleting queue-based distributed data deletion. Background technique [0002] At present, distributed cluster storage is widely used in various big data scenarios, such as video surveillance, radio and television media assets, and high performance. There are massive amounts of data in the distributed file system, and users need to clean up the temporary data generated by calculations in a timely manner, especially in big data scenarios such as genetic calculations. A genetic calculation will generate a large number of intermediate files, and how to delete temporary files in a timely manner has become a computing bottleneck. [0003] It can be seen that how to improve the data deletion efficiency of the distributed cluster is an urgent problem to be solved by those skilled in the art. Contents of the invention ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/16G06F16/182
CPCG06F16/162G06F16/182
Inventor 刘洪栋肖国栋
Owner LANGCHAO ELECTRONIC INFORMATION IND CO LTD