Data cleaning method, device, medium and computing equipment

Abstract

An embodiment of the present disclosure provides a data cleaning method, including: monitoring the working status of the application; the working status includes at least an application end event; in response to the application end event, sending the application identifier of the application to the first component, so that the first component sends a cleaning request to the second component; the cleaning request includes the application identifier; in response to the second component finding a target folder containing the application identifier, performing a cleaning operation on the target folder; the target folder is created based on the application identifier when the application writes the application data to a container. In the above process, on the one hand, targeted cleaning of applications is achieved, avoiding the consumption of disk input and output caused by large-scale cleaning operations, and avoiding affecting other businesses; on the other hand, timely and efficient cleaning is achieved, speeding up resource recovery, and minimizing the impact of data space occupation.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more specifically, to a data cleaning method, apparatus, medium, and computing device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] During the operation of an application, a lot of data is usually generated. For example, when a user uses a browser, some temporary files are generated so that the application can respond quickly to the user when the user uses the browser later.

[0004] However, as these temporary files grow in number, if they are not cleaned up in time, when the data takes up too much space and leads to insufficient storage space, it is very likely to cause the application to freeze. Therefore, by performing data cleaning operations on the above data, the above application freeze problem can be solved and the application running speed can be prevented from being reduced. Summary of the Invention

[0005] In this context, embodiments of the present disclosure are intended to provide a data cleaning method and apparatus.

[0006] In a first aspect of an embodiment of the present disclosure, a data cleaning method is provided, which is applied to a resource scheduler, on which a first component, a second component, and a containerized application are deployed, comprising:

[0007] Monitoring the working status of the application; the working status at least includes an application end event;

[0008] In response to the application end event, sending the application identifier of the application to the first component, so that the first component sends a cleanup request to the second component; the cleanup request includes the application identifier;

[0009] In response to the second component finding the target folder containing the application identifier, a cleanup operation is performed on the target folder; the target folder is created based on the application identifier when the application writes the application data into the container.

[0010] In a second aspect of the embodiments of the present disclosure, a data cleaning device is provided, which is applied to a resource scheduler, on which a first component, a second component, and a containerized application are deployed, including:

[0011] A monitoring module, monitoring the working status of the application; the working status at least includes an application end event;

[0012] a sending module, in response to the application end event, sending the application identifier of the application to the first component, so that the first component sends a cleanup request to the second component; the cleanup request includes the application identifier;

[0013] The cleaning module performs a cleaning operation on the target folder in response to the second component finding the target folder containing the application identifier; the target folder is created based on the application identifier when the application writes the application data into the container.

[0014] In a third aspect of the embodiments of the present disclosure, a storage medium is provided; computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method described below are implemented:

[0015] Monitoring the working status of the application; the working status at least includes an application end event;

[0016] In response to the application end event, sending the application identifier of the application to the first component, so that the first component sends a cleanup request to the second component; the cleanup request includes the application identifier;

[0017] In response to the second component finding the target folder containing the application identifier, a cleanup operation is performed on the target folder; the target folder is created based on the application identifier when the application writes the application data into the container.

[0018] In a fourth aspect of the embodiments of the present disclosure, there is provided a computing device, comprising:

[0019] a processor; and a memory for storing processor-executable instructions;

[0020] The processor executes the executable instructions to implement the following steps:

[0021] Monitoring the working status of the application; the working status at least includes an application end event;

[0022] In response to the application end event, sending the application identifier of the application to the first component, so that the first component sends a cleanup request to the second component; the cleanup request includes the application identifier;

[0023] In response to the second component finding the target folder containing the application identifier, a cleanup operation is performed on the target folder; the target folder is created based on the application identifier when the application writes the application data into the container.

[0024] The above embodiments of the present disclosure have at least the following beneficial effects:.

[0025] By monitoring the working status of the application and sending the application identifier to the first component when the application ends, the first component will send a cleanup request for the application to the second component. Furthermore, the second component will find the folder corresponding to the application and perform the cleanup operation. Through the above technical solution, on the one hand, it is possible to achieve targeted cleanup of applications, avoiding the consumption of disk input and output caused by large-scale cleanup operations and avoiding affecting the normal operation of other businesses; on the other hand, it can achieve timely and efficient cleanup of data, speed up the speed of resource recovery, minimize the impact of data space occupation, and save costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0027] Figure 1 The following schematically shows a flow chart of a data cleaning method according to an embodiment of the present disclosure;

[0028] Figure 2 The following schematically shows a flow chart of a data cleaning method according to an embodiment of the present disclosure;

[0029] Figure 3 The following schematically shows a block diagram of a data cleaning device according to an embodiment of the present disclosure;

[0030] Figure 4 Schematically shows a schematic diagram of a data cleaning medium according to an embodiment of the present disclosure;

[0031] Figure 5 A schematic diagram of an electronic device capable of implementing the above method according to an embodiment of the present disclosure is schematically shown.

[0032] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0033] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer-readable storage medium. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0035] According to an embodiment of the present disclosure, a data cleaning method, apparatus, medium, and computing device are proposed.

[0036] It should be understood herein that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0037] The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0039] The inventors have discovered that data cleanup can usually only be performed on a scheduled basis, with the cleanup operation being performed on all data after a fixed time interval. However, when performing such scheduled cleanup, if the time interval is set too long, it will result in an excessive amount of data being cleaned at a single time, occupying a large amount of disk input and output resources, which may affect the normal operation of other services. Furthermore, although a shorter time interval can be set to avoid excessive data being cleaned at a single time, this may result in the deletion of data required by the application, affecting the normal use of the application, causing inconvenience to the user, and causing resource duplication.

[0040] In view of this, this specification provides a technical solution for targeted cleaning of the application by monitoring the working status of the application and sending the application identifier to the first component when the application ends, and the first component sends a cleanup request for the application to the second component. Furthermore, the second component finds the folder corresponding to the application and performs a cleanup operation.

[0041] The core technical concept of this manual is:

[0042] While writing the application's data into the container, a folder for storing the application data is created based on the application's corresponding application identifier. Furthermore, at the end of the application's lifecycle, the resource scheduler can monitor the application's working status and, in response to an application termination event, send the application identifier to the first component. The first component then controls the second component to clean up the folder containing the application identifier.

[0043] Through the above technical solutions, on the one hand, targeted cleaning of applications can be achieved, avoiding the consumption of disk input and output caused by large-scale cleaning operations, and avoiding affecting the normal operation of other businesses; on the other hand, timely and efficient data cleaning can be achieved, speeding up resource recovery, minimizing the impact of data space occupation, and saving costs.

[0044] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0045] Application Scenario Overview

[0046] As mentioned above, when users use various application clients, temporary files are often generated in the device's storage space so that they can quickly access the application the next time they use it. These temporary files are also called cached data. For example, cached data can be temporary files generated when users use a browser to access the Internet, temporary files generated when users use social media apps to view images or videos, and files downloaded when users update applications.

[0047] With the rapid development of cloud computing, applications no longer need to be installed locally on devices for use. Users can manage and use applications through the resource scheduler provided by the cloud platform. Usually, various types of containerized applications are deployed on the resource scheduler.

[0048] For example, the resource scheduler mentioned above can be Kubernetes, or "k8s" for short. It is a system for deploying, scaling, and managing containerized applications. In Kubernetes, you can create multiple containers, each of which runs an application instance. These containers run in pods, which are the smallest computing unit in Kubernetes. A pod can contain one or more containers, which are relatively tightly coupled together. In Kubernetes, pods are the carriers of applications, and applications can be accessed through the pod's IP address.

[0049] Kubernetes, as described above, is essentially a cluster of servers. A Kubernetes cluster primarily consists of a control node (Master) and worker nodes (Nodes), each of which has different components installed. The Master serves as the gateway and central hub of the cluster, responsible for cluster decision-making. The Nodes receive work instructions from the Master, create and destroy Pod objects accordingly, and adjust network rules for optimal routing and traffic forwarding.

[0050] When using these containerized applications, temporary files will also be left behind, which need to be cleaned up promptly. Furthermore, unlike devices used by ordinary consumers, not only must the performance impact of data usage be considered, but also the cost implications of data usage.

[0051] Taking Kubernetes as an example, it can only perform scheduled cleanup by shutting down containers. However, if the interval is too short, application data required for cloud computing tasks will be deleted, causing some tasks to be recalculated, affecting computing efficiency and increasing computing costs. Furthermore, due to the limited read and write speeds of Kubernetes, a short interval prevents complete cache cleanup. However, if the interval is too long, too much data will be cleaned at a time, occupying a large amount of disk I / O resources, wasting precious computing resources on data cleanup, and also increasing cloud computing costs. Furthermore, when cleaning cached data, Kubernetes is unable to achieve application awareness and distinguish the cached data corresponding to each application.

[0052] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0053] Exemplary Methods

[0054] The technical concept of this specification will be described in detail below through specific embodiments.

[0055] The present disclosure aims to provide a technical solution for targeted cleaning of an application by monitoring the working status of the application and sending the application identifier to a first component when the application ends, and the first component sends a cleanup request for the application to a second component. Furthermore, the second component finds the folder corresponding to the application and performs a cleanup operation.

[0056] See Figure 1 , Figure 1 This is a flowchart of a data cleaning method provided by an exemplary embodiment, wherein the method is applied to a resource scheduler and includes the following steps:

[0057] Step 101: A resource scheduler may monitor the working status of an application; the working status may include at least an application end event; wherein a first component, a second component, and a containerized application are deployed on the resource scheduler;

[0058] For example, the resource scheduler can monitor the event log corresponding to the application to determine the working status of the application and whether the application life cycle has ended.

[0059] Step 102: In response to the application end event, the application identifier of the application may be sent to the first component, so that the first component may send a cleanup request to the second component; the cleanup request may include the application identifier;

[0060] For example, when an application task ends, a preset function can be called to execute the corresponding logic, and the application identifier corresponding to the application can be sent to a specified address and received by the first component; further, the first component can send a cleanup request for the application to the second component based on the application identifier.

[0061] Step 103: In response to the second component finding a target folder containing the application identifier, a cleanup operation may be performed on the target folder; the target folder is created based on the application identifier when the application writes the application data into the container;

[0062] For example, after receiving the cleanup request, the second component may search for a target folder containing the above application identifier in the directory monitored by the second component, and if found, perform a cleanup operation on the target folder.

[0063] Through the above technical solutions, on the one hand, targeted cleaning of applications can be achieved, avoiding the consumption of disk input and output caused by large-scale cleaning operations, and avoiding affecting the normal operation of other businesses; on the other hand, timely and efficient data cleaning can be achieved, speeding up resource recovery, minimizing the impact of data space occupation, and saving costs.

[0064] In the above step 101, the first component, the second component, and the containerized application are deployed on the resource scheduler, and the resource scheduler can monitor the working status of the application.

[0065] The working status at least includes an application end event.

[0066] The resource scheduler can be the aforementioned Kubernetes, or other systems used in the cloud platform that can manage online applications.

[0067] In one embodiment shown, the application is constructed by a computing engine and is used to perform data computing.

[0068] Specifically, the above applications can be built by a computing engine that supports large-scale data processing and used to perform large-scale data analysis applications, such as SQL queries, text processing, or machine learning calculations.

[0069] Obviously, since the above-mentioned data analysis applications involve a very large amount of data when performing large-scale data processing, there is an even greater need for timely, efficient and targeted data cleaning.

[0070] The computing engine may be an Apache Spark computing engine or an Apache Hadoop computing engine.

[0071] Taking Spark as an example, this computing engine defines runtime events for applications to demonstrate the complete lifecycle of the application, including the entire process from application submission to application termination. The computing engine also provides a historical log service that can store application practice logs, so that application performance can be further optimized by analyzing these event logs.

[0072] In the above step 102, when the working status of the application is task completion, the resource scheduler can monitor the application completion event, and in response to the application completion event, send the application identifier corresponding to the application to the first component so that the first component can provide auxiliary services, and send a cleanup request containing the application identifier to the second component.

[0073] As can be seen from the above, the first component and the second component are deployed on the resource scheduler. It should be noted that the first component itself does not perform the cleanup operation, but acts as a bridge between the resource scheduler and the second component to notify the second component to perform the cleanup operation on the application.

[0074] For example, the running event corresponding to the application built by the above Spark at the end of the task is the application end event. By monitoring the running event of the Spark application, the resource scheduler can respond to the Spark application end event and send the application identifier of the Spark application to the first component; further, the first component sends a cleanup request containing the Spark application identifier to the second component.

[0075] In one embodiment shown, in response to the application end event, a preset function may be called to send an end message to the first component via a specified address; the end message includes an application identifier of the application.

[0076] For example, to implement sending the application identifier to the first component in a Spark application, a function ExtraListener can be preset, which can be inherited from the SparkListener class in the Spark source code. Furthermore, the auxiliary listening function of SparkApplication provided by the function ExtraListener can be used to make it possible to send an end message containing the ApplicationId to the service at the specified address, that is, to the first component, by calling the function ExtraListener when the SparkApplication task ends.

[0077] As can be seen from the foregoing, the resource scheduler may include a control node and a computing node. The computing node is used to execute the calculation of the application, so the second component for cleaning the application data may be deployed on the computing node.

[0078] In one embodiment shown, the second component is deployed on multiple computing nodes in the resource scheduler;

[0079] Furthermore, the first component may send the cleanup request to the second component according to a pre-stored second component list; the second component list records the address information corresponding to the second component.

[0080] Since the application sends a task completion message in a short time when the task ends, it is not suitable for the resource scheduler to directly send a cleanup request to the second component. Therefore, the first component can be deployed as a bridge between the resource scheduler and the second component.

[0081] For example, after receiving an application termination message, the first component can send a cleanup request to each second component deployed on each computing node based on the address information corresponding to each second component recorded in a pre-stored second component list. When deployed, the second component can register the host information and IP information of the node where it resides with the first component, so that the first component can add the second component information to the second component list.

[0082] It should be noted that since there may be many nodes in daily operations, using TCP connections to send cleanup requests will cause huge overhead and affect the stability of online cluster resources. Therefore, UDP broadcast can be used to send messages to reduce overhead.

[0083] In one embodiment shown, the second component creates copies of the second component on multiple computing nodes in the resource scheduler in a mirrored manner;

[0084] Furthermore, the second component list records the address information of the computing node where the copy of the second component is located when the copy of the second component is registered with the first component when the copy of the second component is created.

[0085] For example, when deploying the second component, you can mirror it from one compute node to multiple compute nodes, thereby creating a replica of the second component on each compute node. Accordingly, when creating the replica of the second component, it registers the address information of the compute node where the replica of the second component is located with the first component.

[0086] In one embodiment shown, in response to a new computing node being added to the resource scheduler, a copy of the second component is created on the new computing node in a mirrored manner.

[0087] For example, when a new computing node is added to the resource scheduler, the second component can continue to be mirrored to create another copy of the second component on the new computing node; accordingly, when the another copy is created, the address information of the new computing node where the another copy of the second component is located will be registered with the first component.

[0088] Taking Kubernetes as an example, a DaemonSet (daemon set) is started on each node in Kubernetes. DaemonSet can ensure that a copy of a Pod runs on all (or some) nodes. When a node joins the cluster, a new Pod is added for it. When a node is removed from the cluster, these Pods are also recycled. Deleting a DaemonSet will delete all the Pods it created.

[0089] In the above step 103, after receiving the cleanup request sent by the first component, the second component can check whether the target folder contains the application identifier, and if so, perform a cleanup operation on the target folder.

[0090] It should be noted that the target folder is created based on the application identifier when the application writes the application data into the container.

[0091] For example, when shuffling data to a local machine, the data is not accompanied by an application identifier. Therefore, after the Spark executor exits, the resource scheduler cannot determine which application the local data belongs to. Therefore, you can modify createLocalDirs in Spark's DiskBlockManager to prefix the shuffle folder name generated by the Spark executor with the Spark ApplicationId (the application identifier).

[0092] Continuing with the example, the second component may check whether there is a folder containing ApplicationId in the local Shuffle folder, and if so, perform a cleanup operation on the folder.

[0093] In an illustrated embodiment, in response to the second component finding a target folder containing the application identifier in a designated folder directory, a cleanup operation on the target folder may be performed.

[0094] For example, after receiving the cleanup request sent by the first component, the second component may scan whether there is a target folder containing an application identifier under the second component execution directory, and if so, perform a cleanup operation on the target folder.

[0095] From the above, it can be seen that when the first component sends a cleanup request to the second component, it can use UDP broadcast. However, UDP broadcast may cause packet loss. Therefore, a timer can be set for the second component to prevent data from not being cleaned up in time due to packet loss.

[0096] In one embodiment shown, a timer is preset in the second component;

[0097] Furthermore, in response to the triggering of the timer, the second component is called to perform a cleanup operation on the folders that have existed for a period exceeding a threshold.

[0098] For example, a timer may be set in the second component. When the timer is triggered, folders whose existence time exceeds a threshold are deleted, thereby clearing the residual cached data in the event of packet loss.

[0099] In one embodiment shown, the container is mounted to a specified disk;

[0100] Furthermore, when the application writes the application data into the container, a target folder is created in the disk mounted by the resource scheduler based on the application identifier.

[0101] Taking Kubernetes as an example, the cached data generated in daily business may be very large. If the storage space provided by Kubernetes is used, firstly, the storage space is limited and insufficient to support the huge amount of data. When encountering a large amount of data writing, the Pod will self-destruct, resulting in data loss. Secondly, the read and write rate of Kubernetes is low, not as good as the real disk read and write.

[0102] Therefore, the container can be mounted to a specified disk to cache data. Furthermore, when the application writes the application data to the container, a folder can be created based on the application identifier in the mounted specified disk.

[0103] Through the above technical solutions, on the one hand, targeted cleaning of applications can be achieved, avoiding the consumption of disk input and output caused by large-scale cleaning operations, and avoiding affecting the normal operation of other businesses; on the other hand, timely and efficient data cleaning can be achieved, speeding up resource recovery, minimizing the impact of data space occupation, and saving costs.

[0104] Next, we will use Spark on Kubernetes as an example to describe how to clean up residual shuffle files based on the Spark Application status. Figure 2 , Figure 2 The flowchart of a data cleaning method provided by an exemplary embodiment includes the following steps:

[0105] Step 201: Start the first component Cleaner-GateWay on the Kubernetes cluster.

[0106] Among them, the above-mentioned first component Cleaner-GateWay can listen to two pre-configured ports, one port is used to communicate with the resource scheduler and receive application end messages, and the other port is used to communicate with the second component, receive the registration information of the second component and send a cleanup request to the second component.

[0107] Step 202: Start the second component Block-Cleaner on the Kubernetes cluster.

[0108] The second component, Block-Cleaner, can be created as a DaemonSet, generating a Block-Cleaner instance on each scheduled Kubernetes node. When each Block-Cleaner instance is created, it registers its node host information and IP address with the configured first component, Cleaner-GateWay.

[0109] Step 203: The Spark application is started and a folder containing the Spark AppId prefix is ​​created.

[0110] When the Spark application starts, the preset function ExtraListener is loaded, and Spark Executors are started in the Kubernetes cluster. Furthermore, when Spark Executors start, a folder for the Shuffle link is created on the node disk where the Executor is located. The folder uses the Spark AppId as the application identifier.

[0111] The above function ExtraListener inherits from the SparkListener class in the Spark source code. By overriding the onApplicationEnd function, it sends the ApplicationId to the service at the specified address.

[0112] Step 204: The Spark application ends, and an application end event is generated.

[0113] When a Spark application ends, an application end event, SparkApplicationEnd, is generated. When ExtraListener listens to this application end event, it can send an application end message with the Spark AppId to the first component, Cleaner-GateWay.

[0114] In step 205 , the first component Cleaner-GateWay sends a cleaning request to the second component Block-Cleaner.

[0115] After receiving the application end message, the first component Cleaner-GateWay may send a cleaning request to all registered second components Block-Cleaner according to the pre-stored second component list. The cleaning request includes Spark AppId.

[0116] Step 206: The second component Block-Cleaner searches for a target folder.

[0117] After receiving the cleaning request, the second component Block-Cleaner will scan the specified directory corresponding to its own node to find whether there is a folder containing the Spark AppId prefix. If so, it will perform a cleaning operation on the target folder.

[0118] Step 207: trigger a timer to clean up folders whose existence time exceeds a threshold.

[0119] A timer can be preset in the second component Block-Cleaner. When the timer is triggered, a cleaning operation can be performed on folders that have existed for longer than a threshold, thereby preventing the cached data from not being cleaned up in time due to packet loss of the cleaning request.

[0120] In the above process, on the one hand, targeted cleaning of applications can be achieved, avoiding the consumption of disk input and output caused by large-scale cleaning operations, and avoiding affecting the normal operation of other businesses; on the other hand, timely and efficient cleaning of data can be achieved, accelerating the speed of resource recovery, minimizing the impact of data space occupation, and at the same time avoiding the waste of cloud computing platform resources and saving costs.

[0121] Exemplary devices

[0122] After introducing the method of the exemplary embodiment of the present disclosure, see Figure 3 , Figure 3 It is a block diagram of a data cleaning device provided by an exemplary embodiment.

[0123] The implementation process of the functions and effects of each module in the following device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here. For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment.

[0124] like Figure 3 As shown, the data cleaning device 300 may include: a monitoring module 301, a sending module 302, and a cleaning module 303.

[0125] The monitoring module 301 is configured to monitor the working status of the application; the working status at least includes an application end event;

[0126] The sending module 302 is configured to send the application identifier of the application to the first component in response to the application end event, so that the first component sends a cleanup request to the second component; the cleanup request includes the application identifier;

[0127] The cleaning module 303 is configured to perform a cleaning operation on the target folder in response to the second component finding the target folder containing the application identifier; the target folder is created based on the application identifier when the application writes the application data into the container.

[0128] In one embodiment, the application is constructed by a computing engine and is used to perform data computing.

[0129] In one embodiment, the sending module further:

[0130] In response to the application end event, a preset function is called to send an end message to the first component via a specified address; the end message includes an application identifier of the application.

[0131] In one embodiment, the second component is deployed on multiple computing nodes in the resource scheduler;

[0132] The first component sends a cleanup request to the second component, including:

[0133] The first component sends the cleanup request to the second component according to a pre-stored second component list; the second component list records address information corresponding to the second component.

[0134] In one embodiment, the second component creates copies of the second component on multiple computing nodes in the resource scheduler in a mirroring manner;

[0135] The second component list records the address information corresponding to the second component, including:

[0136] The second component list records the address information of the computing node where the copy of the second component is located when the copy of the second component is registered with the first component when the copy of the second component is created.

[0137] In one embodiment, the apparatus 300 further includes:

[0138] The mirroring module 304 , in response to a new computing node being added to the resource scheduler, creates a copy of the second component on the new computing node in a mirroring manner.

[0139] In one embodiment, the cleaning module 303 further:

[0140] In response to the second component finding the target folder containing the application identifier in the specified folder directory, a cleanup operation is performed on the target folder.

[0141] In one embodiment, the container is mounted to a designated disk;

[0142] When the application writes the application data into the container, creating a target folder based on the application identifier includes:

[0143] When the application writes the application data into the container, a target folder is created in the disk mounted by the resource scheduler based on the application identifier.

[0144] In one embodiment, a timer is preset in the second component;

[0145] The apparatus 300 further includes:

[0146] The timing module 305, in response to the triggering of the timer, calls the second component to perform a cleaning operation on the folders that have existed for a period exceeding a threshold.

[0147] The specific details of each module of the above-mentioned data cleaning device 300 have been described in detail in the previous description of the word vector compression method process based on frequency domain transformation, so they will not be repeated here.

[0148] It should be noted that although several modules or units of the data cleaning device 300 are mentioned in the detailed description above, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided to be embodied by multiple modules or units.

[0149] Exemplary media

[0150] After introducing the apparatus of the exemplary embodiment of the present disclosure, see Figure 4 , Figure 4 is a schematic diagram of a data cleaning medium provided by an exemplary embodiment.

[0151] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present disclosure.

[0152] refer to Figure 4 As shown, a readable storage medium 400 for implementing the above method according to an embodiment of the present disclosure is described. The readable storage medium 400 may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the readable storage medium of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0153] The readable storage medium can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0154] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0155] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0156] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0157] Exemplary computing devices

[0158] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, see Figure 5 , Figure 5 It is a schematic diagram of an electronic device capable of implementing the above method provided by an exemplary embodiment.

[0159] Refer to the following Figure 5 hereinafter, an electronic device 500 according to such an embodiment of the present disclosure is described. Figure 5 The electronic device 500 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0160] like Figure 5 As shown, electronic device 500 is implemented as a general-purpose computing device. Components of electronic device 500 may include, but are not limited to, at least one processing unit 501, at least one storage unit 502, and a bus 503 connecting different system components (including storage unit 502 and processing unit 501).

[0161] The storage unit stores program codes, which can be executed by the processing unit 501, so that the processing unit 501 executes the steps of the various embodiments described above in this specification.

[0162] The storage unit 502 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 5021 and / or a cache memory unit 5022 , and may further include a read-only memory unit (ROM) 5023 .

[0163] The storage unit 502 may also include a program / utility 5024 having a set (at least one) of program modules 5025, such program modules 5025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the reality of a network environment.

[0164] Bus 503 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0165] The electronic device 500 may also communicate with one or more external devices 504 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 500, and / or any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may occur via an input / output (I / O) interface 505. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 506. As shown, the network adapter 506 communicates with other modules of the electronic device 500 via a bus 503. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0166] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0167] It should be noted that although several units / modules or sub-units / modules of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0168] Furthermore, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0169] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features in these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A data cleaning method, applied to a resource scheduler of a cloud platform, wherein a first component, a second component, and a containerized application are deployed on the resource scheduler, comprising: Monitor the working status of the application; The working status at least includes an application end event; In response to the application end event, the application identifier of the application is sent to the first component, so that the first component sends a cleanup request to the second component; the second component is deployed on multiple computing nodes in the resource scheduler, and the cleanup request includes the application identifier; In response to the second component finding the target folder containing the application identifier, a cleanup operation is performed on the target folder; the target folder is created based on the application identifier when the application writes the application data into the container.

2. The method according to claim 1, wherein the application is constructed by a computing engine and is used to perform data computing.

3. The method according to claim 1, wherein, in response to the application end event, sending the application identifier of the application to the first component comprises: In response to the application end event, calling a preset function and sending an end message to the first component via a specified address; The end message includes the application identifier of the application.

4. The method according to claim 1, wherein the second component is deployed on multiple computing nodes in the resource scheduler; The first component sends a cleanup request to the second component, including: The first component sends the cleanup request to the second component according to a pre-stored second component list; The second component list records the address information corresponding to the second component.

5. The method according to claim 4, wherein the second component creates copies of the second component on multiple computing nodes in the resource scheduler in a mirroring manner; The second component list records the address information corresponding to the second component, including: The second component list records the address information of the computing node where the copy of the second component is located when the copy of the second component is registered with the first component when the copy of the second component is created.

6. The method according to claim 5, further comprising: In response to a new computing node being added to the resource scheduler, a copy of the second component is created on the new computing node in a mirroring manner.

7. The method according to claim 1, wherein, in response to the second component finding the target folder containing the application identifier, performing a cleanup operation on the target folder comprises: In response to the second component finding the target folder containing the application identifier in the specified folder directory, a cleanup operation is performed on the target folder.

8. The method according to claim 1, wherein the container is mounted to a specified disk; When the application writes the application data into the container, creating a target folder based on the application identifier includes: When the application writes the application data into the container, a target folder is created in the disk mounted by the resource scheduler based on the application identifier.

9. The method according to claim 1, wherein a timer is preset in the second component; The method further comprises: In response to the triggering of the timer, the second component is called to perform a cleanup operation on the folders that have existed for a period exceeding a threshold.

10. A data cleaning device, applied to a resource scheduler of a cloud platform, wherein a first component, a second component, and a containerized application are deployed on the resource scheduler, comprising: A monitoring module, monitoring the working status of the application; The working status at least includes an application end event; a sending module, in response to the application end event, sending the application identifier of the application to the first component, so that the first component sends a cleanup request to the second component; the second component is deployed on multiple computing nodes in the resource scheduler, and the cleanup request includes the application identifier; The cleaning module performs a cleaning operation on the target folder in response to the second component finding the target folder containing the application identifier; the target folder is created based on the application identifier when the application writes the application data into the container. The apparatus according to claim 10 , wherein the application is constructed by a computing engine and is used to perform data computing.

12. The apparatus according to claim 10, wherein the sending module further: In response to the application end event, a preset function is called to send an end message to the first component via a specified address; the end message includes an application identifier of the application.

13. The apparatus according to claim 10, wherein the second component is deployed on a plurality of computing nodes in the resource scheduler; The first component sends a cleanup request to the second component, including: The first component sends the cleanup request to the second component according to a pre-stored second component list; The second component list records the address information corresponding to the second component.

14. The apparatus according to claim 13, wherein the second component creates copies of the second component on multiple computing nodes in the resource scheduler in a mirroring manner; The second component list records the address information corresponding to the second component, including: The second component list records the address information of the computing node where the copy of the second component is located when the copy of the second component is registered with the first component when the copy of the second component is created.

15. The apparatus according to claim 14, further comprising: The mirroring module creates a copy of the second component on the new computing node in a mirroring manner in response to a new computing node being added to the resource scheduler.

16. The apparatus according to claim 10, wherein the cleaning module further: In response to the second component finding the target folder containing the application identifier in the specified folder directory, a cleanup operation is performed on the target folder.

17. The apparatus according to claim 10, wherein the container is mounted to a designated disk; When the application writes the application data into the container, creating a target folder based on the application identifier includes: When the application writes the application data into the container, a target folder is created in the disk mounted by the resource scheduler based on the application identifier.

18. The device according to claim 10, wherein a timer is preset in the second component; The device further comprises: The timing module, in response to the triggering of the timer, calls the second component to perform a cleaning operation on the folders that have existed for a period exceeding a threshold.

19. A storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

20. A computing device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

Images