Disk cleaning method, device and equipment
By automatically identifying and executing disk cleanup strategies during file transfer tasks, the problem of file loading failures due to insufficient disk space was resolved, improving the success rate of file transfers and reducing maintenance costs.
Patent Information
- Application Number
- CN202010585502.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-06-24
AI Technical Summary
During file transfer, insufficient server disk space can cause file loading failures. Existing technologies require manual intervention for troubleshooting, which is costly and results in a poor user experience.
When receiving file transfer tasks, the system obtains file size and disk usage information, pre-configures cleanup strategies, and automatically determines whether disk usage has reached the cleanup threshold. If so, it performs disk file cleanup, including file transfer, deletion, or soft linking, to ensure sufficient disk space.
It improves the success rate of file transfer, reduces manual intervention, ensures stable system operation, and lowers maintenance costs.
Smart Images

Figure CN111767009B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and more particularly to a disk cleaning method, device and equipment. Background Art
[0002] As a basic function for interaction between systems, file transfer is essential in production, especially in scenarios where stock file interaction is common. Upstream and downstream files are traceable and need to be saved for a certain period of time. Generally, files can be saved on the system's disk. In production, insufficient server disk space often occurs, which may cause difficulties in batch service processing and lead to system errors. Usually, files are cleaned up regularly within a period of time, but the disk storage space is limited. If a large file is suddenly received, it may cause a shortage of disk resources, requiring manual intervention for troubleshooting, which has a poor user experience and requires high labor costs. Summary of the invention
[0003] The purpose of the embodiments of this specification is to provide a disk cleaning method, device, equipment and system to ensure that the disk has sufficient memory space and improve the success rate of file transfer.
[0004] On the one hand, an embodiment of the present specification provides a disk cleaning method, the method comprising:
[0005] When receiving a file transfer task, obtaining file transfer information of the file transfer task, the file transfer information including the transfer file size and the file transfer path;
[0006] According to the file transfer path, obtaining a queue file size in a file transfer queue of the disk corresponding to the file transfer task;
[0007] Determine the post-transfer disk usage information after the file transfer task is completed based on the transfer file size, the queue file size and the monitored disk usage information;
[0008] It is determined whether the disk usage reaches a cleaning threshold value according to the disk usage information after the transmission. If so, the files in the disk are cleaned up according to a pre-configured disk cleaning strategy.
[0009] On the other hand, the present specification provides a disk cleaning device, comprising:
[0010] A file transfer task triggering module, used for obtaining file transfer information of the file transfer task upon receiving the file transfer task, wherein the file transfer information includes the transfer file size and the file transfer path;
[0011] A transmission queue information acquisition module, used to acquire the queue file size in the file transmission queue of the disk corresponding to the file transmission task according to the file transmission path;
[0012] A disk occupancy information acquisition module, used to determine the post-transmission disk usage information after the file transfer task is completed based on the transfer file size, the queue file size and the monitored disk usage information;
[0013] The disk cleaning module is used to determine whether the disk occupancy reaches a cleaning threshold according to the disk usage information after the transmission, and if so, clean up the files in the disk according to a pre-configured disk cleaning strategy.
[0014] On the other hand, an embodiment of the present specification provides a disk cleaning device, including at least one processor and a memory for storing processor executable instructions, and the processor implements the above-mentioned disk cleaning method when executing the instructions.
[0015] The disk cleaning method, device and equipment provided in this specification configures the disk cleaning strategy in advance. When receiving a file transfer task, the file size of the file transfer task and the queue file size in the file transfer queue of the disk are combined with the monitoring of the disk usage to obtain the current disk usage information and calculate the disk usage after the file transfer task is completed. Determine whether the disk usage reaches the cleaning threshold after the file transfer task is completed. If so, immediately clean the files in the disk according to the configured disk cleaning strategy to ensure that the file transfer task can be completed accurately, avoid the problem of disk file loading failure due to large file transfer, improve the success rate of file transfer, and ensure that the system can work stably and normally. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0017] Figure 1 It is a flowchart of a disk cleaning method embodiment provided in an embodiment of this specification;
[0018] Figure 2 is a schematic diagram of a disk monitoring and cleaning process in some embodiments of this specification;
[0019] Figure 3 It is a flowchart of disk cleaning during file transfer in one embodiment of the present specification;
[0020] Figure 4 is a schematic diagram of the structure of a disk cleaning system in another embodiment of the present specification;
[0021] Figure 5 is a schematic diagram of the structure of a disk cleaning device in some embodiments of this specification;
[0022] Figure 6 It is a hardware structure block diagram of a disk cleaning server in one embodiment of this specification. DETAILED DESCRIPTION
[0023] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.
[0024] With the development of computer technology, the amount of data that computers need to save and process is constantly increasing, and the number and size of files transferred between systems are also increasing. For files that cannot be deleted immediately, they may need to be saved on disk. When large files are transferred to disk, the disk storage space may be insufficient, resulting in file loading and execution exceptions.
[0025] Figure 1 It is a flowchart of the disk cleanup method embodiment provided in the embodiment of this specification. Although this specification provides the method operation steps or device structure shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures that do not logically have a necessary causal relationship, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure shown in the embodiments or drawings of this specification. When the method or module structure is applied in an actual device, server or terminal product, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiment or drawings (for example, a parallel processor or multi-threaded processing environment, or even a distributed processing, server cluster implementation environment).
[0026] A specific implementation example is Figure 1 As shown, in one embodiment of the disk cleaning method provided in this specification, the method can be applied in a terminal device such as a client, a server, etc., specifically a terminal such as a computer, a tablet computer, a server, etc., and the method may include the following steps:
[0027] Step 102: When a file transfer task is received, file transfer information of the file transfer task is obtained, where the file transfer information includes a transfer file size and a file transfer path.
[0028] In the specific implementation process, when a file transfer task is received, the disk cleaning process in the embodiment of this specification can be triggered, that is, the file transfer information transmitted by the file transfer task is obtained, and the file transfer information can include the size of the file to be transferred, that is, the transfer file size. Of course, the file transfer information can also include other information such as: file name, file transfer time, file transfer path, priority and other information, which are not specifically limited in the embodiment of this specification. Among them, the file transfer path can include information such as which disk the file is transferred to and which queue the file is transferred through.
[0029] In addition, when receiving a file transfer task, you can suspend the file transfer task first, that is, temporarily not execute the transfer task until the disk cleanup is completed or the disk memory space is determined to meet the requirements, so as to avoid transferring files that are too large, resulting in insufficient disk space, and file loading failure during the file transfer process.
[0030] Step 104: According to the file transfer path, obtain the queue file size in the file transfer queue of the disk corresponding to the file transfer task.
[0031] In the specific implementation process, file transfer is usually carried out through queue sorting, that is, each file needs to be loaded into the queue, and the file transfer is carried out according to the sorting in the queue. After receiving the file transfer task, the disk to which the file in the file transfer task is transferred can be obtained according to the file transfer path, and the file size in each file transfer queue of the disk, that is, the queue file size, can be obtained. For example: according to the file transfer path in the received file transfer task, it is known that the file needs to be transferred to disk A. If disk A has 3 file transfer queues for file transfer, then after receiving the file transfer task, the sum of the sizes of the files in the 3 file transfer queues of disk A can be obtained as the queue file size of disk A. There may be files being transferred or waiting to be transferred in each file transfer queue, and the sum of the sizes of each file in the file transfer queue can be obtained. For example: if it is obtained that there are 10 files in file transfer queue A, one of which is being transferred, and the other 9 are being sorted and waiting to be transferred, the size of these 10 files is the queue file size of queue A.
[0032] Of course, since the file transfer time of each queue cannot be determined, it is also possible to only obtain the queue file size of the file transfer queue added to the file in the file transfer task. For example, if the file with the file transfer path in the file transfer task needs to be transferred in file transfer queue A, it is also possible to only obtain the queue file size of file transfer queue A.
[0033] Step 106: Based on the transfer file size, the queue file size, and the monitored disk usage information, determine the post-transfer disk usage information after the file transfer task is completed.
[0034] In the specific implementation process, in some embodiments of the present specification, the disk usage in the system can be monitored to obtain disk usage information, which may include at least one of the disk usage size and disk usage rate. After receiving the file transfer task, the disk usage information after the file transfer in the file transfer task is completed can be calculated based on the acquired transfer file size of the file to be transferred in the file transfer task and the queue file size of the transfer queue, combined with the monitored disk usage information. The disk usage information after the transfer can be the disk usage size and / or disk usage rate of the disk after the file transfer task is completed. The usage size can represent the absolute value of the disk occupancy, and the usage rate can represent the relative value of the disk occupancy.
[0035] For example, after receiving a file transfer task, the size of the file to be transferred is 10G, the queue file size of the disk transfer queue is 30G, and the current disk usage size is monitored to be 50G. It can be determined that when the file transfer task is completed, the disk usage size is 90G.
[0036] Step 108: Determine whether the disk usage reaches a cleaning threshold based on the disk usage information after the transmission. If so, clean up the files in the disk according to a pre-configured disk cleaning policy.
[0037] In the specific implementation process, after calculating the disk usage size after the file transfer task is completed, that is, the disk usage information after the transfer, it can be determined whether the disk occupancy reaches the cleaning threshold after the file transfer task is completed. If so, the files in the disk can be cleaned up according to the pre-configured disk cleaning strategy. Disk occupancy can be understood as the size or proportion of disk memory occupied, and the cleaning threshold can be understood as the limit value that the disk needs to be cleaned up, which can be the upper limit of the disk usage size or the upper limit of the disk usage rate. The specific value can be configured according to actual needs. For example: if the total space of the disk is 100G, the cleaning threshold can be set to the disk usage size reaching 90G or the disk usage rate being 90%. The disk cleaning strategy can be understood as a disk cleaning rule, which can include how to clean up the files in the disk, such as: cleaning up the files generated within a specified time range according to the time when the files in the disk are generated, or starting with the earliest files until the disk usage information meets the requirements, or cleaning up unimportant or low-priority files according to the file name, the importance of the file, etc. The specific content of the disk cleaning strategy can be set according to the actual situation, and the embodiments of this specification are not specifically limited.
[0038] The cleanup threshold can be a single value, or multiple cleanup thresholds can be set according to actual usage needs. For example, two disk cleanup thresholds can be set according to actual needs, namely 80G and 90G. When there are multiple cleanup thresholds, the disk cleanup policy can include the file sizes that need to be cleaned corresponding to different cleanup thresholds, such as 80G corresponding to 10G files that need to be cleaned, 90G corresponding to 20G files that need to be cleaned, etc.
[0039] In addition, when it is determined that the disk needs to be cleaned, a warning message can be returned to prompt relevant personnel to pay attention, or when the disk cleanup fails, such as: the disk cannot be cleaned or the file cannot be deleted, a warning message is returned to prompt relevant personnel to handle it in time.
[0040] In some embodiments of this specification, the cleaning method in the disk cleaning strategy may include at least one of file transfer, file deletion, and soft link, wherein file transfer can be understood as transferring files from the disk to other storage devices for storage such as: cache or database, file deletion can be understood as deleting files directly from the disk, and deleted files may no longer be viewed, and soft links are also called symbolic links, and soft links can contain the path name of another file. Clicking on a soft link can jump to other paths to view the corresponding file. A suitable cleaning method can be selected according to the generation time, file name, importance, etc. of the file, and the disk can be cleaned up while ensuring the safety of the file to ensure that the disk can guarantee sufficient memory space and ensure accurate and successful transmission of the file. For example, for some relatively important files, file transfer can be selected to transfer the file from the disk to other storage devices for subsequent search or use, or soft links can be used to store the file in other paths, and the soft links of the storage path are stored in the disk. For some unimportant files, they can be directly deleted. Of course, other cleaning methods can also be set according to actual use needs, and the embodiments of this specification are not specifically limited.
[0041] Based on the above embodiments, in some embodiments of this specification, the method further includes:
[0042] After the disk is cleaned, the files transferred by the file transfer task are added to a file transfer queue;
[0043] And record the file cleaning information in the disk into the log file.
[0044] In the specific implementation process, when a file transfer task is received, the file transfer task can be suspended first and not executed temporarily. When it is determined that there is insufficient space on the disk after the file transfer task is completed, the disk can be cleaned up first to avoid file loading failure or slow file transfer speed during the file transfer process. When the disk is cleaned up, the files in the file transfer task can be added to the specified file transfer queue according to the file transfer path in the file transfer task to ensure that the files are transferred accurately and on time. At the same time, the file cleaning information on the disk can also be recorded in the log file, such as: which files were cleaned up, the cleaning time, the cleaning method, etc., are all recorded in the log file for subsequent tracing or query.
[0045] The disk cleaning method provided in the embodiment of this specification pre-configures a disk cleaning strategy. When a file transfer task is received, the file size of the file transfer task and the queue file size in the file transfer queue of the disk are combined with the monitoring of the disk usage to obtain the current disk usage information and calculate the disk usage after the file transfer task is completed. Determine whether the disk usage reaches the cleaning threshold after the file transfer task is completed. If so, immediately clean the files in the disk according to the configured disk cleaning strategy to ensure that the file transfer task can be completed accurately, avoid the problem of disk file loading failure due to large file transfer, improve the success rate of file transfer, and ensure the normal operation of the system.
[0046] Based on the above embodiments, in some embodiments of this specification, the method further includes:
[0047] Get disk usage information of the disk at preset intervals;
[0048] Determine whether the disk usage reaches the cleaning threshold according to the disk usage information; if so, clean up the files in the disk according to the disk cleaning policy.
[0049] In the specific implementation process, in some embodiments of this specification, the usage of the disk can be monitored, and the disk usage information of the disk can be obtained every preset time. The disk usage information may include the disk usage size or disk usage rate in the disk. According to the obtained disk usage information, it can be determined whether the disk occupancy reaches the cleaning threshold. If it reaches, the files in the disk can be cleaned according to the preset disk cleaning policy. Among them, the meaning of the cleaning threshold and the disk cleaning policy can refer to the records of the above embodiments, which will not be repeated here. The value of the preset time can be set according to actual needs, such as: 1 second, 5 seconds, etc., and the embodiments of this specification do not make specific limitations.
[0050] The embodiments of this specification monitor disk usage, detect when disk memory space is insufficient, and perform disk cleanup to ensure that the disk memory space is always sufficient, so as to ensure that files can be successfully transferred during file transfer, especially large file transfer.
[0051] Based on the above embodiments, in some embodiments of this specification, the disk usage information includes disk usage information of multiple disks; and determining whether the files in the disk have reached the cleanup threshold according to the disk usage information includes:
[0052] According to the disk usage information of each disk, determine whether the total usage of each disk reaches the cleaning threshold of the total disk;
[0053] And / or, based on the disk usage information of each disk, it is determined whether the occupancy of each disk reaches a clearing threshold of a single disk.
[0054] In the specific implementation process, the embodiments of this specification can monitor multiple disks in the system and obtain disk usage information of each disk at preset time intervals. The multiple disks in the embodiments of this specification can be understood as multiple disks in the system. If a disk is divided into multiple partitions, the multiple disks can also be understood as multiple partitions of a disk, that is, each partition in the disk is monitored, and the disk usage information of each partition disk is obtained at preset time intervals. After obtaining the disk usage information of each disk, it is possible to calculate whether the total occupancy of all disks reaches the cleaning threshold of the total disk, or it is possible to determine whether the disk occupancy of each disk reaches the cleaning threshold of a single disk. Figure 2 is a schematic diagram of a disk monitoring and cleaning process in some embodiments of this specification, such as Figure 2 As shown, you can first determine whether the total usage of each disk reaches the cleaning threshold of the total disk. If it does, you can sort the disks according to the disk usage information of each disk, and give priority to cleaning files in the disks with larger disk usage. If the total usage of each disk does not reach the cleaning threshold of the total disk, then determine whether the disk usage of each disk reaches the cleaning threshold of a single disk, and clean up files on the disks that have reached the cleaning threshold.
[0055] For example, if there are three disks in the system, these three disks can be monitored, and the disk usage information of the three disks can be obtained every 1 second. The disk usage information of disk 1 is 70%, the disk usage information of disk 2 is 80%, and the disk usage information of disk 3 is 60%. It can be determined that the total disk usage of the three disks is 70%. Assuming that the cleaning threshold of the total disk is 60%, it can be determined that the total usage of the three disks has reached the cleaning threshold of the total disk. The disks can be sorted in descending order according to the disk usage information. Disk 2 with relatively large disk usage information is cleaned first. If the total usage of the three disks is still greater than the cleaning threshold after cleaning, then continue to clean disk 1, and so on. Of course, when cleaning files in a disk with relatively large disk usage information, it does not mean that all files in this disk are cleaned. According to the disk cleaning strategy, some files in the disk can be cleaned. If the disk space requirement is still not met, clean other disks. If the cleaning threshold of the total disk is set to 80%, it can be considered that the total disk usage of the three disks has not reached the cleaning threshold of the total disk, and the disk usage of each disk can be judged separately. For example, if the cleanup threshold of a single disk is 80%, the disks can be sorted in descending order according to the disk usage information, and the disk occupancy of each disk can be determined in turn to determine whether it reaches the cleanup threshold of a single disk. After determination, it can be determined that the disk occupancy of Disk 2 reaches the cleanup threshold of a single disk, and the files in Disk 2 can be cleaned up. Disk 1 and Disk 3 have not reached the cleanup threshold of a single disk, and thus they do not need to be cleaned up.
[0056] The cleaning threshold of the total disk and the cleaning threshold of a single disk can be set to different thresholds or the same threshold according to actual needs, and the embodiments of this specification do not specifically limit this.
[0057] The embodiments of this specification comprehensively monitor each disk in the system, and determine whether the total occupancy of each disk and the disk occupancy of a single disk are greater than the cleaning threshold, so as to ensure that the memory space of each disk and the total disk can meet the memory space requirements and ensure that the file can be successfully transferred. By traversing and monitoring each disk, the disks with high occupancy and high occupancy rate are located, which reduces the time of manual positioning.
[0058] like Figure 2As shown, in some embodiments of the present specification, a threshold for canceling cleaning can also be pre-configured. During the disk cleaning process, the disk usage information of each disk is monitored in real time. When the disk usage reaches the threshold for canceling cleaning, cleaning can be stopped to avoid deleting important files on the disk and ensure that a certain amount of space is reserved on the disk to temporarily save important files. For example, if the threshold for canceling cleaning is that the disk usage rate reaches 40%, during the cleaning process of the files on the disk, the size of the files on the disk is monitored in real time. When the files on the disk reach 40% of the total disk space, cleaning can be stopped and the disk usage can continue to be monitored.
[0059] Figure 3 FIG. 1 is a flowchart of disk cleanup during file transfer in one embodiment of the present specification. Figure 3 As shown, the embodiments of this specification can monitor file transfer tasks, and monitor the appearance of new tasks in the file transfer queue to trigger the disk cleanup process, and suspend the current transfer task before completing the processing. According to the current disk occupancy, the size of the existing files in the queue, and the size of the files in the current task, it is calculated whether the disk occupancy will reach the configured cleanup threshold after the transfer task is completed. If so, the files in the disk are cleaned according to the disk cleanup strategy, and the disk occupancy is monitored in real time to determine whether the release cleanup threshold is reached. When the release cleanup threshold is reached, the current task is added to the file transfer queue to complete the file transfer. If it is calculated that the disk occupancy will not reach the cleanup threshold after the file transfer task is completed, the current task can be directly added to the file transfer queue to complete the file transfer.
[0060] The disk cleaning method provided in the embodiments of this specification can monitor file transfer tasks and disks, and provides a method for adaptively cleaning files based on disk usage, which helps environmental operation and maintenance personnel monitor disk usage. At the same time, it pre-checks imported large files and imports them after cleaning, reducing the manual cleaning steps of operation and maintenance personnel. Monitor the disk occupancy, traverse each path, determine the directories with high occupancy size and occupancy rate, and reduce the time for manual positioning. In addition, different disk cleaning strategies can be customized to prioritize the transmission and storage of important files, which improves flexibility. At the same time, it improves log records and early warnings to ensure that all operations can be traced and problems can be quickly located and notified.
[0061] In this specification, each embodiment of the above method is described in a progressive manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments. For relevant parts, refer to the partial description of the method embodiment.
[0062] Based on the disk cleaning method described above, one or more embodiments of this specification also provide a system for disk cleaning. The system may include a system (including a distributed system), software (application), module, component, server, client, etc. that uses the method described in the embodiments of this specification and is combined with a device of necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiments of this specification is as described in the following embodiments. Since the implementation scheme of the device to solve the problem is similar to the method, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the systems and devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0063] Figure 4 is a schematic diagram of the structure of a disk cleaning system in another embodiment of the present specification. Figure 4 As shown, in some embodiments of the present specification, a disk cleaning system may be provided, which may include: a user decision definition module, a policy parsing execution module, a disk usage monitoring module, and a file cleaning module, wherein:
[0064] The user decision definition module can be used to customize the disk trigger cleanup threshold, disk release cleanup threshold, adopted file cleanup strategy, log recording path, number and size of log files saved, warning sending configuration, and polling interval. The relevant definition information can be saved in the project configuration file.
[0065] The policy analysis execution module can read user configuration information from the user decision definition module, which mainly includes the disk trigger cleanup threshold, disk release cleanup threshold, adopted file cleanup strategy, log recording path, number and size of log files saved, warning sending configuration, and polling interval. After that, the policy analysis execution module will adopt the execution logic accordingly. After the policy analysis execution module is executed, the configuration information can be obtained, which is used when the disk usage monitoring module and the file cleanup module are executed.
[0066] The disk usage monitoring module can obtain the total disk usage size, disk usage rate, traverse the path, record the file size under the path, sort and record. When the device starts, poll to determine whether the disk usage has reached the threshold configured by the user. If it has, the corresponding file cleanup strategy is used to clean the files; if it has not, the device is suspended according to the polling interval configured by the user and waits for the next wake-up. After executing the cleanup strategy, if the disk usage reaches the threshold for lifting the cleanup, the file cleanup strategy is stopped and polling is re-entered to determine the disk usage.
[0067] The file cleaning module can clean up files according to the file cleaning strategy configured by the user. The cleaning strategy is configured according to time, file name, etc., and uses file transfer, file deletion, soft link, etc. to perform physical or logical cleaning.
[0068] The log and warning module can record the operation time, operation behavior, operation configuration, operation object, operation path, operation result and other information into the log after one operation is performed when executing the user decision definition module, policy analysis and execution module, disk usage monitoring module and file cleaning module. For operations that trigger the threshold but fail to successfully execute the file cleaning policy, a warning message is sent to notify the environment maintenance personnel.
[0069] In addition, the disk cleaning system in the embodiments of the present specification may also include a file transfer queue monitoring module which is triggered when a file transfer task is created, suspends the current file transfer task, obtains the file size in the file transfer queue plus the current disk usage size plus the size of the file in the current file transfer task, calculates the disk usage after the transfer task is completed, and then triggers the file cleaning module based on user configuration.
[0070] The disk cleaning method provided in the embodiments of this specification can monitor file transfer tasks and disks, and provides a method for adaptively cleaning files based on disk usage, which helps environmental operation and maintenance personnel monitor disk usage. At the same time, it pre-checks imported large files and imports them after cleaning, reducing the manual cleaning steps of operation and maintenance personnel. Monitor the disk occupancy, traverse each path, determine the directories with high occupancy size and occupancy rate, and reduce the time for manual positioning. In addition, different disk cleaning strategies can be customized to prioritize the transmission and storage of important files, which improves flexibility. At the same time, it improves log records and early warnings to ensure that all operations can be traced and problems can be quickly located and notified.
[0071] Figure 5 is a schematic diagram of the structure of a disk cleaning device in some embodiments of this specification, such as Figure 5 As shown, the disk cleaning device provided in this specification may include:
[0072] The file transfer task triggering module 51 is used to obtain the file transfer information of the file transfer task when receiving the file transfer task, wherein the file transfer information includes the transfer file size and the file transfer path;
[0073] The transmission queue information acquisition module 52 is used to obtain the queue file size in the file transmission queue of the disk corresponding to the file transmission task;
[0074] The disk usage information acquisition module 53 is used to determine the post-transmission disk usage information after the file transfer task is completed based on the transfer file size, the queue file size and the monitored disk usage information;
[0075] The disk cleaning module 54 is used to determine whether the disk occupancy reaches a cleaning threshold according to the disk usage information after the transmission, and if so, clean up the files in the disk according to a pre-configured disk cleaning strategy.
[0076] The disk cleaning device provided in the embodiment of the specification pre-configures a disk cleaning strategy. When receiving a file transfer task, the file size of the file transfer task and the queue file size in the file transfer queue of the disk are combined with the monitoring of the disk usage to obtain the current disk usage information and calculate the disk usage after the file transfer task is completed. Determine whether the disk usage reaches the cleaning threshold after the file transfer task is completed. If so, immediately clean the files in the disk according to the configured disk cleaning strategy to ensure that the file transfer task can be completed accurately and avoid the problem of disk file loading failure due to large file transfer.
[0077] In some embodiments of the present specification, the device further comprises a disk monitoring module for: obtaining disk usage information of the disk at preset time intervals;
[0078] The disk cleaning module is also used to determine whether the disk occupancy has reached a cleaning threshold according to the disk usage information, and if so, clean up the files in the disk according to the disk cleaning strategy.
[0079] The disk cleaning device provided in the embodiment of this specification monitors the disk usage, promptly detects when the disk memory space is insufficient, and performs disk cleaning to ensure that the disk memory space is always sufficient, so as to ensure that files can be successfully transferred during file transfer, especially large file transfer.
[0080] It should be noted that the above-mentioned device may also include other implementations according to the description of the corresponding method embodiment. The specific implementation methods can refer to the description of the corresponding method embodiment above, and will not be described one by one here.
[0081] The embodiment of this specification also provides a disk cleaning device, including: at least one processor and a memory for storing processor executable instructions, and the processor implements the information recommendation data processing method of the above embodiment when executing the instructions, such as:
[0082] When receiving a file transfer task, obtaining file transfer information of the file transfer task, the file transfer information including the size of the transferred file;
[0083] Obtaining the queue file size in the file transfer queue of the disk corresponding to the file transfer task;
[0084] Determine the post-transfer disk usage information after the file transfer task is completed based on the transfer file size, the queue file size and the monitored disk usage information;
[0085] It is determined whether the disk usage reaches a cleaning threshold value according to the disk usage information after the transmission. If so, the files in the disk are cleaned up according to a pre-configured disk cleaning strategy.
[0086] It should be noted that the above-mentioned device may also include other implementations according to the description of the method embodiment. The specific implementation methods can refer to the description of the relevant method embodiment, and will not be described one by one here.
[0087] The disk cleaning device provided in this specification can also be used in a variety of data analysis and processing systems. The system or server or terminal or device can be a separate server, or it can include a server cluster, system (including distributed system), software (application), actual operation device, logic gate circuit device, quantum computer, etc. that uses one or more of the methods or one or more embodiments of the system or server or terminal or device of this specification and a terminal device combined with necessary implementation hardware. The detection system for checking difference data can include at least one processor and a memory storing computer executable instructions, and when the processor executes the instructions, the steps of the method described in any one or more of the above embodiments are implemented.
[0088] The method embodiments provided in the embodiments of this specification can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 6 1 is a hardware structure block diagram of a disk cleaning server in one embodiment of this specification. The computer terminal may be a disk cleaning server or a disk cleaning device in the above embodiment. Figure 6 The server 10 shown may include one or more (only one is shown in the figure) processors 100 (the processor 100 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a non-volatile memory 200 for storing data, and a transmission module 300 for communication functions. Those skilled in the art will understand that Figure 6 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 6 More or fewer components shown in the figure may also include other processing hardware, such as a database or multi-level cache, GPU, or other hardware with Figure 6 Different configurations shown.
[0089] The non-volatile memory 200 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the disk cleaning method in the embodiment of this specification. The processor 100 executes various functional applications and resource data updates by running the software programs and modules stored in the non-volatile memory 200. The non-volatile memory 200 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the non-volatile memory 200 may further include a memory remotely arranged relative to the processor 100, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a bureau and a network, a mobile communication network, and combinations thereof.
[0090] The transmission module 300 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission module 300 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 300 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0091] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0092] The methods or devices described in the above embodiments provided in this specification can implement business logic through computer programs and record them on storage media, and the storage media can be read and executed by computers to achieve the effects of the solutions described in the embodiments of this specification, such as:
[0093] When receiving a file transfer task, obtaining file transfer information of the file transfer task, the file transfer information including the size of the transferred file;
[0094] Obtaining the queue file size in the file transfer queue of the disk corresponding to the file transfer task;
[0095] Determine the post-transfer disk usage information after the file transfer task is completed based on the transfer file size, the queue file size and the monitored disk usage information;
[0096] It is determined whether the disk usage reaches a cleaning threshold value according to the disk usage information after the transmission. If so, the files in the disk are cleaned up according to a pre-configured disk cleaning strategy.
[0097] The storage medium may include a physical device for storing information, which is usually a medium that digitizes the information and then stores it in an electrical, magnetic or optical manner. The storage medium may include: a device that stores information in an electrical energy manner, such as various memories, such as RAM, ROM, etc.; a device that stores information in a magnetic energy manner, such as a hard disk, a floppy disk, a magnetic tape, a magnetic core memory, a magnetic bubble memory, a USB flash drive; a device that stores information in an optical manner, such as a CD or a DVD. Of course, there are other readable storage media, such as quantum memory, graphene memory, etc.
[0098] The above-mentioned disk cleaning method or device provided in the embodiments of this specification can be implemented by a processor in a computer executing corresponding program instructions, such as using the C++ language of the Windows operating system to implement it on a PC, on a Linux system, or other systems such as Android and iOS system programming languages to implement it on a smart terminal, as well as based on the processing logic of a quantum computer.
[0099] It should be noted that the device, computer storage medium, and system described in the specification may also include other implementation methods according to the description of the relevant method embodiments. The specific implementation methods can refer to the description of the corresponding method embodiments, which will not be described one by one here.
[0100] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0101] The embodiments of this specification are not limited to complying with industry communication standards, standard computer resource data update and data storage rules, or the situations described in one or more embodiments of this specification. Certain industry standards or slightly modified implementation plans based on the implementation described in the custom method or embodiment can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above-mentioned embodiments. The embodiments obtained by using these modified or deformed data acquisition, storage, judgment, processing methods, etc. can still fall within the scope of the optional implementation plans of the embodiments of this specification.
[0102] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0103] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0104] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0105] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, an environment of a parallel processor or multi-threaded processing, or even a distributed resource data update environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. The first, second, etc. words are used to represent the name, and do not represent any particular order.
[0106] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0107] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable resource data update device to produce a machine, so that instructions executed by the processor of the computer or other programmable resource data update device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0108] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable resource data update device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0109] These computer program instructions may also be loaded onto a computer or other programmable resource data updating device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0111] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0112] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0113] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0114] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0115] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. In the description of this specification, the description of the reference term "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.
[0116] The above description is only an example of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims.
Claims
1. A disk cleaning method, characterized in that: The method comprises: When receiving a file transfer task, obtaining file transfer information of the file transfer task, the file transfer information including the transfer file size and the file transfer path; According to the file transfer path, obtaining a queue file size in a file transfer queue of the disk corresponding to the file transfer task; Based on the transfer file size, the queue file size and the monitored disk usage information, determine the post-transfer disk usage information after the file transfer task is completed; the disk usage information includes disk usage information of multiple disks; Determining whether the disk usage reaches a cleaning threshold according to the disk usage information after the transmission, and if so, cleaning the files in the disk according to a pre-configured disk cleaning strategy; During the disk cleaning process, the disk usage information is monitored in real time to determine whether the disk usage has reached the cleaning threshold. If so, the cleaning is stopped. Determine whether the disk usage reaches a cleaning threshold according to the disk usage information after the transmission, and if so, clean up the files in the disk according to a pre-configured disk cleaning strategy, including: Determine whether the total occupancy of each disk reaches the cleaning threshold of the total disk; if so, then: sort the disks according to the disk usage information of each disk, clean up some files in the disks with large disk usage information, and clean up other disks when the total occupancy after cleaning still reaches the cleaning threshold of the total disk; if not, then: determine whether the disk occupancy of each disk reaches the cleaning threshold of a single disk, and clean up files on the disks that have reached the cleaning threshold; When cleaning files on the disk, at least one cleaning method is selected from file transfer, file deletion, and soft connection according to the generation time, file name, and importance of the files in the disk to clean the files.
2. The method according to claim 1, characterized in that The method further comprises: Get disk usage information of the disk at preset intervals; Determine whether the disk usage reaches the cleaning threshold according to the disk usage information; if so, clean up the files in the disk according to the disk cleaning policy.
3. The method according to claim 1, characterized in that The method further comprises: After the disk is cleaned, the files transferred by the file transfer task are added to a file transfer queue; And record the file cleaning information in the disk into the log file.
4. The method according to claim 1, characterized in that The method further comprises: Before the disk cleanup is completed, the file transfer task is suspended.
5. A disk cleaning device, characterized in that: include: A file transfer task triggering module, used for obtaining file transfer information of the file transfer task upon receiving the file transfer task, wherein the file transfer information includes the transfer file size and the file transfer path; A transmission queue information acquisition module, used to acquire the queue file size in the file transmission queue of the disk corresponding to the file transmission task according to the file transmission path; A disk occupancy information acquisition module, used to determine the post-transmission disk usage information after the file transfer task is completed based on the transfer file size, the queue file size and the monitored disk usage information; The disk cleaning module is used to determine whether the disk usage reaches the cleaning threshold according to the disk usage information after the transmission, and if so, clean up the files in the disk according to the pre-configured disk cleaning strategy; during the disk cleaning process, the disk usage information of the disk is monitored in real time to determine whether the disk usage reaches the cleaning release threshold, and if so, stop cleaning; The disk cleanup module is specifically used to: determine whether the total occupancy of each disk has reached the cleanup threshold of the total disk; if reached, then: sort the disks according to the disk usage information of each disk, clean up some files in the disks with large disk usage information, and when the total occupancy after cleaning still reaches the cleanup threshold of the total disk, clean up other disks; if not reached, then: determine whether the disk occupancy of each disk has reached the cleanup threshold of a single disk, and perform file cleanup on the disk that has reached the cleanup threshold; when performing file cleanup on the disk, select at least one cleaning method from file transfer, file deletion, and soft connection to perform file cleanup based on the generation time, file name, and importance of the files in the disk.
6. The device according to claim 5, characterized in that The device also includes a disk monitoring module, which is used to: obtain disk usage information of the disk at preset time intervals; The disk cleaning module is also used to determine whether the disk occupancy has reached a cleaning threshold according to the disk usage information, and if so, clean up the files in the disk according to the disk cleaning strategy.
7. A disk cleaning device, characterized in that: include: At least one processor and a memory for storing processor-executable instructions, wherein when the processor executes the instructions, the method according to any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Portable cloud storage method and device
CN102202044A
Method for clearing disk space
CN102333079A
File storage method and device
CN103500077A
File automatic-cleaning method and device
CN104182439A
File storage method and device
CN105956035A
