Distributed file cleaning method and system, storage medium and computer equipment

Abstract

The invention relates to the technical field of internet and finance, and particularly discloses a distributed file cleaning method and system, a storage medium and computer device.The method comprises the steps that in the operation process of a target distributed cluster, in response to a file cleaning strategy dynamic configuration instruction, target configuration data are obtained, the target configuration data comprises at least one of custom file cleaning strategy configuration data, preset file cleaning strategy selection data and preset file cleaning strategy modification data; analyzing the target configuration data, determining a target file cleaning strategy based on an analysis result, and generating a corresponding file cleaning strategy execution script according to the target file cleaning strategy; and determining a first application object file corresponding to the target file cleaning strategy, and distributing a file cleaning strategy execution script to a first data node in which the first application object file is stored, so that the first data node starts to execute the target file cleaning strategy after receiving the file cleaning strategy execution script.

Description

Technical Field

[0001] The present application relates to the fields of Internet and financial technology, and in particular to a distributed file cleaning method and system, storage medium, and computer equipment. Background Art

[0002] With the rapid development of information technology, distributed file systems have become the cornerstone of big data storage and processing. Distributed file systems achieve high availability and scalability of data by distributing and storing data on multiple physical nodes. They are widely used in cloud computing, big data analysis, content distribution networks and other fields. In the financial field, many financial institutions have accumulated a large amount of customer data, transaction data, market data, etc. These data are not only large in scale, but also highly sensitive and time-sensitive. In order to ensure business continuity and data security, financial institutions usually use distributed file systems to store and manage these data. However, with the continuous growth of data volume, the management and maintenance of file systems has become increasingly complex, especially file cleaning, which has become a key link in ensuring system performance and effective use of storage resources.

[0003] Traditional distributed file cleaning methods often rely on manual cleaning and cleaning based on static configuration files. For manual cleaning methods, cleaning errors are prone to occur, resulting in the deletion of important data files, resulting in data loss or business interruption; for cleaning methods based on static configuration files, flexibility is poor and it is difficult to meet the diverse needs of users. Summary of the invention

[0004] In view of this, the present application provides a distributed file cleaning method and system, storage medium, and computer equipment, which allow users to dynamically configure file cleaning policies according to actual needs during the operation of the target distributed cluster, without modifying the system code or restarting the service, thereby greatly improving the flexibility and timeliness of file cleaning policy revision. In addition, a file cleaning policy execution script is generated according to the target file cleaning policy, and the script is distributed to the first data node. Subsequently, the first data node directly executes the newly added file cleaning policy according to the script, which can greatly improve the efficiency and accuracy of file cleaning.

[0005] According to one aspect of the present application, a distributed file cleaning method is provided, which is applied to a management node and includes:

[0006] During the operation of the target distributed cluster, in response to the file cleaning policy dynamic configuration instruction, obtaining target configuration data, wherein the target configuration data includes at least one of custom file cleaning policy configuration data, preset file cleaning policy selection data, and preset file cleaning policy modification data;

[0007] Parsing the target configuration data, determining a target file cleaning strategy based on the parsing result, and generating a corresponding file cleaning strategy execution script according to the target file cleaning strategy;

[0008] Determine the first application object file corresponding to the target file cleaning policy, and distribute the file cleaning policy execution script to the first data node storing the first application object file, so that the first data node starts to execute the target file cleaning policy after receiving the file cleaning policy execution script.

[0009] According to another aspect of the present application, there is provided a distributed file cleaning system, comprising a management node and a first data node;

[0010] The management node is used to obtain target configuration data in response to a file cleaning policy dynamic configuration instruction during the operation of the target distributed cluster, wherein the target configuration data includes at least one of custom file cleaning policy configuration data, preset file cleaning policy selection data, and preset file cleaning policy modification data;

[0011] The management node is further configured to parse the target configuration data, determine a target file cleaning strategy based on the parsing result, and generate a corresponding file cleaning strategy execution script according to the target file cleaning strategy; determine a first application object file corresponding to the target file cleaning strategy, and distribute the file cleaning strategy execution script to a first data node storing the first application object file;

[0012] The first data node is used to start executing the target file cleaning policy after receiving the file cleaning policy execution script.

[0013] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned distributed file cleaning method is implemented.

[0014] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned distributed file cleaning method when executing the program.

[0015] By means of the above technical scheme, the present application provides a method and system for cleaning distributed files, a storage medium, and a computer device. During the operation of the target distributed cluster, the management node can respond to the file cleaning policy dynamic configuration instruction and obtain the target configuration data according to the instruction. Here, the target configuration data may include custom file cleaning policy configuration data, preset file cleaning policy selection data, preset file cleaning policy modification data, etc. The management node can further parse the acquired target configuration data and determine the target file cleaning policy based on the parsing result. After determining the target file cleaning policy, the management node can generate a corresponding file cleaning policy execution script according to the target file cleaning policy. In addition, the management node can also determine the first application object file corresponding to the target file cleaning policy. Subsequently, the management node distributes the generated file cleaning policy execution script to the first data node storing the first application object file. When the first data node receives the file cleaning policy execution script, it can start to execute the target file cleaning policy on the basis of normal operation. The embodiment of the present application allows the user to dynamically configure the file cleaning policy according to actual needs during the operation of the target distributed cluster without modifying the system code or restarting the service, thereby greatly improving the flexibility and timeliness of the file cleaning policy correction. In addition, a file cleaning policy execution script is generated according to the target file cleaning policy, and the script is distributed to the first data node. Subsequently, the first data node directly executes the newly added file cleaning policy according to the script, which can greatly improve the efficiency and accuracy of file cleaning.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A schematic diagram of a flow chart of a distributed file cleaning method provided in an embodiment of the present application is shown;

[0019] Figure 2 A schematic diagram of the structure of a distributed file cleaning system provided in an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0022] In this embodiment, a distributed file cleaning method is provided, which is applied to a management node, such as Figure 1 As shown, the method includes:

[0023] Step 101, during the operation of the target distributed cluster, in response to a file cleaning policy dynamic configuration instruction, obtain target configuration data, wherein the target configuration data includes at least one of custom file cleaning policy configuration data, preset file cleaning policy selection data, and preset file cleaning policy modification data.

[0024] A method for cleaning distributed files provided in an embodiment of the present application can be applied to a management node in a target distributed cluster. The target distributed cluster may include multiple nodes, including a management node and a data node for storing data. The target distributed cluster may be HDFS (Hadoop Distributed File System), etc., which is not required here. In a distributed cluster, the management node is responsible for interacting with users, and the data node communicates with the management node, and the file cleaning strategy can be updated under the management of the management node. In this embodiment, during the operation of the target distributed cluster, the management node can respond to the file cleaning strategy dynamic configuration instruction and obtain the target configuration data according to the instruction. Here, the target configuration data may include custom file cleaning strategy configuration data, preset file cleaning strategy selection data, preset file cleaning strategy modification data, etc. In one embodiment, the file cleaning strategy dynamic configuration instruction can be generated based on the front end corresponding to the management node. Specifically, during the operation of the target distributed cluster, when the user wants to dynamically configure the file cleaning strategy, he can log in to the Web page for dynamic configuration through the front end, and the pre-set file cleaning strategy can be displayed on the Web page, and a custom editing area can also be displayed. Among them, the pre-set file cleaning strategies can be some commonly used cleaning strategies, which can be directly selected by users, and users can also make certain modifications on this basis to meet specific cleaning needs; the custom editing area allows users to edit new file cleaning strategies according to specific needs.

[0025] For example, a large investment bank uses a distributed file system to store various documents generated by its daily operations, including transaction records, customer identity information, market research reports, etc. As the business volume grows, the number of files in the system increases dramatically. Some old files (such as completed transaction records and expired market analysis reports) no longer need to be frequently accessed, but they still occupy storage space. In order to optimize the use of storage resources and ensure data security, the investment bank can implement the above-mentioned distributed file cleaning method. Users can generate file cleaning policy dynamic configuration instructions through the Web front end corresponding to the management node, and the management node can receive and respond to the file cleaning policy dynamic configuration instructions to obtain the target configuration data. Here, the custom file cleaning policy configuration data may be the data corresponding to the custom file cleaning policy configured by the administrator through the Web management page. The custom file cleaning policy may be a file cleaning policy that meets specific needs, such as "delete transaction records older than 3 years" and "retain market analysis reports for the last 12 months"; the preset file cleaning policy selection data may be the data generated by the administrator selecting the preset file cleaning policy provided by the Web management page; the preset file cleaning policy modification data may be the data generated by the administrator selecting the preset file cleaning policy provided by the Web management page and then modifying its parameters, such as adjusting the cleaning period of "documents that have not been accessed for more than 6 months" to 9 months.

[0026] Step 102: parse the target configuration data, determine a target file cleaning strategy based on the parsing result, and generate a corresponding file cleaning strategy execution script according to the target file cleaning strategy.

[0027] In this embodiment, the management node can further parse the acquired target configuration data to understand its meaning and requirements. Based on the parsing results, the management node can determine the target file cleaning policy. This policy is used to supplement the existing file cleaning policy. Specifically, when the target configuration file includes custom file cleaning policy configuration data, the target file cleaning policy may include a user-defined file cleaning policy; when the target configuration file includes preset file cleaning policy selection data, the target file cleaning policy may include a file cleaning policy selected by the user from the preset file cleaning policy; when the target configuration file includes preset file cleaning policy modification data, the target file cleaning policy may include a file cleaning policy obtained after the user modifies the selected preset file cleaning policy. After determining the target file cleaning policy, the management node can generate a corresponding file cleaning policy execution script according to the target file cleaning policy. The script can be an automated script containing a series of commands and parameters, which is used to guide the data node to perform file cleaning tasks.

[0028] Step 103, determine the first application object file corresponding to the target file cleaning policy, and distribute the file cleaning policy execution script to the first data node storing the first application object file, so that the first data node starts to execute the target file cleaning policy after receiving the file cleaning policy execution script.

[0029] In this embodiment, in addition, the management node may also determine the first application object file corresponding to the target file cleanup policy. The first application object file refers to the file targeted by the target file cleanup policy, that is, the application object of the policy. Subsequently, the management node distributes the generated file cleanup policy execution script to the first data node storing the first application object file. Here, the first data node may be one or more. After the first data node receives the file cleanup policy execution script, it may start to execute the target file cleanup policy on the basis of normal operation, that is, delete or process the specified files according to the instructions and parameters in the script.

[0030] By applying the technical solution of this embodiment, during the operation of the target distributed cluster, the management node can respond to the file cleaning policy dynamic configuration instruction and obtain the target configuration data according to the instruction. Here, the target configuration data may include custom file cleaning policy configuration data, preset file cleaning policy selection data, preset file cleaning policy modification data, etc. The management node can further parse the acquired target configuration data and determine the target file cleaning policy based on the parsing result. After determining the target file cleaning policy, the management node can generate a corresponding file cleaning policy execution script according to the target file cleaning policy. In addition, the management node can also determine the first application object file corresponding to the target file cleaning policy. Subsequently, the management node distributes the generated file cleaning policy execution script to the first data node storing the first application object file. When the first data node receives the file cleaning policy execution script, it can start to execute the target file cleaning policy on the basis of normal operation. The embodiment of the present application allows the user to dynamically configure the file cleaning policy according to actual needs during the operation of the target distributed cluster without modifying the system code or restarting the service, thereby greatly improving the flexibility and timeliness of the file cleaning policy correction. In addition, a file cleaning policy execution script is generated according to the target file cleaning policy, and the script is distributed to the first data node. Subsequently, the first data node directly executes the newly added file cleaning policy according to the script, which can greatly improve the efficiency and accuracy of file cleaning.

[0031] In an embodiment of the present application, optionally, the method further includes: obtaining configuration file modification data in response to a configuration file modification instruction; parsing the configuration file modification data, and when the configuration file modification data is parsed to create a file lifetime, creating at least one file lifetime in the current configuration file based on the configuration file modification data, and binding a second application object file to each file lifetime to obtain an updated configuration file; distributing the updated configuration file to a second data node storing the second application object file, so that after the second data node is restarted, the updated configuration file is loaded, the lifetime of the second application object file is monitored based on the updated configuration file, and when the lifetime of the second application object file is consistent with the configured file lifetime, a file cleanup operation is triggered.

[0032] In this embodiment, the user can also modify or add, delete, etc. the file cleaning strategy by modifying the configuration file in the management node. When the management node receives the configuration file modification instruction from the administrator or the automation tool, the specific configuration file modification data can be extracted from the configuration file modification instruction. The configuration file modification instruction can include the modification requirements for the configuration file, such as adding, deleting or modifying configuration items. The instruction can be submitted through the Web management interface or sent through API calls. Then, the management node parses the acquired configuration file modification data to determine the specific content and intention of the modification. Here, the management node can check whether the configuration file modification data includes an instruction for creating a file life time (TTL, Time To Live). If included, the management node further parses the number of file life times to be created, the specific value of each life time (such as 30 days, 60 days, etc.), and which second application object files these life times are to be bound to. The second application object file refers to the file whose life time is subsequently limited by these file life times. Afterwards, the management node can create at least one file life time in the current configuration file based on the parsing result, and bind a second application object file to each file life time. After completing the creation and binding of the file life time, the management node can obtain the updated configuration file and distribute the updated configuration file to the second data node storing the second application object file. Here, the distribution process can ensure that each relevant data node can receive the latest configuration file. It can be implemented specifically through network communication, such as using the file transfer protocol (FTP), the secure copy protocol (SCP) or a distributed file system (such as HDFS). After receiving the updated configuration file, the second data node can store the updated configuration file, and can read and load the latest configuration file after restarting, thereby realizing the update of the internal file cleaning strategy. In this way, the subsequent second data node can monitor the life time of the second application object file based on the updated configuration file, and calculate whether the storage time of these files has reached the file life time indicated by the updated configuration file. When the second data node detects that the life time of a certain file has expired, it can trigger a file cleaning operation, such as deleting the file or moving it to a recycle bin. In addition, the second data node can also record the log information of the cleaning operation for subsequent auditing and tracking. The embodiment of the present application allows the user to modify the configuration file to add a new file life time, and different life times can be flexibly set for different files, thereby improving the flexibility of life time setting.

[0033] In an embodiment of the present application, optionally, after the data nodes in the target distributed cluster start running, they listen to file cleaning operations. When a file cleaning operation is listened to, the number of files to be cleaned corresponding to the file cleaning operation is determined, and current load data and predicted resource occupancy data within a first preset time period after the current time are obtained. According to the current load data and the predicted resource occupancy data, idle resources are determined. Based on the number of files to be cleaned, the occupied resources when cleaning the files to be cleaned are calculated. According to the idle resources and the occupied resources, it is determined whether to delete the files to be cleaned.

[0034] In this embodiment, the data node may be the first data node mentioned above, or the second data node, or other data nodes. When the data node is started, the necessary configuration information may be loaded and the data node may start to perform its duties. At the same time, a monitoring mechanism may be enabled to monitor the file cleaning operation automatically generated from the management node or by its own file cleaning strategy. When a file cleaning operation is monitored, the data node first determines the number of files to be cleaned corresponding to the operation. Specifically, the file cleaning operation includes an identifier or path list of files to be cleaned. The data node may parse these identifiers or paths and count the total number of files to be cleaned. For example, a financial institution has launched multiple financial products, and stores the data files of each popular financial product in a corresponding data node. For the data node corresponding to financial product A, when it monitors the file cleaning operation indication to clean all transaction record files that have not been accessed for more than 30 days, the data node may search and count the number of these files.

[0035] Afterwards, the data node can obtain its own current load data and predict the resource occupancy in the first preset time period after the current time. For example, if the first preset time period is 5 hours, then the predicted resource occupancy data can be the data obtained by predicting the resource occupancy in the next 5 hours. Among them, the current load data includes key performance indicators such as CPU usage, memory occupancy, and disk I / O. The predicted resource occupancy data can be predicted based on historical business demand data, current load trends and other information. Specifically, the current data processing volume and the data processing volume in the third preset time period before the current time (such as the past hour) can be used as input data. Using the input data, the data processing volume in the future (first preset time period) is predicted by the preset data processing volume prediction model. The preset data processing volume prediction model can be obtained based on machine learning training, and is used to predict the data processing volume in the first preset time period in the future. The predicted resource occupancy data can be calculated based on the predicted data processing volume. These predicted resource occupancy data help data nodes more accurately evaluate current and future resource conditions.

[0036] The data node can also calculate the idle resources in the current and predicted time periods based on the current load data and the predicted resource occupancy data. Here, idle resources refer to the resources that the data node can use additionally without affecting the execution of the current and predicted services. Specifically, they may include unused CPU cycles, idle memory, unsaturated disk I / O channels, etc. The data node can comprehensively consider these factors and calculate the amount of idle resources that can be used for file cleaning operations. In addition, the data node can also calculate the occupied resources required to perform file cleaning operations based on the number and size of the files to be cleaned. Among them, the file cleaning operation can be operations such as file deletion and recycle bin movement, which will consume certain system resources. Subsequently, the data node decides whether to perform the file cleaning operation based on the idle resources and occupied resources. If the idle resources are sufficient to support the execution of the file cleaning operation (that is, the idle resources are greater than or equal to the occupied resources), the data node can perform the cleaning operation and delete the files to be cleaned. If the idle resources are insufficient, the data node can choose to delay the cleaning operation and wait until the idle resources are larger before executing it. In the embodiment of the present application, the data node can comprehensively consider its own current and future resource status, as well as the resource requirements of the file cleaning operation, so as to make more reasonable and efficient file cleaning decisions, which helps to optimize the resource usage of the distributed cluster and improve the stability and performance of the system.

[0037] In an embodiment of the present application, optionally, the "determining whether to delete the file to be cleaned according to the idle resources and the occupied resources" includes: when the idle resources are less than the occupied resources, adding the file identifier corresponding to the file to be cleaned to the file cleanup list, and calculating the idle resources in real time, until the idle resources are greater than or equal to the occupied resources, performing a file cleanup operation according to the file cleanup list; or, dividing the files to be cleaned according to the cleaning priority, determining the target cleanup file from the files to be cleaned with the highest cleaning priority according to the idle resources, performing a file cleanup operation on the target cleanup file, and after the target cleanup file is cleaned, recalculating the idle resources, and re-determining the target cleanup file from the remaining files to be cleaned according to the cleaning priority, cleaning the re-determined target cleanup file, until the cleaning of all the files to be cleaned is completed; when the idle resources are greater than or equal to the occupied resources, performing a file cleanup operation on the files to be cleaned.

[0038] In this embodiment, the data node can determine whether to delete the files to be cleaned according to the idle resources and occupied resources in the following manner: Specifically, if the idle resources are less than the occupied resources, it means that the idle resources are insufficient to perform the cleaning operation of all the files to be cleaned. At this time, the data node can adopt one of two strategies.

[0039] The first is to add the file identifiers (such as file names, paths, etc.) of the files to be cleaned to a file cleanup list. For example, in a financial system, a data node can simultaneously process multiple transaction requests, financial report generation, financial data analysis, and other tasks, resulting in tight resource usage. When files need to be cleaned, if the current idle resources of the data node are insufficient to support the operation of all files to be cleaned, the data node can add the file identifiers (such as file names, paths) of the files to be cleaned (such as old transaction records, processed financial reports, etc.) to a file cleanup list. After that, the data node can continuously monitor its own resource usage, including CPU usage, memory occupancy, disk I / O, etc., as well as the predicted resource occupancy data within the first preset time period after the current time, and calculate the current idle resources in real time based on its own resource usage and the predicted resource occupancy data. Once the idle resources become sufficient (i.e., greater than or equal to the occupied resources required to perform the cleanup operation), the data node can start to perform the file cleanup operation according to the order (or priority) in the file cleanup list, and clean up all the files to be cleaned indicated in the file cleanup list.

[0040] The second is to divide the files to be cleaned according to the cleaning priority. For example, determine which files to be cleaned need to be cleaned urgently and which files to be cleaned can be cleaned later. The files to be cleaned that need to be cleaned urgently can be set with a higher cleaning priority. Then, according to the current idle resources, determine the target cleaning files from the files to be cleaned with the highest cleaning priority, and perform the file cleaning operation. For example, in the case of tight resources, financial institutions may want to prioritize the cleaning of user privacy data. At this time, the data node can prioritize the files to be cleaned stored in itself, and the urgency of file cleaning can be considered. According to the current idle resources and the priority of the files to be cleaned, select a batch of files with the highest priority from the files to be cleaned as the target cleaning files. Here, the number of cleaning files allowed for file cleaning operations can be calculated based on the idle resources, and then the same number of files to be cleaned can be determined as target cleaning files according to the order of cleaning priority from high to low. After the first batch of target cleaning files are cleaned, the data node can recalculate the idle resources and re-determine the next batch of target cleaning files based on the new idle resources and the priority of the remaining files to be cleaned, until all files to be cleaned are cleaned. It should be noted that the method used by the data node to calculate the idle resources each time can be the same.

[0041] If the free resources are greater than or equal to the occupied resources, the data node can immediately perform a file cleanup operation on the file to be cleaned up.

[0042] The data nodes in the embodiment of the present application can accurately evaluate their own idle resource situation by acquiring the current load data in real time and predicting the future resource occupancy, so as to make more reasonable cleaning decisions. Among them, when idle resources are insufficient and there are no files to be cleaned that need to be cleaned urgently, cleaning can be carried out when there are sufficient idle resources; when there are files to be cleaned that need to be cleaned urgently, the files to be cleaned can be cleaned according to the cleaning priority to ensure that the files that need to be deleted urgently can be cleaned first, while improving the cleaning efficiency, it also reduces the impact on the performance of the data node. The embodiment of the present application provides two file cleaning strategies for selection when idle resources are insufficient, and the data node can be flexibly switched according to the actual situation to adapt to different business scenarios and needs.

[0043] In an embodiment of the present application, optionally, after the data nodes in the target distributed cluster start running, the method also includes: real-time monitoring of the current remaining storage space, and when the current remaining storage space is less than the first preset storage space, screening out files whose storage space is less than the first preset threshold from the current storage files as target merged files, classifying the target merged files according to the storage space occupied by the target merged files and the first preset threshold, and performing file merging processing on the target merged files under each category through a preset file merging tool.

[0044] In this embodiment, the space occupied by files in the data node can also be reduced by file merging. This is because each file requires certain metadata to store its attribute information (such as file name, size, location, etc.). When there are a large number of small files, these metadata occupy a large amount of storage space. After the small files are merged into a large file, the number of metadata will be reduced accordingly, thereby reducing the storage space occupied. In addition, small files are prone to fragmentation when stored, and these fragments will occupy additional storage space and affect storage performance. Merging large files can reduce the generation of file fragments and improve the utilization of storage space. For example, in a bank's transaction system, a large number of transaction record files are generated every day. Through file merging, these transaction record files can be merged into larger files to reduce the storage space occupied. Specifically, when a data node detects that its remaining storage space is less than the first preset storage space, it can trigger the file merging process, and select the files with a storage space occupied less than the first preset threshold from the currently stored files as the target merged files. These target merged files are usually some small, independent data files, especially for those files less than 1M, which occupy a large amount of memory resources, and the merger is conducive to improving the performance of the data node. For example, in the HDFS system, files smaller than 128M are considered small files by default. These small files can be merged into a large file, thereby reducing the storage space occupied. Here, the first preset threshold and the first preset storage space can be determined according to actual needs. After determining the target merged file, further, the target merged file can be classified according to the storage space occupied by the target merged file and the first preset threshold. The purpose of classification is to better merge the target merged file and ensure that the file size after the merge of the files under each classification is close to the first preset threshold. For example, the first preset threshold is 128M, and the target merged file includes 5, namely file 1 (occupied storage space 31M), file 2 (occupied storage space 77M), file 3 (occupied storage space 100M), file 4 (occupied storage space 21M), and file 5 (occupied storage space 11M). The data node can automatically classify these target merged files to seek the best classification result, so that the file size after the merge of the target merged files under each category after classification is close to the first preset threshold, and the total storage space occupied is minimized. For example, file 1, file 2, and file 5 can be classified into one category, and file 3 and file 4 can be classified into another category. After all target merge files are classified, the preset file merge tool can be used to merge the target merge files under each category. Here, commonly used file merge tools include Hadoop's MapReduce, Spark and other distributed processing frameworks, which can efficiently process large-scale data sets.With these tools, you can merge multiple small files into one large file, thereby reducing the number of files and reducing space usage.

[0045] In an embodiment of the present application, optionally, after the data nodes in the target distributed cluster start running, the method also includes: real-time monitoring of the current remaining storage space, and when the current remaining storage space is less than the first preset storage space, selecting files whose storage space is greater than a second preset threshold from the current storage files as target compressed files, and performing file compression processing on the target compressed files through a preset file compression tool.

[0046] In this embodiment, the space occupied by files in the data node can also be reduced by file compression. For example, in financial risk management, financial institutions need to store a large amount of risk indicator data and model data. When the storage space of a data node is insufficient, the risk indicator data that occupies a large amount of storage space can be screened out for compression processing to ensure the normal operation of the data node. Specifically, when a data node detects that its remaining storage space is less than the first preset storage space, the file compression process can be triggered to filter out files that occupy a storage space greater than the second preset threshold from the currently stored files as target compressed files. These target compressed files are usually some large, independent data files. Here, the second preset threshold can be determined according to actual needs. After determining the target compressed file, a preset file compression tool can be used to compress each target compressed file. The preset file compression tool has the characteristics of high efficiency, security, and reliability, and can ensure that the compressed file still maintains the original data integrity and readability. Here, commonly used file compression tools include Gzip and the like. Through these tools, the size of the file can be significantly reduced. The embodiment of the present application can significantly reduce the size of the file through compression processing, thereby releasing storage space and improving the performance and stability of the data node.

[0047] In an embodiment of the present application, optionally, during the operation of the target distributed cluster, the method further includes: real-time monitoring of the remaining storage space of each data node in the target distributed cluster, and when there is a third data node whose remaining storage space is less than the second preset storage space, triggering an early warning mechanism to send an early warning message containing the node identifier of the third data node to a preset terminal through the early warning mechanism; and / or, real-time monitoring of the fourth data node on which a file cleanup operation is to be performed, and when there is any fourth data node that has not performed the file cleanup operation after the second preset time period, triggering an early warning mechanism to send an early warning message containing the node identifier of any fourth data node to the preset terminal through the early warning mechanism.

[0048] In this embodiment, the management node can also ensure that each data node in the cluster can maintain sufficient storage space through real-time monitoring and early warning mechanisms so that it can operate normally and process business. Specifically, the management node can monitor the remaining storage space of each data node in the target distributed cluster in real time to ensure that no data node affects the normal operation of the cluster due to insufficient storage space. When it is found that the remaining storage space of a certain data node (referred to as the third data node) is lower than the second preset storage space, the management node can immediately trigger the early warning mechanism. Among them, the early warning mechanism can be a method such as sending an email, a text message, a system log record, or triggering an alarm in the cluster. In addition, the early warning information can also include the node identifier (such as IP address, host name, etc.) of the third data node, so that the relevant staff who receive the early warning information can quickly locate the problem data node. The early warning information can also include the specific value of the remaining storage space, the duration below the preset value, and other information to help the relevant staff understand the severity of the problem.

[0049] In addition, the management node can also monitor the file cleaning operation corresponding to each data node in real time. Specifically, when a certain data node automatically triggers a file cleaning operation, or receives a file cleaning instruction from the outside, the execution of the next file cleaning operation of the data node can be monitored to verify whether the data node can normally perform the file cleaning operation. For example, after monitoring that a certain fourth data node triggers a file cleaning operation, the management node can start a timer, and when the timing time reaches the second preset time period, check whether the fourth data node has started to perform the file cleaning operation. If the file cleaning operation is still not started, then the early warning mechanism can be triggered at this time. Among them, the early warning mechanism can be a method such as sending an email, a text message, a system log record, or triggering an alarm in the cluster. The early warning information can include the node identification of the fourth data node that did not perform the cleaning operation, the specific reason for not performing the cleaning operation (if it can be detected), such as a disk failure, a network problem, or an error in the cleaning task itself, etc., which can help relevant staff quickly locate the problem data node and understand the reason why the fourth data node did not perform the file cleaning operation in time.

[0050] The management node in the embodiment of the present application uses a real-time monitoring and early warning mechanism, which enables relevant staff to promptly discover and solve problems such as insufficient storage space and untimely file cleaning, thereby avoiding the risk of cluster performance degradation; through an automated monitoring and early warning mechanism, the workload of relevant staff is reduced and management efficiency is improved.

[0051] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a distributed file cleaning system, such as Figure 2 As shown, the system includes a management node and a first data node;

[0052] The management node is used to obtain target configuration data in response to a file cleaning policy dynamic configuration instruction during the operation of the target distributed cluster, wherein the target configuration data includes at least one of custom file cleaning policy configuration data, preset file cleaning policy selection data, and preset file cleaning policy modification data;

[0053] The management node is further configured to parse the target configuration data, determine a target file cleaning strategy based on the parsing result, and generate a corresponding file cleaning strategy execution script according to the target file cleaning strategy; determine a first application object file corresponding to the target file cleaning strategy, and distribute the file cleaning strategy execution script to a first data node storing the first application object file;

[0054] The first data node is used to start executing the target file cleaning policy after receiving the file cleaning policy execution script.

[0055] Optionally, the management node is further configured to obtain configuration file modification data in response to a configuration file modification instruction; parse the configuration file modification data, and when the configuration file modification data is parsed to be used for creating a file lifetime, create at least one file lifetime in a current configuration file based on the configuration file modification data, and bind a second application object file to each file lifetime to obtain an updated configuration file; and distribute the updated configuration file to a second data node storing the second application object file;

[0056] The second data node is used to load the updated configuration file after restarting, monitor the lifetime of the second application object file based on the updated configuration file, and trigger a file cleanup operation when the lifetime of the second application object file is consistent with the configured file lifetime.

[0057] Optionally, the data nodes in the target distributed cluster are used to listen to file cleaning operations after starting operation. When a file cleaning operation is listened to, the data nodes determine the number of files to be cleaned corresponding to the file cleaning operation, and obtain current load data and predicted resource occupancy data within a first preset time period after the current time. According to the current load data and the predicted resource occupancy data, the idle resources are determined, and based on the number of files to be cleaned, the occupied resources when cleaning the files to be cleaned are calculated. According to the idle resources and the occupied resources, it is determined whether to delete the files to be cleaned.

[0058] Optionally, the data node is further used to:

[0059] When the idle resources are less than the occupied resources, the file identifier corresponding to the file to be cleaned is added to the file cleaning list, and the idle resources are calculated in real time. When the idle resources are greater than or equal to the occupied resources, the file cleaning operation is performed according to the file cleaning list; or, the files to be cleaned are divided according to the cleaning priority, and according to the idle resources, the target cleaning file is determined from the files to be cleaned with the highest cleaning priority, and the file cleaning operation is performed on the target cleaning file, and after the cleaning of the target cleaning file is completed, the idle resources are recalculated, and the target cleaning file is re-determined from the remaining files to be cleaned according to the cleaning priority, and the re-determined target cleaning file is cleaned until the cleaning of all the files to be cleaned is completed;

[0060] When the idle resources are greater than or equal to the occupied resources, a file cleaning operation is performed on the file to be cleaned up.

[0061] Optionally, the data node is also used to monitor the current remaining storage space in real time after starting operation. When the current remaining storage space is less than a first preset storage space, files that occupy a storage space less than a first preset threshold are screened out from the current storage files as target merged files. According to the storage space occupied by the target merged files and the first preset threshold, the target merged files are classified, and file merger processing is performed on the target merged files under each classification through a preset file merger tool.

[0062] Optionally, the data node is also used to monitor the current remaining storage space in real time after starting operation. When the current remaining storage space is less than the first preset storage space, the file occupying the storage space greater than the second preset threshold is screened out from the current storage files as the target compressed file, and the target compressed file is compressed by a preset file compression tool.

[0063] Optionally, the management node is further used to:

[0064] monitoring the remaining storage space of each data node in the target distributed cluster in real time, and when there is a third data node whose remaining storage space is less than the second preset storage space, triggering an early warning mechanism to send an early warning message including a node identifier of the third data node to a preset terminal through the early warning mechanism; and / or,

[0065] The fourth data nodes to be subjected to file cleaning operations are monitored in real time, and when any fourth data node fails to perform the file cleaning operation after a second preset time period, an early warning mechanism is triggered to send an early warning message including a node identifier of any fourth data node to a preset terminal through the early warning mechanism.

[0066] It should be noted that for other corresponding descriptions of the functional units involved in the distributed file cleaning system provided in the embodiment of the present application, reference can be made to Figure 1 The corresponding description in the method will not be repeated here.

[0067] The present application also provides a computer device, which may be a personal computer, a server, a network device, etc. Figure 3 As shown, the computer device includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.

[0068] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0069] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0070] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0071] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0072] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0073] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A distributed file cleaning method, applied to a management node, characterized in that: include: During the operation of the target distributed cluster, in response to the file cleaning policy dynamic configuration instruction, obtaining target configuration data, wherein the target configuration data includes at least one of custom file cleaning policy configuration data, preset file cleaning policy selection data, and preset file cleaning policy modification data; Parsing the target configuration data, determining a target file cleaning strategy based on the parsing result, and generating a corresponding file cleaning strategy execution script according to the target file cleaning strategy; Determine the first application object file corresponding to the target file cleaning policy, and distribute the file cleaning policy execution script to the first data node storing the first application object file, so that the first data node starts to execute the target file cleaning policy after receiving the file cleaning policy execution script.

2. The method according to claim 1, characterized in that The method further comprises: In response to the configuration file modification instruction, obtaining configuration file modification data; Parsing the configuration file modification data, when parsing out that the configuration file modification data is used to create a file lifetime, creating at least one file lifetime in the current configuration file based on the configuration file modification data, and binding a second application object file to each file lifetime to obtain an updated configuration file; Distribute the updated configuration file to the second data node storing the second application object file, so that after the second data node is restarted, the updated configuration file is loaded, and the lifetime of the second application object file is monitored based on the updated configuration file. When the lifetime of the second application object file is consistent with the configured file lifetime, a file cleanup operation is triggered.

3. The method according to claim 1 or 2, characterized in that: After the data nodes in the target distributed cluster start running, they listen to the file cleaning operation. When the file cleaning operation is listened, the number of files to be cleaned corresponding to the file cleaning operation is determined, and the current load data and the predicted resource occupancy data within the first preset time period after the current time are obtained. According to the current load data and the predicted resource occupancy data, the idle resources are determined. Based on the number of files to be cleaned, the occupied resources when cleaning the files to be cleaned are calculated. According to the idle resources and the occupied resources, it is determined whether to delete the files to be cleaned.

4. The method according to claim 3, characterized in that The determining whether to delete the to-be-cleaned file according to the idle resources and the occupied resources includes: When the idle resources are less than the occupied resources, the file identifier corresponding to the file to be cleaned is added to the file cleaning list, and the idle resources are calculated in real time. When the idle resources are greater than or equal to the occupied resources, the file cleaning operation is performed according to the file cleaning list; or, the files to be cleaned are divided according to the cleaning priority, and according to the idle resources, the target cleaning file is determined from the files to be cleaned with the highest cleaning priority, and the file cleaning operation is performed on the target cleaning file, and after the cleaning of the target cleaning file is completed, the idle resources are recalculated, and the target cleaning file is re-determined from the remaining files to be cleaned according to the cleaning priority, and the re-determined target cleaning file is cleaned until the cleaning of all the files to be cleaned is completed; When the idle resources are greater than or equal to the occupied resources, a file cleaning operation is performed on the file to be cleaned up.

5. The method according to claim 3, characterized in that: After the data nodes in the target distributed cluster start running, the method further includes: The current remaining storage space is monitored in real time. When the current remaining storage space is less than the first preset storage space, files whose storage space is less than the first preset threshold are selected from the current storage files as target merged files. The target merged files are classified according to the storage space occupied by the target merged files and the first preset threshold, and the target merged files under each classification are merged through a preset file merging tool.

6. The method according to claim 3, characterized in that After the data nodes in the target distributed cluster start running, the method further includes: The current remaining storage space is monitored in real time. When the current remaining storage space is less than the first preset storage space, files that occupy a storage space greater than a second preset threshold are selected from the current storage files as target compressed files, and the target compressed files are compressed using a preset file compression tool.

7. The method according to claim 1, characterized in that During the operation of the target distributed cluster, the method further includes: monitoring the remaining storage space of each data node in the target distributed cluster in real time, and when there is a third data node whose remaining storage space is less than the second preset storage space, triggering an early warning mechanism to send an early warning message including a node identifier of the third data node to a preset terminal through the early warning mechanism; and / or, The fourth data nodes to be subjected to file cleaning operations are monitored in real time, and when any fourth data node fails to perform the file cleaning operation after a second preset time period, an early warning mechanism is triggered to send an early warning message including a node identifier of any fourth data node to a preset terminal through the early warning mechanism.

8. A distributed file cleaning system, characterized in that: including a management node and a first data node; The management node is used to obtain target configuration data in response to a file cleaning policy dynamic configuration instruction during the operation of the target distributed cluster, wherein the target configuration data includes at least one of custom file cleaning policy configuration data, preset file cleaning policy selection data, and preset file cleaning policy modification data; The management node is further configured to parse the target configuration data, determine a target file cleaning strategy based on the parsing result, and generate a corresponding file cleaning strategy execution script according to the target file cleaning strategy; determine a first application object file corresponding to the target file cleaning strategy, and distribute the file cleaning strategy execution script to a first data node storing the first application object file; The first data node is used to start executing the target file cleaning policy after receiving the file cleaning policy execution script.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images