A system and method for timely cleaning of data lake files based on storage and computing separation

Abstract

This invention relates to the field of file cleanup technology, and in particular to a data lake file timely cleanup system and method based on storage-compute separation. The timely cleanup method includes: scanning each file in the data lake, determining the files to be cleaned based on the scan results to obtain a number of isolated files; accumulating the isolated files to obtain the actual number of markings, and performing batch cleanup on the data lake according to the actual number of markings or the initial cleanup cycle; for any cleanup batch, generating a cleanup set from the isolated files corresponding to the batch cleanup, and arranging the files to be cleaned in the cleanup set in a cleanup order; during cleanup based on the cleanup order arrangement, monitoring the cleanup process in real time, sequentially determining the consistency of the cleanup target data, cleanup process data, and cleanup operations, and determining the corresponding process control mode based on the determination results; periodically acquiring cleanup results, and adjusting the cleanup conditions based on the cleanup results.

Description

Technical Field

[0001] The present invention relates to the technical field of file cleaning, and in particular to a system and method for timely cleaning of data lake files based on storage and computing separation. Background Art

[0002] With the advent of the big data era, data lakes have been widely used as a solution for storing and managing massive data. However, the number of files in the data lake is huge. If they are not cleaned up in time, it will lead to a waste of storage space and management difficulties. Therefore, how to clean up files in the data lake on time has become an urgent problem to be solved.

[0003] Chinese patent application publication number CN103886059A discloses a file cleaning method, comprising: obtaining a request to delete a current application; obtaining record information corresponding to the current application according to the request, wherein the record information includes files and / or folders related to the current application; and deleting files and / or folders related to the current application according to the record information.

[0004] This shows that the current file cleaning cannot quickly respond to and process the files that need to be cleaned. Summary of the invention

[0005] To this end, the purpose of the present invention is to provide a data lake file timely cleaning system and method based on storage and computing separation, so as to overcome the problem that the current file cleaning cannot quickly respond to and process the files that need to be cleaned.

[0006] To achieve the above objectives, the present invention provides a data lake file timely cleaning system based on storage and computing separation, including:

[0007] An identification module is used to perform real-time scanning of each file in the data lake, identify target files in turn according to the scanning results, determine the cleaning type of the target files, and record the actual number of the target files, and determine the actual cleaning mode according to the cleaning type and the actual number;

[0008] A preprocessing module, used to generate a cleaning set for each target file obtained by scanning according to the actual cleaning mode, evaluate the cleaning urgency of each file to be cleaned in the cleaning set in turn, and sort each file to be cleaned according to the evaluation result to obtain an actual cleaning sequence;

[0009] A cleaning module, used for cleaning according to the actual cleaning sequence, obtaining actual cleaning parameters during the cleaning process, analyzing the actual cleaning parameters to determine the consistency of the cleaning process, and determining a corresponding process control mode according to the determination result;

[0010] The feedback correction module is used to periodically obtain cleaning results and perform feedback adjustment on cleaning conditions according to the cleaning results.

[0011] Furthermore, the identification module includes: a scanning unit and a judgment unit;

[0012] The scanning unit is used to scan each file in the data lake in real time according to the scanning standard to obtain actual scanning results;

[0013] The judgment unit is used to analyze each file according to the actual scanning result in combination with a preset target cleaning strategy to identify a number of target files;

[0014] Among them, the scanning standards include: time standards, frequency standards, capacity standards, format standards and reference standards; the target cleaning strategy includes: standard deadline strategy, standard usage strategy, standard capacity strategy and standard format strategy.

[0015] Furthermore, the identification module further comprises: a marking unit and a dividing unit;

[0016] The marking unit is used to mark each target file according to the judgment result to obtain isolated files of different cleaning types;

[0017] The division unit is used to divide each isolated file according to the cleaning type to obtain a plurality of subsets to be cleaned.

[0018] Furthermore, the identification module further includes: a counting unit and a first determining unit;

[0019] The counting unit is used to accumulate the actual number of marks of each subset to be cleaned;

[0020] The first determining unit is used to determine the actual cleaning mode as a threshold cleaning mode or a periodic cleaning mode according to the actual number of marks of each to-be-cleaned subset combined with the cleaning type.

[0021] Further, the preprocessing module includes: a generating unit, a calculating unit and a comparing unit;

[0022] The generating unit is used to determine the single cleaning number according to the actual cleaning mode, and generate the cleaning set based on the single cleaning number;

[0023] The calculation unit is used to calculate the cleaning urgency of each isolated file according to the marking result combined with the actual number of marks;

[0024] The comparison unit is used to sort the isolated files according to the cleaning urgency values ​​to obtain the actual cleaning sequence.

[0025] Further, the cleaning module includes: a monitoring unit and a first determination unit;

[0026] The monitoring unit is used to monitor the cleaning process in real time to obtain the actual cleaning parameters;

[0027] The first determination unit is used to determine whether the actual cleaning process data and the metadata are synchronized, and to issue a first alarm signal when they are not synchronized;

[0028] The actual cleaning parameters include: actual cleaning target data, actual cleaning operation results and the actual cleaning process data.

[0029] Furthermore, the cleaning module further includes: a second determination unit and a third determination unit;

[0030] The second determination unit is used to determine a single cleaning success rate according to the actual cleaning operation result and the target cleaning strategy, and to issue a second alarm signal when the single cleaning success rate is not equal to 100%;

[0031] The third determination unit is used to determine whether conflicting data exists according to the actual cleaning target data combined with the currently executed task data, and to issue a third alarm signal when the conflicting data exists.

[0032] Further, the cleaning module further includes: a second determining unit;

[0033] The second determination unit can determine that the process control mode is data re-reading according to the first alarm signal;

[0034] The second determination unit can determine that the process control mode is data rollback recovery according to the second alarm signal;

[0035] The second determination unit can determine that the process control mode is a data locking mechanism according to the third alarm signal.

[0036] Further, the feedback correction module includes: an acquisition unit, an evaluation unit and an adjustment unit;

[0037] The acquisition unit is used to periodically acquire the cleaning result;

[0038] The evaluation unit is used to determine the cleaning level according to the absolute value of the first difference combined with a preset first evaluation value;

[0039] The regulating unit is used to determine the feedback regulating mode as single feedback regulating or overall feedback regulating according to the cleaning level;

[0040] The first absolute value of the difference is the absolute value of the difference between the actual storage release amount and the preset standard storage release amount.

[0041] Another aspect of the present invention further provides a method for timely cleaning of data lake files based on storage and computing separation, comprising:

[0042] Step S1, scanning each file in the data lake, determining the files to be cleaned up according to the scanning results, and marking them to obtain a number of isolated files;

[0043] Step S2, accumulating the isolated files to obtain an actual number of tags, and performing batch cleaning on the data lake according to the actual number of tags or a preset initial cleaning cycle;

[0044] Step S3, for any cleaning batch, generating a cleaning set from the orphan files corresponding to the cleaning batch, and arranging the files to be cleaned in the cleaning set in a cleaning order;

[0045] Step S4, when cleaning is performed based on the cleaning sequence arrangement result, the cleaning process is monitored in real time, the consistency of the cleaning target data, the cleaning process data and the cleaning operation is determined in turn, and the corresponding process control mode is determined according to the determination result;

[0046] Step S5, periodically obtaining cleaning results, and performing feedback adjustment on cleaning conditions according to the cleaning results;

[0047] The cleaning conditions include: the initial cleaning period, the total mark threshold and the sub-mark threshold.

[0048] Compared with the prior art, the beneficial effects of the present invention are that, by real-time scanning of files in the data lake, the target files can be found in time; it helps to quickly respond to and process files that need to be cleaned, reducing data redundancy and storage space waste; according to the scanning results and the cleaning type, the actual cleaning mode is determined in combination with the actual quantity, and the dynamic adjustment capability makes the cleaning strategy more flexible and targeted, and can adapt to different cleaning needs and scenarios; by evaluating the cleaning urgency, the files to be cleaned are sorted, and the actual cleaning sequence is generated, which ensures that important or urgent files are cleaned first, and optimizes resource utilization and processing efficiency; in the cleaning process, the actual cleaning parameters are obtained and analyzed to determine the consistency of the cleaning process, and the corresponding process control mode is determined according to the determination result, which helps to ensure the stability and reliability of the cleaning process and reduce the risk of errors and data loss; by periodically obtaining the cleaning results and feedback-adjusting the cleaning conditions, the closed-loop feedback mechanism enables the system to continuously optimize the cleaning strategy according to the actual cleaning effect, improve the cleaning efficiency and effect, reduce the need for manual intervention, and improve management efficiency and accuracy; it can effectively manage the files in the data lake, keep the data lake clean and orderly, improve storage and retrieval efficiency, and extend the service life of the data lake.

[0049] In particular, the files in the data lake are scanned in real time by time, frequency, capacity, format and reference to multiple standards to ensure comprehensive coverage and accurate identification. Multi-dimensional scanning can capture various file types that may need to be cleaned; the scan results are analyzed in combination with preset target cleaning strategies (such as standard deadlines, usage, capacity and format strategies) to accurately identify the target files that need to be cleaned. The cleaning strategies can be flexibly adjusted according to different management needs to improve the accuracy of judgment; the target files are marked according to the judgment results, and files of different cleaning types (such as expired files, infrequently used files, large-capacity files, and files in non-compliant formats) are clearly distinguished. Classification and marking will help to carry out the subsequent cleaning work in an orderly manner; the marked isolated files are divided according to the cleaning type to form several subsets to be cleaned. The detailed division makes the cleaning work more organized and is conducive to the use of different cleaning strategies for different types of files; the actual number of marks for each subset to be cleaned is accumulated and counted to provide data support, which helps to quantify the workload of the cleaning task and facilitate Resource allocation and task scheduling; determine the actual cleaning mode (threshold cleaning mode or periodic cleaning mode) based on the actual number of subsets to be cleaned and the type of cleaning. The dynamic mode determination mechanism can flexibly adjust the cleaning strategy according to the actual situation to improve the cleaning efficiency and effect; by comprehensively considering time, frequency, capacity, format and reference standards, as well as different cleaning strategies, the system can conduct all-round and multi-angle analysis and identification of files, thereby improving the comprehensiveness and accuracy of identification; each unit works together to automatically perform operations such as scanning, judgment, marking, division and counting, reducing manual intervention and improving the automation level and processing efficiency of the cleaning work; determine the cleaning mode based on different cleaning types and actual quantities, so that the system can flexibly respond to different cleaning needs and has strong adaptability; through precise identification and classification marking, the system can effectively reduce data redundancy, optimize the storage structure and management efficiency of the data lake, extend the service life of the data lake, improve the overall data management level, and help improve the accuracy, efficiency and flexibility of data lake file cleaning.

[0050] In particular, by using time, frequency, capacity, format and reference multi-dimensional scanning standards, combined with specific target cleaning strategies (such as deadline, frequency of use, capacity limit, format requirements), different types of target files can be accurately identified, and the detailed condition setting ensures the accuracy and pertinence of the cleaning operation; according to the different characteristics of the file creation time, frequency of use, capacity and format, combined with their own standard strategies, it is judged whether the file meets the cleaning standards. The multi-dimensional target identification strategy effectively covers various file types that may need to be cleaned in the data lake, and improves the cleaning efficiency; according to the analysis results of the judgment unit, different types of tags are applied to the target files (such as expired, infrequently used, unpopular, abnormal capacity, and format inconsistency), so that subsequent cleaning work can be carried out in an orderly manner, and flexible classification tags help optimize the cleaning process and resource allocation; the marked files are divided into different subsets to be cleaned according to the cleaning type, and each subset has a clear cleaning target The precise division method makes the cleaning work more efficient and can take appropriate cleaning measures for different types of files. The cleaning mode (such as threshold cleaning mode or periodic cleaning mode) is dynamically determined according to the actual number of tags and the preset cleaning cycle. The flexible mode determination mechanism can adjust the cleaning strategy according to the actual situation to ensure the maximum cleaning effect. By setting the total tag threshold and sub-tag threshold, and combining the number of items that meet a single judgment condition, the system can automatically trigger cleaning when certain conditions are met, reducing the need for manual intervention and improving the efficiency and timeliness of cleaning decisions. Taking into account time, frequency, capacity, format and reference factors, the system can effectively manage files in the data lake, optimize storage structure and management efficiency, and extend the service life of the data lake, thereby improving the overall data management level and operating cost-effectiveness. It realizes efficient and intelligent cleaning operations, providing strong support for the long-term operation and management of the data lake.

[0051] In particular, the actual cleaning mode is used to automatically determine the number of single cleanups and generate a cleaning set, which reduces manual intervention and improves the automation and efficiency of the cleaning task. By combining the marking results and the actual number of markings, the cleaning urgency of each isolated file is calculated to ensure that high-urgency files are cleaned first, optimize resource usage, and ensure the effectiveness and timeliness of the cleaning work. The cleaning urgency of each isolated file is sorted to generate an actual cleaning sequence. The priority-based cleaning sequence ensures that the files that need cleaning most are processed first, improving the overall efficiency and effectiveness of the cleaning work. The cleaning sequence can be automatically generated according to the cleaning urgency, providing clear guidance for subsequent cleaning operations and reducing the complexity and uncertainty in the decision-making process. Determinism; by calculating and sorting the urgency of cleaning, better use can be made of system resources, avoiding low-priority files from occupying too many resources, thereby improving the overall performance of the system; the number of single cleanups is dynamically adjusted according to the actual cleaning mode, so that the system can flexibly respond to different cleaning needs and has strong adaptability; the entire preprocessing module makes decisions based on data (marking results, actual number of marks, cleaning urgency), with a high degree of data-driven characteristics, ensuring the scientificity and rationality of decisions; by calculating and sorting the urgency of cleaning, important cleaning targets can be avoided from being missed, improving the accuracy and comprehensiveness of the cleaning work; efficient, accurate and intelligent cleaning task scheduling is achieved, significantly improving the efficiency and effectiveness of data lake management.

[0052] In particular, by real-time monitoring of the cleaning process, the actual cleaning parameters are obtained, including the cleaning target data, the cleaning operation results and the cleaning process data, thereby ensuring the transparency and traceability of the cleaning process, timely discovering and handling problems, and improving the reliability of the cleaning operation; by judging whether the actual cleaning process data and metadata are synchronized, a first alarm signal is issued when they are not synchronized, thereby ensuring the consistency and integrity of the data and avoiding errors and confusion caused by data asynchrony; the single cleaning success rate is determined based on the actual cleaning operation results and the target cleaning strategy, and a second alarm signal is issued when the success rate is not equal to 100%. This function ensures the effectiveness and accuracy of the cleaning operation, timely identifies and handles unsuccessful cleaning operations, and improves the overall cleaning effect; based on the actual cleaning target data and the current execution task data, it determines whether there is conflicting data. If so, a third alarm signal is issued, which can effectively prevent and resolve data conflict problems and ensure the smooth progress of data cleaning; according to different alarm signals, the corresponding process control mode is determined: the first alarm signal triggers data re-reading to ensure the correctness and consistency of the data; the second alarm signal triggers data rollback recovery to avoid data loss caused by cleaning failure; the third alarm signal triggers the data locking mechanism to prevent conflicting data from affecting subsequent operations; the flexible control mode improves the system's resilience and operational stability; by issuing alarm signals for different types of errors and taking corresponding control measures, the system can quickly respond to and handle various abnormal situations, reduce the downtime during the cleaning process, and improve the stability and reliability of the cleaning operation; the early warning mechanism and the corresponding control mode are driven by data to achieve intelligent decision support, reduce manual intervention, and improve operational efficiency and accuracy; the automation, reliability and intelligence level of the cleaning operation are significantly improved, ensuring the efficiency and stability of data lake management.

[0053] In particular, by periodically obtaining the cleaning results, it is ensured that the actual effect of the cleaning operation and the amount of data released are obtained in time. The periodic acquisition function ensures that the feedback correction module can evaluate and adjust based on the latest data, improving the timeliness and accuracy of the feedback; according to the absolute value of the difference between the actual storage release amount and the preset standard storage release amount (the first absolute value of the difference), combined with the preset first evaluation value, the cleaning level is determined. The intelligent evaluation method can objectively judge the pros and cons of the cleaning effect and provide a scientific basis for subsequent adjustments; the feedback adjustment method is determined according to the cleaning level obtained by the evaluation, and a single feedback adjustment or an overall feedback adjustment can be selected. This flexible feedback adjustment method can be adjusted in a targeted manner according to different cleaning situations and needs to optimize the cleaning effect; by periodically obtaining the cleaning results and according to the intelligent judgment of the evaluation unit, the adjustment unit can effectively optimize the utilization of storage resources to ensure that the maximum storage release effect is achieved under the premise of minimizing the impact on the normal operation of the data lake; through intelligent evaluation and adjustment, the scientificity and effectiveness of the cleaning strategy are ensured, providing strong support for the long-term management of the data lake; the optimization and continuous improvement of the cleaning operation are achieved, bringing significant advantages to the management and operation of the data lake.

[0054] In particular, through scanning, files that need to be cleaned can be quickly located and marked, which improves the efficiency of the cleaning work and avoids the time wasted in checking files one by one; batch cleaning is performed according to the actual number of tags or the initial cleaning cycle, providing administrators with flexible cleaning strategies, and the frequency and scale of cleaning can be adjusted according to the actual situation of the files in the data lake; by generating a cleaning set and arranging the files in sequence, the orderly progress of the cleaning process is ensured, and confusion and conflicts during file cleaning are avoided; through real-time monitoring and consistency judgment, the accuracy of the cleaning process and the security of the data are guaranteed; through real-time monitoring and adjustment, it can be ensured that the cleaning operation is performed in accordance with the established rules, avoiding the accidental deletion or omission of important data; adjusting the cleaning conditions based on the feedback of the cleaning results can enable the cleaning strategy to be adaptively adjusted according to the actual use of the data lake, ensuring the effectiveness of the cleaning work and the optimization of the data lake performance; through regular and on-demand cleaning, space that is no longer needed in the data lake can be released, thereby improving the storage space utilization of the data lake; regular cleaning of isolated and obsolete files helps to maintain the performance of the data lake and avoid performance degradation caused by the accumulation of invalid data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a structural diagram of a system for timely cleaning of data lake files based on storage and computing separation according to an embodiment of the present invention;

[0056] Figure 2 This is a structural schematic diagram of a preprocessing module in a data lake file timely cleaning system based on storage and computing separation in an embodiment of the present invention;

[0057] Figure 3 It is a structural diagram of a feedback correction module in a data lake file timely cleanup system based on storage and computing separation according to an embodiment of the present invention;

[0058] Figure 4 This is a flow chart of a method for timely cleaning of data lake files based on storage and computing separation according to an embodiment of the present invention;

[0059] The figure includes: 1-recognition module, 2-preprocessing module, 3-cleaning module, 4-feedback correction module. DETAILED DESCRIPTION

[0060] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0061] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0062] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0063] See also Figure 1-Figure 4 As shown, Figure 1 This is a structural diagram of a system for timely cleaning of data lake files based on storage and computing separation according to an embodiment of the present invention; Figure 2 This is a structural schematic diagram of a preprocessing module in a data lake file timely cleaning system based on storage and computing separation in an embodiment of the present invention; Figure 3 It is a structural diagram of a feedback correction module in a data lake file timely cleanup system based on storage and computing separation according to an embodiment of the present invention; Figure 4 The present invention is a flowchart of a method for timely cleaning of data lake files based on storage and computing separation according to an embodiment of the present invention.

[0064] On one hand, the present invention provides a data lake file on-time cleaning system based on storage and computing separation, comprising:

[0065] Identification module 1 is used to scan each file in the data lake in real time, identify the target file in turn according to the scanning results, determine the cleaning type of the target file, and record the actual number of the target file, and determine the actual cleaning mode according to the cleaning type and the actual number;

[0066] A preprocessing module 2 is used to generate a cleaning set for each target file obtained by scanning according to the actual cleaning mode, evaluate the cleaning urgency of each file to be cleaned in the cleaning set in turn, and sort each file to be cleaned according to the evaluation result to obtain an actual cleaning sequence;

[0067] A cleaning module 3 is used to perform cleaning according to the actual cleaning sequence, obtain actual cleaning parameters during the cleaning process, analyze the actual cleaning parameters to determine the consistency of the cleaning process, and determine a corresponding process control mode according to the determination result;

[0068] The feedback correction module 4 is used to periodically obtain the cleaning results and perform feedback adjustment on the cleaning conditions according to the cleaning results.

[0069] Specifically, the embodiments of the present invention ensure timely discovery of target files by performing real-time scanning of files in the data lake; help to quickly respond to and process files that need to be cleaned, reduce data redundancy and storage space waste; determine the actual cleaning mode based on the scanning results and the cleaning type, combined with the actual quantity, and dynamically adjust the cleaning strategy to make it more flexible and targeted, and adapt to different cleaning needs and scenarios; by evaluating the cleaning urgency, sorting the files to be cleaned, and generating an actual cleaning sequence, it is ensured that important or urgent files are cleaned first, and resource usage and processing efficiency are optimized; in the cleaning process, the actual cleaning parameters are obtained and analyzed to determine the consistency of the cleaning process, and the corresponding process control mode is determined according to the determination result, which helps to ensure the stability and reliability of the cleaning process and reduce the risk of errors and data loss; by periodically obtaining the cleaning results and feedback-adjusting the cleaning conditions, the closed-loop feedback mechanism enables the system to continuously optimize the cleaning strategy according to the actual cleaning effect, improve the cleaning efficiency and effect, reduce the need for manual intervention, and improve management efficiency and accuracy; it can effectively manage the files in the data lake, keep the data lake clean and orderly, improve storage and retrieval efficiency, and extend the service life of the data lake.

[0070] Specifically, the identification module 1 in this embodiment includes: a scanning unit, a judgment unit, a marking unit, a division unit, a counting unit and a first determination unit;

[0071] The scanning unit is used to scan each file in the data lake in real time according to the scanning standard to obtain actual scanning results;

[0072] The judgment unit is used to analyze each file according to the actual scanning result in combination with a preset target cleaning strategy to identify a number of target files;

[0073] The scanning standards include: time standards, frequency standards, capacity standards, format standards and reference standards; the target cleaning strategy includes: standard deadline strategy, standard usage strategy, standard capacity strategy and standard format strategy;

[0074] The marking unit is used to mark each target file according to the judgment result to obtain isolated files of different cleaning types;

[0075] The division unit is used to divide each isolated file according to the cleaning type to obtain a plurality of subsets to be cleaned;

[0076] The counting unit is used to accumulate the actual number of marks of each subset to be cleaned;

[0077] The first determining unit is used to determine that the actual cleaning mode is a threshold cleaning mode or a periodic cleaning mode according to the actual number, the actual number of marks of each subset to be cleaned, the cleaning type and a preset initial cleaning period.

[0078] Specifically, the embodiments of the present invention perform real-time scanning of files in the data lake by time, frequency, capacity, format and reference to multiple standards to ensure comprehensive coverage and accurate identification. Multi-dimensional scanning can capture various file types that may need to be cleaned; the scanning results are analyzed in combination with preset target cleaning strategies (such as standard deadlines, usage, capacity and format strategies) to accurately identify target files that need to be cleaned, and the cleaning strategies can be flexibly adjusted according to different management needs to improve the accuracy of judgment; the target files are marked according to the judgment results, and files of different cleaning types (such as expired files, infrequently used files, large-capacity files, and files in non-compliant formats) are clearly distinguished. Classification and marking facilitate the orderly progress of subsequent cleaning work; the marked isolated files are divided according to the cleaning type to form several subsets to be cleaned. The detailed division makes the cleaning work more organized and is conducive to the use of different cleaning strategies for different types of files; the actual number of marks for each subset to be cleaned is accumulated and counted to provide data support, which helps to quantify the work of the cleaning task. The system can analyze and identify files from all angles and improve the comprehensiveness and accuracy of identification by comprehensively considering time, frequency, capacity, format and reference standards, as well as different cleaning strategies; each unit works together to automatically perform operations such as scanning, judging, marking, dividing and counting, which reduces manual intervention and improves the automation level and processing efficiency of cleaning work; the cleaning mode is determined by combining different cleaning types and actual quantities, so that the system can flexibly respond to different cleaning needs and has strong adaptability; through accurate identification and classification marking, the system can effectively reduce data redundancy, optimize the storage structure and management efficiency of the data lake, extend the service life of the data lake, improve the overall data management level, and help improve the accuracy, efficiency and flexibility of data lake file cleaning.

[0079] Specifically, in this embodiment, for any file in the data lake, based on the scanning standard being the time standard and the target cleanup policy being the standard deadline policy, the scanning unit obtains the creation time of the file, and calculates the actual creation duration based on the creation time combined with the current time;

[0080] The judging unit identifies whether the file is the target file according to the actual creation time and the standard deadline policy;

[0081] If the actual creation time exceeds the standard time limit policy, the judgment unit determines that the file is the target file;

[0082] The marking unit marks the target file as an expired orphan file;

[0083] If the actual creation time does not exceed the standard time limit policy, the judgment unit determines that the file is not the target file;

[0084] In this embodiment, the standard term policy is: the creation period is three months;

[0085] Based on the target cleaning policy being the standard usage policy when the scanning standard is the frequency standard, the scanning unit acquires the actual usage frequency of the file;

[0086] The judging unit identifies whether the file is the target file according to the actual usage frequency combined with the standard usage policy;

[0087] If the actual usage frequency is less than or equal to the standard usage policy, the judgment unit determines that the file is the target file;

[0088] The marking unit marks the target file as an uncommonly used orphan file;

[0089] If the actual usage frequency is greater than the standard usage policy, the judgment unit determines that the file is not the target file;

[0090] Based on the fact that the scanning standard is the reference standard and the target cleaning policy is the standard use policy, the scanning unit obtains the actual reference count of the file;

[0091] The judging unit identifies whether the file is the target file according to the actual number of citations combined with the standard usage policy;

[0092] If the actual number of references is less than or equal to the standard usage policy, the judgment unit determines that the file is the target file;

[0093] The marking unit marks the target file as an unpopular orphan file;

[0094] If the actual number of citations is greater than the standard usage policy, the judgment unit determines that the file is not the target file;

[0095] In this embodiment, the standard usage strategy is: within three months, the usage frequency is 10 and the number of citations is 8;

[0096] Based on the target cleaning policy being the standard capacity policy when the scanning standard is the capacity standard, the scanning unit acquires the actual capacity of the file;

[0097] The judging unit identifies whether the file is the target file according to the actual capacity combined with the standard capacity policy;

[0098] If the actual capacity is less than or equal to the minimum capacity value in the standard capacity policy, the judgment unit determines that the file is the target file;

[0099] If the actual capacity is greater than or equal to the maximum capacity value in the standard capacity policy, the judgment unit determines that the file is the target file;

[0100] The marking unit marks the target file as an isolated file with abnormal capacity;

[0101] If the actual capacity is greater than the minimum capacity value and less than the maximum capacity value, the determination unit determines that the file is not the target file;

[0102] In this embodiment, the standard capacity policy is: [minimum capacity value = 3KB, maximum capacity value = 3GB];

[0103] Based on the fact that the target cleaning strategy is the standard format strategy when the scanning standard is the format standard, the scanning unit acquires the actual format of the file;

[0104] The judging unit identifies whether the file is the target file according to the actual format combined with the standard format strategy;

[0105] If the actual format is the standard format policy, the judgment unit determines that the file is the target file;

[0106] The marking unit marks the target file as a failed orphan file;

[0107] If the actual format is not the standard format policy, the judgment unit determines that the file is not the target file;

[0108] In this embodiment, the standard format strategy is: run failure format;

[0109] The division unit divides each target file according to the cleaning type,

[0110] Based on the fact that the cleaning type is the expired orphan file, a first subset to be cleaned is formed, and the counting unit accumulates the actual number of the first mark of the first subset to be cleaned;

[0111] Based on the cleaning type being the infrequently used isolated files, a second subset to be cleaned is formed, and the counting unit accumulates the actual number of the second mark of the second subset to be cleaned;

[0112] Based on the cleaning type being the unpopular isolated file, a third subset to be cleaned is formed, and the counting unit accumulates the actual number of the third mark of the third subset to be cleaned;

[0113] Based on the cleaning type being the isolated files with abnormal capacity, a fourth subset to be cleaned is formed, and the counting unit accumulates the actual number of the fourth mark of the fourth subset to be cleaned;

[0114] Based on the cleaning type being the failed isolated file, a fifth subset to be cleaned is formed, and the counting unit accumulates the actual number of the fifth mark of the fifth subset to be cleaned;

[0115] When the actual number exceeds a preset total mark threshold and does not reach the initial cleaning period, the first determining unit determines that the actual cleaning mode is a first threshold cleaning mode;

[0116] When the actual number does not exceed the preset total mark threshold and does not reach the initial cleaning period, the first determining unit determines not to start cleaning at this time;

[0117] Based on the fact that the actual number does not exceed the preset total mark threshold and reaches the initial cleaning period, the first determining unit determines that the actual cleaning mode is the periodic cleaning mode;

[0118] The first threshold cleaning mode is: when the number of marked target files reaches the total marking threshold, cleaning is started;

[0119] The periodic cleaning mode is: the data lake starts cleaning once each time the initial cleaning cycle is run;

[0120] For any subset to be cleaned, if the actual number of the tags corresponding to the subset to be cleaned exceeds the preset sub-tag threshold, it is determined that the subset to be cleaned meets the single determination condition;

[0121] If the actual number of the tags corresponding to the subset to be cleaned does not exceed the sub-tag threshold, it is determined that the subset to be cleaned does not meet the single determination condition;

[0122] Based on the actual number not exceeding the total marking threshold and not reaching the initial cleaning cycle,

[0123] If the number of items in the first to-be-cleaned subset, the second to-be-cleaned subset, the third to-be-cleaned subset, the fourth to-be-cleaned subset, and the fifth to-be-cleaned subset that meet the single determination condition is greater than or equal to 2, the first determination unit determines that the actual cleaning mode is the second threshold cleaning mode;

[0124] If the number of items in the first subset to be cleaned, the second subset to be cleaned, the third subset to be cleaned, the fourth subset to be cleaned, and the fifth subset to be cleaned that meet the single determination condition is less than 2, the first determination unit determines to start cleaning at this time;

[0125] The second threshold cleaning mode is: when the number of tags in any two subsets to be cleaned exceeds the sub-tag threshold, cleaning is started;

[0126] In this embodiment, the initial cleaning period is set to 2 hours, the total mark threshold is set to 30, and the sub-mark threshold is set to 10.

[0127] Specifically, the embodiments of the present invention can accurately identify different types of target files by using time, frequency, capacity, format and reference multi-dimensional scanning standards, combined with specific target cleaning strategies (such as deadline, frequency of use, capacity limit, format requirements), and the detailed condition setting ensures the accuracy and pertinence of the cleaning operation; according to the different characteristics of the file such as creation time, frequency of use, capacity and format, combined with the respective standard strategies, it is judged whether the file meets the cleaning standards. The multi-dimensional target identification strategy effectively covers various file types that may need to be cleaned in the data lake, thereby improving the cleaning efficiency; according to the analysis results of the judgment unit, different types of markings are performed on the target files (such as expired, infrequently used, unpopular, abnormal capacity, and format non-compliance), so that subsequent cleaning work can be carried out in an orderly manner, and flexible classification marking helps to optimize the cleaning process and resource allocation; the marked files are divided into different subsets to be cleaned according to the cleaning type, and each subset has a clear Cleaning goals and strategies, precise division makes the cleaning work more efficient, and can take appropriate cleaning measures for different types of files; according to the actual number of tags and the preset cleaning cycle, the cleaning mode (such as threshold cleaning mode or periodic cleaning mode) is dynamically determined, and the flexible mode determination mechanism can adjust the cleaning strategy according to the actual situation to ensure the maximum cleaning effect; by setting the total tag threshold and sub-tag threshold, and combining the number of single judgment conditions, the system can automatically trigger cleaning when certain conditions are met, reducing the need for manual intervention and improving the efficiency and timeliness of cleaning decisions; taking into account time, frequency, capacity, format and reference factors, the system can effectively manage files in the data lake, optimize storage structure and management efficiency, and extend the service life of the data lake, thereby improving the overall data management level and operating cost-effectiveness; it realizes efficient and intelligent cleaning operations, providing strong support for the long-term operation and management of the data lake.

[0128] Specifically, the preprocessing module 2 in this embodiment includes: a generating unit, a calculating unit and a comparing unit;

[0129] The generating unit is used to determine the single cleaning number according to the actual cleaning mode, and generate the cleaning set based on the single cleaning number;

[0130] The calculation unit is used to calculate the cleaning urgency of each isolated file according to the marking result combined with the actual number of marks;

[0131] The comparison unit is used to sort the isolated files according to the cleaning urgency values ​​to obtain the actual cleaning sequence.

[0132] Specifically, the cleaning urgency = the characteristic value corresponding to the cleaning type × the actual number of the marks of the cleaning type;

[0133] In this embodiment, the characteristic value corresponding to the expired isolated file is set to 1.5; the characteristic value corresponding to the infrequently used isolated file is set to 1.3; the characteristic value corresponding to the unpopular isolated file is set to 1.2; the characteristic value corresponding to the capacity abnormal isolated file is set to 1.2; and the characteristic value corresponding to the failed isolated file is set to 1.3.

[0134] Specifically, the embodiment of the present invention automatically determines the number of single cleanups and generates a cleanup set through an actual cleanup mode, thereby reducing manual intervention and improving the automation and efficiency of the cleanup task; by combining the marking results and the actual number of markings, the cleaning urgency of each isolated file is calculated, which can ensure that high-urgency files are cleaned first, optimize resource usage, and ensure the effectiveness and timeliness of the cleanup work; the cleaning urgency of each isolated file is sorted to generate an actual cleanup sequence, and the priority-based cleanup sequence ensures that the files that need to be cleaned most are processed first, thereby improving the overall efficiency and effect of the cleanup work; the cleanup sequence can be automatically generated according to the cleaning urgency, providing clear guidance for subsequent cleanup operations and reducing the complexity in the decision-making process. Complexity and uncertainty; by calculating and sorting the cleaning urgency, it can better utilize system resources and avoid low-priority files occupying too many resources, thereby improving the overall performance of the system; dynamically adjust the number of single cleanups according to the actual cleaning mode, so that the system can flexibly respond to different cleaning needs and has strong adaptability; the entire preprocessing module makes decisions based on data (marking results, actual number of marks, cleaning urgency), which has a high degree of data-driven characteristics to ensure the scientific and rationality of decision-making; by calculating and sorting the cleaning urgency, it can avoid missing important cleaning targets and improve the accuracy and comprehensiveness of the cleaning work; it realizes efficient, accurate and intelligent cleaning task scheduling, significantly improving the efficiency and effectiveness of data lake management.

[0135] Specifically, the cleaning module 3 in this embodiment includes: a monitoring unit, a first determination unit, a second determination unit, a third determination unit, and a second determination unit;

[0136] The monitoring unit is used to monitor the cleaning process in real time to obtain the actual cleaning parameters;

[0137] The first determination unit is used to determine whether the actual cleaning process data and the metadata are synchronized, and to issue a first alarm signal when they are not synchronized;

[0138] Wherein, the actual cleaning parameters include: actual cleaning target data, actual cleaning operation results and the actual cleaning process data;

[0139] The second determination unit is used to determine a single cleaning success rate according to the actual cleaning operation result and the target cleaning strategy, and to issue a second alarm signal when the single cleaning success rate is not equal to 100%;

[0140] The third determination unit is used to determine whether conflicting data exists according to the actual cleaning target data combined with the currently executed task data, and to issue a third alarm signal when the conflicting data exists;

[0141] The second determination unit can determine that the process control mode is data re-reading according to the first alarm signal;

[0142] The second determination unit can determine that the process control mode is data rollback recovery according to the second alarm signal;

[0143] The second determination unit can determine that the process control mode is a data locking mechanism according to the third alarm signal.

[0144] Specifically, the embodiment of the present invention obtains actual cleaning parameters, including cleaning target data, cleaning operation results and cleaning process data, by real-time monitoring of the cleaning process, thereby ensuring the transparency and traceability of the cleaning process, timely discovering and handling problems, and improving the reliability of the cleaning operation; by judging whether the actual cleaning process data and metadata are synchronized, a first alarm signal is issued when they are not synchronized, thereby ensuring the consistency and integrity of the data and avoiding errors and confusion caused by data asynchrony; the single cleaning success rate is determined based on the actual cleaning operation results and the target cleaning strategy, and a second alarm signal is issued when the success rate is not equal to 100%. This function ensures the effectiveness and accuracy of the cleaning operation, timely identifies and handles unsuccessful cleaning operations, and improves the overall cleaning effect; based on the actual cleaning target data and the current execution task data, it determines whether there is conflicting data. If so, a third alarm signal is issued, which can effectively prevent and resolve data conflict problems and ensure the smooth progress of data cleaning; according to different alarm signals, the corresponding process control mode is determined: the first alarm signal triggers data re-reading to ensure the correctness and consistency of the data; the second alarm signal triggers data rollback recovery to avoid data loss caused by cleaning failure; the third alarm signal triggers the data locking mechanism to prevent conflicting data from affecting subsequent operations; the flexible control mode improves the system's resilience and operational stability; by issuing alarm signals for different types of errors and taking corresponding control measures, the system can quickly respond to and handle various abnormal situations, reduce the downtime during the cleaning process, and improve the stability and reliability of the cleaning operation; the early warning mechanism and the corresponding control mode are driven by data to achieve intelligent decision support, reduce manual intervention, and improve operational efficiency and accuracy; the automation, reliability and intelligence level of the cleaning operation are significantly improved, ensuring the efficiency and stability of data lake management.

[0145] Specifically, the feedback correction module 4 in this embodiment includes: an acquisition unit, an evaluation unit and an adjustment unit;

[0146] The acquisition unit is used to periodically acquire the cleaning result;

[0147] The evaluation unit is used to determine the cleaning level according to the absolute value of the first difference combined with a preset first evaluation value;

[0148] The regulating unit is used to determine the feedback regulating mode as single feedback regulating or overall feedback regulating according to the cleaning level;

[0149] The first absolute value of the difference is the absolute value of the difference between the actual storage release amount and the preset standard storage release amount.

[0150] Specifically, the absolute value of the first difference=|actual storage release amount-standard storage release amount|;

[0151] If the absolute value of the first difference is less than or equal to the first evaluation value, the cleaning level is determined to be level 2; and the feedback adjustment method is determined to be the single feedback adjustment;

[0152] If the absolute value of the first difference is greater than the first evaluation value and the actual storage release amount is greater than the standard storage release amount, then the cleaning level is determined to be level one;

[0153] If the absolute value of the first difference is greater than the first evaluation value and the actual storage release amount is less than the standard storage release amount, it is determined that the cleaning level is level three, and the feedback adjustment mode is determined to be the overall feedback adjustment;

[0154] In this embodiment, the standard storage release amount is set to 40%; and the first evaluation value is set to 8%.

[0155] Specifically, the embodiment of the present invention periodically obtains the cleaning results to ensure that the actual effect of the cleaning operation and the amount of data released are obtained in time. The periodic acquisition function ensures that the feedback correction module can evaluate and adjust based on the latest data, thereby improving the timeliness and accuracy of the feedback; the cleaning level is determined according to the absolute value of the difference between the actual storage release amount and the preset standard storage release amount (the first absolute value of the difference), combined with the preset first evaluation value, and the intelligent evaluation method can objectively judge the pros and cons of the cleaning effect, providing a scientific basis for subsequent adjustments; the feedback adjustment method is determined according to the cleaning level obtained by the evaluation, and a single feedback adjustment or an overall feedback adjustment can be selected. This flexible feedback adjustment method can be adjusted in a targeted manner according to different cleaning situations and needs to optimize the cleaning effect; by periodically obtaining the cleaning results and according to the intelligent judgment of the evaluation unit, the adjustment unit can effectively optimize the utilization of storage resources, ensuring that the maximum storage release effect is achieved under the premise of minimizing the impact on the normal operation of the data lake; through intelligent evaluation and adjustment, the scientific nature and effectiveness of the cleaning strategy are ensured, providing strong support for the long-term management of the data lake; the optimization and continuous improvement of the cleaning operation are achieved, bringing significant advantages to the management and operation of the data lake.

[0156] On the other hand, the present invention also provides a method for timely cleaning of data lake files based on storage and computing separation, comprising:

[0157] Step S1, scanning each file in the data lake, determining the files to be cleaned up according to the scanning results, and marking them to obtain a number of isolated files;

[0158] Step S2, accumulating the isolated files to obtain an actual number of tags, and performing batch cleaning on the data lake according to the actual number of tags or the initial cleaning cycle;

[0159] Step S3, for any cleaning batch, generating a cleaning set from the orphan files corresponding to the cleaning batch, and arranging the files to be cleaned in the cleaning set in a cleaning order;

[0160] Step S4, when cleaning is performed based on the cleaning sequence arrangement result, the cleaning process is monitored in real time, the consistency of the cleaning target data, the cleaning process data and the cleaning operation is determined in turn, and the corresponding process control mode is determined according to the determination result;

[0161] Step S5, periodically obtaining cleaning results, and performing feedback adjustment on cleaning conditions according to the cleaning results;

[0162] The cleaning conditions include: the initial cleaning period, the total mark threshold and the sub-mark threshold.

[0163] Specifically, the embodiments of the present invention can quickly locate and mark files that need to be cleaned through scanning, thereby improving the efficiency of the cleaning work and avoiding the time waste of checking files one by one; batch cleaning is performed according to the actual number of tags or the initial cleaning cycle, providing the administrator with a flexible cleaning strategy, and the frequency and scale of cleaning can be adjusted according to the actual situation of the files in the data lake; by generating a cleaning set and arranging the files in sequence, the orderly progress of the cleaning process is ensured, and confusion and conflict during file cleaning are avoided; through real-time monitoring and consistency determination, the accuracy of the cleaning process and the security of the data are guaranteed; through real-time monitoring and adjustment, it can be ensured that the cleaning operation is performed in accordance with the established rules, avoiding the accidental deletion or omission of important data; by adjusting the cleaning conditions according to the feedback of the cleaning results, the cleaning strategy can be adaptively adjusted according to the actual use of the data lake, ensuring the effectiveness of the cleaning work and the optimization of the data lake performance; through regular and on-demand cleaning, the space no longer needed in the data lake can be released, thereby improving the storage space utilization of the data lake; regular cleaning of isolated and obsolete files helps to maintain the performance of the data lake and avoid performance degradation due to the accumulation of invalid data.

[0164] The calculation compensation parameters and the calculation adjustment parameters described in the present invention have two functions: one is to balance the left and right dimensions of the formula, and the other is to adjust the numerical results. No specific assignment is performed in this embodiment. In addition, the calculation formulas in this embodiment are used to intuitively reflect the adjustment relationship between the numerical values, such as positive correlation and negative correlation. Unless otherwise specified, the parameter values ​​that are not specifically limited to numerical values ​​are all positive.

[0165] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data lake file scheduled cleaning system based on storage and computing separation, characterized in that: include: An identification module is used to scan each file in the data lake in real time, identify the target file in turn according to the scanning result, determine the cleaning type of the target file, and record the actual number of the target file, and determine the actual cleaning mode according to the cleaning type and the actual number; the identification module includes a counting unit and a first determining unit; the counting unit is used to accumulate the actual number of marks of each subset to be cleaned; The first determining unit is used to determine whether the actual cleaning mode is a threshold cleaning mode or a periodic cleaning mode according to the actual number, the actual number of marks of each to-be-cleaned subset, the cleaning type and a preset initial cleaning period; A preprocessing module, used to generate a cleaning set for each target file obtained by scanning according to the actual cleaning mode, evaluate the cleaning urgency of each file to be cleaned in the cleaning set in turn, and sort each file to be cleaned according to the evaluation result to obtain an actual cleaning sequence; Wherein, the cleaning urgency = the characteristic value corresponding to the cleaning type × the actual number of the marks of the cleaning type; A cleaning module is used to clean the actual cleaning sequence, including obtaining actual cleaning parameters, analyzing the actual cleaning parameters to determine the consistency of the cleaning process, and determining the corresponding process control mode according to the determination result; the cleaning module includes a monitoring unit, a first determination unit, a second determination unit, a third determination unit and a second determination unit; the monitoring unit is used to monitor the cleaning process in real time to obtain the actual cleaning parameters; the first determination unit is used to determine whether the actual cleaning process data and metadata are synchronized, and issue a first alarm signal based on the lack of synchronization; the second determination unit is used to determine a single cleaning success rate according to the actual cleaning operation result and the preset target cleaning strategy, and issue a second alarm signal based on the single cleaning success rate is not equal to 100%; the third determination unit is used to determine whether there is conflicting data according to the actual cleaning target data combined with the current execution task data, and issue a third alarm signal based on the existence of the conflicting data; the second determination unit is used to determine the corresponding process control mode according to different alarm signals, including data re-reading, data rollback recovery or data locking mechanism; Wherein, the actual cleaning parameters include: the actual cleaning target data, the actual cleaning operation result and the actual cleaning process data; the second determination unit can determine that the process control mode is data re-reading according to the first alarm signal, can determine that the process control mode is data rollback recovery according to the second alarm signal, and can determine that the process control mode is data locking mechanism according to the third alarm signal; A feedback correction module, used to periodically obtain cleaning results and perform feedback adjustment on cleaning conditions according to the cleaning results; The cleaning conditions include: the initial cleaning period, the total mark threshold and the sub-mark threshold.

2. The data lake file scheduled cleaning system based on storage and computing separation according to claim 1 is characterized in that: The identification module includes a scanning unit and a judgment unit; The scanning unit is used to scan each file in the data lake in real time according to the scanning standard to obtain actual scanning results; The judgment unit is used to analyze each file according to the actual scanning result and the target cleaning strategy to identify a number of target files; Among them, the scanning standards include: time standards, frequency standards, capacity standards, format standards and reference standards; the target cleaning strategy includes: standard deadline strategy, standard usage strategy, standard capacity strategy and standard format strategy.

3. The data lake file on-time cleaning system based on storage and computing separation according to claim 2 is characterized in that: The identification module also includes a marking unit and a dividing unit; The marking unit is used to mark each target file according to the judgment result to obtain isolated files of different cleaning types; The division unit is used to divide each isolated file according to the cleaning type to obtain a plurality of subsets to be cleaned.

4. The data lake file on-time cleaning system based on storage and computing separation according to claim 3 is characterized in that: The preprocessing module includes a generating unit, a calculating unit and a comparing unit; The generating unit is used to determine the single cleaning number according to the actual cleaning mode, and generate the cleaning set based on the single cleaning number; The calculation unit is used to calculate the cleaning urgency of each isolated file according to the marking result combined with the actual number of marks; The comparison unit is used to sort the isolated files according to the cleaning urgency values ​​to obtain the actual cleaning sequence.

5. The data lake file on-time cleaning system based on storage and computing separation according to claim 4 is characterized in that: The feedback correction module includes an acquisition unit, an evaluation unit and an adjustment unit; The acquisition unit is used to periodically acquire the cleaning result; The evaluation unit is used to determine the cleaning level according to the absolute value of the first difference combined with a preset first evaluation value; The regulating unit is used to determine the feedback regulating mode as single feedback regulating or overall feedback regulating according to the cleaning level; The first absolute value of the difference is the absolute value of the difference between the actual storage release amount and the preset standard storage release amount.

6. A method for timely cleaning of data lake files based on storage and computing separation, using the system for timely cleaning of data lake files based on storage and computing separation according to any one of claims 1 to 5, characterized in that: include: Step S1, scanning each file in the data lake, determining the files to be cleaned up according to the scanning results, and marking them to obtain a number of isolated files; Step S2, accumulating the isolated files to obtain an actual number of tags, and performing batch cleaning on the data lake according to the actual number of tags or the initial cleaning cycle; Step S3, for any cleaning batch, generating a cleaning set from the orphan files corresponding to the cleaning batch, and arranging the files to be cleaned in the cleaning set in a cleaning order; Step S4, when cleaning is performed based on the cleaning sequence arrangement result, the cleaning process is monitored in real time, the consistency of the cleaning target data, the cleaning process data and the cleaning operation is determined in turn, and the corresponding process control mode is determined according to the determination result; Step S5, periodically obtaining cleaning results, and performing feedback adjustment on cleaning conditions according to the cleaning results; The cleaning conditions include: the initial cleaning period, the total mark threshold and the sub-mark threshold.

Images